[FP8] Fix Benchmarking for certain Priors #155722
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/155722
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 08ad237 with merge base ffac0de ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
This pull request was exported from Phabricator. Differential Revision: D76092551 |
|
This pull request was exported from Phabricator. Differential Revision: D76092551 |
oniononion36
force-pushed
the
export-D76092551
branch
from
77d0e0d to
b31b1ab
Compare
|
This pull request was exported from Phabricator. Differential Revision: D76092551 |
oniononion36
force-pushed
the
export-D76092551
branch
from
b31b1ab to
3b8ff60
Compare
|
This pull request was exported from Phabricator. Differential Revision: D76092551 |
oniononion36
force-pushed
the
export-D76092551
branch
from
3b8ff60 to
b26ec80
Compare
|
This pull request was exported from Phabricator. Differential Revision: D76092551 |
oniononion36
force-pushed
the
export-D76092551
branch
from
b26ec80 to
0ec1c44
Compare
|
This pull request was exported from Phabricator. Differential Revision: D76092551 |
oniononion36
force-pushed
the
export-D76092551
branch
from
0ec1c44 to
f017a2f
Compare
|
This pull request was exported from Phabricator. Differential Revision: D76092551 |
oniononion36
force-pushed
the
export-D76092551
branch
from
f017a2f to
1778fec
Compare
oniononion36
added a commit
to oniononion36/pytorch
that referenced
this pull request
Jun 11, 2025
Summary: Pull Request resolved: pytorch#155722 For priors like layer norm, the order of the weight quantization kernel might be different and therefore have a different suffix, so we use regular expression instead. Test Plan: Trying this on model id 737772166 with ``` buck2 run mode/opt mode/inplace -c fbcode.platform010_cuda_version=12 -c fbcode.nvcc_arch=h100 caffe2/torch/fb/model_transform/experimental/benchmark:mts_gpu_benchmark -- --lower-backend=AOT_INDUCTOR --model-snapshot-id=737772166_0 --trace-aot-inductor-module=True --disable-acc-tracer=False --batch-size=1024 --node_replacement_dict "{'(autotune)':{'(1000+,1000+)':'fp8_float_model_dynamic_quantization_rowwise'}" ``` will allow more linears to be correctly replaced with fp8. An example of the gpu trace can be found in https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree/hpc/new/models/feed/benchmark/libkineto_activities_773108_f58b57e208c04787acd3bcb01a3e8771.json.gz&bucket=gpu_traces. Rollback Plan: Differential Revision: D76092551
|
This pull request was exported from Phabricator. Differential Revision: D76092551 |
oniononion36
force-pushed
the
export-D76092551
branch
from
1778fec to
27642e3
Compare
|
This pull request was exported from Phabricator. Differential Revision: D76092551 |
oniononion36
force-pushed
the
export-D76092551
branch
from
27642e3 to
82cfb48
Compare
oniononion36
force-pushed
the
export-D76092551
branch
from
b068f71 to
f9d518f
Compare
pytorch-bot bot
pushed a commit
that referenced
this pull request
Jun 16, 2025
Summary: Pull Request resolved: #155722 For priors like layer norm, the order of the weight quantization kernel might be different and therefore have a different suffix, so we use regular expression instead. Test Plan: Trying this on model id 737772166 with ``` buck2 run mode/opt mode/inplace -c fbcode.platform010_cuda_version=12 -c fbcode.nvcc_arch=h100 caffe2/torch/fb/model_transform/experimental/benchmark:mts_gpu_benchmark -- --lower-backend=AOT_INDUCTOR --model-snapshot-id=737772166_0 --trace-aot-inductor-module=True --disable-acc-tracer=False --batch-size=1024 --node_replacement_dict "{'(autotune)':{'(1000+,1000+)':'fp8_float_model_dynamic_quantization_rowwise'}" ``` will allow more linears to be correctly replaced with fp8. An example of the gpu trace can be found in https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree/hpc/new/models/feed/benchmark/libkineto_activities_773108_f58b57e208c04787acd3bcb01a3e8771.json.gz&bucket=gpu_traces. Rollback Plan: Reviewed By: frank-wei Differential Revision: D76092551
|
This pull request was exported from Phabricator. Differential Revision: D76092551 |
oniononion36
force-pushed
the
export-D76092551
branch
from
f9d518f to
27b1eb8
Compare
|
This pull request was exported from Phabricator. Differential Revision: D76092551 |
oniononion36
force-pushed
the
export-D76092551
branch
from
27b1eb8 to
01af5c1
Compare
|
This pull request was exported from Phabricator. Differential Revision: D76092551 |
oniononion36
force-pushed
the
export-D76092551
branch
from
01af5c1 to
4a40cae
Compare
|
This pull request was exported from Phabricator. Differential Revision: D76092551 |
oniononion36
force-pushed
the
export-D76092551
branch
from
4a40cae to
08fef07
Compare
oniononion36
added a commit
to oniononion36/pytorch
that referenced
this pull request
Jun 16, 2025
Summary: Pull Request resolved: pytorch#155722 For priors like layer norm, the order of the weight quantization kernel might be different and therefore have a different suffix, so we use regular expression instead. Test Plan: Trying this on model id 737772166 with ``` buck2 run mode/opt mode/inplace -c fbcode.platform010_cuda_version=12 -c fbcode.nvcc_arch=h100 caffe2/torch/fb/model_transform/experimental/benchmark:mts_gpu_benchmark -- --lower-backend=AOT_INDUCTOR --model-snapshot-id=737772166_0 --trace-aot-inductor-module=True --disable-acc-tracer=False --batch-size=1024 --node_replacement_dict "{'(autotune)':{'(1000+,1000+)':'fp8_float_model_dynamic_quantization_rowwise'}" ``` will allow more linears to be correctly replaced with fp8. An example of the gpu trace can be found in https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree/hpc/new/models/feed/benchmark/libkineto_activities_773108_f58b57e208c04787acd3bcb01a3e8771.json.gz&bucket=gpu_traces. Rollback Plan: Reviewed By: frank-wei Differential Revision: D76092551
oniononion36
force-pushed
the
export-D76092551
branch
from
08fef07 to
9ae78df
Compare
|
This pull request was exported from Phabricator. Differential Revision: D76092551 |
pytorch-bot bot
pushed a commit
that referenced
this pull request
Jun 23, 2025
Summary:
For priors like layer norm, the order of the weight quantization kernel might be different and therefore have a different suffix, so we use regular expression instead.
Test Plan:
Trying this on model id 737772166 with
```
buck2 run mode/opt mode/inplace -c fbcode.platform010_cuda_version=12 -c fbcode.nvcc_arch=h100 caffe2/torch/fb/model_transform/experimental/benchmark:mts_gpu_benchmark -- --lower-backend=AOT_INDUCTOR --model-snapshot-id=737772166_0 --trace-aot-inductor-module=True --disable-acc-tracer=False --batch-size=1024 --node_replacement_dict "{'(autotune)':{'(1000+,1000+)':'fp8_float_model_dynamic_quantization_rowwise'}"
```
will allow more linears to be correctly replaced with fp8.
An example of the gpu trace can be found in https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree/hpc/new/models/feed/benchmark/libkineto_activities_773108_f58b57e208c04787acd3bcb01a3e8771.json.gz&bucket=gpu_traces.
Rollback Plan:
Reviewed By: frank-wei
Differential Revision: D76092551
oniononion36
force-pushed
the
export-D76092551
branch
from
9ae78df to
b45fee9
Compare
pytorch-bot bot
pushed a commit
that referenced
this pull request
Jun 24, 2025
Summary:
For priors like layer norm, the order of the weight quantization kernel might be different and therefore have a different suffix, so we use regular expression instead.
Test Plan:
Trying this on model id 737772166 with
```
buck2 run mode/opt mode/inplace -c fbcode.platform010_cuda_version=12 -c fbcode.nvcc_arch=h100 caffe2/torch/fb/model_transform/experimental/benchmark:mts_gpu_benchmark -- --lower-backend=AOT_INDUCTOR --model-snapshot-id=737772166_0 --trace-aot-inductor-module=True --disable-acc-tracer=False --batch-size=1024 --node_replacement_dict "{'(autotune)':{'(1000+,1000+)':'fp8_float_model_dynamic_quantization_rowwise'}"
```
will allow more linears to be correctly replaced with fp8.
An example of the gpu trace can be found in https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree/hpc/new/models/feed/benchmark/libkineto_activities_773108_f58b57e208c04787acd3bcb01a3e8771.json.gz&bucket=gpu_traces.
Rollback Plan:
Reviewed By: frank-wei
Differential Revision: D76092551
|
This pull request was exported from Phabricator. Differential Revision: D76092551 |
Summary:
For priors like layer norm, the order of the weight quantization kernel might be different and therefore have a different suffix, so we use regular expression instead.
Test Plan:
Trying this on model id 737772166 with
```
buck2 run mode/opt mode/inplace -c fbcode.platform010_cuda_version=12 -c fbcode.nvcc_arch=h100 caffe2/torch/fb/model_transform/experimental/benchmark:mts_gpu_benchmark -- --lower-backend=AOT_INDUCTOR --model-snapshot-id=737772166_0 --trace-aot-inductor-module=True --disable-acc-tracer=False --batch-size=1024 --node_replacement_dict "{'(autotune)':{'(1000+,1000+)':'fp8_float_model_dynamic_quantization_rowwise'}"
```
will allow more linears to be correctly replaced with fp8.
An example of the gpu trace can be found in https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree/hpc/new/models/feed/benchmark/libkineto_activities_773108_f58b57e208c04787acd3bcb01a3e8771.json.gz&bucket=gpu_traces.
Rollback Plan:
Reviewed By: frank-wei
Differential Revision: D76092551
oniononion36
force-pushed
the
export-D76092551
branch
from
b45fee9 to
08ad237
Compare
|
This pull request was exported from Phabricator. Differential Revision: D76092551 |
|
@pytorchbot merge (Initiating merge automatically since Phabricator Diff has merged) |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary: For priors like layer norm, the order of the weight quantization kernel might be different and therefore have a different suffix, so we use regular expression instead.
Test Plan:
Trying this on model id 737772166 with
will allow more linears to be correctly replaced with fp8.
An example of the gpu trace can be found in https://www.internalfb.com/intern/perfdoctor/trace_view?filepath=tree/hpc/new/models/feed/benchmark/libkineto_activities_773108_f58b57e208c04787acd3bcb01a3e8771.json.gz&bucket=gpu_traces.
Rollback Plan:
Differential Revision: D76092551
cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov