test rowwise fp32 #2431
Open
test rowwise fp32 #2431
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/2431
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 New FailureAs of commit b0240a2 with merge base 2025b75 ( NEW FAILURE - The following job has failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
facebook-github-bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D73552660 |
y-sq
requested review from
vkuzo,
danielvegamyhre and
drisspg
and removed request for
vkuzo
y-sq
added
float8
topic: not user facing
vkuzo
reviewed
Jun 24, 2025
|
|
||
| if convert_dtypes_for_rowwise_scaled_mm and is_rowwise_scaling: | ||
| output_dtype = torch.bfloat16 | ||
|
|
There was a problem hiding this comment.
instead of adding a flag, TBH I think we can just enable this on-by-default, like this:
file issue in PyTorch core to add float32 output to scaled_mm
output_dtype_to_use = output_dtype
if is_rowwise_scaling:
# work around torch._scaled_mm not having float32 output type
# TODO(issue number): remove this once torch._scaled_mm supports float32 output
output_dtype_to_use = torch.bfloat16
output = torch._scaled_mm(..., output_dtype_to_use, ...)
...
if is_rowwise_scaling and output_dtype == torch.float32:
# work around torch._scaled_mm not having float32 output type
# TODO(issue number): remove this once torch._scaled_mm supports float32 output
output = output.to(orig_dtype)There was a problem hiding this comment.
makes sense, I'll change to enable by default and file an issue.
vkuzo
reviewed
Jun 24, 2025
|
Updated to enable the workaround by default. Included fp16 and fp32 dtypes in the existing test cases. The additional changes are formatting things generated by linter. The pytorch issue: pytorch/pytorch#156771 |
|
@y-sq , maybe export again? |
y-sq
added a commit
to y-sq/ao
that referenced
this pull request
Jul 3, 2025
…ytorch#2431) Summary: Running rowwise scaling on fp32 tensors got the error, P1794222725 ``` RuntimeError: Only bf16 high precision output types are supported for row-wise scaling. ``` This pr adds an option to explicitly use bfloat16 as the output of rowwise_scaled, and cast it back to the original precision. It can be enabled by setting ``` config = dataclasses.replace(config, convert_dtypes_for_rowwise_scaled_mm=True) ``` Differential Revision: D73552660
|
This pull request was exported from Phabricator. Differential Revision: D73552660 |
|
@vkuzo sorry there were some un-synced files between github and fbcode so the previous export all failed. The pr should be updated now. |
y-sq
added a commit
to y-sq/ao
that referenced
this pull request
Jul 3, 2025
…ytorch#2431) Summary: Running rowwise scaling on fp32 tensors got the error, P1794222725 ``` RuntimeError: Only bf16 high precision output types are supported for row-wise scaling. ``` This pr adds an option to explicitly use bfloat16 as the output of rowwise_scaled, and cast it back to the original precision. It can be enabled by setting ``` config = dataclasses.replace(config, convert_dtypes_for_rowwise_scaled_mm=True) ``` Differential Revision: D73552660
|
This pull request was exported from Phabricator. Differential Revision: D73552660 |
y-sq
added a commit
to y-sq/ao
that referenced
this pull request
Jul 3, 2025
…ytorch#2431) Summary: Running rowwise scaling on fp32 tensors got the error, P1794222725 ``` RuntimeError: Only bf16 high precision output types are supported for row-wise scaling. ``` This pr adds an option to explicitly use bfloat16 as the output of rowwise_scaled, and cast it back to the original precision. It can be enabled by setting ``` config = dataclasses.replace(config, convert_dtypes_for_rowwise_scaled_mm=True) ``` Differential Revision: D73552660
|
This pull request was exported from Phabricator. Differential Revision: D73552660 |
y-sq
added a commit
to y-sq/ao
that referenced
this pull request
Jul 3, 2025
…ytorch#2431) Summary: Running rowwise scaling on fp32 tensors got the error, P1794222725 ``` RuntimeError: Only bf16 high precision output types are supported for row-wise scaling. ``` This pr adds an option to explicitly use bfloat16 as the output of rowwise_scaled, and cast it back to the original precision. It can be enabled by setting ``` config = dataclasses.replace(config, convert_dtypes_for_rowwise_scaled_mm=True) ``` Differential Revision: D73552660
y-sq
force-pushed
the
export-D73552660
branch
2 times, most recently
from
abbed3a to
303d6a6
Compare
y-sq
added a commit
to y-sq/ao
that referenced
this pull request
Jul 3, 2025
…ytorch#2431) Summary: Running rowwise scaling on fp32 tensors got the error, P1794222725 ``` RuntimeError: Only bf16 high precision output types are supported for row-wise scaling. ``` This pr adds an option to explicitly use bfloat16 as the output of rowwise_scaled, and cast it back to the original precision. It can be enabled by setting ``` config = dataclasses.replace(config, convert_dtypes_for_rowwise_scaled_mm=True) ``` Differential Revision: D73552660
|
This pull request was exported from Phabricator. Differential Revision: D73552660 |
…ytorch#2431) Summary: Pull Request resolved: pytorch#2431 Running rowwise scaling on fp32 tensors got the error, P1794222725 ``` RuntimeError: Only bf16 high precision output types are supported for row-wise scaling. ``` This pr adds an option to explicitly use bfloat16 as the output of rowwise_scaled, and cast it back to the original precision. It can be enabled by setting ``` config = dataclasses.replace(config, convert_dtypes_for_rowwise_scaled_mm=True) ``` Differential Revision: D73552660
|
This pull request was exported from Phabricator. Differential Revision: D73552660 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Running rowwise scaling on fp32 tensors got the error, P1794222725
This pr adds an option to explicitly use bfloat16 as the output of rowwise_scaled, and cast it back to the original precision.
It can be enabled by setting
Differential Revision: D73552660