Improve API for f4f4bf16 #5163
Closed
Improve API for f4f4bf16 #5163
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
cthi
force-pushed
the
export-D87655845
branch
from
a3644a2 to
c7c8473
Compare
Summary: Pull Request resolved: pytorch#5163 X-link: https://github.com/facebookresearch/FBGEMM/pull/2162 We add some improvements for FP4 gemm. - Remove the need to pass `use_mx`, we can infer this based on `global_scale` - As a follow up we should improve the assertions on the proper FP4 dtypes, similar to what we have with [FP4 group gemm](https://www.internalfb.com/code/fbsource/[addad803d330]/fbcode/deeplearning/fbgemm/fbgemm_gpu/experimental/gen_ai/src/quantize/cutlass_extensions/f4f4bf16_grouped.cu?lines=388-409). - Add optional `output` to API, which is in-line with other torch APIs. - Move function declaration to `torch_ops.h` which will remove the need for the forward declaration in Blas.cpp - Small code cleans ups Misc - Later we should likely clean-up & re-evaluate the heuristic for the kernel, right now its almost identical (and duplicated) for MX and NV FP4, and we are likely instantiating more instances than needed. Reviewed By: slayton58 Differential Revision: D87655845
cthi
force-pushed
the
export-D87655845
branch
from
c7c8473 to
4ba9997
Compare
Contributor
|
This pull request has been merged in 903002a. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
We add some improvements for FP4 gemm.
use_mx, we can infer this based onglobal_scaleoutputto API, which is in-line with other torch APIs.Misc
Differential Revision: D87655845