Finegrained FP8 v4: fused grouepd/batched by IlyasMoutawwakil · Pull Request #46957 · huggingface/transformers

IlyasMoutawwakil · 2026-06-29T10:25:04Z

What does this PR do?

Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
we ask that new users do not submit pure code agent PRs at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read CONTRIBUTING.md.

I confirm that this is not a pure code agent PR.

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline and the
Pull Request checks?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes according to the guidelines?
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

HuggingFaceDocBuilderDev · 2026-06-29T10:45:25Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Copilot

Pull request overview

This PR updates the finegrained-fp8 kernel integration to v4 and extends FP8 MoE support with new fused grouped/batched expert dispatch paths, while also refining decode-time experts implementation switching based on batch size.

Changes:

Bump the finegrained-fp8 kernel mapping to version 4 and extend the Python wrapper to expose new v4 entry points (including fused MoE ops).
Add new fused MoE expert forward implementations (fused_grouped_mm, fused_batched_mm) and register them in the experts interface mapping.
Update generation decode optimization to conditionally switch *_grouped_mm -> *_batched_mm only for small batches (<= 32), and update call sites to pass batch_size.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
src/transformers/models/higgs_audio_v2/generation_higgs_audio_v2.py	Passes `batch_size` into the updated `_optimize_model_for_decode` context manager.
src/transformers/integrations/hub_kernels.py	Bumps `finegrained-fp8` kernel mapping version from 3 → 4.
src/transformers/integrations/finegrained_fp8.py	Updates kernel API bindings to v4 naming and adds fused MoE expert dispatch implementations.
src/transformers/generation/utils.py	Makes decode-time experts switching batch-size aware and updates `_sample()` to pass `batch_size`.

vasqu

Some smaller comments but looks overall good!

vasqu · 2026-06-29T20:18:51Z

+                "any issues with this, please open an issue on the Hugging Face Transformers GitHub repository.",
            )
-            self.set_experts_implementation("batched_mm")
+            self.set_experts_implementation(decode_experts_implementation)


It looks a bit weird with the variables now, can we add a nudge comment that this is to note for fp8

its more general where each grouped has a batched alternative

Yep yep, just a small comment to note this is to cover all grouped mm etc cases including fp8

vasqu · 2026-06-29T20:21:04Z

+        )
+
+    finegrained_fp8 = load_finegrained_fp8_kernel()
+    fused_op = getattr(finegrained_fp8, op_name)


feels a bit unsafe :D

vasqu · 2026-06-29T20:21:27Z

+        raise NotImplementedError(
+            f"{op_name} experts dispatch requires gated experts (a combined gate_up projection)."
+        )
+    if self.swiglu_alpha is not None or self.swiglu_limit is not None or self.limit is not None:


hmm didnt we ignore it for the base grouped mm etc? Why make an exception here then

no we didn't actually ignore it in base grouped, we normally have _apply_gate which applies these in between grouped gemms, i guess you mean sonicmoe or some other backend.

imo better have an error that nudges users to open a pr in kernels-community rather than wrong outputs

Ah oops yes it was sonicmoe I think. But yes agree, probably better to error out early

Should we introduce something similar for sonicmoe then?

yess will do

ArthurZucker

do you have any bench you can publish in this PR desc before merging?

IlyasMoutawwakil · 2026-07-01T07:03:31Z

was actually making them when i realized m3 has swiglu alpha/limlit 😅 publishing after kernels pr merge

github-actions · 2026-07-01T16:57:46Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: deepseek_v4, gpt_oss, higgs_audio_v2, minimax_m3_vl, openai_privacy_filter, finegrained_fp8

github-actions · 2026-07-01T17:17:57Z

CI recap

Dashboard: View test results in Grafana
Latest run: 28534097273:1
Result: success | Jobs: 13 | Tests: 46,605 | Failures: 0 | Duration: 18h 24m

IlyasMoutawwakil · 2026-07-02T08:12:13Z

did some changes, along deprecating alpha and limit

vasqu

Just some smaller comments but overall LGTM

vasqu · 2026-07-02T13:54:09Z



-@functools.cache
+_DEEPGEMM: DeepGEMM | None = None


Ah it was still loading multiple times? lazy load kernels should pretty much interact as cache already. Maybe we should rather cache that we are compatible device wise etc

no no it was loading once, it just thrashes the stdout with dynamo warnings about it

vasqu · 2026-07-02T13:56:02Z

+    matmul_batched: Callable
+    matmul_grouped: Callable
+    moe_fused_grouped: Callable
+    moe_fused_batched: Callable


This is kind of public facing, should we add a 🚨 just in case? Usually people shouldnt use it but you never know

vasqu · 2026-07-02T13:56:52Z

+    Cached in the module-global ``_FINEGRAINED_FP8`` (populated on first call) instead of via
+    ``functools.cache``: Dynamo traces through the lru wrapper and warns on every compiled call,
+    whereas a plain global check/return traces cleanly.


Ah this explanation was missing in deepgemm

vasqu · 2026-07-02T13:58:06Z

        self.swiglu_alpha = getattr(config, "swiglu_alpha", None)
        self.swiglu_limit = getattr(config, "swiglu_limit", None)
-        self.act_fn = ACT2FN[_first_attr(config, "hidden_activation", "hidden_act")]
-        self.limit = getattr(config, "swiglu_limit", None)


need to deprecate here as well no?

vasqu · 2026-07-02T13:58:59Z

        raise ValueError("sonicmoe requires gated experts (has_gate=True)")
    if hidden_states.device.type != "cuda":
        raise ValueError("sonicmoe requires CUDA device")
+    if getattr(self, "swiglu_alpha", None) is not None or getattr(self, "swiglu_limit", None) is not None:


Yes thanks 🙏

vasqu · 2026-07-02T14:01:10Z

+            # Match a full module path or a trailing component (a bare name given under a submodel);
+            # keep the renamed entry as-is if it names no module (a regex / leaf-param pattern).
+            matches = [m for m in module_names if m == entry or m.endswith(f".{entry}")]
+            remapped.extend(matches or [entry])


hmm interesting, where did you encounter this?

m3 mxfp8 config has vision_tower without the "model." prefix, not sure whats the best way to handle them tho

IlyasMoutawwakil added 2 commits June 29, 2026 03:15

v4

43ce463

rename

0314297

IlyasMoutawwakil requested review from ArthurZucker, Copilot and vasqu and removed request for vasqu June 29, 2026 17:35

IlyasMoutawwakil marked this pull request as ready for review June 29, 2026 17:35

Copilot started reviewing on behalf of IlyasMoutawwakil June 29, 2026 17:35 View session

Copilot AI reviewed Jun 29, 2026

View reviewed changes

Comment thread src/transformers/integrations/finegrained_fp8.py

vasqu approved these changes Jun 29, 2026

View reviewed changes

ArthurZucker approved these changes Jul 1, 2026

View reviewed changes

IlyasMoutawwakil added 6 commits July 1, 2026 02:43

support swiglu stuff

821acbe

Merge branch 'main' into finegrained-fp8-v4

08a6ed9

scale fmt with mxfp8

a26c2db

honor the dtype

555123f

fix vision tower quantization

3b806ca

avoid cache to avoid dynamo thrashing warnings

6009484

IlyasMoutawwakil requested review from ArthurZucker and vasqu July 2, 2026 08:11

vasqu approved these changes Jul 2, 2026

View reviewed changes

vasqu mentioned this pull request Jul 8, 2026

Implement VibeVoice #40546

Open

Uh oh!

Conversation

IlyasMoutawwakil commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Code Agent Policy

Before submitting

Who can review?

Uh oh!

HuggingFaceDocBuilderDev commented Jun 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

IlyasMoutawwakil commented Jul 1, 2026

Uh oh!

github-actions Bot commented Jul 1, 2026

Uh oh!

github-actions Bot commented Jul 1, 2026

CI recap

Uh oh!

IlyasMoutawwakil commented Jul 2, 2026

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

IlyasMoutawwakil commented Jun 29, 2026 •

edited

Loading