[AudioFlamingo3] Fix bfloat16 dtype mismatch in audio encoder positional embedding

[snkii]

What does this PR do?

Fixes a RuntimeError when loading AudioFlamingo3ForConditionalGeneration with torch_dtype=torch.bfloat16.

AudioFlamingo3ForConditionalGeneration inherits _keep_in_fp32_modules_strict = ["embed_positions"] from VoxtralForConditionalGeneration, keeping embed_positions in float32 even when the model is loaded in bfloat16. In AudioFlamingo3AudioEncoder.forward(), the positional embedding addition was missing the dtype cast that VoxtralEncoder already applies:

# Before (AudioFlamingo3) — missing cast
hidden_states = inputs_embeds + self.embed_positions.weight
# inputs_embeds: bfloat16, embed_positions.weight: float32
# → hidden_states upcast to float32
# → LayerNorm(float32 input, bfloat16 weight) → RuntimeError: expected scalar type Float but found BFloat16

# After — consistent with VoxtralEncoder.forward()
hidden_states = (inputs_embeds + self.embed_positions.weight).to(inputs_embeds.dtype)

Verification

MMAU-test (9,000 samples), bfloat16 + FlashAttention-2, scored via the official sonalkum/MMAU-Eval space:

	accuracy
bs=5, before fix (dtype mismatch causes audio embedding contamination across samples)	57.08%
bs=5, after fix	72.40%

Who can review?

@eustlb @vasqu

[vasqu]

I checked the original weights in one of the models

Imo, doesn't it mean that we should rather override the keep module flag instead to not have the embedding there? It should have never been casted to float.

cc @ebezzam as well

[vasqu]

run-slow: audioflamingo3

[github-actions]

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/audioflamingo3"]
quantizations: []

[github-actions]

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	756eb3d5	workflow commit (merge commit)
PR	88f842f1	branch commit (from PR)
main	f1a2a70f	base commit (on main)

⚠️ Model CI failed to report results

The test failure analysis could not be completed. Please check the workflow run for details.

[vasqu]

Run slow failed but the tests passed all 🤷 waiting for the others on a final confirmation but should be good

[ebezzam]

Thanks @snkii for spotting this! It seems like this bug snuck in with #45534 as overwriting _keep_in_fp32_modules_strict was kept with the base model here as was done for the original AudioFlamingo3ForConditionalGeneration. But the PR did not do it for the refactored AudioFlamingo3ForConditionalGeneration

[ebezzam]

@vasqu, @snkii can we check if other models that were refactor by #45534 need this fix? E.g. musicflamingo, glmasr, and vibevoice ASR come to mind as they did modular from audioflamingo3

You'll probably have to regenerate the modeling files anyway

[vasqu]

Sounds good! Best to group the fixes if possible indeed

[ebezzam]

You'll have to regenerate the modeling file, e.g. with

python utils/modular_model_converter.py \
--files-to-parse src/transformers/models/audioflamingo3/modular_audioflamingo3.py

And the modeling of other models will also need to be regenerated as they inherited from AudioFlamingo3

python utils/modular_model_converter.py \
--files-to-parse src/transformers/models/glmasr/modular_glmasr.py

python utils/modular_model_converter.py \
--files-to-parse src/transformers/models/musicflamingo/modular_musicflamingo.py

python utils/modular_model_converter.py \
--files-to-parse src/transformers/models/qwen3_asr/modular_qwen3_asr.py

[vasqu]

You can also just run make fix-repo or just python utils/modular_model_converter.py (that will run all conversions for you)

[ebezzam]

@snkii can you run python utils/modular_model_converter.py and delete this line from Qwen3 ASR modular? that will resolve the failing repo consistency check

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: audioflamingo3, glmasr, musicflamingo, qwen3_asr

[ebezzam]

run-slow: audioflamingo3, glmasr, musicflamingo, qwen3_asr

[github-actions]

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/audioflamingo3", "models/glmasr", "models/musicflamingo", "models/qwen3_asr"]
quantizations: []

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 29349983865:2
Result: success | Jobs: 5 | Tests: 918 | Failures: 1 | Duration: 1m 27s

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[github-actions]

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

• The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
• A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/
• A new high-severity security issue was detected in the changed Python files (Bandit check)

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

[github-actions]

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	8446c858	workflow commit (merge commit)
PR	b652c8a4	branch commit (from PR)
main	0f33294a	base commit (on main)

✅ No failing test specific to this PR 🎉 👏 !

[vasqu]

Thanks for the fix(es), merging now 🤗