Gemma 4 E2B/E4B audio produces gibberish — two issues found

[fredchu]

Summary

Gemma 4 E2B/E4B audio input produces gibberish output for all model variants (bf16, 8bit, 4bit). Two independent issues were found:

1. All mlx-community Gemma 4 E2B/E4B models are missing feature_extractor in processor_config.json — this causes Gemma4Processor.feature_extractor = None and audio is silently ignored.
2. Even after patching feature_extractor, audio output is garbage — the audio tower loads correctly but embeddings don't reach the language model properly.

Environment

• mlx-vlm: 0.4.3 (also tested with main @ 43b9b20, includes Fix Gemma 4 embedding scaling #893 and Fix Gemma 4 chunked prefill for KV-shared models and thinking  #901)
• macOS, M1 Max 32GB
• transformers: 5.5.0
• Audio: 28s WAV, 16kHz mono PCM16

Issue 1: Missing feature_extractor in processor_config.json

The original Google model (google/gemma-4-e2b-it) has feature_extractor with Gemma4AudioFeatureExtractor config in its processor_config.json. All mlx-community converted models are missing this section:

from mlx_vlm import load
model, processor = load("mlx-community/gemma-4-e2b-it-8bit")
print(processor.feature_extractor)  # None

Workaround: manually copy the feature_extractor block from the Google model's processor_config.json into the mlx-community model's cached processor_config.json. After patching:

print(type(processor.feature_extractor))
# <class 'mlx_vlm.models.gemma4.audio_feature_extractor.Gemma4AudioFeatureExtractor'>

This affects all E2B and E4B models in the mlx-community/gemma-4 collection. The 31B and 26B models don't support audio, so they are unaffected.

Issue 2: Audio output is gibberish after patching

After patching feature_extractor, audio features are correctly produced (input_features shape (1, 2998, 128) for 28s audio), and the audio tower exists with non-zero weights. However, the model output is garbage:

Model	Output	Peak RAM
E2B-IT bf16	Multi-language gibberish: صling用のling損失mont这款ling-につき...	11.04 GB
E2B-IT 8bit	Hallucinated ML metrics: "Weight: 1.5, precision: 0.9..."	6.95 GB
E4B-IT 4bit	All <pad> tokens	6.34 GB

Tested with both temperature=0.0 and temperature=1.0, top_p=0.95, top_k=64 (as recommended in the Gemma 4 README after #901). Same garbage output.

Reproduce

from mlx_vlm import load, generate
from huggingface_hub import hf_hub_download
import json

model_id = "mlx-community/gemma-4-e2b-it-bf16"  # or any E2B/E4B variant

# Step 1: Patch processor_config.json (only needed once)
proc_path = hf_hub_download(model_id, "processor_config.json")
with open(proc_path) as f:
    config = json.load(f)
if "feature_extractor" not in config:
    config["feature_extractor"] = {
        "feature_extractor_type": "Gemma4AudioFeatureExtractor",
        "feature_size": 128, "fft_length": 512, "fft_overdrive": False,
        "frame_length": 320, "hop_length": 160, "sampling_rate": 16000,
        "max_frequency": 8000.0, "min_frequency": 0.0, "mel_floor": 0.001,
        "dither": 0.0, "input_scale_factor": 1.0, "padding_side": "right",
        "padding_value": 0.0, "preemphasis": 0.0, "preemphasis_htk_flavor": True,
        "return_attention_mask": True, "per_bin_mean": None, "per_bin_stddev": None,
    }
    config["audio_ms_per_token"] = 40
    with open(proc_path, "w") as f:
        json.dump(config, f, indent=2)

# Step 2: Run ASR
model, processor = load(model_id)
print(f"Feature extractor: {type(processor.feature_extractor)}")  # Should not be None

prompt = (
    "<bos><|turn>user\n"
    "<|audio|>Transcribe the following speech segment in its original language. "
    "Only output the transcription. Add proper punctuation.<turn|>\n"
    "<|turn>model\n"
)

output = generate(
    model, processor, prompt,
    audio=["path/to/any_audio.wav"],  # any 16kHz WAV, <30s
    max_tokens=200, temperature=1.0, top_p=0.95,
)
print(output.text)  # Expect transcription, get gibberish

Diagnosis

• processor.feature_extractor is None without patching → audio silently skipped
• After patching, input_features shape is correct (mel spectrogram)
• audio_tower has 12 conformer layers, output_proj weight shape (1536, 1024), mean≈0, std≈0.031 — looks loaded correctly
• audio_token_id = 258881 is present in input_ids (750 audio tokens for 30s audio)
• masked_scatter in get_input_embeddings() should replace audio token positions with audio encoder output, but the final output suggests audio embeddings are not reaching the language model correctly

Suggested fixes

1. Issue 1: Update the convert command or mlx-community model cards to include feature_extractor in processor_config.json for E2B/E4B models.
2. Issue 2: May need debugging in Model.get_input_embeddings() → audio_tower() → embed_audio() → masked_scatter() path.

[Blaizzy]

Hey thanks for sharing!

This issue is fixed on the main branch and I will make a patch release today

[FakeRocket543]

I ran into the same issues and found 4 bugs total in mlx-vlm's Gemma 4 support. #893 fixed bug 1,
but 3 remain:

1. ScaledLinear inherits nn.Module instead of nn.Linear — quantizer can't find these
   layers (fixed in Fix Gemma 4 embedding scaling #893)
2. PLE layers get quantized — nn.quantize() has no predicate to skip
   PLE/ScaledLinear/vision/audio layers → garbage output
3. processor.save_pretrained() strips feature_extractor from config — audio silently
   dropped, model says "no audio provided"
4. SwitchLinear (MoE experts) not quantized — 26B-A4B ends up 49 GB instead of 16 GB

I've published PLE-safe quantized weights for all 12 variants (4 models × 3 precisions), fully
validated on vision + audio + text:
https://github.com/FakeRocket543/mlx-gemma4

Patches are in mlx_vlm_patches/. Happy to open a PR if helpful.

[Blaizzy]

Hey @FakeRocket543

Thanks for reaching out!

PLE layers get quantized — nn.quantize() has no predicate to skip
PLE/ScaledLinear/vision/audio layers → garbage output

This not true, we have a predicate to skip vision and audio layer. And scaledLinear doesn't get quantized because it doesn't have to_quantize. You can inspect the weights.

processor.save_pretrained() strips feature_extractor from config — audio silently
dropped, model says "no audio provided"

True, thanks! Fixed in #906

SwitchLinear (MoE experts) not quantized — 26B-A4B ends up 49 GB instead of 16 GB

This is not true. Here is ane example of a model I personally converted and uploaded.
https://huggingface.co/mlx-community/gemma-4-26b-a4b-5bit/tree/main