[Bug] Qwen3.5-35B-A3B (Hybrid Mamba+MoE) fails with CPU offload due to Mamba conv_states dtype mismatch

[JasonNing96]

[Bug] Qwen3.5-35B-A3B (Hybrid Mamba+MoE) fails with CPU offload due to Mamba conv_states dtype mismatch

Summary

When running Qwen3.5-35B-A3B-GPTQ-Int4 model with CPU offload on a 16GB GPU, the server starts successfully but inference fails with a dtype mismatch error in the Mamba causal_conv1d_fwd kernel. This appears to be a compatibility issue between SGLang's CPU offload mechanism and hybrid linear attention (Mamba) models.

Environment

• GPU: NVIDIA RTX 5070 Ti, 16GB VRAM (Blackwell architecture)
• OS: Linux 6.17.0-14-generic
• Python: 3.11
• SGLang: v0.5.9 (pip installed)
• Model: Qwen3.5-35B-A3B-GPTQ-Int4 (~23.8GB)
• CUDA: 12.x
• PyTorch: Latest stable

Reproduction

Step 1: Launch server with CPU offload

python -m sglang.launch_server \
    --model-path /path/to/Qwen3.5-35B-A3B-GPTQ-Int4 \
    --quantization gptq_marlin \
    --dtype float16 \
    --attention-backend triton \
    --cpu-offload-gb 12 \
    --offload-group-size 1 \
    --mem-fraction-static 0.80 \
    --kv-cache-dtype fp8_e4m3 \
    --disable-cuda-graph \
    --skip-server-warmup \
    --trust-remote-code \
    --port 30000

Step 2: Send a test request

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "default",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 50
  }'

Error

The server starts successfully and loads the model, but when processing inference requests:

RuntimeError: Expected conv_states_.scalar_type() == input_type to be true, but got false.  (Could this error message be improved?  If so, please report an error request to PyTorch at https://github.com/pytorch/pytorch/issues/new?assignees=&labels=&template=report-error-message.yaml&title=Report+an+improvement+for+RuntimeError%3A+Expected+conv_states_.scalar_type%28%29+%3D%3D+input_type+to+be+true)

This error occurs in causal_conv1d_fwd kernel, indicating that the conv_states tensor has a different dtype than the input.

Root Cause Analysis

The Qwen3.5-35B-A3B model uses a hybrid architecture combining Mamba (linear attention) and MoE layers. When CPU offload is enabled:

1. Model parameters are moved between CPU and GPU during inference
2. The Mamba state management (conv_states) appears to have its dtype changed during this process
3. The causal_conv1d_fwd kernel expects conv_states to match the input dtype, but they differ

This is a deep compatibility issue between:

• SGLang's CPU offload mechanism (offloader.py)
• Mamba/hybrid linear attention state management

Workarounds Attempted

1. ✅ Fixed: tie_weights issue

Error:

ValueError: functional_call got multiple values for keys
['linear_attn.A_log', 'linear_attn.attn.A_log'], which are tied.

Fix: Added tie_weights=False to functional_call in offloader.py:

# Line 144
output = functional_call(module, device_state, args=args, kwargs=kwargs, tie_weights=False)

# Lines 264-267
output = functional_call(
    module, get_parameter_and_buffer_dicts(), args=args, kwargs=kwargs,
    tie_weights=False,
)

2. ❌ Not Fixed: Mamba conv_states dtype mismatch

Tried the following without success:

• Using bfloat16 instead of float16
• Different --cpu-offload-gb values (10, 12, 14)
• Different --mem-fraction-static values
• With/without --kv-cache-dtype fp8_e4m3
• With/without --disable-cuda-graph

None of these workarounds resolve the underlying dtype mismatch in Mamba state management.

Related Issues and PRs

• Issue [Bug] Qwen3.5 dtype half occur error #19773: Qwen3.5 dtype half occur error
• Issue [Bug] GPTQ-Int4 Hybrid (Mamba+MoE) model crashes with AssertionError: moe_wna16_marlin_gemm assumes hidden_states.dtype == w1_scale.dtype #20599: GPTQ-Int4 Hybrid (Mamba+MoE) model crash
• PR [Bug Fix] Fix mamba conv_states dtype mismatch when using --dtype flo… #20217: Fix mamba conv_states dtype mismatch (potential fix in progress)

Proposed Solution

The fix likely needs to ensure that Mamba state tensors (conv_states) maintain consistent dtype handling during CPU offload operations. This may require changes to:

1. sglang/srt/utils/offloader.py - CPU offload logic
2. Mamba state management code - dtype preservation during state transfers
3. causal_conv1d kernel integration - proper dtype casting

Impact

This issue prevents users with limited GPU memory (16GB) from running large hybrid Mamba+MoE models like Qwen3.5-35B-A3B, even with CPU offload feature enabled.

Additional Context

Model Architecture

Qwen3.5-35B-A3B is a hybrid model combining:

• Mamba blocks (linear attention) - efficient long-context handling
• MoE (Mixture of Experts) - sparse activation for efficiency
• Sliding window attention - for local context

This architecture is increasingly common in modern LLMs for efficiency, so fixing this issue would benefit multiple models.

Test Logs

Server startup log shows successful model loading:

[offloader] offload module_index=X submodule=<class 'X'> params=[...] memory_allocated=...

Error occurs only during inference when Mamba conv1d kernel is invoked.

[marcorobiati]

The same happens to me without cpu offloading

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.