[Bug]: ROCm ROCMAiterMLASparseMetadata missing num_decode_tokens, sparse MLA (DSA) models fail to start on MI300X on main

[haosdent]

Your current environment

Environment

vLLM:     built from main @ ef9975d021448b99a5408e8c78a4c4f6b63443c7
PyTorch:  2.11.0+gitd0c8b1f
HIP:      7.2.53211
Platform: 8x AMD MI300X (gfx942 / CDNA3), ROCm
Python:   3.12
Model:    zai-org/GLM-5.2-FP8  (model_type=glm_moe_dsa, DeepSeek-V3.2-style
                                sparse attention / lightning indexer)
Attn backend: ROCm AITER sparse MLA (vllm/v1/attention/backends/mla/rocm_aiter_mla_sparse.py)

Built for gfx942 with PYTORCH_ROCM_ARCH=gfx942 VLLM_TARGET_DEVICE=rocm python3 setup.py bdist_wheel
inside rocm/vllm-dev:ci_base-ef9975d021448b99a5408e8c78a4c4f6b63443c7.

🐛 Describe the bug

On current main, any sparse-MLA (DSA) model fails during engine startup on ROCm with:

AttributeError: 'ROCMAiterMLASparseMetadata' object has no attribute 'num_decode_tokens'.
Did you mean: 'num_actual_tokens'?

The failure happens in determine_available_memory() (the startup profiling dummy run), so the
server never becomes ready.

Traceback (abridged)

(EngineCore) File "vllm/v1/engine/core.py", line 291, in _initialize_kv_caches
(EngineCore) File "vllm/v1/executor/abstract.py", line 147, in determine_available_memory
...
(Worker_TP0) File "vllm/model_executor/models/deepseek_v2.py", line 1425, in forward
(Worker_TP0) File "vllm/compilation/piecewise_backend.py", line 380, in __call__
...
(Worker_TP0) File "vllm/model_executor/layers/attention/mla_attention.py", line 1171,
                  in unified_mla_kv_cache_update
(Worker_TP0)     attn_metadata.num_decode_tokens if attn_metadata is not None else None,
(Worker_TP0) AttributeError: 'ROCMAiterMLASparseMetadata' object has no attribute 'num_decode_tokens'

RuntimeError: Engine core initialization failed. See root cause above.

Root cause

unified_mla_kv_cache_update reads attn_metadata.num_decode_tokens unconditionally
(vllm/model_executor/layers/attention/mla_attention.py:1171), in order to pass it to
maybe_gather_mla_latent_cache_inputs():

kv_c_normed, k_pe, layer_slot_mapping = maybe_gather_mla_latent_cache_inputs(
    kv_c_normed,
    k_pe,
    layer_slot_mapping,
    attn_metadata.num_decode_tokens if attn_metadata is not None else None,   # <-- here
    attn_layer.use_pcp,
)

This read was introduced by b6ff8a2 ("[Core] Add MRV2 virtual-batch PCP for MLA", #46570).
That PR did not touch vllm/v1/attention/backends/mla/rocm_aiter_mla_sparse.py (0 hits in its diff).

All three CUDA sparse-MLA backends already declare the field, so they are unaffected:

backend	declaration
flashmla_sparse.py	num_decode_tokens: int = 0 (L162)
flashinfer_mla_sparse.py	num_decode_tokens: int (L256)
flashattn_mla_sparse.py	num_decode_tokens: int = 0 (L134)
rocm_aiter_mla_sparse.py	absent (0 occurrences in the file)

Note that the value itself is only consumed when PCP is enabled
(vllm/model_executor/layers/attention/pcp.py:48 returns its inputs unchanged when
not use_pcp or num_decode_tokens is None), so for a non-PCP ROCm deployment the attribute
merely has to exist. But exist it must, and it should be populated correctly so that PCP is
not silently wrong if it is ever enabled on ROCm.

Likely follow-on gaps (static analysis, not yet confirmed at runtime)

Startup dies at the first missing attribute, so I could not empirically reach the rest.
Comparing every attn_metadata.<attr> access in mla_attention.py against the fields
ROCMAiterMLASparseMetadata actually declares, these are also missing and look reachable:

attribute	access site	guard
num_decodes, num_prefills, num_decode_tokens	MLAAttention.forward_impl L757	unconditional assert
prefill_max_seq_len	L765	if self.impl.is_sparse and num_mha_tokens > 0
prefill (MLACommonPrefillMetadata, uses .chunked_context, .flatten)	L765/L780	same sparse-prefill branch
decode (MLACommonDecodeMetadata)	L891	assert attn_metadata.decode is not None

ROCMAiterMLASparseMetadata never constructs the nested MLACommonPrefillMetadata /
MLACommonDecodeMetadata objects at all, so fully re-syncing this backend with the shared
MLA attention layer looks like more than a one-field fix.

(Caveat: prefill/decode/prefill_max_seq_len are behind branches I have not verified are
taken for this configuration, so please treat that table as a lead, not a confirmed list. Only the
num_decode_tokens failure above is reproduced.)

Suggested fix

Minimum, to unbreak startup: declare the fields and populate them the same way the CUDA
backends do:

# in ROCMAiterMLASparseMetadata
num_decode_tokens: int = 0
num_prefill_tokens: int = 0

# in ROCMAiterMLASparseMetadataBuilder.build(), before constructing the metadata
from vllm.v1.attention.backends.utils import split_decodes_and_prefills

(_, _, num_decode_tokens, num_prefill_tokens) = split_decodes_and_prefills(
    common_attn_metadata,
    decode_threshold=self.reorder_batch_threshold,
)

A more complete fix would also provide num_decodes / num_prefills / prefill_max_seq_len
and the nested prefill/decode metadata, so the ROCm sparse backend satisfies the same interface
the shared MLA layer expects from the CUDA ones.

It would also help to have CI coverage for a sparse-MLA/DSA model on ROCm, since this path appears to
have no upstream coverage, which is presumably why the PCP change could land without noticing it.

Reproduction

vllm serve zai-org/GLM-5.2-FP8 \
  --kv-cache-dtype fp8_e4m3 --tensor-parallel-size 8 \
  --linear-backend aiter --moe-backend aiter \
  --max-model-len 204800 -O3
# env: VLLM_ROCM_USE_AITER=1

Fails during engine init with the AttributeError above.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the docs chatbot to help.

[github-actions]

CC @hongxiayang @tjtanaa @vllmellm @giuseppegrossi for ROCm-related issue

[giuseppegrossi]

Hi @haosdent I was able to reproduce this issue. A recently merged PR#48886 addresses this and fixes it. Can I close this issue?

cc @hongxiayang

[haosdent]

Thanks @giuseppegrossi !

[dvellon]

Hit this exact crash today on the current release image, and can also
confirm the fix works — noting both because #48886 (merged 2026-07-27)
is not contained in v0.26.0 or the published ROCm image:

• Release image vllm/vllm-openai-rocm@sha256:5709fafe4712...
  (:latest, 2026-08-09), vLLM 0.26.0, 8x MI325X (gfx942),
  zai-org/GLM-5.2-FP8, TP8: identical
  AttributeError: 'ROCMAiterMLASparseMetadata' object has no attribute 'num_decode_tokens'
  on all 8 workers during warmup. Also tried VLLM_ROCM_USE_AITER=0
  (refused by the sparse indexer, as designed) and
  VLLM_ROCM_USE_AITER_MLA=0 (same AttributeError) — no working
  configuration from the release image.
• Fix confirmed: nightly-f8d03e77416bf90c49acbe50e233275722f02c4b
  (2026-08-09) serves this model on the same hardware with the full
  recipe flags, clean output.

Could #48886 / 28158b2 be backported to a 0.26.x patch release or the
image rebuilt from post-fix main? Until then GLM-5.2-FP8 is unservable
on ROCm from the published image.