Skip to content

fix(trtllm_mha): clamp page_table to k_cache page range to prevent SWA crash - #5

Open
pyc96 wants to merge 1 commit into
pyc/sota-gemma4-mtp-trtllm-mha-debugfrom
pyc/sota-gemma4-mtp-trtllm-swa-windowed
Open

fix(trtllm_mha): clamp page_table to k_cache page range to prevent SWA crash#5
pyc96 wants to merge 1 commit into
pyc/sota-gemma4-mtp-trtllm-mha-debugfrom
pyc/sota-gemma4-mtp-trtllm-swa-windowed

Conversation

@pyc96

@pyc96 pyc96 commented May 23, 2026

Copy link
Copy Markdown
Owner

Summary

Defensive clamp that prevents the deterministic CUDA Warp Illegal
Address crash in fmhaSm100fKernel_*SlidingOrChunkedCausal* when
running Gemma-4 + --attention-backend trtllm_mha + MTP +
summarization workloads.

Stacked on #3 (debug trap). Staged on pyc96/sglang only.

What this fixes

Without this PR, the same workload that the bounds trap (#3)
catches in 30 s on E4B crashes the SGLang server with
cudaErrorIllegalAddress and SIGQUIT. With this PR, the same
workload completes cleanly.

How

In trtllm_mha_backend.py::forward_decode and forward_extend, right
after _get_layer_page_table and before the flashinfer kernel call,
clamp every page index to [0, k_cache.shape[0] - 1].

page_table = self._get_layer_page_table(layer, forward_batch)
num_pages_in_cache = k_cache.shape[0]
if num_pages_in_cache > 0:
    page_table = page_table.clamp(min=0, max=num_pages_in_cache - 1)

Three lines per call site, two call sites.

Benchmark — Gemma-4-E4B-IT, trtllm_mha, MTP, summarization 8 k/1 k × 80

metric before this PR after this PR
outcome CRASH ~30 s in COMPLETED
peak output token throughput n/a 4,032 tok/s
median TTFT n/a 1,437 ms
median TPOT n/a 12.16 ms
accept length n/a 1.61

Benchmark — Gemma-4-26B-A4B-IT, trtllm_mha, MTP, summarization 8 k/1 k × 80

metric triton + Patch 1+2 (best safe baseline) trtllm_mha + this PR Δ
output throughput 1,097 tok/s 1,832 tok/s +67 %
median TPOT 37.87 ms 24.97 ms −34 %
median TTFT 8,763 ms 2,887 ms −67 %
accept length 2.76 1.69 −39 % (see limitation below)

This unlocks the trtllm_mha attention backend for Gemma-4 MTP, which is
otherwise unusable.

Quality — MMLU @ 500 questions (Gemma-4-26B-A4B-IT, seed 0, temp 0)

server accuracy
Patch 2 baseline (triton + MTP) 0.706
trtllm_mha + this PR 0.718
vLLM nightly (for comparison) 0.710

Within MMLU sampling noise; no regression.

Known limitation

Accept length drops from 2.76 (triton) → 1.69 (trtllm_mha + this PR)
on 26B summarization. Investigation:

  • The clamp replaces OOB page indices with the LAST valid SWA page.
  • For positions in the sliding-window range, the kernel will use that
    page's K/V instead of the correct one, producing slightly wrong
    attention values, which lowers MTP draft acceptance.
  • This is a defensive safety net, not a complete fix. The
    underlying off-by-one in either full_to_swa_index_mapping or the
    SWA paged allocator's edge cases needs upstream investigation
    (filed as Patch E in humanize/source-idea-ledger.md).

For workloads where the lower acceptance is acceptable (the ~50 %
throughput improvement still significantly beats the triton baseline),
this fix is a net win.

Cost

One clamp() per kernel call. Few microseconds per forward. No
measurable performance impact.

Tests

No new unit test — the test is the reproducer:

agent-pad/runs/.../crash_repro/repro_e4b_bounds.sh         # used to crash
agent-pad/runs/.../crash_repro/repro_e4b_trtllm_eager_fix.sh   # used to crash, also fixed

Both reproduce no-crash behavior with this PR applied.


CI States

Latest PR Test (Base): ❌ Missing run-ci label -- add it to run CI tests.
Latest PR Test (Extra): ❌ Blocked -- run-ci is required first.

…A crash

Prevents the deterministic CUDA Warp Illegal Address crash in
'fmhaSm100fKernel_*SlidingOrChunkedCausal*' that triggers under
Gemma-4 + --attention-backend trtllm_mha + MTP + summarization
workloads at ~85% SWA pool utilization (see
crash_repro/TRIAGE_REPORT.md).

Root cause: the full_to_swa_index_mapping accumulates entries that
become invalid in certain MTP draft-token allocation patterns; after
//page_size, the resulting swa_page_table can contain values >=
num_swa_pages, which the trtllm SWA kernel TMA-prefetches and traps on.

Fix: clamp page_table values to [0, k_cache.shape[0] - 1] right
before the kernel call in both forward_decode and forward_extend.
Applies to BOTH the regular page_table and swa_page_table paths.

Verification on Gemma-4-E4B-IT + trtllm_mha + MTP + summarization
(8 k/1 k x 80 prompts, max_concurrency=64):
  before this fix: CRASH at ~85% SWA fill, ~30 s into bench
  after this fix:  COMPLETED, output 4032 tok/s peak, no trap events

Verification on Gemma-4-26B-A4B-IT + trtllm_mha + MTP + summarization
(8 k/1 k x 80 prompts, max_concurrency=64):
  before: CRASH (same kernel, same SWA fill trigger)
  after:  COMPLETED, output 1832 tok/s peak (vs Patch 1+2 triton
          1097 tok/s = +67%), TPOT 25 ms (vs triton 38 ms = -34%),
          TTFT 2.9 s (vs triton 8.8 s = -67%)

MMLU @ 500 questions on 26B with this fix: 0.718 (vs Patch 2 baseline
0.706, vLLM 0.710) -- within noise, no regression.

KNOWN LIMITATION: accept length drops vs triton backend (1.69 vs 2.76
on 26B summarization).  Clamped page indices that fall in the attention
window cause the kernel to read the LAST valid SWA page's K/V instead
of the correct one, producing slightly wrong attention values for
those positions.  The clamp is a defensive safety net, not a complete
fix; the underlying ownership of stale full_to_swa_index_mapping
entries needs upstream investigation (filed in
humanize/source-idea-ledger.md as Patch E).  For workloads where the
quality regression is acceptable (or workloads that don't hit the
near-pool-full edge), this fix unlocks the trtllm_mha attention
backend with MTP -- which is otherwise unusable.

Cost: one clamp() per kernel call (~few microseconds, no measurable
perf impact).

See crash_repro/TRIAGE_REPORT.md.

Co-authored-by: Claude
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant