Skip to content

b10730

Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 01 Sep 05:08
09412af

qwen4exp: sum the indexer heads by slices (#28023)

  • qwen4exp: sum the indexer heads by slices

The head reduction went through a transpose and a sum_rows over ne[1],
which left sum_rows with ne0 = 4, one block per row for a four element
reduction, and the transpose copied the whole block by token surface
twice on the way in.

The heads are adjacent on ne[1], so each one is a strided view and the
sum is a short chain of adds.

RTX PRO 6000, Qwen3.8-Flash-Next UD-Q4_K_XL, fa on, 55k context, warm
runs on top of #28011:

prompt processing 2170 -> 2366 t/s

Generation is unaffected. The removed work scales with n_blocks by
n_tokens, so the gain grows with context and with ubatch size.

  • qwen4exp: drop the redundant cont on the indexer query

rope returns a freshly allocated, contiguous tensor, so the reshape that
feeds the matmul does not need a copy. ggml_reshape_3d asserts
contiguity, so a layout that would need the cont cannot slip through
silently.

Greedy output is unchanged token for token.

Address review from @ggerganov

Website:

Attestations:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI: