Skip to content

b11569

Pre-release
Pre-release

Choose a tag to compare

View all tags
@github-actions github-actions released this 12 Oct 13:16
b11569
073b740
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

metal : fix mul_mat_id with duplicate ids (#30373)

  • metal : fix mul_mat_id limits and kernel name

  • metal : fix mul_mat_id with duplicate ids

kernel_mul_mm_id_map0 assumed at most one match per token and a fixed
capacity of ne21 entries per expert, which produced wrong or dropped
output rows and could overflow into the slice of the next expert when
the ids contain duplicates.

build a compact per-expert list of the (token, slot) entries instead:

  • pass 1 counts the entries of every expert, thread 0 computes the
    offsets of the expert slices (tpe now holds [base, count])
  • pass 2 writes the entries with one unconditional store per token
    (conditional stores of divergent lanes are very slow), the dummy
    value of the tokens without a match lands on a padding slot of the
    slice that is never read - duplicates are handled by a rare branch
  • kernel_mul_mm_id reads the base/count of its expert and loops over
    the entry tiles that do not fit into the grid

The ids buffer is sized as ne20*ne21 + ne02 (entries + padding).

883/883 test-backend-ops MUL_MAT_ID tests pass, llama-bench is within
run-to-run variance of the previous behavior.

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD

  • metal : make mul_mat_id map0 ne02 a function constant

use FOR_UNROLL for the id loops of kernel_mul_mm_id_map0 and turn ne02
into the function constant FC_mul_mm_id_map0_ne02 (FC_MUL_MM + 7), so
that the threadgroup offsets of cnt/bas/sids and the prefix-sum bound
fold at compile time.

the pipeline name now encodes ne02 (same convention as the ssm_conv
nc/silu constants) and the map0 pipeline is compiled with the constant
value set. ne02 is no longer part of ggml_metal_kargs_mul_mm_id_map0 -
the dispatch grid still comes from the tensor ne[2] on the host.

no measurable perf change: 883/883 test-backend-ops MUL_MAT_ID tests
pass, op-level and llama-bench results are within run variance.

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD

  • cont : clean-up + TODO

  • metal : dispatch mul_mat_id entry tiles from the prepass

follow the approach of the vulkan implementation (#29998): instead of
launching a grid over experts x ne21 entry tiles and letting most of
the threadgroups early exit, kernel_mul_mm_id_map0 now also emits a
row map - the total tile count plus one (expert, first entry, valid
rows) descriptor per 32-entry tile that has work.

kernel_mul_mm_id is launched with a static upper bound on the number
of tiles (full tiles over all entries plus at most one partial tile
per expert), reads its descriptor and processes a single tile - the
tail early exits. This removes the round loop that was needed to cover
duplicate ids past the grid (register pressure: th_max 768 -> back to
the plain mul_mm level) and duplicates are now handled implicitly,
since they only produce more descriptors within the bound.

The grid shrinks by ~8-16x on typical shapes (e.g. 128 experts,
8 used, 512 tokens: 128 tiles instead of 2048 entry tiles, of which
94% only launched to exit) and ne21 is no longer needed by the mm
kernel kargs.

883/883 test-backend-ops MUL_MAT_ID tests pass. llama-bench
(Qwen3.6-35B-A3B Q8_0) is now faster than master on all pp sizes
(pp2048/4096/8192 +2%, pp512 +0.5%), tg32 unchanged.

Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD

  • cont : add smem assert

Website:

Attestations:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI:

Assets 38
Loading