Skip to content

Make get_qmv_batch_limit consider which vector kernel it guards - #3987

Open
katlun-lgtm wants to merge 2 commits into
ml-explore:mainfrom
katlun-lgtm:feat/qmv-batch-limit-mode
Open

Make get_qmv_batch_limit consider which vector kernel it guards#3987
katlun-lgtm wants to merge 2 commits into
ml-explore:mainfrom
katlun-lgtm:feat/qmv-batch-limit-mode

Conversation

@katlun-lgtm

Copy link
Copy Markdown
Contributor

Proposed changes

Follow-up from #3863 (@zcbenz: "get_qmv_batch_limit does not consider use_qmv_wide").

The limit picks the vector→matmul crossover from (D, O) and the architecture alone, while the vector kernel it guards differs by quantization mode (use_qmv_wide: fp modes everywhere, affine on gen 15+, plain qmv otherwise). This passes the mode in and retunes the one cell I can measure — affine with qmv_wide active, gen-15 non-d — leaving fp modes, the gen ≤ 14 tables, and the d tiers unchanged for measurements on other hardware.

Measured crossovers (M3 Max g15s, both routings forced at the same M via a local env-override build, cold weights round-robin so nothing stays cache-resident, min of 8 trials × 30 reps, M ∈ 3–20, 4-bit gs 128, bf16 activations):

shape D·O old limit measured crossover new limit
2048×2048 4.2M 18 17 17
2048×4096 8.4M 12 16 16
3072×4096 12.6M 12 13 13
4096×4096 16.8M 12 13 13
3072×8192 25.2M 10 13 13
4096×8192 33.6M 10 13 13
2048×16384 33.6M 10 13 13
4096×14336 58.7M 10 13 13
5120×17408 89.1M 10 13 13

The crossover tracks D·O for affine — 4096×8192 and 2048×16384 (same D·O) cross at the same M — and settles on a floor of 13 that holds across a 5× range of sizes. Below it qmv_wide's margin is large (1.2–3.5× at M ≤ 10 on these shapes); above it qmm pulls ahead steadily (1.5–1.7× by M=20).

End-to-end effect (default routing, this branch vs 0.32.0, same harness):

shape class M=10 M=11–12 elsewhere
≥ 25M elements (4 shapes) 1.22–1.25× 1.04–1.08× parity
4096×4096 1.11× at M=12 parity
2048×2048 1.10× at M=17, parity else

No cell regresses beyond ±2% noise (full sweep M ∈ 3–20 × 7 shapes).

I ran mxfp4 through the same harness: its crossover is K-dependent rather than D·O-dependent (4096×8192 crosses at 11 where 2048×16384 crosses at 13), so fp wants its own cell rather than inheriting this one — data below if useful.

mxfp4 crossovers (same harness)

2048×2048: 17 · 4096×4096: 13 · 3072×8192: 13 · 4096×8192: 11 · 2048×16384: 13 · 4096×14336: 11 · 5120×17408: 11

One correction to the sweep I posted on #3863: it put the large-shape crossover near 7. This finer forced-routing sweep on current main puts the floor at 13; the earlier number was inconsistent with its own native-qmv_wide measurement at M=10 (0.49 ms measured, vs the ~0.72 ms at M=9 the crossover claim implied). The table above supersedes it.

test_quantized.py passes in full (32 tests, 2896 subtests). Routing-only change — both kernels are already covered by those tests.

Checklist

Put an x in the boxes that apply.

  • I have read the CONTRIBUTING document
  • I have run pre-commit run --all-files to format my code / installed pre-commit prior to committing changes (clang-format v21.1.8 as pinned)
  • I have added tests that prove my fix is effective or that my feature works (routing-only change; existing test_quantized.py covers both kernels)
  • I have updated the necessary documentation (if needed)

The qmv batch limit picked the matmul crossover from (D, O) and the
architecture alone, while the vector kernel it protects differs by
quantization mode (use_qmv_wide: fp modes everywhere, affine on gen 15+,
plain qmv otherwise) and the two scale very differently in M.

Give the limit the quantization mode and retune the one cell measured so
far — affine with qmv_wide active on gen-15 non-desktop parts — where the
measured crossover tracks D * O rather than the dims separately: identical
D * O shapes (4096x4096 and 2048x8192) cross at the same M. fp modes, the
gen <= 14 tables, and the desktop tiers are unchanged pending measurements
on those paths.
2048x2048 crosses at 17, 2048x4096 at 16, and everything from ~12M
elements up (3072x4096 through 5120x17408, six shapes) crosses at 13.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant