metal: mlx's preferred implicit-GEMM conv kernel, plus grouped convolutions (1.2-75x) - #2551
Open
czoli1976 wants to merge 5 commits into
Open
metal: mlx's preferred implicit-GEMM conv kernel, plus grouped convolutions (1.2-75x)#2551czoli1976 wants to merge 5 commits into
czoli1976 wants to merge 5 commits into
Conversation
The Metal convolution was a direct kernel computing one output position per thread, which leaves most of the GPU idle: on this M1 Pro it runs a 56x56x64 -> 128 3x3 layer at about 20 GFLOP/s. Port mlx's tiled implicit-GEMM conv as owned .metal source and route NHWC f16/f32 single-group 2D convolutions to it, leaving every other shape on the direct kernel. The shared rewrite puts kernels in OIHW, which the direct kernel indexes, so a metal-local rule reorders eligible ones into the OHWI layout the ported kernel wants - from whichever layout the exporter used, and into a constant, since the metal transform does not declutter afterwards. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Depthwise convolutions were left on the direct kernel by the implicit-GEMM path, which is most of a mobile vision model: MobileNet v2 is 17 depthwise convolutions out of 18. Port mlx's depthwise kernel and route the shapes it covers - one channel per group in and out, kernel up to 7x7, stride up to 2, channels a multiple of 16. The OIHW kernel tract already hands over is laid out the way this kernel indexes it, so no reorder is needed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The port took mlx's general implicit-GEMM kernel, which is its fallback. mlx prefers a second kernel whenever the channel counts are aligned, specialised on small channel counts and short filters, and it is faster on every shape measured here - 4.5x on a 3-channel first layer, 1.2x to 2.5x elsewhere. Port it too and route the way mlx does. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Author
|
Added a large-activation case (1024x1024x32 -> 64) to the conv tests. mlx has been fixing 32-bit shape arithmetic in this area upstream (ml-explore/mlx#3938, #3893); this port shares that 32-bit arithmetic, and the headroom is wide — the worst realistic shape I tried gives ~131k tiles against i32's 2.1e9 — but the case is covered now rather than assumed. |
Grouped convolutions that are not depthwise stayed on the direct kernel. mlx sends them to the same specialised implicit-GEMM kernel, with the group on the grid's z axis and the implicit GEMM sized per group, when each group's channel counts suit it. Route them the same way; groups mlx would not take stay where they were. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2549 and #2550 — their commits are the first two here.
#2549 ported mlx's general implicit-GEMM convolution. That is mlx's fallback:
its own routing prefers a second kernel whenever the channel counts are aligned
(
C ≤ 4 or C%16 == 0, andO ≤ 16 or O%16 == 0), which is most real shapes.This ports that one too and routes the way mlx does, so the general kernel now
only takes what the specialised one declines.
The specialised kernel drops the per-output-position jump tables and is
instantiated on small channel counts (1–4) and short filters, which is where the
difference mostly comes from.
Numbers
bench_conv_specialized(added here), M1 Pro, ms per dispatch, best of 10.Both kernels are already checked against the CPU op by the existing tests.
f16 tracks within a few percent. Every shape measured improves; the first-layer
case gains most, since three input channels hit the channel specialisation.
End to end on Inception v3 (TF, NHWC, 299×299),
--metal, M1 Pro:So 1.10× on top of the stack, 11.8× against main. Same predicted class as
the CPU path.
Grouped convolutions
The same kernel takes grouped (non-depthwise) convolutions — group on the grid's
z axis, implicit GEMM sized per group — under mlx's gate:
C/group ≤ 4or amultiple of 16, and
O/group ≤ 16or a multiple of 16. Those were on the directkernel before, so the gap is wide (M1 Pro, 3×3, ms/dispatch):
No end-to-end number for these: I have no grouped-conv model to hand, and none
of the vision models I used has them. Depthwise keeps its own kernel (#2550) and
groups outside mlx's gate stay on the direct one.
Worth knowing if you review the tests: for a grouped convolution tract's
OHWIputs the group on the I axis, so its kernel is
[O/group, kH, kW, C]whilethis kernel indexes
[O, kH, kW, C/group]. The metal rule produces the latterfrom
OIHW. The grouped tests driveMetalTransformend to end rather thanhand-building a layout, because I got that distinction wrong first time and a
hand-built test just compares two different convolutions.
Validation
cargo test -p tract-metal --release: 98 passed, one failure —test_mfa_attention_causal_const_is_noop, pre-existing on main and unrelated(#2546). The existing conv tests now exercise both kernels: aligned channels go
to the specialised one, and the unaligned case (C=5, O=7) still covers the
general path, plus grouped cases (2/4/8 groups, stride 2, 1×1, f16) and a
gate test for the grouped shapes mlx would decline. A large-activation case
(1024×1024×32 → 64) covers the shape arithmetic mlx has been fixing upstream
(ml-explore/mlx#3938, #3893). fmt and clippy clean.
M1 Pro only, like #2550 — my M4 was asleep. Happy to add second-device numbers
before this lands if you would rather have them.
mlx (ml-explore/mlx) is MIT, Copyright (c) 2023-2025 Apple Inc.; attributed in
the source header, flattened from the same pinned commit as #2549.
🍍