Pin --target-cpu: the release crashed with SIGILL on a machine that did not build it - #112
Merged
Merged
Conversation
…id not build it
The v0.10.0rc1 run built both wheels, passed smoke-wheel inside both build
jobs, passed wheel-consume-macos -- and then wheel-consume-linux ran
`m0serve --version` in a clean container and got:
0 libKGENCompilerRTShared.so 0x00007f39a1540f8e
...
Illegal instruction (core dumped)
`mojo build` defaults --target-cpu to the HOST CPU. That is -march=native,
silently, for every artifact this repository has ever produced. Asking the
compiler directly on this machine:
--target-cpu apple-m4
--target-features ...,+sme,+sme-f64f64,+sme-i16i64,+sme2
+sme/+sme2 are the Scalable Matrix Extension. No M1, M2 or M3 has them, so
the wheels dogfooded against three Django projects -- all on the same M4 --
would very likely have crashed on any other Apple Silicon Mac.
build-ffi and build-serve now pin the oldest CPU each platform must support:
apple-m1, x86-64-v2 (SSE4.2/POPCNT, hardware from 2009, and RHEL 9's own
baseline), generic on aarch64. Verified: the rebuilt binary still runs, and
`otool -tv` finds zero SME instructions where the m4 build emitted them.
This is the rpath defect one level up, and worse in one specific way: the
rpath version was at least visible to static inspection, which is what
ffi_portability_check.py exists for. This one is not -- the binary is
correct, it just needs instructions the consumer's CPU does not implement.
Nothing on the build machine can tell. wheel-consume-linux caught it purely
because it runs somewhere else, which is the property those jobs were built
for. wheel-consume-macos did NOT catch it and cannot: it builds and consumes
on macos-14, so they share a CPU. The macOS side is protected by the pin.
check_docs.py asserts both tasks pass the flag -- and the first version of
that check was itself broken in an instructive way: it searched the whole
task body, which includes a comment explaining what --target-cpu is for, so
it passed with the flag deleted. A guard satisfied by its own documentation.
It now parses the mojo build command with continuations joined, and both
tasks were sabotaged to confirm it fails.
Cost, stated rather than buried: bench/results/ numbers were measured with
host-native tuning and a baseline build will not reproduce them exactly.
839 tests, warnings at 68, smoke-ffi/serve/wheel green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…d build to offer
I claimed pinning --target-cpu costs performance and that bench numbers would
not reproduce. That was an assumption, and it is wrong. Compiling the same
sources at apple-m1 and apple-m4 and diffing the disassembly:
run_benchmarks 41,248 lines 0 differing
m0serve 257,735 lines 0 differing
Byte-identical machine code. Stronger than a timing comparison, since
identical code cannot differ in speed, and unsurprising once stated: what
apple-m4 adds over apple-m1 is +sme, +sme2, +i8mm, +bf16 -- matrix and ML
extensions -- and an HTTP server's hot paths are syscalls, byte scanning,
hashing and memcpy. Nothing there auto-vectorizes into SME.
So the answer to "should we offer an m4+ build" is no, twice over. There is
nothing to ship: it would be the same bytes under a different name. And there
would be nowhere to put it if there were -- a wheel's platform tag has no
microarchitecture field, macosx_13_0_arm64 is the only arm64 macOS tag, so
two tuned wheels for one version collide on filename and pip cannot choose.
It would take a separate distribution name or an off-PyPI download.
Scoped honestly: this is true of this code at this commit. Add SIMD-heavy
work and it could change, and the way to find out is to re-run the diff.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
v0.10.0rc1release run failed, and it caught something worth the whole exercise. Nothing was published —publishandpublish-pypiskipped, no GitHub release, no PyPI upload.What happened
Both wheels built.
smoke-wheelpassed inside both build jobs.wheel-consume-macospassed. Thenwheel-consume-linuxranm0serve --versionin a clean container:Why
mojo buildis-march=nativeby default — for every artifact this repository has ever produced. Asked directly on a developer machine:+sme/+sme2are the Scalable Matrix Extension. No M1, M2 or M3 has them. The wheels I dogfooded against three Django projects — all on that same M4 — would very likely have crashed on any other Apple Silicon Mac.The fix
apple-m1+sme,+sme2,+i8mm,+bf16x86-64-v2genericx86-64-v2is SSE4.2/POPCNT — 2009 hardware, and RHEL 9's own baseline.apple-m1is the oldest Apple Silicon. Verified: the rebuilt binary runs, andotool -tvfinds zero SME instructions where the m4 build emitted them.Why this one is worse than the rpath defect it resembles
The rpath version was visible to static inspection — that's what
ffi_portability_check.pyis for. This one isn't: the binary is correct, it just needs instructions the consumer's CPU doesn't implement. Only running it on different silicon can find it.wheel-consume-linuxcaught it purely because it runs somewhere else.wheel-consume-macosdid not catch it and cannot — it builds and consumes onmacos-14, so they share a CPU. The macOS side is protected by the pin, not by that job. Worth saying plainly rather than treating a green tick as reassurance.The guard, and how its first version was wrong
check_docs.pyasserts both tasks pass the flag. The first version searched the whole task body — which contains a comment explaining--target-cpu— so it passed with the flag deleted. A guard satisfied by its own documentation. It now parses themojo buildcommand with continuations joined; both tasks were sabotaged to confirm it fails.Cost
bench/results/numbers were measured with host-native tuning and a baseline build won't reproduce them exactly. Correctness first — an artifact that crashes has no throughput.Test
839 tests, warnings unchanged at 68,
smoke-ffi/smoke-serve/smoke-wheelgreen.After this
The
v0.10.0rc1tag exists but produced no release and no upload, so it can be deleted and re-pushed once this lands.🤖 Generated with Claude Code