A PyPI wheel for m0serve, and the two blind spots found on the way - #110
Merged
Conversation
…ere blind the same way Scoping the PyPI wheel meant pointing the libm0core machinery at bin/m0serve. It answered SELF-CONTAINED, exit 0, for a binary that resolves the Mojo runtime through a path inside one developer's .venv. `otool -L` prints the file's own LC_ID_DYLIB install name before its dependencies -- for a dylib. An MH_EXECUTE has no LC_ID_DYLIB at all, so `entries[1:]` discarded @rpath/libKGENCompilerRTShared.dylib and left only libSystem, which is a system library and therefore nothing to report. bundle_ffi.py carried its own copy of the same four lines, so it would have copied zero runtime libraries and called the bundle complete: two tools agreeing with each other about an artifact neither could see. Same shape as the defect that shipped for seven releases, one level up. - scripts/binfmt.py: the parse, once, keyed on LC_ID_DYLIB's presence rather than the header filetype (an MH_BUNDLE may carry one either way, and the question is only ever "is entries[0] this file's own name?"). inspect_elf moves here unchanged -- it was always correct for executables. --selftest runs both cases against canned otool output, in CI, beside the warning parser's for the same reason. Fat archives now exit 2 instead of being parsed as one arbitrary slice. - scripts/relocate.py: build-ffi's post-link surgery, lifted out of the task so build-serve can share it instead of not having it. bin/m0serve gets two self-relative rpaths -- @loader_path and @loader_path/../_lib -- so one build output serves both the tarball and the wheel layout. - build-serve then completes the bundle in place, making bin/ the shipped shape rather than a development arrangement that worked for a different reason. Removing the venv rpath broke every serve-*/smoke-* task until it did, which measures how much the dev path had been leaning on it. - bundle_ffi.py -> bundle_artifact.py: executable-aware (no -id where there is no install name; codesign kept, because arm64 invalidates a signature on any Mach-O edit), --layout flat|wheel, --also, in-place staging. Two new refusals: a macOS bundle that copied nothing while the artifact names non-system dependencies, and a Linux bundle that copied anything at all (those artifacts are statically linked; copies would need a $ORIGIN RUNPATH nothing here sets). - Licensing: m0serve statically embeds the lightbug fork, so the MIT notice has to travel with the binary -- the first artifact here that redistributes it in object form. licenses/LICENSE.lightbug_http.txt and licenses/NOTICE.m0serve.txt ship in the bundle; NOTICE gains a table saying which notice describes which artifact, and stops implying the Linux builds contain no Modular code (no Modular *file*; the code is linked in). - A third machine dependency the checker never looked at: LC_BUILD_VERSION. bin/m0serve records minos 26.0 because a Mojo binary inherits the build host's SDK, while the toolchain wheel is tagged macosx_13_0. A wheel tagged from uv.lock would install on macOS 15 and fail in dyld, so the wheel's platform tag has to be measured, not copied. Reported now, with --require-min-os. - ROADMAP gate 1 said "the binary links libpython". It does not; std.python dlopens it. That is what lets the wheel carry no ABI tag. Verified by sabotage, transcripts in docs/FFI_DISTRIBUTION.md: the checker now reports BROKEN for bin/m0serve and is UNCHANGED for libm0core.dylib (the row that distinguishes fixing it from breaking it); the bundler copies 3 dylibs where it copied 0; reintroducing entries[1:] fails the self-test by name. The bundled binary runs from /tmp under env -i, and fails when its dylibs are moved away -- naming @loader_path, not the venv, which is what proves the surgery replaced the build path rather than adding to it. 839 tests pass, warning count unchanged at 68, smoke-serve/wsgi/django/flask/ asgi/hybrid/blocking-threads/reload all green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`pip install m0serve && m0serve myapp:application`, with no Mojo toolchain
on the machine. ROADMAP gate 1's wheel, scoped and built.
**One wheel per platform covers every supported CPython.** m0serve does not
link libpython -- Mojo dlopens it at run time -- so there is no ABI in the
archive to be compatible with. Verified rather than argued: one
macosx_13_0_arm64 wheel installs and runs on 3.10, 3.11, 3.12, 3.14 and
free-threaded 3.14t, and serves Django 5.2.17 on CPython 3.11 from a venv
that has nothing to do with the 3.13/Django 6.1 venv that built it. Two
wheels, not ten.
- packaging/m0serve/: a separate project with the only [build-system] in the
repo. Putting one in the root would make uv treat the repo as installable,
and that build needs bin/m0serve, which needs the .venv uv is creating.
Wheels only; an sdist here cannot build.
- The console script is a shim rather than a binary dropped into
.data/scripts, for a reason that is not stylistic: from .data/scripts the
relative path from the binary to its libraries contains the Python minor
version, so the rpath would have to be baked per interpreter and the
one-wheel-per-platform property would be lost. It also lets the exec set
PATH/MOJO_PYTHON_LIBRARY/M0_CORE_LIB, so the embedded interpreter is the
one m0serve was installed beside instead of whatever `python3` means in
the caller's shell. os.execve keeps the pid, so the drain still works.
- scripts/wheel_tag.py measures the tag instead of copying the toolchain's.
It reports the spread it finds: Modular's own runtime is built for macOS
11.0, ours for 13.0, strictest wins.
- MACOSX_DEPLOYMENT_TARGET turns out to work (measured: 13.0 in, minos 13.0
out), so build-serve and build-ffi pin it to 13.0 -- the version the Mojo
toolchain's own wheel is tagged for. Without the pin the floor was macOS
26, i.e. a property of whichever image GitHub currently calls macos-latest
rather than of this repository.
- smoke-wheel installs the wheel outside the tree on a different Python
minor, serves a real app with the environment scrubbed, and asserts the
binary STOPS working once its bundled runtime is moved away. If it kept
running, the bundle would be a placebo and the binary would be resolving
the runtime from the build venv.
**And the thing CI found.** The first Linux run refused to build, from the
guard added in the previous commit:
bundle-artifact: bin/m0serve needed 3 runtime libraries
(libKGENCompilerRTShared.so, ...), but the ELF path assumes static
linking and sets no RUNPATH on the copies.
The repo has recorded since the original FFI work that "the Linux artifact is
statically linked". That was measured on libm0core.so and then carried as a
platform fact; the m0serve EXECUTABLE links the Mojo runtime dynamically on
Linux exactly as on macOS. So nothing in the tooling decides by platform any
more, only by what the file records: relocate_elf reads DT_NEEDED and sets a
$ORIGIN DT_RUNPATH, patchelf becomes a real Linux build requirement with an
error message that says so, and the notices stop claiming Linux ships no
Modular file -- the m0serve bundle and wheel carry all three there, with
their DT_RUNPATH rewritten, which is an Apache 4(b) modification like the
macOS one.
nightly-canary.yml runs test-all, and so builds m0serve, with no apt step at
all -- it was relying on the runner image for libsqlite3-dev and would now
have failed on patchelf. Given one.
- Four new check-docs ratchets, each sabotage-verified: version drift between
pyproject.toml and M0SERVE_VERSION; a literal version creeping into the
wheel's metadata as a third copy; a README that points users at 3.13t (a
dead end pyproject.toml and WSGI_VS_ASGI.md both record, and only 3.14t is
tested); and a README documenting `pip install` that no CI job backs with
an artifact.
- README gains an install section with an architecture-qualified platform
table. "Linux and macOS" was unqualified, and once wheels exist that is a
promise pip enforces: macOS Intel is not untested but impossible, Linux
aarch64 is buildable and unshipped, and the glibc floor rules out
musl/Alpine. It also becomes the PyPI front page at first upload, which
cannot be edited without a new version.
839 tests pass, warnings unchanged at 68, smoke-ffi/serve/django/asgi/wheel
green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tchet on their cleanliness smoke-wheel runs inside the repo, where .venv/.../modular/lib is on disk -- the exact directory a stale rpath names. It cannot establish that the wheel works anywhere else, and its docstring says so. This adds the half that can. - build-wheels on PINNED runners (macos-14, ubuntu-22.04), because a Mojo binary inherits the build host's SDK and glibc: with `-latest` the wheel's reach would change whenever GitHub rotated an image. The pin IS the floor. - wheel-inspect: static, the only consume-side job with a checkout (it needs scripts/) and it never runs the binary. - wheel-consume-linux: `docker run --network none` across four CPython minors. --network none is not decoration -- it asserts the wheel needs nothing fetched, which is the whole "pip install m0serve" story. Plus a permanent negative control: a manylinux2014 container must have pip REFUSE the wheel, so the glibc floor is enforced by a refusal rather than by a runtime GLIBC_2.x-not-found crash. That one needs no sabotage because it is one. - wheel-consume-macos on macos-14, the oldest arm64 image, which turns the declared macosx_ floor into a measurement rather than a claim. Every consume job asserts its own cleanliness (no repo, no mojo, no DYLD_/LD_/MOJO_PYTHON_LIBRARY) BEFORE it asserts anything about the wheel. That property is the entire value of these jobs and is invisible when it breaks -- someone adds actions/checkout to get a test fixture, the job still passes, and the proof reverts to build-machine conditions with a green tick. So check_docs.py now enforces it: a wheel-consume-* job that acquires a checkout, uv, or a poe task fails the build, as does one that drops its cleanliness assertion. Verified by doing both. (It then caught me reverting this file with a stray `git checkout`, which is a fair test of a guard.) publish now needs the three wheel jobs, so a release cannot carry a wheel that was only ever installed where it was built, and it records SHA256SUMS. publish-pypi is present but INERT: it runs only when the repository variable PUBLISH_TO_PYPI is "true", needs the GitHub release to have succeeded first, and refuses a set that is not exactly two platforms at one version matching the tag. A GitHub release is fixable; a PyPI filename is burned permanently, and a yank leaves the file installable by exact pin. The one-time Trusted Publisher setup is written down in the job's own comment. Also fixed, found while writing wheel-inspect and before CI could hit it: --require-self-contained would have failed every Linux binary. ELF names its dependencies by bare soname, so the SYSTEM_PREFIXES path rule never matched libc.so.6 and the checker filed it as unresolvable. It never showed because libm0core.so has no DT_NEEDED at all; bin/m0serve is the first ELF artifact here with real ones. binfmt now owns SYSTEM_SONAMES, relocate.py uses it instead of its own copy, and --selftest asserts both directions -- glibc classified as the platform's, the three Mojo runtime .so files as ours. Getting that backwards either ships libc or omits what the binary cannot start without. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ot prepended Serving three portfolio Django projects through the installed wheel turned up a pre-existing discrepancy, visible in Django's own debug page: `--app-dir` lands AFTER site-packages, so an application module can be shadowed by an installed package with the same name. gunicorn, uvicorn and runserver all sys.path.insert(0, ...); m0serve.mojo calls Python.add_to_path, which appends. The flag's help, cli.mojo:136 and app.mojo:96 all say "prepended". Confirmed pre-existing rather than introduced by the wheel's console-script shim: the repo's own bin/m0serve behaves identically. Recorded rather than fixed. Changing import precedence can break an application that accidentally depends on the current order, so it wants its own change and its own test rather than a drive-by edit inside a packaging change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The install table named manylinux_2_34_x86_64, copied from the Mojo toolchain's own wheel tag. That is not what the artifact requires: a binary linked on the build image needs that image's glibc through symbol versioning, and wheel_tag.py measures and declares the real number. Naming 2_34 in prose while the filename says something higher is the same class of error as tagging the wheel from uv.lock. The table now points at the filename, which carries the measured floors, and the tradeoff is recorded as a known issue: RHEL 9 and its rebuilds sit at glibc 2.34, so a wheel built on a newer image excludes them. Reaching them means building inside a manylinux_2_34 container, not relabelling — deferred, because it adds a container build to the release path for reach the first quiet 0.x does not need. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It was guarded on Darwin, written while the repo still recorded "the Linux artifact is statically linked". That holds for libm0core.so and not for the m0serve executable, which links the three Mojo runtime .so files on Linux exactly as on macOS -- so the assertion that matters most (the binary must STOP working once its bundled runtime is moved away) was skipped on the platform where it is equally load-bearing. LD_LIBRARY_PATH is scrubbed alongside the DYLD_ ones for the same reason. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The wheel section said what is true about publishing; this says what to type. Three one-time steps only the account holder can do (pending Trusted Publisher on both indexes, the `pypi` environment, the PUBLISH_TO_PYPI variable), then a TestPyPI rehearsal whose last step is the one that actually matters: pip choosing the right file out of several platform wheels, from an index, in a container that never built them. Local checks install a wheel FILE and cannot test that. Also written down: collect the other platform's wheel from a CI run rather than rebuilding it. Rebuilding between the rehearsal and the release means the artifact that was tested is not the artifact that ships. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI settled three things that were inference until now. The ELF relocation produces what the Mach-O one does: `relocated bin/m0serve (search paths: ['$ORIGIN', '$ORIGIN/../_lib'])`, the m0serve bundle assembles with its three .so files, and smoke-wheel builds, installs and serves from `m0serve-0.9.0-py3-none-manylinux_2_35_x86_64.whl`. And the glibc floor is not where I said it was. The README claimed the wheel "declares the build image's glibc"; the measurement says 2.35 on a runner whose own glibc is 2.39. The floor is what the Mojo toolchain's OUTPUT requires, so — unlike the macOS deployment target, which really did follow the build host until it was pinned — building on an older image would not lower it. Both docs now say the measured number and where it comes from. The practical consequence is unchanged and now stated precisely: 2.35 covers Ubuntu 22.04 and Debian 12, and misses RHEL 9 and its rebuilds by one minor. Reaching them needs a manylinux_2_34 container build, still deferred. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ROADMAP gate 1: the install matrix.
pip install m0serve && m0serve myapp:application, with no Mojo toolchain on the machine.Two commits — fix the instrument, then build the wheel on it. They landed together because the second is what proved the first incomplete.
The headline: one wheel per platform, every CPython
m0servedoes not link libpython — Mojodlopens the interpreter at run time — so there is no CPython ABI in the archive to be compatible with. That collapses the matrix from ~10 wheels to 2. (ROADMAP gate 1 asserted the opposite; corrected.)Verified rather than argued. One
macosx_13_0_arm64wheel:m0serve 0.9.0m0serve 0.9.0and, serving a real app across a version boundary — a wheel built on CPython 3.13 with Django 6.1 in the build venv, serving from a separate 3.11 venv:
Blind spot 1: the checker could not see an executable
otool -Lprints the file's ownLC_ID_DYLIBbefore its dependencies — for a dylib.bin/m0serveisMH_EXECUTEand has none, soentries[1:]discarded the Mojo runtime.bundle_ffi.pycarried the same four lines and would have copied zero runtime libraries: two tools agreeing about an artifact neither could see.The parse now lives in
scripts/binfmt.py, keyed onLC_ID_DYLIB's presence, with a--selftestcovering both cases in CI.Separately,
build-servehad no post-link surgery at all, so everybin/m0serveever built recorded a search path into the venv that produced it.Blind spot 2: "the Linux artifact is statically linked"
The first Linux CI run refused to build, from the guard added in commit 1:
That claim was measured on
libm0core.soand then carried as a platform fact. The m0serve executable links the Mojo runtime dynamically on Linux exactly as on macOS. Nothing in the tooling decides by platform any more, only by what the file records:relocate_elfreadsDT_NEEDEDand sets a$ORIGINDT_RUNPATH,patchelfbecomes a real Linux build requirement with an error saying so, and the notices stop claiming Linux ships no Modular file.nightly-canary.ymlrunstest-all— and so builds m0serve — with no apt step at all. It was relying on the runner image forlibsqlite3-devand would now have failed onpatchelf. Given one.The rest
packaging/m0serve/— a separate project with the only[build-system]in the repo. One in the root would make uv treat the repo as installable, and that build needsbin/m0serve, which needs the.venvuv is creating..data/scripts— from there the relative path to the libraries contains the Python minor version, so the rpath would be per-interpreter and the one-wheel property lost. It also lets the exec setPATH/MOJO_PYTHON_LIBRARY/M0_CORE_LIB.os.execvekeeps the pid, so the drain still works.scripts/wheel_tag.pymeasures the tag instead of copying the toolchain's, and reports the spread: Modular's runtime is built for macOS 11.0, ours for 13.0, strictest wins.MACOSX_DEPLOYMENT_TARGETworks (measured), so builds pin 13.0. Without it the floor was macOS 26 — a property of whichever image GitHub callsmacos-latest, not of this repo.licenses/LICENSE.lightbug_http.txt+NOTICE.m0serve.txtship inside; NOTICE gains a table of which notice describes which artifact.check-docsratchets, each sabotage-verified: version drift, a third version copy in wheel metadata, a README pointing at 3.13t (a dead end onlypyproject.tomlandWSGI_VS_ASGI.mdrecorded), and a README documentingpip installthat no CI job backs.Test
839 tests, warnings unchanged at 68,
smoke-ffi/serve/django/asgi/wheelgreen.smoke-wheelinstalls outside the tree on a different Python minor, serves a real app scrubbed, and asserts the binary stops working once its runtime is moved away — if it kept running the bundle would be a placebo.Not in scope, deliberately: publishing anything. Linux aarch64 wheels, the clean-consumer release jobs, and the actual TestPyPI/PyPI upload are next.
🤖 Generated with Claude Code