Skip to content

A PyPI wheel for m0serve, and the two blind spots found on the way - #110

Merged
codetalcott merged 8 commits into
mainfrom
claude/wheel-phase1-binfmt
Aug 25, 2026
Merged

A PyPI wheel for m0serve, and the two blind spots found on the way#110
codetalcott merged 8 commits into
mainfrom
claude/wheel-phase1-binfmt

Conversation

@codetalcott

@codetalcott codetalcott commented Aug 25, 2026

Copy link
Copy Markdown
Owner

ROADMAP gate 1: the install matrix. pip install m0serve && m0serve myapp:application, with no Mojo toolchain on the machine.

Two commits — fix the instrument, then build the wheel on it. They landed together because the second is what proved the first incomplete.

The headline: one wheel per platform, every CPython

m0serve does not link libpython — Mojo dlopens the interpreter at run time — so there is no CPython ABI in the archive to be compatible with. That collapses the matrix from ~10 wheels to 2. (ROADMAP gate 1 asserted the opposite; corrected.)

Verified rather than argued. One macosx_13_0_arm64 wheel:

interpreter result
CPython 3.10, 3.11, 3.12, 3.14 m0serve 0.9.0
free-threaded 3.14t m0serve 0.9.0

and, serving a real app across a version boundary — a wheel built on CPython 3.13 with Django 6.1 in the build venv, serving from a separate 3.11 venv:

$ curl localhost:8177/
{"django": "5.2.17", "python": "3.11.16"}

Blind spot 1: the checker could not see an executable

$ python3 scripts/ffi_portability_check.py bin/m0serve
install name  : @rpath/libKGENCompilerRTShared.dylib
dependencies  : ['/usr/lib/libSystem.B.dylib']
SELF-CONTAINED — nothing is resolved at load time.        exit 0

otool -L prints the file's own LC_ID_DYLIB before its dependencies — for a dylib. bin/m0serve is MH_EXECUTE and has none, so entries[1:] discarded the Mojo runtime. bundle_ffi.py carried the same four lines and would have copied zero runtime libraries: two tools agreeing about an artifact neither could see.

The parse now lives in scripts/binfmt.py, keyed on LC_ID_DYLIB's presence, with a --selftest covering both cases in CI.

Separately, build-serve had no post-link surgery at all, so every bin/m0serve ever built recorded a search path into the venv that produced it.

Blind spot 2: "the Linux artifact is statically linked"

The first Linux CI run refused to build, from the guard added in commit 1:

bundle-artifact: bin/m0serve needed 3 runtime libraries
(libKGENCompilerRTShared.so, ...), but the ELF path assumes static linking
and sets no RUNPATH on the copies.

That claim was measured on libm0core.so and then carried as a platform fact. The m0serve executable links the Mojo runtime dynamically on Linux exactly as on macOS. Nothing in the tooling decides by platform any more, only by what the file records: relocate_elf reads DT_NEEDED and sets a $ORIGIN DT_RUNPATH, patchelf becomes a real Linux build requirement with an error saying so, and the notices stop claiming Linux ships no Modular file.

nightly-canary.yml runs test-all — and so builds m0serve — with no apt step at all. It was relying on the runner image for libsqlite3-dev and would now have failed on patchelf. Given one.

The rest

  • packaging/m0serve/ — a separate project with the only [build-system] in the repo. One in the root would make uv treat the repo as installable, and that build needs bin/m0serve, which needs the .venv uv is creating.
  • A console-script shim, not .data/scripts — from there the relative path to the libraries contains the Python minor version, so the rpath would be per-interpreter and the one-wheel property lost. It also lets the exec set PATH/MOJO_PYTHON_LIBRARY/M0_CORE_LIB. os.execve keeps the pid, so the drain still works.
  • scripts/wheel_tag.py measures the tag instead of copying the toolchain's, and reports the spread: Modular's runtime is built for macOS 11.0, ours for 13.0, strictest wins.
  • MACOSX_DEPLOYMENT_TARGET works (measured), so builds pin 13.0. Without it the floor was macOS 26 — a property of whichever image GitHub calls macos-latest, not of this repo.
  • Licensing — the wheel is the first artifact to redistribute the lightbug fork in object form, so its MIT notice must travel. licenses/LICENSE.lightbug_http.txt + NOTICE.m0serve.txt ship inside; NOTICE gains a table of which notice describes which artifact.
  • Four new check-docs ratchets, each sabotage-verified: version drift, a third version copy in wheel metadata, a README pointing at 3.13t (a dead end only pyproject.toml and WSGI_VS_ASGI.md recorded), and a README documenting pip install that no CI job backs.
  • README gains an architecture-qualified platform table. "Linux and macOS" unqualified is a promise pip enforces: macOS Intel is impossible (no toolchain wheel), Linux aarch64 buildable but unshipped, glibc floor excludes musl. It also becomes the PyPI front page at first upload, uneditable without a new version.

Test

839 tests, warnings unchanged at 68, smoke-ffi/serve/django/asgi/wheel green. smoke-wheel installs outside the tree on a different Python minor, serves a real app scrubbed, and asserts the binary stops working once its runtime is moved away — if it kept running the bundle would be a placebo.

Not in scope, deliberately: publishing anything. Linux aarch64 wheels, the clean-consumer release jobs, and the actual TestPyPI/PyPI upload are next.

🤖 Generated with Claude Code

codetalcott and others added 2 commits August 25, 2026 15:09
…ere blind the same way

Scoping the PyPI wheel meant pointing the libm0core machinery at bin/m0serve.
It answered SELF-CONTAINED, exit 0, for a binary that resolves the Mojo
runtime through a path inside one developer's .venv.

`otool -L` prints the file's own LC_ID_DYLIB install name before its
dependencies -- for a dylib. An MH_EXECUTE has no LC_ID_DYLIB at all, so
`entries[1:]` discarded @rpath/libKGENCompilerRTShared.dylib and left only
libSystem, which is a system library and therefore nothing to report.
bundle_ffi.py carried its own copy of the same four lines, so it would have
copied zero runtime libraries and called the bundle complete: two tools
agreeing with each other about an artifact neither could see. Same shape as
the defect that shipped for seven releases, one level up.

- scripts/binfmt.py: the parse, once, keyed on LC_ID_DYLIB's presence rather
  than the header filetype (an MH_BUNDLE may carry one either way, and the
  question is only ever "is entries[0] this file's own name?"). inspect_elf
  moves here unchanged -- it was always correct for executables. --selftest
  runs both cases against canned otool output, in CI, beside the warning
  parser's for the same reason. Fat archives now exit 2 instead of being
  parsed as one arbitrary slice.

- scripts/relocate.py: build-ffi's post-link surgery, lifted out of the task
  so build-serve can share it instead of not having it. bin/m0serve gets two
  self-relative rpaths -- @loader_path and @loader_path/../_lib -- so one
  build output serves both the tarball and the wheel layout.

- build-serve then completes the bundle in place, making bin/ the shipped
  shape rather than a development arrangement that worked for a different
  reason. Removing the venv rpath broke every serve-*/smoke-* task until it
  did, which measures how much the dev path had been leaning on it.

- bundle_ffi.py -> bundle_artifact.py: executable-aware (no -id where there
  is no install name; codesign kept, because arm64 invalidates a signature
  on any Mach-O edit), --layout flat|wheel, --also, in-place staging. Two
  new refusals: a macOS bundle that copied nothing while the artifact names
  non-system dependencies, and a Linux bundle that copied anything at all
  (those artifacts are statically linked; copies would need a $ORIGIN
  RUNPATH nothing here sets).

- Licensing: m0serve statically embeds the lightbug fork, so the MIT notice
  has to travel with the binary -- the first artifact here that redistributes
  it in object form. licenses/LICENSE.lightbug_http.txt and
  licenses/NOTICE.m0serve.txt ship in the bundle; NOTICE gains a table saying
  which notice describes which artifact, and stops implying the Linux builds
  contain no Modular code (no Modular *file*; the code is linked in).

- A third machine dependency the checker never looked at: LC_BUILD_VERSION.
  bin/m0serve records minos 26.0 because a Mojo binary inherits the build
  host's SDK, while the toolchain wheel is tagged macosx_13_0. A wheel tagged
  from uv.lock would install on macOS 15 and fail in dyld, so the wheel's
  platform tag has to be measured, not copied. Reported now, with
  --require-min-os.

- ROADMAP gate 1 said "the binary links libpython". It does not; std.python
  dlopens it. That is what lets the wheel carry no ABI tag.

Verified by sabotage, transcripts in docs/FFI_DISTRIBUTION.md: the checker
now reports BROKEN for bin/m0serve and is UNCHANGED for libm0core.dylib (the
row that distinguishes fixing it from breaking it); the bundler copies 3
dylibs where it copied 0; reintroducing entries[1:] fails the self-test by
name. The bundled binary runs from /tmp under env -i, and fails when its
dylibs are moved away -- naming @loader_path, not the venv, which is what
proves the surgery replaced the build path rather than adding to it.

839 tests pass, warning count unchanged at 68, smoke-serve/wsgi/django/flask/
asgi/hybrid/blocking-threads/reload all green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`pip install m0serve && m0serve myapp:application`, with no Mojo toolchain
on the machine. ROADMAP gate 1's wheel, scoped and built.

**One wheel per platform covers every supported CPython.** m0serve does not
link libpython -- Mojo dlopens it at run time -- so there is no ABI in the
archive to be compatible with. Verified rather than argued: one
macosx_13_0_arm64 wheel installs and runs on 3.10, 3.11, 3.12, 3.14 and
free-threaded 3.14t, and serves Django 5.2.17 on CPython 3.11 from a venv
that has nothing to do with the 3.13/Django 6.1 venv that built it. Two
wheels, not ten.

- packaging/m0serve/: a separate project with the only [build-system] in the
  repo. Putting one in the root would make uv treat the repo as installable,
  and that build needs bin/m0serve, which needs the .venv uv is creating.
  Wheels only; an sdist here cannot build.

- The console script is a shim rather than a binary dropped into
  .data/scripts, for a reason that is not stylistic: from .data/scripts the
  relative path from the binary to its libraries contains the Python minor
  version, so the rpath would have to be baked per interpreter and the
  one-wheel-per-platform property would be lost. It also lets the exec set
  PATH/MOJO_PYTHON_LIBRARY/M0_CORE_LIB, so the embedded interpreter is the
  one m0serve was installed beside instead of whatever `python3` means in
  the caller's shell. os.execve keeps the pid, so the drain still works.

- scripts/wheel_tag.py measures the tag instead of copying the toolchain's.
  It reports the spread it finds: Modular's own runtime is built for macOS
  11.0, ours for 13.0, strictest wins.

- MACOSX_DEPLOYMENT_TARGET turns out to work (measured: 13.0 in, minos 13.0
  out), so build-serve and build-ffi pin it to 13.0 -- the version the Mojo
  toolchain's own wheel is tagged for. Without the pin the floor was macOS
  26, i.e. a property of whichever image GitHub currently calls macos-latest
  rather than of this repository.

- smoke-wheel installs the wheel outside the tree on a different Python
  minor, serves a real app with the environment scrubbed, and asserts the
  binary STOPS working once its bundled runtime is moved away. If it kept
  running, the bundle would be a placebo and the binary would be resolving
  the runtime from the build venv.

**And the thing CI found.** The first Linux run refused to build, from the
guard added in the previous commit:

    bundle-artifact: bin/m0serve needed 3 runtime libraries
    (libKGENCompilerRTShared.so, ...), but the ELF path assumes static
    linking and sets no RUNPATH on the copies.

The repo has recorded since the original FFI work that "the Linux artifact is
statically linked". That was measured on libm0core.so and then carried as a
platform fact; the m0serve EXECUTABLE links the Mojo runtime dynamically on
Linux exactly as on macOS. So nothing in the tooling decides by platform any
more, only by what the file records: relocate_elf reads DT_NEEDED and sets a
$ORIGIN DT_RUNPATH, patchelf becomes a real Linux build requirement with an
error message that says so, and the notices stop claiming Linux ships no
Modular file -- the m0serve bundle and wheel carry all three there, with
their DT_RUNPATH rewritten, which is an Apache 4(b) modification like the
macOS one.

nightly-canary.yml runs test-all, and so builds m0serve, with no apt step at
all -- it was relying on the runner image for libsqlite3-dev and would now
have failed on patchelf. Given one.

- Four new check-docs ratchets, each sabotage-verified: version drift between
  pyproject.toml and M0SERVE_VERSION; a literal version creeping into the
  wheel's metadata as a third copy; a README that points users at 3.13t (a
  dead end pyproject.toml and WSGI_VS_ASGI.md both record, and only 3.14t is
  tested); and a README documenting `pip install` that no CI job backs with
  an artifact.

- README gains an install section with an architecture-qualified platform
  table. "Linux and macOS" was unqualified, and once wheels exist that is a
  promise pip enforces: macOS Intel is not untested but impossible, Linux
  aarch64 is buildable and unshipped, and the glibc floor rules out
  musl/Alpine. It also becomes the PyPI front page at first upload, which
  cannot be edited without a new version.

839 tests pass, warnings unchanged at 68, smoke-ffi/serve/django/asgi/wheel
green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@codetalcott codetalcott changed the title The portability checker could not see an executable, and both tools were blind the same way A PyPI wheel for m0serve, and the two blind spots found on the way Aug 25, 2026
codetalcott and others added 6 commits August 25, 2026 15:30
…tchet on their cleanliness

smoke-wheel runs inside the repo, where .venv/.../modular/lib is on disk --
the exact directory a stale rpath names. It cannot establish that the wheel
works anywhere else, and its docstring says so. This adds the half that can.

- build-wheels on PINNED runners (macos-14, ubuntu-22.04), because a Mojo
  binary inherits the build host's SDK and glibc: with `-latest` the wheel's
  reach would change whenever GitHub rotated an image. The pin IS the floor.

- wheel-inspect: static, the only consume-side job with a checkout (it needs
  scripts/) and it never runs the binary.

- wheel-consume-linux: `docker run --network none` across four CPython
  minors. --network none is not decoration -- it asserts the wheel needs
  nothing fetched, which is the whole "pip install m0serve" story. Plus a
  permanent negative control: a manylinux2014 container must have pip REFUSE
  the wheel, so the glibc floor is enforced by a refusal rather than by a
  runtime GLIBC_2.x-not-found crash. That one needs no sabotage because it
  is one.

- wheel-consume-macos on macos-14, the oldest arm64 image, which turns the
  declared macosx_ floor into a measurement rather than a claim.

Every consume job asserts its own cleanliness (no repo, no mojo, no
DYLD_/LD_/MOJO_PYTHON_LIBRARY) BEFORE it asserts anything about the wheel.
That property is the entire value of these jobs and is invisible when it
breaks -- someone adds actions/checkout to get a test fixture, the job still
passes, and the proof reverts to build-machine conditions with a green tick.
So check_docs.py now enforces it: a wheel-consume-* job that acquires a
checkout, uv, or a poe task fails the build, as does one that drops its
cleanliness assertion. Verified by doing both. (It then caught me reverting
this file with a stray `git checkout`, which is a fair test of a guard.)

publish now needs the three wheel jobs, so a release cannot carry a wheel
that was only ever installed where it was built, and it records SHA256SUMS.

publish-pypi is present but INERT: it runs only when the repository variable
PUBLISH_TO_PYPI is "true", needs the GitHub release to have succeeded first,
and refuses a set that is not exactly two platforms at one version matching
the tag. A GitHub release is fixable; a PyPI filename is burned permanently,
and a yank leaves the file installable by exact pin. The one-time Trusted
Publisher setup is written down in the job's own comment.

Also fixed, found while writing wheel-inspect and before CI could hit it:
--require-self-contained would have failed every Linux binary. ELF names its
dependencies by bare soname, so the SYSTEM_PREFIXES path rule never matched
libc.so.6 and the checker filed it as unresolvable. It never showed because
libm0core.so has no DT_NEEDED at all; bin/m0serve is the first ELF artifact
here with real ones. binfmt now owns SYSTEM_SONAMES, relocate.py uses it
instead of its own copy, and --selftest asserts both directions -- glibc
classified as the platform's, the three Mojo runtime .so files as ours.
Getting that backwards either ships libc or omits what the binary cannot
start without.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ot prepended

Serving three portfolio Django projects through the installed wheel turned up
a pre-existing discrepancy, visible in Django's own debug page: `--app-dir`
lands AFTER site-packages, so an application module can be shadowed by an
installed package with the same name. gunicorn, uvicorn and runserver all
sys.path.insert(0, ...); m0serve.mojo calls Python.add_to_path, which
appends. The flag's help, cli.mojo:136 and app.mojo:96 all say "prepended".

Confirmed pre-existing rather than introduced by the wheel's console-script
shim: the repo's own bin/m0serve behaves identically.

Recorded rather than fixed. Changing import precedence can break an
application that accidentally depends on the current order, so it wants its
own change and its own test rather than a drive-by edit inside a packaging
change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The install table named manylinux_2_34_x86_64, copied from the Mojo
toolchain's own wheel tag. That is not what the artifact requires: a binary
linked on the build image needs that image's glibc through symbol
versioning, and wheel_tag.py measures and declares the real number. Naming
2_34 in prose while the filename says something higher is the same class of
error as tagging the wheel from uv.lock.

The table now points at the filename, which carries the measured floors, and
the tradeoff is recorded as a known issue: RHEL 9 and its rebuilds sit at
glibc 2.34, so a wheel built on a newer image excludes them. Reaching them
means building inside a manylinux_2_34 container, not relabelling — deferred,
because it adds a container build to the release path for reach the first
quiet 0.x does not need.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It was guarded on Darwin, written while the repo still recorded "the Linux
artifact is statically linked". That holds for libm0core.so and not for the
m0serve executable, which links the three Mojo runtime .so files on Linux
exactly as on macOS -- so the assertion that matters most (the binary must
STOP working once its bundled runtime is moved away) was skipped on the
platform where it is equally load-bearing. LD_LIBRARY_PATH is scrubbed
alongside the DYLD_ ones for the same reason.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The wheel section said what is true about publishing; this says what to type.
Three one-time steps only the account holder can do (pending Trusted
Publisher on both indexes, the `pypi` environment, the PUBLISH_TO_PYPI
variable), then a TestPyPI rehearsal whose last step is the one that actually
matters: pip choosing the right file out of several platform wheels, from an
index, in a container that never built them. Local checks install a wheel
FILE and cannot test that.

Also written down: collect the other platform's wheel from a CI run rather
than rebuilding it. Rebuilding between the rehearsal and the release means
the artifact that was tested is not the artifact that ships.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI settled three things that were inference until now.

The ELF relocation produces what the Mach-O one does:
`relocated bin/m0serve (search paths: ['$ORIGIN', '$ORIGIN/../_lib'])`, the
m0serve bundle assembles with its three .so files, and smoke-wheel builds,
installs and serves from `m0serve-0.9.0-py3-none-manylinux_2_35_x86_64.whl`.

And the glibc floor is not where I said it was. The README claimed the wheel
"declares the build image's glibc"; the measurement says 2.35 on a runner
whose own glibc is 2.39. The floor is what the Mojo toolchain's OUTPUT
requires, so — unlike the macOS deployment target, which really did follow
the build host until it was pinned — building on an older image would not
lower it. Both docs now say the measured number and where it comes from.

The practical consequence is unchanged and now stated precisely: 2.35 covers
Ubuntu 22.04 and Debian 12, and misses RHEL 9 and its rebuilds by one minor.
Reaching them needs a manylinux_2_34 container build, still deferred.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@codetalcott
codetalcott merged commit 654b2d2 into main Aug 25, 2026
4 checks passed
@codetalcott
codetalcott deleted the claude/wheel-phase1-binfmt branch August 25, 2026 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant