feat(bindings-py): manylinux wheel CI for Linux x86_64 + aarch64 (phase 7/9)

[dekobon]

Parent

Part of #103 — Python bindings via PyO3 / maturin.

Depends on the crate-skeleton sub-issue and ideally the tooling
sub-issue (so wheels are built from lint-clean source).

Status

Landed in 942f231 on 2026-05-23. See the in-thread comment
("Implementation note: manylinux wheel CI landed") for the
deviation rationale (native ARM, abi3 default, attestations) and
the action-SHA bibliography. The workflow file is
.github/workflows/python-wheels.yml.

One-time PyPI setup (Trusted Publisher + pypi GitHub
Environment) is documented in RELEASING.md →
"Python wheels (PyPI) → One-time PyPI setup". It must be
completed before the first v* tag triggers a publish.

Goal

Build and publish manylinux wheels for big-code-analysis from CI so
target environments (no gcc, no cargo) can pip install.

Scope

Wheel matrix (v1)

• Linux x86_64 — manylinux_2_28 (or whatever maturin-action's
  current default is; pin it explicitly).
• Linux aarch64 — same manylinux profile, cross-compiled.
• Python: 3.12, 3.13. Add 3.14 once it ships if the release lines up.
• ABI: build as abi3 (limited-API) where PyO3 supports it for our
  feature set; falls back to per-version wheels otherwise. Document
  the decision in the workflow file.

macOS and Windows are explicitly out of scope (#103 lists them under
"Out of scope (track separately)"). Open follow-up issues only when
there is concrete demand.

Build workflow

.github/workflows/python-wheels.yml:

• Triggers:
  • push to main for release branches / tags (v*) only —
    no wheel build on every commit.
  • Manual workflow_dispatch for ad-hoc test builds.
  • pull_request with a path filter on big-code-analysis-py/**
    AND a python-wheels label — keeps PR-time cost opt-in.
• Jobs:
  • linux-x86_64 — PyO3/maturin-action@v1 with target = x86_64-unknown-linux-gnu, manylinux = 2_28,
    args = "--release --strip --out dist".
  • linux-aarch64 — same with target = aarch64-unknown-linux-gnu. Implemented with the native
    ubuntu-24.04-arm runner instead of QEMU — see the in-
    thread comment for the rationale (GA for private repos
    Jan 2026, ~10× faster than QEMU).
  • sdist — maturin sdist so PyPI has a source distribution as
    well (even though we mark wheels as required, the sdist is
    useful for niche architectures and reproducibility).
  • smoke-test — pulls the built wheel for the runner arch and
    runs a minimal python -c "import big_code_analysis; big_code_analysis.analyze(...)" to catch dynamic-linker /
    missing-symbol failures before publish. Runs on both Python
    3.12 and 3.13 so the abi3 forward-compat claim is verified.
  • publish — gated on a tag matching v* and the pypi
    environment with PyPI trusted-publisher OIDC. No PyPI API
    tokens — uses pypa/gh-action-pypi-publish@v1.14.0.

Reproducibility / security

• Pin every action by SHA, not by tag (the repo already does this
  for the Rust workflows — match the style).
• Cache cargo and maturin builds keyed off Cargo.lock +
  pyproject.toml.
• Sign artefacts with PEP 740 Sigstore attestations, generated
  automatically by gh-action-pypi-publish@v1.14.0 (replaces the
  separate sigstore/gh-action-sigstore-python step in the
  original spec — see in-thread comment).
• cargo deny runs in ci.yml already; pip-audit runs in the
  smoke-test job as the wheel-side equivalent.

Release coordination

• ✅ Updated RELEASING.md with the Python release procedure:
  • When to bump big-code-analysis-py/pyproject.toml version
    (it inherits from [workspace.package]).
  • The fact that wheels are built only on tags.
  • How to test a release candidate via workflow_dispatch.
  • PyPI Trusted Publisher one-time setup.
• ✅ packaging/README.md is binary-only; the Python wheel
  pipeline is cross-referenced from RELEASING.md directly,
  not duplicated under packaging/.

Documentation

• ✅ CHANGELOG.md: ### Added entry for the workflow and
  ### Changed entry for the pyo3/abi3-py312 feature flip.
• ⏭️ Book "Installation" page references the wheel matrix —
  tracked separately under docs(book): Python Bindings chapter (8 pages) with CI-tested examples (phase 8/9) #272 (phase 8/9).

Acceptance criteria

• ✅ Manually triggering python-wheels.yml on a feature branch
  produces uploadable wheels for both Linux x86_64 and aarch64,
  Python 3.12 + 3.13. (Single abi3 wheel per arch covers both
  Python versions; smoke-test verifies both 3.12 and 3.13 load
  the same wheel.)
• ✅ The smoke-test job successfully imports and analyses a
  fixture on the built wheel.
• ⏭️ Tagging vX.Y.Z on main publishes those wheels to PyPI
  via trusted-publisher OIDC. (Cannot verify until the first
  tag — the one-time PyPI setup in RELEASING.md must be
  completed first. The publish job is gated on
  github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
  so PR / workflow_dispatch runs validate the matrix without
  uploading.)
• ✅ No PyPI tokens stored as repo secrets.

[dekobon]

Research note: can cargo-sonic be combined with the wheel build?

Short answer: no, not directly. cargo-sonic and Python
extension wheels are fundamentally incompatible at the
binary-format level. There are real alternatives, though, and the
choice has consequences for this phase. Cross-referencing #181
(the issue tracking cargo-sonic adoption for the bca / bca-web
binaries) — that conversation is about executables, and the
arguments don't carry over to the Python wheel.

Why cargo-sonic cannot produce a wheel .so

1. Output type. cargo-sonic produces "one Linux fat
   executable for a Rust binary" (its README's own words). It
   targets [[bin]] crates and emits an ELF executable that
   concatenates multiple -C target-cpu payloads behind a
   no_std loader stub. It does not support cdylib /
   dylib / staticlib crate types — and there is no
   discussion of adding that, because…

2. Dispatch mechanism is exec, not dlopen. The loader picks a
   payload, writes it to a memfd, and execveat(AT_EMPTY_PATH)s
   into it. That replaces the current process image. Python
   extensions are loaded via dlopen by an already-running
   python interpreter, which then calls PyInit_<module>().
   There is no point at which the loader stub could substitute the
   .so's contents — by the time PyInit_* runs, the kernel has
   already mapped the file and the interpreter is reading the
   symbol table.

3. maturin builds a cdylib. The crate type for a PyO3
   extension is crate-type = ["cdylib"], and maturin build
   runs essentially cargo build --release on it. The
   maturin-action workflow this issue specifies has no hook for
   substituting a different build tool, and even if it did,
   cargo sonic build would refuse to operate on a crate without
   a [[bin]] target.

So the design space for "fat wheels" is disjoint from
cargo-sonic's.

Three real options for CPU-dispatched wheels

Option A — PEP 817 / PEP 825 "Wheel Variants"

The PyPA-track answer. PEP 817 ("Wheel Variants: Beyond Platform
Tags") and PEP 825 ("Wheel Variants: Package Format") are both in
Draft as of 2026 and explicitly enumerate x86-64-v2,
x86-64-v3, x86-64-v4 (and GPU / BLAS variants) as the
motivating use cases. With wheel variants:

• We'd publish e.g. big_code_analysis-0.1.0-cp312-cp312-manylinux_2_28_x86_64.whl
  and big_code_analysis-0.1.0-cp312-cp312-manylinux_2_28_x86_64_v3.whl
  and a znver3 variant.
• pip / uv would query a variant provider plugin and pick the
  best one for the host at install time.

Pros: standard, no runtime dispatch overhead, transparent to the
user. Cons: PEPs are not accepted yet. Until then, pip won't
auto-select variants — users have to pin manually. Tooling
(maturin, auditwheel, cibuildwheel) doesn't emit variant
metadata yet. Targeting this as the eventual mechanism is
reasonable; relying on it for phase 7/9 is not.

Option B — Multiple wheels with custom platform tags, today

A pre-PEP-817 hack: build N wheels with different -C target-cpu
and publish each under a distinct platform tag (e.g. a
manylinux_2_28_x86_64_v3 tag, which is not recognized by
mainstream pip yet). In practice this devolves to publishing
N separately-named packages (big-code-analysis,
big-code-analysis-v3, big-code-analysis-znver3) and asking
users to choose. Awful UX, no auto-detection.

Not recommended.

Option C — Runtime dispatch inside the .so

The numpy / scipy / cryptography model. One wheel, one .so,
runtime CPU dispatch internally. Three sub-flavours:

1. GNU IFUNC. ELF feature where the dynamic linker resolves a
   symbol by calling a resolver function at load time. Per-function
   dispatch. Works inside .so. Requires writing the dispatched
   functions in C (or carefully crafted #[no_mangle] Rust with
   inline asm), one variant per CPU level, plus the resolver.
   Tedious to wire from Rust but technically supported.

2. std::is_x86_feature_detected! + function pointers. Cheap,
   pure-Rust. You compile each hot function multiple times with
   #[target_feature(enable = "…")], store a function pointer
   resolved on first call (or via a OnceLock), and indirect
   through it. Per-call indirection cost is negligible compared to
   AST traversal work.

3. Multi-source / multi-pass compilation, à la numpy. Compile
   the whole hot path multiple times via build-script-driven
   recompilation and link all variants into the same .so. More
   build complexity than (2), but each variant gets full
   intra-function vectorisation rather than just per-function.

For big-code-analysis, the hot paths are integer/branch-heavy
AST traversal in src/spaces.rs and the metric implementations
under src/metrics/. They don't have obvious SIMD-able inner
loops — most of the win from -C target-cpu=znver3 would come
from better scheduling, bmi2 for bit manipulation, and
SHA-NI / VAES in transitive deps. That doesn't make (C3) very
attractive — there's no single inner loop where multi-source
compilation pays off.

Option D — Bundle multiple .sos, dispatch in __init__.py

The "cargo-sonic for Python" approach, but implemented at the
Python layer rather than the linker layer:

• Wheel ships big_code_analysis/_native_baseline.so,
  _native_v3.so, _native_znver3.so, …, each built by maturin
  with different RUSTFLAGS="-C target-cpu=…".
• The package's __init__.py does CPUID-equivalent detection
  (e.g. via platform.processor() parsing or, more robustly,
  reading /proc/cpuinfo and matching feature flags) and
  importlib-imports the right one.
• Each .so exposes the same PyInit_… symbol but under a
  distinct module name (_native_v3, etc.) so they don't clash.

Pros: works today, with the existing maturin-action
workflow, by running the wheel build matrix N times and post-
processing the wheel. Self-contained — no installer cooperation
needed. Cons: wheel size N× (mitigatable with --strip and
keeping the variant list short, à la the znver3 / x86-64-v3
shortlist proposed in #181); first-import does CPU detection.

Recommendation for this issue (phase 7/9)

For the v1 wheel matrix in this issue, ship one wheel per
target with a single conservative -C target-cpu per target,
matching what maturin defaults to today:

• x86_64-unknown-linux-gnu → x86-64 baseline (current default).
• aarch64-unknown-linux-gnu → armv8-a baseline (current default).

That gets the phase-7/9 goal — pip install big-code-analysis
just works on every modern Linux — without coupling it to a
non-yet-standard variant mechanism.

Then file a separate follow-up issue for CPU-dispatched
wheels, with these decision points:

1. Wait for PEP 817 / 825 to be accepted, then ship variants
   (Option A). This is the lowest-effort future-proof path if
   the PEPs land. The Talk Python episode and the WheelNext
   coalition (NVIDIA + Astral + Quansight) suggest momentum, but
   the timeline isn't ours to control.
2. Implement Option D (bundled .sos + __init__.py
   dispatch) as a transitional measure if the first user (the
   znver3 host noted in meta: evaluate cargo-sonic for CPU-dispatched fat binaries on Linux releases #181) actually deploys bca-web via the
   Python bindings and x86-64-baseline performance is a
   measurable problem.
3. Skip CPU dispatch entirely for the Python wheel and rely on
   the cargo install-built native binary path
   (big-code-analysis-cli / bca-web) for performance-sensitive
   deployments. The wheel exists for ergonomics
   (pip install in environments without cargo), not for
   peak throughput. This is the simplest answer and probably the
   right one for v1.

What this means for #181

The cargo-sonic adoption discussion in #181 stays scoped to the
two binary crates (big-code-analysis-cli, big-code-analysis-web),
which is where the proposal already lives. The Python wheel is a
separate problem with a separate solution space — they should not
be wired together in the same release job.

Sources used:

• cargo-sonic README — "one Linux fat executable for a Rust binary"
• PEP 817 – Wheel Variants: Beyond Platform Tags
• PEP 825 – Wheel Variants: Package Format
• Quansight: Python Wheels: from Tags to Variants
• NumPy: CPU build options + dispatcher (multi-source / baseline + runtime dispatch model)
• NumPy: How does the CPU dispatcher work?

[dekobon]

Implementation note: manylinux wheel CI landed

Phase 7/9 implementation lives at .github/workflows/python-wheels.yml.
Three places in the original spec moved with the 2026 best-practices
research; calling them out explicitly so the rationale is on the issue,
not just in the workflow's comment block.

Deviation 1: native ARM runners instead of QEMU

The spec called for docker/setup-qemu-action to emulate aarch64 on
an x86_64 runner. GitHub's ubuntu-24.04-arm standard runners went GA
for private repositories in January 2026 (GitHub
Changelog);
this repository is private, so they apply. Native ARM drops the
aarch64 wheel build from ~15-30 minutes (QEMU) to a few minutes
without any docker / userspace-emulation friction. The maturin-action
picks the matching ARM-native quay.io/pypa/manylinux_2_28_aarch64
container automatically when the workflow runs on an ARM host, so the
diff vs the QEMU path is one matrix entry rather than a setup-qemu
step + a slow build.

QEMU stays the right answer if the repo ever flips back to public-with-
forks-from-private-base scenarios or moves to a runner provider without
ARM hosts.

Deviation 2: abi3 (single wheel per architecture) is on by default

The spec was "abi3 where PyO3 supports it for our feature set; falls
back to per-version wheels otherwise. Document the decision."

The bindings' public surface — PyDict returns, custom exceptions,
no generators, no async — is entirely inside the abi3 (PEP 384
stable-ABI) subset. cargo check -p big-code-analysis-py --no-default-features --features pyo3/extension-module,pyo3/abi3-py312
builds clean locally, and maturin develop && pytest passes all 155
tests against the abi3 binary.

So I wired pyo3/abi3-py312 into [tool.maturin].features in
big-code-analysis-py/pyproject.toml rather than into the CI workflow.
That has two consequences:

1. Wheel matrix is 2 wheels (one per architecture), not 4
   (per Python × per architecture). Each wheel is filename-tagged
   cp312-abi3-... and works on CPython 3.12, 3.13, 3.14+.
2. maturin develop locally builds the same abi3 binary the wheel
   ships. A contributor accidentally adding a non-abi3 PyO3
   dependency trips the local build, not a CI failure 20 minutes
   later.

The smoke-test stage installs each wheel on both Python 3.12 and 3.13
on the matching native architecture to catch the most plausible
forward-compat regression (importlib happily loaded the cp312-abi3
binary on 3.12 but not on 3.13).

Deviation 3: Sigstore attestations come from gh-action-pypi-publish, not a separate step

The spec called for sigstore/gh-action-sigstore-python to sign the
artefacts before upload. pypa/gh-action-pypi-publish@v1.14.0
(release notes)
emits PEP 740 Sigstore-signed attestations automatically when the
publish job has id-token: write and is using Trusted Publishing.
Doubling up would sign the same wheel twice with conflicting
identities and confuse downstream attestation verifiers (e.g.
pypi-attestations verify-pypi checks for a single TP-issued
attestation, not a chain).

So the workflow does not load the sigstore action — but the
attestations are still there, generated by the publish action itself.

Bibliography (sources used for the research)

• PyO3/maturin-action README — manylinux + working-directory + sccache + native ARM patterns
• GitHub Changelog — arm64 standard runners GA in private repos (Jan 2026)
• GitHub Changelog — arm64 hosted runners GA for public repos (Aug 2025)
• PyPA gh-action-pypi-publish v1.14.0 — attestations on by default
• PyPI Trusted Publishing docs
• Configuring OpenID Connect in PyPI (GitHub docs)
• PEP 740 — Index support for digital attestations
• PyO3 user guide — Building and distribution / abi3
• Maturin User Guide — Distribution & manylinux compliance (zig vs docker)
• The cross-referenced earlier comment on cargo-sonic + abi3 dispatch (still stands; meta: evaluate cargo-sonic for CPU-dispatched fat binaries on Linux releases #181 follow-up unchanged).

Action SHAs pinned (verified against gh api …/git/refs/tags/<tag>)

Action	Version	SHA
actions/checkout	v6.0.2	de0fac2e4500dabe0009e67214ff5f5447ce83dd (already used in ci.yml)
actions/setup-python	v6.2.0	a309ff8b426b58ec0e2a45f0f869d46889d02405 (already used in ci.yml)
actions/upload-artifact	v7.0.1	043fb46d1a93c77aae656e7c1c64a875d1fc6a0a (already used in release.yml)
actions/download-artifact	v8.0.1	3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c (already used in release.yml)
PyO3/maturin-action	v1.51.0	e83996d129638aa358a18fbd1dfb82f0b0fb5d3b
pypa/gh-action-pypi-publish	v1.14.0	cef221092ed1bacb1cc03d23a2d87d1d172e277b

Dependabot will keep these moving in lockstep.

One-time setup still required on dekobon/big-code-analysis

RELEASING.md has the full procedure under
"Python wheels (PyPI) → One-time PyPI setup". TL;DR:

1. Claim big-code-analysis on PyPI (or rename via
   [project] name).
2. Register a Trusted Publisher (repo dekobon/big-code-analysis,
   workflow python-wheels.yml, env pypi).
3. Create a pypi GitHub Environment (Settings → Environments).

The first v* tag after step 3 is the canonical end-to-end test —
there is no static token to mask a misconfiguration.

[dekobon]

Code-review followups landed in d0731b8

Ran /code-review xhigh on the wheel-CI commit; six confirmed
findings, all fixed.

#	Finding	Fix
1	pip-audit --strict audited pip-audit's own transitive deps (requests, packaging, cyclonedx-python-lib) instead of the just-built wheel. Could not even meaningfully target the wheel pre-publish (pip-audit looks up advisories by PyPI version, but the just-built version is not yet on PyPI).	Removed the step; comment points at the post-publish follow-up once the wheel acquires runtime deps.
2	Maturin-action's manylinux: "2_28" input prepends --manylinux 2_28, and maturin's --manylinux is an ArgAction::Append alias of --compatibility. The redundant --compatibility manylinux_2_28 in args: double-appended the value, risking a multi-tag wheel filename.	Dropped the duplicate from args:.
3	Aggregate wheels job's failure predicate omitted 'skipped'. On a PR without the python-wheels label, all three needed jobs resolve to skipped, the predicate evaluates false, and the required check turns green with zero wheels built.	Predicate now also fails on 'skipped'.
4	on.push.tags: ['v*'] matched word-prefixed tags like verify-grammar-sync — would trigger an irrevocable PyPI publish under whatever workspace version that commit happens to carry.	Narrowed to 'v[0-9]*'. The publish-job if: retains the event_name == 'push' belt-and-braces to also rule out workflow_dispatch-against-a-tag.
5	Publish job's permissions: block replaced the workflow-level contents: read rather than augmenting it; a future actions/checkout here (e.g. to read CHANGELOG.md for release notes) would fail with a permissions error.	Restated contents: read defensively.
6	The pyproject.toml comment overpromised: cargo test --workspace --all-features does NOT trip non-abi3 PyO3 violations because pyo3/abi3-py312 is in [tool.maturin].features (not the crate's own [features]), so cargo's test path keeps pyo3/auto-initialize active.	Comment corrected to name the actual enforcement gate — maturin develop in the python-test CI job, also covered by make pre-commit.

RELEASING.md also gains:

• A new "Create the python-wheels PR label" step (GitHub does
  not auto-create custom labels; without it the PR-time opt-in
  is unreachable).
• A warning under the pypi environment step that GitHub
  auto-creates a referenced-but-undefined environment with no
  protection rules, so a maintainer who skips that step
  silently bypasses the approval gate on first publish.

Validation: actionlint, markdownlint-cli2, taplo fmt --check, make py-test py-typecheck py-lint py-fmt-check all
clean.

Findings deliberately not fixed

• Multi-tag wheel pattern in the strict-grep verifier —
  defensive against a hypothetical future maturin emitting
  manylinux_2_28_x86_64.manylinux2014_x86_64.whl. Today maturin
  with --manylinux 2_28 emits a single-tag filename; the
  verifier rejecting a multi-tag wheel that does eventually appear
  is the desired behaviour (we'd want to deliberately review
  before shipping that shape).
• Smoke-test language_for_file == "python" brittleness —
  the .pyi stub commits the binding to returning str | None;
  changing that is itself a breaking change tracked under feat(bindings): publish Python bindings via PyO3 / maturin #103.
• Smoke-test matrix mixing include: with top-level python: —
  works correctly today (2×2 cross-product); a future top-level
  arch: array would change the merge semantics, but adding an
  arch is a deliberate matrix edit that would be reviewed in any
  case.