Source-build darwin/iOS Python + Android dep-wheel layout for 3.13+#10
Merged
Conversation
…3/3.13.13/3.14.5 Move iOS/macOS off the beeware Python-Apple-support repo onto CPython's standard build mechanism, and stop hand-pinning the Linux python-build-standalone release. macOS (build_macos.py): universal2 Python.framework built from source for all versions. OpenSSL + xz (+ zstd on 3.14) built universal2 from source (the macOS SDK ships neither OpenSSL nor lzma.h); OpenSSL is shared and bundled into the framework with @rpath install names, matching the released layout; binaries stripped. iOS (build_ios.py): one unified path using CPython's in-tree `Apple build iOS` tool. 3.14 is native; 3.13/3.12 apply a vendored back-port patch (ios_patches/) that adds the Apple tooling (+ the PEP 730 runtime for 3.12) -- no dependency on the Python-Apple-support repo. Reshapes the cross-build output into both the dart bundle and the mobile-forge install/support tree (per-arch dep dirs, VERSIONS, bin stub, platform-config sysconfig), and embeds the iOS deployment version into HOST_GNU_TYPE so mobile-forge's crossenv resolves the iOS release. Linux (resolve_pbs.py): auto-resolve the newest python-build-standalone release for the target micro+arch (PYTHON_DIST_RELEASE kept as optional override); keep x86_64_v2. Workflows: bump matrix to 3.12.13/3.13.13/3.14.5, darwin runner -> macos-26, and only publish GitHub release assets from the main branch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
serious_python loads the stdlib from python-stdlib/ and C extensions from python-xcframeworks/*.fwork, so the full un-pruned stdlib carried inside the iOS Python.xcframework (lib/python3.X, ~200 MB incl. the CPython test suite) and the per-slice lib-dynload/*.so were dead weight in shipped apps. Drop them in package-ios-for-dart.sh: the iOS dart artifact goes from ~89 MB to ~22 MB compressed (308 MB -> 70 MB extracted), with all runtime essentials intact. Also extend the per-platform stdlib excludes (darwin/android/linux/windows) to drop a few more never-needed items: lib2to3 (deprecated; removed upstream in 3.13+), the curses C extensions (the curses package is already excluded), and xxsubtype. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Apple build tool leaves the iOS framework binary and extension modules unstripped. Strip local symbols (strip -x, keeps exported PyInit_/C-API symbols needed for dynamic linking) in build_ios.py's reshape, before they become python-xcframeworks. Xcode re-signs on embed. ~4 MB off the extracted iOS dart artifact (70 -> 66 MB). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Running an app in embedded mode never enters the interactive REPL, so the REPL/dev-only modules are dead weight. Confirmed via import trace (a plain script imports none of them; site.py only touches _pyrepl/rlcompleter inside the interactive hook). Remove from all platform excludes: _pyrepl (~300 KB), rlcompleter, tabnanny, the easter eggs this/ antigravity, and the frozen demo modules __hello__/__phello__. Deliberately kept (a library or breakpoint() may import them): unittest (mock), pdb/bdb, pydoc, doctest, code/codeop, multiprocessing, venv. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CPython's official Android tooling, used by python-build's 3.13+ build
path, builds OpenSSL/bzip2/libffi/xz/SQLite as part of the host build
and installs them intermixed under $PREFIX/{include,lib}/. mobile-forge
make_dep_wheels.py expects each as a sibling per-lib install dir with
its own include/+lib/ — which is what python-build's 3.12 path
produces (by downloading separate prebuilt tarballs).
For 3.13/3.14 today, the sibling install dirs don't exist and the
generated support/<X.Y>/android/VERSIONS file lists no deps, so
make_dep_wheels.py emits zero Android dep wheels. Recipes with host
requirements like `openssl>=3.0.12` (cryptography) then fail at
`pip install --only-binary=:all:` with "No matching distribution found
for openssl".
Add extract_mobile_forge_deps.py: after the CPython host build copy,
detect each bundled lib's version from its header/pkgconfig (or the
known-pinned value for bzip2), materialize a sibling install dir at
install/android/<abi>/<lib>-<ver>-<N>/{include,lib}/, and append a
corresponding `<lib>: <ver>-<N>` entry to VERSIONS. Build number
starts at 1 for each dep, independent of the 3.12 counter.
Idempotent — re-running replaces sibling dirs and de-duplicates VERSIONS
entries so a tree can be re-extracted in place without manual cleanup.
After this lands, `source ./setup.sh 3.13.13` will produce Android dep
wheels for arm64-v8a + x86_64, and recipes that host-depend on
openssl/bzip2/libffi/xz/sqlite resolve normally.
CPython 3.14's Android tooling emits `lib/python<X.Y>/build-details.json` that downstream cross-compile tooling (notably `maturin`) reads to decide where libpython lives, where the headers are, and which pkg-config dir to consult. The values are absolute build-time paths rooted at `/usr/local`, so on every consumer machine the file claims libpython sits at `/usr/local/lib/libpython3.14.so` — which is empty. That makes `maturin` add `-L/usr/local/lib` and the linker fails with `unable to find library -lpython3` even when libpython is sitting right there at the install's actual `lib/`. Add a one-shot rewrite to `normalize_mobile_forge_install.py` so the JSON gets re-anchored to the install path at the same time we append the sysconfigdata relocation block. Idempotent (no `/usr/local` remaining after first pass). 3.13 is unaffected because it doesn't ship build-details.json.
# Conflicts: # .github/workflows/build-python-version.yml
`rewrite_build_details_json` hard-coded `/usr/local` as the build-time prefix to substitute. That works for the only consumer schema we ship today (3.14's CPython Android tooling installs to /usr/local), but it violates the narrow-substitution discipline #9 introduced for the sysconfig relocator: an upstream CPython change that moves the install root to anything else would silently miss the rewrite, and unrelated strings under /usr/local that future schema fields might carry would get clobbered. Switch the substitution source to the JSON's own `base_prefix` field. That field is the canonical record of where Python was installed at build time — by construction every other absolute path in the file shares it as a prefix, and reading it from the data instead of from this script's constants means CPython tooling drift gets handled automatically. Idempotent for the same reason as before: once `base_prefix` has been overwritten with the consumer's install path, the build-time string no longer appears anywhere in the file. The early-return on `build_time_prefix == prefix_str` makes that explicit so the function can be safely re-invoked on an already-rewritten tree. Adds android/tests/test_built_build_details_json.py covering: - at least one build-details.json is shipped per ABI (catches the function getting dropped from main() or CPython silently stopping emission of the file) - no `/usr/local` strings survive (narrower than the anchored-at-prefix check below, but targeted at the historical pre-fix shape) - every documented absolute-path field is anchored at the install prefix (catches partial rewrites and build-time-prefix changes) - libpython.dynamic_stableabi points at a file that exists on disk (the linker `-L` target — catches the tarball shipping with the path right but the file missing) Gated `@requires_python("3.14")` because 3.13 and 3.12 don't ship build-details.json (CPython's Android tooling added it in 3.14).
Merge of origin/main into darwin-std-pkg landed two copies of the `if: github.ref == 'refs/heads/main'` guard on the publish-release job: one above `needs:` and one below it. YAML mappings can't have duplicate keys, so GitHub Actions rejected the workflow at parse time: Invalid workflow file: .github/workflows/build-python-version.yml#L1 (Line: 204, Col: 5): 'if' is already defined Both copies were the same condition (added independently on each side of the merge), so removing the second one is purely syntactic — no behavior change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two largely independent strands rolled into one branch:
darwin/build_ios.py+darwin/build_macos.pyand a set of vendored CPython patches. This was the original motivation for thedarwin-std-pkgbranch.build-details.jsonrewrite for 3.13+ so mobile-forge'smake_dep_wheels.pycan produce Android wheels for native deps (openssl, bzip2, libffi, xz, sqlite) on the official CPython Android tooling path, and somaturin's linker actually findslibpython3.soon Mac developer machines.Highlights
darwin / iOS source-build flow
darwin/build_ios.py+darwin/build_macos.py— new entry points for the darwin-side build, replacing the beeware Python-Apple-support recipe path on 3.12 and adding a fresh source-build flow for 3.13 / 3.14. Emitsinstall/+support/trees in the layout mobile-forge expects.darwin/ios_patches/3.12/anddarwin/ios_patches/3.13/— CPython patches needed to build for iOS from source on each version.darwin/macos_support/app-store-compliance.patch— App Store compliance fixup for the macOS framework.darwin/package-{ios,macos}-for-dart.sh— refresh the dart-side packaging to match the new layout.Android dep-wheel layout (3.13+)
android/extract_mobile_forge_deps.py(new) +android/build.sh(wires it in) — CPython's official Android tooling drops OpenSSL / bzip2 / libffi / xz / sqlite intermixed insidepython-<ver>/lib/. mobile-forge expects them as siblinginstall/android/<abi>/<lib>-<ver>-<N>/install trees (the 3.12 layout). The new script reorganizes the post-install tree to match, detecting each lib's version from headers / pkgconfig, copying the right files, and appending<lib>: <ver>-<N>entries tosupport/<X.Y>/android/VERSIONS. Build numbers start at 1 per lib for 3.13+, independent of the 3.12 counters.android/normalize_mobile_forge_install.py— addsrewrite_build_details_jsonso thebuild-details.jsonshipped on 3.14 has its absolute paths re-anchored at the consumer's install prefix. Uses the JSON's ownbase_prefixfield as the substitution source rather than hard-coding/usr/local, matching the narrow-prefix discipline Fix sysconfig relocator mangling NDK paths under/usr/local#9 introduced for the sysconfig relocator. Without this,maturinreadslibpython.dynamic_stableabi = /usr/local/lib/libpython3.14.sostraight through and the consumer linker fails withunable to find library -lpython3.Test framework follow-ups
android/tests/test_built_build_details_json.py(new) —@post_build,@requires_python("3.14"). Asserts every shippedbuild-details.jsonhas its paths anchored at the install prefix, that no/usr/localstrings leak through, and thatlibpython.dynamic_stableabipoints at a file that exists on disk.CI / workflow
.github/workflows/build-python-version.yml—macos-26runner (needed for the newer Xcode required by 3.14 iOS source-build) and the matrix expansion across 3.12 / 3.13 / 3.14 for darwin. De-duplicatedif:key on thepublish-releasejob from the merge with main.Merge of main
Includes upstream changes from main (PR #8: NDK relocation fix, PR #9: relocator narrowing + Android test framework). My
rewrite_build_details_jsonadapted to follow the narrow-substitution model #9 introduced.Test plan
rewrite_build_details_jsonis idempotent.android/tests/test_built_build_details_json.pypasses against a real built tree.🤖 Generated with Claude Code