[cpython] Migrate to Nanvix SDK#782
Merged
Merged
Conversation
Build the static interpreter with SDK Clang and exact SDK-built dependencies while keeping Nanvix downloads runtime-only. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3a354ddf-0f65-44b2-a2cb-81d4662a3587
There was a problem hiding this comment.
Pull request overview
This PR migrates the Nanvix CPython port from the legacy GCC/toolchain-python flow to a pinned Nanvix Clang/LLVM SDK image, with build-time headers/libs provided via a separate dependency buildroot and a runtime sysroot kept execution-only. It also updates CI packaging and expands/adjusts the validated test set via targeted skips for Nanvix runtime limitations and libc tolerance deviations.
Changes:
- Switch
Makefile.nanvixand.nanvix/*orchestration to use the pinned Nanvix SDK (Clang/LLVM tools, SDK manifest checks, dependency buildroot under/mnt/buildroot). - Adjust packaging, Docker build behavior, and docs/CI workflows to reflect the SDK + runtime/buildroot split and updated Nanvix version.
- Add/extend Nanvix-specific test skips for FIFO/device-node/group APIs and math/cmath fixture tolerances.
Show a summary per file
| File | Description |
|---|---|
| Makefile.nanvix | Use SDK-provided clang/LLVM tools and dependency buildroot for Nanvix builds; updated configure/build/clean flows. |
| Lib/test/test_tarfile.py | Skip FIFO-related tarfile test on Nanvix. |
| Lib/test/test_stat.py | Skip FIFO test on Nanvix. |
| Lib/test/test_shutil.py | Skip named-pipe related tests on Nanvix. |
| Lib/test/test_posix.py | Skip unsupported identity/FIFO/device/group tests on Nanvix. |
| Lib/test/test_pathlib.py | Skip FIFO test on Nanvix. |
| Lib/test/test_math.py | Skip strict ULP fixture-based tests on Nanvix libc 0.20.0. |
| Lib/test/test_glob.py | Skip named-pipe glob test on Nanvix. |
| Lib/test/test_cmath.py | Skip tolerance-sensitive cmath tests on Nanvix libc 0.20.0. |
| .nanvix/z.py | Update z-script build/setup behavior for SDK + buildroot separation; stage lxml runtime payload. |
| .nanvix/package.py | Adjust release staging; copy static libpython archive directly. |
| .nanvix/nanvix.toml | Bump Nanvix version to 0.20.0. |
| .nanvix/NANVIX.md | Update documentation for SDK build flow, dependency versions, and expanded test set. |
| .nanvix/lxml.py | Generate Modules/Setup.local using buildroot paths (not sysroot). |
| .nanvix/config.py | Define pinned SDK image digest, target triple, SDK ABI/version, and docker paths. |
| .nanvix/build.py | Pass buildroot into make invocation; adjust lxml setup generation accordingly. |
| .nanvix/_test.py | Formatting-only change. |
| .nanvix/_docker.py | Mount buildroot, set container HOME/TMPDIR, and fingerprint SDK+buildroot to force clean rebuilds. |
| .gitignore | Add ignores for new Nanvix build artifacts. |
| .github/workflows/nanvix-ci.yml | Update reusable workflow inputs to the pinned SDK image; tighten release asset handling and mkramfs sourcing. |
| .nanvix/docker/Dockerfile | Remove legacy toolchain-python Dockerfile. |
| .github/workflows/docker-image.yml | Remove legacy toolchain-python image build workflow. |
Review details
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 21/22 changed files
- Comments generated: 3
- Review effort level: Low
Comment on lines
+67
to
+71
| # Cross ``make install`` does not reliably stage the static interpreter | ||
| # archive, so copy the just-built SDK archive directly. | ||
| python_archive = paths.repo_root() / f"libpython{config.PYTHON_VERSION}.a" | ||
| if python_archive.is_file(): | ||
| shutil.copy2(python_archive, br / "lib" / python_archive.name) |
Comment on lines
121
to
+127
| $(CONFIGURED_MARKER): | ||
| ifdef CONFIG_NANVIX_DOCKER | ||
| @echo "Running configure inside Docker..." | ||
| $(DOCKER_RUN) sh -c '$(CONFIGURE_ENV) ./configure $(CONFIGURE_OPTS)' | ||
| else | ||
| @grep -Eq '"sdk_version"[[:space:]]*:[[:space:]]*"v0.20.0-sdk.1"' $(SDK_MANIFEST) || { \ | ||
| echo "SDK version mismatch in $(SDK_MANIFEST)"; exit 1; \ | ||
| } | ||
| @grep -Eq '"c_abi"[[:space:]]*:[[:space:]]*"i686-nanvix-sysv-1"' $(SDK_MANIFEST) || { \ | ||
| echo "SDK C ABI mismatch in $(SDK_MANIFEST)"; exit 1; \ | ||
| } |
| --disable-shared \ | ||
| --build=x86_64-pc-linux-gnux32 \ | ||
| --build=x86_64-pc-linux-gnu \ | ||
| --host=i686-nanvix \ |
Transfer the complete SDK-built install tree to Windows CI instead of mixing the new runtime with an older published interpreter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3a354ddf-0f65-44b2-a2cb-81d4662a3587
Comment on lines
+147
to
+151
| rm -rf build | ||
| rm -f pybuilddir.txt _bootstrap_python Programs/_testembed Programs/_freeze_module | ||
| rm -f Python/deepfreeze/*.[co] Python/frozen_modules/*.h | ||
| rm -f Python/frozen_modules/MANIFEST Include/pydtrace_probes.h | ||
| rm -f $(CONFIGURED_MARKER) python python$(EXE) libpython3.12.a |
Comment on lines
+292
to
+304
| cache_dir = nanvix_root() / "cache" | ||
| candidates = ( | ||
| list(cache_dir.glob(f"lxml-{self.config.machine}-*")) | ||
| if cache_dir.is_dir() | ||
| else [] | ||
| ) | ||
| if not candidates: | ||
| raise FileNotFoundError( | ||
| "lxml release archive is missing from .nanvix/cache" | ||
| ) | ||
| try: | ||
| br.install_dep( | ||
| fallback_dep, | ||
| machine=platform, | ||
| deployment_mode=mode, | ||
| memory_size=memory, | ||
| gh_token=gh_token, | ||
| _release=release, | ||
| ) | ||
| installed = True | ||
| break | ||
| except SystemExit: | ||
| continue | ||
|
|
||
| if not installed: | ||
| log.warning(f"No compatible fallback asset for {dep_name}") | ||
| return | ||
|
|
||
| # --- CPython-specific: extract python-packages/ (e.g. lxml) --- | ||
| cache_dir = buildroot.parent / "cache" | ||
| asset_prefix = f"{dep_name}-{platform}-" | ||
| for cached in sorted(cache_dir.iterdir()) if cache_dir.is_dir() else []: | ||
| if not cached.name.startswith(asset_prefix): | ||
| continue | ||
| self._extract_python_packages(cached, buildroot) | ||
| break | ||
|
|
||
| def _extract_python_packages(self, asset_path: Path, buildroot: Path) -> None: | ||
| """Extract ``python-packages/`` from an archive into *buildroot*.""" | ||
| import tarfile | ||
| import zipfile | ||
|
|
||
| with tempfile.TemporaryDirectory() as tmpdir: | ||
| extract_dir = Path(tmpdir) / "extracted" | ||
| extract_dir.mkdir() | ||
|
|
||
| if zipfile.is_zipfile(asset_path): | ||
| with zipfile.ZipFile(asset_path) as zf: | ||
| for member in zf.namelist(): | ||
| if "python-packages" not in member: | ||
| continue | ||
| if os.path.isabs(member) or ".." in member.split("/"): | ||
| continue | ||
| dest = (extract_dir / member).resolve() | ||
| if not dest.is_relative_to(extract_dir.resolve()): | ||
| continue | ||
| zf.extract(member, extract_dir) | ||
| else: | ||
| with tarfile.open(str(asset_path), "r:*") as tf: | ||
| pkg_members = [ | ||
| m | ||
| for m in tf.getmembers() | ||
| if "python-packages" in m.name | ||
| and not os.path.isabs(m.name) | ||
| and ".." not in m.name.split("/") | ||
| ] | ||
| if not pkg_members: | ||
| return | ||
| try: | ||
| tf.extractall( | ||
| str(extract_dir), members=pkg_members, filter="data" | ||
| ) | ||
| except TypeError: | ||
| tf.extractall(str(extract_dir), members=pkg_members) | ||
|
|
||
| for pkg_src in extract_dir.rglob("python-packages"): | ||
| if not pkg_src.is_dir(): | ||
| continue | ||
| pkg_dst = buildroot / "python-packages" | ||
| pkg_dst.mkdir(parents=True, exist_ok=True) | ||
| for item in pkg_src.iterdir(): | ||
| target = pkg_dst / item.name | ||
| if item.is_dir(): | ||
| if target.exists(): | ||
| shutil.rmtree(target) | ||
| shutil.copytree(item, target) | ||
| else: | ||
| shutil.copy2(item, target) | ||
| log.info(f"Installed python packages from {asset_path.name}") | ||
| break | ||
| archive = max(candidates, key=lambda path: path.stat().st_mtime_ns) | ||
| destination = nanvix_root() / "buildroot" / "python-packages" |
Comment on lines
54
to
+56
| f"CONFIG_NANVIX=y", | ||
| f"NANVIX_HOME={nanvix_home}", | ||
| f"NANVIX_TOOLCHAIN={nanvix_toolchain}", | ||
| f"NANVIX_SDK_ROOT={config.DOCKER_SDK_PATH}", | ||
| f"NANVIX_BUILDROOT={buildroot}", |
Preserve the SDK-built test tree while staging nanvixd.exe and image tools from the matching Windows runtime. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3a354ddf-0f65-44b2-a2cb-81d4662a3587
| --disable-shared \ | ||
| --build=x86_64-pc-linux-gnux32 \ | ||
| --build=x86_64-pc-linux-gnu \ | ||
| --host=i686-nanvix \ |
Comment on lines
121
to
+125
| $(CONFIGURED_MARKER): | ||
| ifdef CONFIG_NANVIX_DOCKER | ||
| @echo "Running configure inside Docker..." | ||
| $(DOCKER_RUN) sh -c '$(CONFIGURE_ENV) ./configure $(CONFIGURE_OPTS)' | ||
| else | ||
| @grep -Eq '"sdk_version"[[:space:]]*:[[:space:]]*"v0.20.0-sdk.1"' $(SDK_MANIFEST) || { \ | ||
| echo "SDK version mismatch in $(SDK_MANIFEST)"; exit 1; \ | ||
| } | ||
| @grep -Eq '"c_abi"[[:space:]]*:[[:space:]]*"i686-nanvix-sysv-1"' $(SDK_MANIFEST) || { \ |
Comment on lines
+292
to
+296
| cache_dir = nanvix_root() / "cache" | ||
| candidates = ( | ||
| list(cache_dir.glob(f"lxml-{self.config.machine}-*")) | ||
| if cache_dir.is_dir() | ||
| else [] |
This was referenced Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Local validation