Build source tar.gz (sdist) alongside wheels (bdist_wheel) #285
Conversation
This is a first pass. Probably enough to build conda packages, but not enough for pip or uv to build from. I've tried to be minimally invasive - Keep setuptools as the packaging backend - Keep `setup.py` as the source of truth A more complete overhaul might consider moving to a more recent/focused build library (e.g. PyO3, Maturin); and/or a more recent (e.g. `setup.cfg`, `pyproject.toml`). A diff of file listings before and after show the wheel contents are unchanged, except for version specific paths ```console $ diff -u \ <(zipinfo -1 dist-before/wasmtime-33.0.0.dev362-py3-none-macosx_11_0_arm64.whl|sort) \ <(zipinfo -1 dist.after/wasmtime-33.0.0.dev363-py3-none-macosx_11_0_arm64.whl|sort) --- /dev/fd/11 2025-06-10 09:53:26 +++ /dev/fd/12 2025-06-10 09:53:26 @@ -1,8 +1,8 @@ -wasmtime-33.0.0.dev362.dist-info/LICENSE -wasmtime-33.0.0.dev362.dist-info/METADATA -wasmtime-33.0.0.dev362.dist-info/RECORD -wasmtime-33.0.0.dev362.dist-info/top_level.txt -wasmtime-33.0.0.dev362.dist-info/WHEEL +wasmtime-33.0.0.dev363.dist-info/LICENSE +wasmtime-33.0.0.dev363.dist-info/METADATA +wasmtime-33.0.0.dev363.dist-info/RECORD +wasmtime-33.0.0.dev363.dist-info/top_level.txt +wasmtime-33.0.0.dev363.dist-info/WHEEL wasmtime/__init__.py wasmtime/_bindings.py wasmtime/_config.py ``` refs bytecodealliance#231
| prune tests/bindgen/generated | ||
| prune tests/codegen/generated | ||
| prune wasmtime/bindgen/generated |
There was a problem hiding this comment.
These are the lines I'm least sure of. Whether a source distribution should include generated .py files. For now I've leaned to no, on the presumption they're platform specific and it's easier to add them back later.
| "*-*/*.dll", | ||
| "*-*/*.dylib", | ||
| "*-*/*.so", |
There was a problem hiding this comment.
This is a bit of a kludge, to maintain the pre-existing layout that has directories which are not valid Python module names (due to the dash). It's probably good enough, but I'm open to alternatives.
|
The sdist was built in GHA, and uploaded to Github artifact store https://github.com/bytecodealliance/wasmtime-py/actions/runs/15555224565/artifacts/3294624002 $ zipinfo -1 ~/Downloads/wheels.zip
wasmtime-33.0.0.dev364-py3-none-android_26_arm64_v8a.whl
...
wasmtime-33.0.0.dev364-py3-none-win_arm64.whl
wasmtime-33.0.0.dev364.tar.gzThe sdist included these files from source control, plus generated metadata file
|
|
This probably requires fixing before this PR can be merged. The sdist is not buildable with just uv or pip (as I expected), but for an unexpected reason $ uv venv
Using CPython 3.13.0
Creating virtual environment at: .venv
Activate with: source .venv/bin/activate
$ . .venv/bin/activate
$ /tmp uv pip install ~/Downloads/wheels/wasmtime-33.0.0.dev364.tar.gz
Resolved 2 packages in 60ms
× Failed to build `wasmtime @ file:///Users/alex/Downloads/wheels/wasmtime-33.0.0.dev364.tar.gz`
╰─▶ Package metadata version `33.0.0.dev0` does not match given version `33.0.0.dev364` |
There was a problem hiding this comment.
Thanks for this! FWIW the current setup is an amalgamation of understanding of various folks over time of how to do this, and you seem to be quite knowledgable in this area so if you have recommendations for what to change and how to change things it'd be very much appreciated and I'd be fine changing things!
The main basic property I've been trying to preserve is that if you install the wasmtime package on a supported platform it shouldn't require a Rust compiler but should instead come with a precompiled dynamic library. Ideally this also wouldn't download the precompiled binary for any other platform either. Beyond that though I'm fine to ship whatever works, and I'm by no means wed to the current solution.
Question on this though: is the goal to be able to build this from source? If so wouldn't this need to include Wasmtime's own source code or a script of some form to build the Wasmtime C API? (plus the Rust bindings from this package)
I'm most familiar with the older ways (setuptools). I've used some of the newer ones (e.g. PyO3), but not all (e.g. cibuildwheel)
I may take you up on that, but I'm conscious of keeping this PR reviewable, and not changing the entire build process at once.
agreed, I'm taking care to leave the existing wheels unchanged in layout/contents.
My initial goal was: MVP source distribution for someone building a Conda1 package (as requested by @rxm7706 in #231), as a springboard to familiarise myself with this project and wasmtime. However on closer reading of #231 (comment) a Conda-MVP-sdist may be of limited benefit (given that a Conda build recipe could reportedly just get the source directly from Github).
Building from a source distribution would certainly need some or all of what the scripts in ci/*.py are currently supplying, perhaps present in
|
|
Ideas/options for computing/applying
There is overlap between some of these options. Edit: Changed to numbered list |
|
Initial attempts with option 1 suggest it's not viable. I'll explore 2 and 3. |
|
FWIW the versioning scheme there is purely intended for testing via test.pypi.org on each commit so, in theory, each commit can be tested against the testing registry there. It's a bit of a hack so I wouldn't be opposed to dropping it altogether. I think it'd be reasonable to require some sort of explicit opt-in to publish to test.pypi.org via CI and that would necessitate bumping a hardcoded version or something like that perhaps. I'll be honest in that I haven't really given any thought to what it might look like for this package to use system |
|
Option 2 ( ➜ /tmp uv pip install ~/src/wasmtime-py/dist/wasmtime-33.0.365.tar.gz
Resolved 2 packages in 451ms
Built wasmtime @ file:///Users/alex/src/wasmtime-py/dist/wasmtime-33.0.365.tar.gz
Prepared 1 package in 387ms
Installed 2 packages in 2ms
+ importlib-resources==6.5.2
+ wasmtime==33.0.365 (from file:///Users/alex/src/wasmtime-py/dist/wasmtime-33.0.365.tar.gz)The files that get installed aren't usable, but that's another matter. |
|
Exploratory branch that incorporates running build-rust.py into setup.py https://github.com/moreati/wasmtime-py/tree/setuptools-crimes, with some ugly kludges specifically in moreati@69aa974. |
|
Note to self: possible alternative to fiddling with setuptools cmdclass https://setuptools.pypa.io/en/latest/build_meta.html |
I have integrated running of the ci/download-wasmtime.py and ci/build-rsyt.py into the PEP-517 `build_wheel()` step. This allows a wheel to be built from just a source distribution (as opposed to a git checkout), so e.g. ``` pip install --no-binary=:any: wasmtime-py ``` has a chance of working. The integration mechanism is ci/_custom_build/backend.py, an in-tree PEP-517 build backend that extends the build backend provided by setuptools. This file is excluded from pytest discovery/scanning for expediency and to keep the scope of this change smaller. A future PR could remove this restriction to improve test and style guide coverage. I elected to keep most project metadata in setup.py for now, again to minimize the size of the PR. I recommend a future PR to move the metadata to pyproject.toml. This change incorporates the changes begun in PR bytecodealliance#285, and supercedes it. As with that PR wheels builtin in wasmtime-pr should be unchanged - so in the common case end users do not require a rust compiler, or to build from source.
|
Superceded by #287 |
…p.py (#287) * Use Python "build" package as build frontend, don't directly run setup.py Running setup.py directly is deprecated, from 2025-10-31 it won't be supported. The Python ecosystem is moving toward seperation of build frontend and backend. This commit switches to the "build" frontend, and keeps setuptools as the backend. Two `setup()` keywords have been altered `packages` was incomplete, but happened to work. Setuptools now warns about such a condition, which added noise in the build output. `test_suite` was unnnecessary because pytest is used for testing. It has been deprecated since by setuptools 41.5.0 (Oct 2019), and its presence emitted a warning that added noise to build output. This change should - allow wasmtime-py wheels to be built, published, and installed. It should not - alter versioning behaviour (specifically .devNNN tagging) - alter wheel contents It does not - allow building of an sdist (source tar.gz distribution) Risks - `-C=--build-option=--plat-name -C=--build-option=...` is ugly and possibly fragile. A future release of build or setuptools may break it. - No version constraints are placed on setuptools in pyproject.toml. I think this is the minimum viable change that fixes #286. Other improvements can be made seperately (e.g. migrating more metadata from setup.py to pyproject.toml). See - https://build.pypa.io/en/stable/ - https://packaging.python.org/en/latest/guides/writing-pyproject-toml/ - https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-test-suite - pypa/pip#11859 - pypa/setuptools#2491 - pypa/setuptools#3882 * Revert change to packages keyword in setup.py CI was failing because the "Setup wasmtime-py CI" action in .github/actions/ setup.yml installs `.[testing]` before ci/build-rust.py has been run. Thus wasmtime/bindgen/generated and wasmtime/bindgen/generated/imports didn't exist yet. See - https://github.com/bytecodealliance/wasmtime-py/actions/runs/15640817846/job/44067456534 * Build source distributions (sdist) alongside wheels (bdist_wheel) I have integrated running of the ci/download-wasmtime.py and ci/build-rsyt.py into the PEP-517 `build_wheel()` step. This allows a wheel to be built from just a source distribution (as opposed to a git checkout), so e.g. ``` pip install --no-binary=:any: wasmtime-py ``` has a chance of working. The integration mechanism is ci/_custom_build/backend.py, an in-tree PEP-517 build backend that extends the build backend provided by setuptools. This file is excluded from pytest discovery/scanning for expediency and to keep the scope of this change smaller. A future PR could remove this restriction to improve test and style guide coverage. I elected to keep most project metadata in setup.py for now, again to minimize the size of the PR. I recommend a future PR to move the metadata to pyproject.toml. This change incorporates the changes begun in PR #285, and supercedes it. As with that PR wheels builtin in wasmtime-pr should be unchanged - so in the common case end users do not require a rust compiler, or to build from source.
This is a first pass. Probably enough to build conda packages, but not enough for pip or uv to build from. I've tried to be minimally invasive
setup.pyas the source of truthA more complete overhaul might consider moving to a more recent/focused build library (e.g. PyO3, Maturin); and/or a more recent (e.g.
setup.cfg,pyproject.toml).A diff of file listings before and after show the wheel contents are unchanged, except for version specific paths
refs #231
TODO
<major>.0.0.dev0does not match given version<major>.0.0.dev364