Use Pyodide from uv python #2327
Description
Description
I was helping someone, and wanted to show them that they could build pyodide wheels from cibuildwheel. They were on Ubuntu, so didn't have quick access to Python 3.12, so I recommended uv python install 3.12. Unfortunately, when trying to then run cibuildwheel (I think with uvx --python 3.12 cibuildwheel, they got an error when cibuildwheel was trying to make a venv. It seems the stand-alone Python installers can't make virtual environments if they've been symlinked, or something like that. I'll try to reproduce soon, but wanted to drop an issue here before I forgot about it. I hope we can work around the issue.
I think the issue is astral-sh/python-build-standalone#380.
I think this only affects pyodide, since the other builds are in Docker.
Reproducer
$ docker run --rm -it ubuntu:24.10
# apt update && apt install -y curl git
# curl -LsSf https://astral.sh/uv/install.sh | sh
# source $HOME/.local/bin/env
# uv python install 3.12
# git clone https://github.com/scikit-hep/boost-histogram
# cd boost-histogram
# uvx cibuildwheel --only cp312-pyodide_wasm32
_ _ _ _ _ _ _
___|_| |_ _ _|_| |_| |_ _ _| |_ ___ ___| |
| _| | . | | | | | . | | | | | -_| -_| |
|___|_|___|___|_|_|___|_____|_|_|___|___|_|
cibuildwheel version 2.23.1
Build options:
platform: pyodide
allow_empty: False
architectures: wasm32
build_selector:
build_config: cp312-pyodide_wasm32
skip_config:
requires_python: >=3.8
enable: ['cpython-freethreading', 'cpython-prerelease', 'pypy']
output_dir: /boost-histogram/wheelhouse
package_dir: /boost-histogram
test_selector:
skip_config: cp*-musllinux_* cp313t-*win* pp311-*
before_all:
before_build:
before_test:
build_frontend:
*: build[uv]
cp312-pyodide_wasm32:
name: build
args: ['--exports', 'whole_archive']
build_verbosity: 0
config_settings:
container_engine: docker
dependency_constraints: pinned
environment:
PIP_ONLY_BINARY="numpy"
PIP_PREFER_BINARY="1"
manylinux_images: None
musllinux_images: None
repair_command:
test_command:
*: pytest -n auto --benchmark-disable {project}/tests
cp312-pyodide_wasm32: pytest --benchmark-disable {project}/tests
test_extras:
test_groups:
test
test_requires:
cloudpickle
hypothesis>=6.0
pytest-benchmark
pytest>=6.0
pytest-xdist
Cache folder: /root/.cache/cibuildwheel
Here we go!
Building cp312-pyodide_wasm32 wheel
CPython 3.12 Pyodide
Setting up build environment...
+ Download https://github.com/pypa/get-virtualenv/blob/20.29.3/public/virtualenv.pyz?raw=true to /root/.cache/cibuildwheel/virtualenv-20.29.3.pyz
+ /root/.cache/uv/archive-v0/m8zF1t0bWCOV1lDOploP5/bin/python -sS /root/.cache/cibuildwheel/virtualenv-20.29.3.pyz --activators= --no-periodic-update --pip=embed --no-setuptools --no-wheel --python /root/.cache/uv/archive-v0/m8zF1t0bWCOV1lDOploP5/bin/python /tmp/cibw-run-zmv0cabc/cp312-pyodide_wasm32/build/venv
created virtual environment CPython3.12.9.final.0-64 in 259ms
creator CPython3Posix(dest=/tmp/cibw-run-zmv0cabc/cp312-pyodide_wasm32/build/venv, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=embed, via=copy, app_data_dir=/root/.local/share/virtualenv)
added seed packages: pip==25.0.1
+ python -m pip install --upgrade pip -c /root/.cache/uv/archive-v0/m8zF1t0bWCOV1lDOploP5/lib/python3.12/site-packages/cibuildwheel/resources/constraints-pyodide312.txt
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
PYTHONHOME = (not set)
PYTHONPATH = (not set)
program name = '/tmp/cibw-run-zmv0cabc/cp312-pyodide_wasm32/build/venv/bin/python'
isolated = 0
environment = 1
user site = 1
safe_path = 0
import site = 1
is in build tree = 0
stdlib dir = '/install/lib/python3.12'
sys._base_executable = '/root/.local/share/uv/python/cpython-3.12.9-linux-x86_64-gnu/bin/python3.12'
sys.base_prefix = '/install'
sys.base_exec_prefix = '/install'
sys.platlibdir = 'lib'
sys.executable = '/tmp/cibw-run-zmv0cabc/cp312-pyodide_wasm32/build/venv/bin/python'
sys.prefix = '/install'
sys.exec_prefix = '/install'
sys.path = [
'/install/lib/python312.zip',
'/install/lib/python3.12',
'/install/lib/python3.12/lib-dynload',
]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'
Current thread 0x00007f8c21d3c740 (most recent call first):
<no Python frame>
Traceback (most recent call last):
File "/root/.cache/uv/archive-v0/m8zF1t0bWCOV1lDOploP5/bin/cibuildwheel", line 12, in <module>
sys.exit(main())
^^^^^^
File "/root/.cache/uv/archive-v0/m8zF1t0bWCOV1lDOploP5/lib/python3.12/site-packages/cibuildwheel/__main__.py", line 49, in main
main_inner(global_options)
File "/root/.cache/uv/archive-v0/m8zF1t0bWCOV1lDOploP5/lib/python3.12/site-packages/cibuildwheel/__main__.py", line 184, in main_inner
build_in_directory(args)
File "/root/.cache/uv/archive-v0/m8zF1t0bWCOV1lDOploP5/lib/python3.12/site-packages/cibuildwheel/__main__.py", line 351, in build_in_directory
platform_module.build(options, tmp_path)
File "/root/.cache/uv/archive-v0/m8zF1t0bWCOV1lDOploP5/lib/python3.12/site-packages/cibuildwheel/pyodide.py", line 244, in build
env = setup_python(
^^^^^^^^^^^^^
File "/root/.cache/uv/archive-v0/m8zF1t0bWCOV1lDOploP5/lib/python3.12/site-packages/cibuildwheel/pyodide.py", line 131, in setup_python
call(
File "/root/.cache/uv/archive-v0/m8zF1t0bWCOV1lDOploP5/lib/python3.12/site-packages/cibuildwheel/util.py", line 154, in call
result = subprocess.run(
^^^^^^^^^^^^^^^
File "/root/.local/share/uv/python/cpython-3.12.9-linux-x86_64-gnu/lib/python3.12/subprocess.py", line 573, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/tmp/cibw-run-zmv0cabc/cp312-pyodide_wasm32/build/venv/bin/python', '-m', 'pip', 'install', '--upgrade', 'pip', '-c', '/root/.cache/uv/archive-v0/m8zF1t0bWCOV1lDOploP5/lib/python3.12/site-packages/cibuildwheel/resources/constraints-pyodide312.txt']' returned non-zero exit status 1.