cpython install directory missing when building package out of tree

[DavidHaintz]

Hi,

since I have the opportunity now, first many thanks for this awesome project!

But sadly I'm stuck with a bug while trying to build a package for pyodide.

What's my goal?

I wanted to use tiktoken from openai, which doesn't have any none wheel. So I want to build it myself, as described in https://pyodide.org/en/stable/development/new-packages.html#building-python-wheels-out-of-tree

What's the problem?

When running pyodide build I'm getting following error:

ERROR [Errno 2] No such file or directory: '/some/path/pyodide/cpython/installs/python-3.11.2/sysconfigdata/_sysconfigdata__emscripten_wasm32-emscripten.py'

How to reproduce?

Step 1:

Run Ubuntu 20 on Windows WSL (I hope that's not the problem, but if yes I can switch to a server/vm).

Step 2: Install stuff

Install Python3.10

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install python3.10

Install Python3.10 Pip

curl -sS https://bootstrap.pypa.io/get-pip.py | python3.10

Install pyodide

git clone https://github.com/pyodide/pyodide
cd pyodide
python3.10 -m pip install -e pyodide-build
make -C emsdk
make -C cpython
source emsdk/emsdk/emsdk_env.sh
cd ..

Install rust/cargo (for tiktoken build)

curl https://sh.rustup.rs -sSf | sh

Step 3: Build package

git clone https://github.com/openai/tiktoken.git
cd tiktoken
pyodide build

Here the mentioned error appears.
The directory pyodide/cpython/installs doesn't exist at all on my environment.

[ryanking13]

Could you try using pyodide-build 0.22.1 installed from PyPI, not installing from the Pyodide main branch? I think you are mixing in-tree build and out-of-tree build.

[DavidHaintz]

Sadly I had to install it from source, because the PyPi package didn't provide me a pyodide executable.
BUT you were completely right. I checked out a stable version branch and already saw a change when running pyodide (it installed some stuff, like cpython).

This fixed my problem. Thanks a lot!

Lesson learned: PEBKAC, use stable branch.