Learning PyO3

[Hamatti]

Roughly a week ago, I asked in Mastodon:

Any (non-corporate) open source Python projects out there that could use help with technical documentation?

One of the people who replied was David who's the maintainer of PyO3 project.

Before I can tackle any of the documentation projects for it, I need to learn how to use it.

Contributing to these notes

If someone ends up here through some of the issues / pull requests / randomly and has constructive advice and willingness to help solve technical or documentation related issues, feel free to participate in the discussion. Other type of commentary I'd kindly ask to refrain from (toot at me at @hamatti@mastodon.world instead) so this thread can stay on topic and be a potentially helpful thing to read in the future.

[Hamatti]

A good place to start learning something new is from the front page of the documentation!

Especially when I'm interested in improving the documentation, I like to follow the get started guide to the letter to see what the current experience is.

Set up a Python project and install maturin

(note: ➜ is my zsh prompt)

➜ mkdir string_sum
➜ cd string_sum
➜ python -m venv .env
➜ source .env/bin/activate
➜ pip install maturin
Successfully installed maturin-1.4.0

maturin is another project of PyO3 that is used to

Build and publish crates with pyo3, rust-cpython, cffi and uniffi bindings as well as rust binaries as python packages with minimal configuration.

➜ maturin init
✔ 🤷 Which kind of bindings to use?
  📖 Documentation: https://maturin.rs/bindings.html · pyo3
  ✨ Done! Initialized project /pyo3/string_sum

I chose pyo3 bindings and now I have a couple of files in my filesystem:

➜ tree
.
├── Cargo.toml
├── pyproject.toml
└── src
    └── lib.rs

2 directories, 3 files

I then ran

➜ maturin develop

which took a minute or two on the first run to download and compile all the dependencies and the string_sum itself.

Let's test if it worked:

➜ python
Python 3.12.1 (main, Dec 16 2023, 15:35:28) [Clang 12.0.0 (clang-1200.0.32.29)] on darwin
>>> import string_sum
>>> string_sum.sum_as_string(42, 144)
'186'

Yay! I managed to get through the first tutorial of installing, setting up, building and using a Python module, written in Rust.

[Hamatti]

Now that I know I'm able to use the project, I want to look into setting up a development environment so I can see how to make changes to the documentation and build the docs.

Following the Contributing guidelines, I cloned the project. When I tried to build the docs with the instructed

➜ nox -s docs -- open

I ran into an issue of not having nox installed.

I took me a while to figure out that I need to install nox separately as a global tool (or rather what I ended up doing, running nox with pipx.

Note

I'll consider adding a clearer mention of that into the documentation once I get that far. It is explicitly mentioned in the user guide part but it's used already one section before.

Next, I wanted to build the user guide with pipx run nox -s build-guide -- --open and to do that, I installed mdBook:

➜ cargo install mdbook
[ -- snip bunch of deps -- ]
Installed package `mdbook v0.4.37` (executable `mdbook`)

and the ran

pipx run nox -s build-guide -- --open

to see the user guide!

I think I have now all the three things set up: I can use PyO3, I can build its docs and I can build its user guide.

[Hamatti]

One more thing I need to figure out how to do: how to call Python from Rust. There's a doc page for it so I created a new demo project:

➜ cargo new demo

and copied the first example from the docs into main.rs.

I then added PyO3 to the project with

➜ cargo add pyo3

and tried to run it with

➜ cargo run

and was greeted with an error:

thread 'main' panicked at ~/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.20.2/src/gil.rs:199:21:
assertion `left != right` failed: The Python interpreter is not initialized and the `auto-initialize` feature is not enabled.

Consider calling `pyo3::prepare_freethreaded_python()` before attempting to use Python APIs.
  left: 0
 right: 0
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Rewinding a bit in the docs, I found this:

If you intend to run Python from Rust (for example in unit tests) you should set the following environment variable when installing a new Python version using pyenv:
PYTHON_CONFIGURE_OPTS="--enable-shared"

So I installed 3.11.7 to my virtual environment (I went with 3.11. because I already had 3.12 installed and wanted to make sure I'd install a fresh Python with the --enable-shared to filter out a potential issue)

➜ env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.11
➜ pyenv local 3.11.7
➜ cargo run
thread 'main' panicked at~/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.20.2/src/gil.rs:199:21: assertion `left != right` failed: The Python interpreter is not initialized and the `auto-initialize` feature is not enabled.

Still not working.

Based on a helpful Stack Overflow answer, I ran

➜ python
Python 3.11.7 (main, Dec  4 2023, 18:10:11) [Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin
>>> import sysconfig
>>> sysconfig.get_config_vars('Py_ENABLE_SHARED')
[0]

in a Python REPL to find out, that it had not installed with the flag enabled. Back to the drawing board.

After a lot of trials and errors, I managed to create a Python with the correct flag set up. After this, I recreated a virtualenvironment for my demo project using that one

➜ ~/.pyenv/versions/3.11.7/bin/python3 -m venv venv
➜ source venv/bin/activate
➜ python
Python 3.11.7 (main, Feb  8 2024, 20:21:21) [Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysconfig
>>> sysconfig.get_config_vars('Py_ENABLE_SHARED')
[1]

Yay! I feel like I'm making progress.

However, running cargo run still fails:

➜ cargo run
   Compiling pyo3-build-config v0.20.2
   Compiling pyo3-ffi v0.20.2
   Compiling pyo3 v0.20.2
   Compiling demo v0.1.0 (/code/pyo3/demo)
    Finished dev [unoptimized + debuginfo] target(s) in 10.57s
     Running `target/debug/demo`
thread 'main' panicked at ~/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.20.2/src/gil.rs:199:21:
assertion `left != right` failed: The Python interpreter is not initialized and the `auto-initialize` feature is not enabled.

Next, I wanted to check if I could see in Rust which Python is being run, when I ended up in features page that says:

Features for embedding Python in Rust

auto-initialize

This feature changes Python::with_gil to automatically initialize a Python interpreter (by calling prepare_freethreaded_python) if needed.

If you do not enable this feature, you should call pyo3::prepare_freethreaded_python() before attempting to call any other Python APIs.

I followed that link to https://pyo3.rs/main/doc/pyo3/fn.prepare_freethreaded_python from where I copied and added to my code this line

pyo3::prepare_freethreaded_python();

before my Python::with_gil call and voilà: it runs!

Feels good to get something like this figured out and working.

[Hamatti]

Looking at the error message now with the new information, it did say

Consider calling `pyo3::prepare_freethreaded_python()` before attempting to use Python APIs.

which should have given me a clue. However, because I was working with something completely new to me, I didn't realize I would need to run it myself, I somehow figured that this was something wrong down the stack.

[Hamatti]

I'll consider adding a clearer mention of that into the documentation once I get that far. It is explicitly mentioned in the user guide part but it's used already one section before.

I made a fix and submitted a pull request in PyO3 for this.

A first step has been taken!