Pyright & pybind11

Run ./demo.sh to execute this example from start to finish.

---

Trying to figure out how to package type information together with pybind11 modules.

This example builds and packages a calculator module using pybind. The package has the following structure:

• module calculator
  • add(a: int, b: int) -> int
  • submodule subtract
    • sub(a: int, b: int) -> int

Tested on macOS 13.5.1 (Apple Silicon) and in a Dev Container with Debian bullseye.

Prerequisites

• Conda, to manage Python environments and dependencies.
• Your system should also meet the minimum requirements for building C++ (make; gcc or clang, etc.)

Building the bindings

Create a new conda environment in .venv using environment.yml, and then activate it.

conda env create --file environment.yml --prefix .venv \
  && conda activate $(realpath .venv)

The environment uses Python 3.8.17, and installs the following packages:

• pybind11
• CMake, for generating the build system
• MyPy, for generating type stubs

Conda also installs the necessary Python/pybind11 headers as well as pybind's CMake input files, which CMake will need.

Generate the build system:

cmake .

CMake will use CMakelists.txt. Your command output should look similar to the following (paths and compiler versions may vary):

-- The CXX compiler identification is AppleClang 14.0.3.14030022
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PythonInterp: ... (found suitable version "3.8.17", minimum required is "3.6")
-- Found PythonLibs: .../.venv/lib/libpython3.8.dylib
-- Found pybind11: .../.venv/lib/python3.8/site-packages/pybind11/include (found version "2.11.1")
-- Configuring done (0.5s)
-- Generating done (0.0s)
-- Build files have been written to: ...

Build the bindings:

make

This creates a dynamic library calculator.*.so in the current directory. Your command output should look similar to the following:

[ 50%] Building CXX object CMakeFiles/calculator.dir/calculator.cpp.o
[100%] Linking CXX shared module calculator.cpython-38-darwin.so
[100%] Built target calculator

It is now possible to import the module from Python:

>>> import calculator
>>> calculator.add(1, 2)
... 3

Generating type stubs

Generate type stubs from the built bindings using MyPy's stubgen command:

stubgen --package calculator --output typings

The stubs will be written to typings/calculator.

Processed 2 modules
Generated files under typings/calculator/

Installing & testing

Create a new conda environment inside example/.venv using example/environment.yml, and then activate it:

conda env create --file example/environment.yml --prefix example/.venv \
  && conda activate $(realpath example/.venv)

This time, the environment installs MyPy, Pyright, and the calculator package we just built.

See what files were installed:

pip show -f calculator

Name: calculator
...
Files:
  ...
  calculator.*.so
  calculator/__init__.pyi
  calculator/py.typed
  calculator/subtract.pyi

Run the example script:

python example/main.py

41 + 1 = 42
43 - 1 = 42

Comparing MyPy and Pyright

The example script contains a typing error:

# def add(a: int, b: int) -> int:
add(41, "1")

Type-check with MyPy:

mypy example/main.py

example/main.py:10: error: Argument 2 to "sub" has incompatible type "str"; expected "int"  [arg-type]
Found 1 error in 1 file (checked 1 source file)

Type-check with Pyright:

pyright example/main.py

.../example/main.py
  .../example/main.py:3:6 - warning: Import "calculator.subtract" could not be resolved from source (reportMissingModuleSource)
  .../example/main.py:10:37 - error: Argument of type "Literal['1']" cannot be assigned to parameter "arg1" of type "int" in function "sub"
    "Literal['1']" is incompatible with "int" (reportGeneralTypeIssues)
1 error, 1 warning, 0 informations

calculator.subtract is a submodule generated by pybind11 which has no corresponding source on the filesystem (the entire calculator package is in a single .so file).

Here, Pyright warns about not being able to resolve it from source, but MyPy does not. Notice also that Pyright still reports the type error correctly thanks to our type stubs.