Poetry (or other Python dependency manager) project with Rust module.

[TiemenSch]

I'm trying to find out the simplest path towards substituting a module of a Poetry managed Python project with some Rust code. I get the feeling that Maturin does great things, but that there's some ease-of-use missing. What would be a sensible default setup for the following case:

Case

Let's say my project structure looks like:

my_project
├── my_project
│   ├── __init__.py
│   └── my_module.py
├── pyproject.toml
├── README.md
└── tests
    └── __init__.py

Let's assume that my_module.py does not need access to Python's standard library per se, but is rather a mathematical exercise like

def double(x):
    return 2 * x

E.g. Python wants the Rust lib to do something, but the Rust lib couldn't care less if it were invoked by Python or any other means.

Goal

How would I replace my_module.py with one or more *.so, *.pyd, *.pyi or whatever files are necessary to have:

• A module that can be pre-built and published for any common OS/arch.
• Python typing stubs (I wouldn't mind having to do this manually per replaced module, but it should be able to be included for ease of use and documentation purposes).
• Preferably leave the Python dependency management with Poetry or some other tool and not have to replicate their tool's work that they already do well.
• In short end-users shouldn't even know that a module has been substituted.

I'm imagining that at least for some OS/arch the tree would become:

my_project
├── my_module
│   ├── Cargo.toml
│   └── src
│       └── lib.rs
├── my_project
│   ├── __init__.py
│   ├── my_module.so
│   └── my_module.pyi
├── pyproject.toml
├── README.md
└── tests
    └── __init__.py

Method

• What build and deploy scripts would I need? Preferable CI platform agnostic (so no GitHub workflow type), just command-line / scripting like bash. Additional development dependencies are fine.
• How would I go about publishing for different architectures at once? Most examples seem to cover building code on your own machine, which is bound to leave my code with Works On My Machine and Your Mileage May Vary warnings or create unwieldy build matrices with different Python versions, environments, and what not floating around.
• It feels as if it should only bundle a different my_module.so per architecture, whereas the other files could remain the same.

[messense]

Before I go through every question you asked, I recommend you read the maturin docs first, particularly tutorial and distribution, I hope it will answer some of your questions.

[messense]

I think what you want is a mixed Rust/Python layout project, if you insist the file structure your described, you can tweak it with some options.

Using Poetry to manage dependencies is fine, but you need to use maturin as the build backend, here is an example: https://github.com/kfields/hello-maturin/blob/main/pyproject.toml

[TiemenSch]

Thanks for your prompt reply! Great to see that Poetry and Maturin can live side-by-side, will take that project for a spin. It's great to see that the install/build is fairly straightforward there. I think I only came across some projects where they switched entirely to Maturin or had (pipeline) setups that were difficult to follow before.

I get a bit lost with the Github workflows. It's at that abstraction point that I get lost if I would like to build/publish (locally) myself or do it with another provider such as GitLab CI. How could I replicate the functionality?

[messense]

You just need to make sure you are building in manylinux compatible docker containers, install Rust via rustup and install maturin via pip3 install maturin, then you are good to go with maturin build or maturin publish, pip wheel should also work.

Or you can just use cibuildwheel since maturin supports PEP 517 cibuildwheel should work out of the box.

[TiemenSch]

Ah, I think cibuildwheel is the part I was missing! So many build tools :)

For a local attempt I should then just install Rust's toolchain rustup with the CIBW_BEFORE_ALL_LINUX environment variable.

Should be able to get this running now. Thanks a lot! Caching/optimization is something for another day.

[RaulTrombin]

you can try hatch

[BartoszJanuszNA]

Hello! Do you have any success in building a wheel using poetry build with maturin as backend? For the hello-maturin sample project you mentioned above it does not include the .so file in the wheel. I described the issue in details here