Static Build of pyo3 (embedding python interpreter)

[uncotion]

I would like to run python from rust, but in my situation, end users who will run the program, haven't preinstalled python and it's not possible to install (their system is extra restricted).
Is it possible build a rust program that uses pyo3 will embed python interpreter? (So that users don't need to install a python interpreter separately)?
Is it possible please give me some hit, In documentation there wasn't any point to this.

[konstin]

That's currently not supported, though it would be possible. I've written the necessary instructions for linux in the fourth of #276

[konstin]

To implement this cargo:rustc-link-search=native must be set to the value of LIBPL in the python sysconfig and cargo:rustc-link-lib=static= must be used instead of cargo:rustc-link-lib. It should be possible to enable this with a feature flag.

[hfingler]

Did anyone have any success related to this? I'm about to try this and any information would be helpful. Will report back with results.

[tr4cks]

Hello everyone,

I was also interested in this feature. I looked into it and I thought of directly integrating the embedded library provided by python rather than compiling it statically. I think it's much easier to deploy since you just have to download the python library in portable format and integrate it into your own project.
I first tried to indicate the python executable via the PYTHON_SYS_EXECUTABLE environment variable but this solution doesn't work. It doesn't work because the method used to know the path of the python library is to execute a python script and get the variable returned by the function named sysconfig.get_config_var('LIBDIR'). However, in the case of the portable python library, this function returns nothing.

To do this, I thought of providing another environment variable allowing to indicate the archive containing the portable python executable. This would then be unzipped into the target/{...}/out folder and automatically linked to the project. Or much more simply leaving the possibility to overload the location of the python library. I think the latter solution would not require much change.

Tell me what you think about this?

[jmwright]

Has there been any progress on this? I see cargo:rustc-link-lib=static= in a couple of places in build.rs.

https://github.com/PyO3/pyo3/blob/master/build.rs#L333

[davidhewitt]

I investigated this briefly, but I think even with the static linking "supported" we have plenty of missing symbol errors which are seen in other tickets #763 #742 .

I believe that the problem is that unused symbols get removed at link-time, which is problematic when loading other Python extensions from a statically embedded Python interpreter.

I asked about this on URLO, but got no response: https://users.rust-lang.org/t/embed-whole-python-interpreter-using-static-linking/42509

Not sure if to get this working properly we need to add changes to cargo / rustc.

[jmwright]

@davidhewitt Thanks for the update.

[stuhood]

FTR: https://pyoxidizer.readthedocs.io/en/stable/rust.html seems to work well for embedding, although there are some limitations around which native python extension modules can be embedded along with the interpreter.

[neoblackcap]

IMHO, I think if someone want to distribue application with embedding python interpreter, he/she doesn't need to static link python interpreter with rust-base application.
Write a bash script set LD_LIBRARY_PATH and name it as wrapper and distribute all shared-libraries with it just like what sublime text do.

[davidhewitt]

If you really do want to embed statically, I've managed to get a very basic POC to function on unix. It might have issues I haven't run across yet; it does successfully build (using nightly) and can import numpy.

I had to emit this slew of cargo options in pyo3's build.rs:

println!("cargo:rustc-link-arg-bins=-Wl,-Bstatic");
println!("cargo:rustc-link-arg-bins=-Wl,--whole-archive");
println!("cargo:rustc-link-arg-bins=-lpython3.9");
println!("cargo:rustc-link-arg-bins=-Wl,-Bdynamic");
println!("cargo:rustc-link-arg-bins=-Wl,--no-whole-archive");
println!("cargo:rustc-link-arg-bins=-lz");
println!("cargo:rustc-link-arg-bins=-lexpat");
println!("cargo:rustc-link-arg-bins=-lutil");
println!("cargo:rustc-link-arg-bins=-lm");
println!("cargo:rustc-link-arg-bins=-Wl,--export-dynamic");

With those in place, I could compile this code:

fn main() {
    unsafe { pyo3::ffi::Py_Main(0, std::ptr::null_mut()); }
}

using this command line:

RUSTFLAGS="-C relocation-model=dynamic-no-pic" cargo +nightly -Zextra-link-arg build

The resulting program worked like a normal python interpreter:

$ target/debug/pyo3-scratch
Python 3.9.1 (default, Dec  8 2020, 02:26:20)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.$ target/debug/pyo3-scratch
Python 3.9.1 (default, Dec  8 2020, 02:26:20)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.__version__
'1.20.1'
>>> x = numpy.ones((100, 100))
>>> x
array([[1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.],
       ...,
       [1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.]])

So what I think about this: while we're still a long way from supporting static linking properly, this at least proves that it is possible. What would need to be improved to make this something that I'd consider ready to ship in PyO3:

• Understand if there's a better set of linker flags to generate. Can we read at least some of them from Python in PyO3's build.rs? (Maybe it's in sysconfig somewhere.)
• Those linker flags probably want to apply only to a specific bin target. Can we do that rather than unconditionally put libpython.a into all bins?
• RUSTFLAGS="-C relocation-model=dynamic-no-pic" forces full rebuilds all the time because (at least in my setup) it conflicts with the flags from rust-analyzer. Maybe this can be set through a linker flag too.
• At the moment the cargo flags require nightly Rust. I'm not totally against only supporting this on nightly Rust, but it's a little painful.
• -Wl,--export-dynamic add all symbols in the executable to the exported symbol table. This is needed to be able to import C extensions like numpy. In practice we don't need to expose all symbols - just the same set of symbols from libpython.a. Is there a different flag to use?

Hopefully the notes here serve as a useful reference for the future.

[vojtechkral]

@davidhewitt Chances are you could get (at least some of) those flags from pkg-config.
On my system:

$ pkg-config --static --libs python3-embed
-lpython3.9 -lcrypt -lpthread -ldl -lutil -lm 

I think Rust has official bindings...