pyo3 crate recompiled on (every?) build

[cswinter]

I don't have a minimal repro yet, but the pyo3 crate seems to be recompiled on most builds which significantly increases compilation times. I'm on pyo3 version 0.13.2, rustc version 1.53.0.

[davidhewitt]

See also #1551

This might be fixed on main / upcoming 0.14 release depending on your build environment.

I think that rust-analyzer runs things like cargo check which can invalidate pyo3's build config unless the environments carefully match.

This is worthy of a topic in the guide, and probably there could be more discussion whether we need to make further changes.

[cswinter]

Ah good to know! I tried setting PYO3_PYTHON both when building manually and in rust-analyzer.server.extraEnv but that doesn't seem to have done the trick yet. I think I'll just remove PyO3 completely from Rust builds for now which I have mostly set up that way anyway already because of that linker issue. Happy to do some more testing/debugging on this once 0.14 lands.

[davidhewitt]

Thanks. I've also been considering reaching out to the rust-analyzer folks who might be more familiar with all the cases when pyo3's build.rs is invoked, however haven't gotten around to that yet.

[mejrs]

I have pyo3 as an optional dependency for this reason.

[dbr]

I have this happening semi-randomly (happens "almost all the time", but, not always). I am running rust-analyzer (in emacs via lsp-mode), and mostly notice this when switching to terminal to cargo run. It was mysterious how rust-analyzer was causing the command-line builds to change, so I looked into it a bit:

Running build with

CARGO_LOG=cargo::core::compiler::fingerprint=trace cargo build

I get the following output (can probably change 'trace' to 'info')

INFO  cargo::core::compiler::fingerprint]     err: env var `PATH` changed: previously Some("/home/dbr/.cargo/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games"), now Some("/home/dbr/.cargo/bin:/home/dbr/.cargo/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games")

Basically ~/.cargo/bin is in $PATH twice in rust-analyzer but only once in regular build (I assume, might be other way around), which then triggers a re-build of PyO3 as intended (as per the #1551 issue David mentioned above)

rust-analyzer uses the same target/ directory, which is how it can interfere with running cargo build. Specifically, the files like target/debug/.fingerprint/pyo3-a2b72c2ee6b7ccd3 stores the previous value of $PATH like so:

[..snip..] {"RerunIfEnvChanged":{"var":"PATH","val":"/home/dbr/.cargo/bin:/home/dbr/.cargo/bin:[..snip..]"}}],"rustflags":[],"metadata":0,"config":0,"compile_kind":0} [..snip..]

So I guess this issue basically becomes: why is ~/.cargo/bin getting added to $PATH twice (which is most certainly an issue for either rust-analyzer, or possibly cargo?)

[dbr]

After much poking around, I think the problem lies in rustup (not rust-analyzer nor cargo), so have filed a bug report in that project:

rust-lang/rustup#2848

For now I think configuring CARGO_TARGET_DIR for rust-analyzer would seem to be the most reliable workaround (at the cost of having two separate builds)

[davidhewitt]

Thanks for the investigation! If the rustup bug getting fixed does indeed remove this frequently-encountered papercut, I'll be very happy 😄

[rbozan]

The PR that fixes it in rustup (rust-lang/rustup#2849) is merged

[xnerhu]

The PR that fixes it in rustup (rust-lang/rustup#2849) is merged

When I run yarn build from Nodejs where in package.json I have scripts

"scripts": {
   "build": "cargo build"
}

It recompiles pyo3 every time. But when I run cargo build directly in the terminal, it doesn't rebuild pyo3.

windows 11
pyo3 0.18.0
rustup 1.25.2 (17db695f1 2023-02-01)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.67.0 (fc594f156 2023-01-24)`

I think it's because yarn adds temp path, which changes every time. Tho running via npm run build doesn't work too.

rustup\toolchains\stable-x86_64-pc-windows-msvc\bin;C:\Users\xnerhu\AppData\Local\Temp\yarn--1675712873242-0.

[bazhenov]

Unfortunately, problem still remains. In my case it is only manifest itself when rust-analyzer is running in IDE.

[antimora]

I can confirm too. I use rust-analyzer (extension v0.3.1402) with vscode and the py03 gets recompiled every time.

use pyo3::{types::PyDict, types::PyModule, PyResult, Python};

   Compiling pyo3-build-config v0.18.0
   Compiling pyo3-ffi v0.18.0
   Compiling pyo3 v0.18.0

[davidhewitt]

Can you try running your builds with CARGO_LOG=cargo::core::compiler::fingerprint=info as suggested above, and show here what is causing the cache invalidation?

[bazhenov]

I managed to solve a problem. Or at least found a decent workaround.

Can you try running your builds with CARGO_LOG=cargo::core::compiler::fingerprint=info as suggested above, and show here what is causing the cache invalidation?

Seems like the same issue: err: env var 'PATH' changed.

Full log:

[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] dependency on `build_script_build` is newer than we are 1676466118.936326620s > 1676466053.631031496s "/Users/bazhenov/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-ffi-0.18.1"
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] dependency on `build_script_build` is newer than we are 1676466119.140438444s > 1676466056.897465668s "/Users/bazhenov/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.18.1"
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] fingerprint error for crab v0.1.0 (/Users/bazhenov/Developer/crab)/Build/TargetInner { ..: lib_target("crab", ["lib"], "/Users/bazhenov/Developer/crab/src/lib.rs", Edition2021) }
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint]     err: current filesystem status shows we're outdated
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] fingerprint error for pyo3 v0.18.1/Build/TargetInner { ..: lib_target("pyo3", ["lib"], "/Users/bazhenov/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.18.1/src/lib.rs", Edition2018) }
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint]     err: current filesystem status shows we're outdated
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] fingerprint error for pyo3 v0.18.1/RunCustomBuild/TargetInner { ..: custom_build_target("build-script-build", "/Users/bazhenov/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.18.1/build.rs", Edition2018) }
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint]     err: unit dependency information changed
    
    Caused by:
        new (build_script_build/b73abd80d1b5c6e5) != old (build_script_build/7d829398a44f12b4)
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] fingerprint error for pyo3 v0.18.1/Build/TargetInner { ..: custom_build_target("build-script-build", "/Users/bazhenov/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.18.1/build.rs", Edition2018) }
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint]     err: unit dependency information changed
    
    Caused by:
        new (pyo3_build_config/1743a4d849f4181b) != old (pyo3_build_config/b34bbfa27ea017eb)
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] fingerprint error for pyo3-build-config v0.18.1/Build/TargetInner { ..: lib_target("pyo3-build-config", ["lib"], "/Users/bazhenov/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-build-config-0.18.1/src/lib.rs", Edition2018) }
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint]     err: unit dependency information changed
    
    Caused by:
        new (build_script_build/5fe9711f85c67c02) != old (build_script_build/d8fecd9d3b0d8efe)
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] fingerprint error for pyo3-build-config v0.18.1/RunCustomBuild/TargetInner { ..: custom_build_target("build-script-build", "/Users/bazhenov/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-build-config-0.18.1/build.rs", Edition2018) }
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint]     err: env var `PATH` changed: previously Some("/Users/bazhenov/.cargo/bin:/Users/bazhenov/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Users/bazhenov/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/local/Cellar/openjdk/19.0.2/libexec/openjdk.jdk/Contents/Home/bin:/usr/local/opt/fzf/bin"), now Some("/Users/bazhenov/.cargo/bin:/Users/bazhenov/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Users/bazhenov/.cargo/bin:/Users/bazhenov/bin:/usr/local/sbin:/usr/local/Cellar/openjdk/19.0.2/libexec/openjdk.jdk/Contents/Home/bin:/usr/local/opt/fzf/bin:/usr/local/sbin:/usr/local/bin:/usr/local/Cellar/openjdk/19.0.2/libexec/openjdk.jdk/Contents/Home/bin")
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] fingerprint error for pyo3-ffi v0.18.1/RunCustomBuild/TargetInner { ..: custom_build_target("build-script-build", "/Users/bazhenov/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-ffi-0.18.1/build.rs", Edition2018) }
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint]     err: unit dependency information changed
    
    Caused by:
        new (build_script_build/d41ac0404f890cd7) != old (build_script_build/9b9073f7bf4231be)
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] fingerprint error for pyo3-ffi v0.18.1/Build/TargetInner { ..: custom_build_target("build-script-build", "/Users/bazhenov/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-ffi-0.18.1/build.rs", Edition2018) }
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint]     err: unit dependency information changed
    
    Caused by:
        new (pyo3_build_config/1743a4d849f4181b) != old (pyo3_build_config/b34bbfa27ea017eb)
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] fingerprint error for pyo3-ffi v0.18.1/Build/TargetInner { ..: lib_target("pyo3-ffi", ["lib"], "/Users/bazhenov/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-ffi-0.18.1/src/lib.rs", Edition2018) }
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint]     err: current filesystem status shows we're outdated
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint] fingerprint error for crab v0.1.0 (/Users/bazhenov/Developer/crab)/Build/TargetInner { name: "crab", doc: true, ..: with_path("/Users/bazhenov/Developer/crab/src/main.rs", Edition2021) }
[2023-02-15T13:02:15Z INFO  cargo::core::compiler::fingerprint]     err: current filesystem status shows we're outdated
   Compiling pyo3-build-config v0.18.1
   Compiling pyo3-ffi v0.18.1
   Compiling pyo3 v0.18.1
   Compiling crab v0.1.0 (/Users/bazhenov/Developer/crab)
    Finished dev [unoptimized + debuginfo] target(s) in 7.76s

Looking at src/impl_.rs in the pyo3-build-config I realize that cargo:rerun-if-env-changed=PATH directive is applied only if no PYO3_PYTHON environment variable is provided. So I've added following configuration to .cargo/config.toml

[env]
PYO3_PYTHON = "/usr/local/bin/python3"

And now everything works like a charm 🍾. Hope this helps.

[max-sixty]

Thanks for the repro and suggestion @bazhenov !

[env]
PYO3_PYTHON = "/usr/local/bin/python3"

Would there be an approach that generalizes to others who have different locations for their Python executable?

[bazhenov]

Would there be an approach that generalizes to others who have different locations for their Python executable?

It's kinda is. Cargo config can be located not in project but home directory – https://doc.rust-lang.org/cargo/reference/config.html

[alex]

We see:

     Dirty pyo3-build-config v0.18.2: the env variable PYO3_PYTHON changed

on basically every CI build. I believe I can explain what's happening:

We use pyo3 + setuptools-rust. In CI we do pip install . and then use CARGO_TARGET_DIR to ensure we have a consistent target/ dir so we can cache already built Rust dependencies between builds.

Because we are a PEP517 package, when pip install . is executed, pip will create an sdist for the package, create a temporary virtualenv, unpack the sdist, and then install the package there. The python that is used to invoke setuptools (and setuptools-rust) which in turn shells out to cargo rustc is the python from the temporary virtualenv, and therefore changes on every invocation.

This leads to invalidating the PYO3_PYTHON env var on every single build.

[alex]

Disregard this, it was entirely wrong 😭. Turns out we were sharing caches between different tox (or nox) jobs (e.g. lint vs. docs) and the environment name is in the path.

[tadeohepperle]

Is there any update on this? I still experience PyO3 recompiling every time when using rust-analyzer.

[Jgfrausing]

They seem to have found a workaround. The documentation in ipetkov/crane#486 specifies setting a path variable for the python binary in NIX. It can also just be put into the [env] section of .cargo/config.toml

...

[env]
PYO3_PYTHON = "path/to/python"

[wyfo]

@Jgfrausing This workaround was already described in a previous comment.
It seems someone needs to open a PR to update the FAQ, I will do it immediately.

[Jgfrausing]

Thanks. Maybe also mark this as solved, so that others know to look for a solution within this issue.

EDIT: I see, you've also covered that. Thanks, mate.

[tadeohepperle]

I tried the workaround with

[env]
PYO3_PYTHON = "./venv/bin/python"

but it does not change anything for me. When rust-analyzer is active, every code edit I make, results in a complete (~20 sec) rebuild of pyo3-macros and other dependencies. :(

[henryiii]

# .cargo/config.toml
[env]
PYO3_PYTHON = "./.venv/bin/python"

Worked for me. Had to make it (uv venv) and vscode asked if I wanted to use it, I said yes. I also manually activated it in the existing terminal pane (new ones are fine, too, and it does add a little "request to relaunch" warning in the sidebar). Reran both cargo test in the terminal and saved the file, and now it's not recompiling PyO3 every time.

[metasim]

FWIW, I'm having the same problem when using RustRover. Thanks to everyone for the workaround suggestions.

[danking]

For any other Emacs users out there, what I ultimately settled on was, in my .emacs

(setenv "PYO3_ENVIRONMENT_SIGNATURE" "cpython-3.11-64bit")
(setenv "PYO3_PYTHON" "/path/to/.venv/bin/python")

And I set the same variables with the same values in my .zprofile so that anytime I invoked Cargo from the terminal it would reuse the cached build. I couldn't convince the rust-analyzer started by Emacs to respect the values in $PROJECT_DIR/.cargo/config.toml 🤷 .