pixi run pip install is overridden by subsequent call to pixi

[abey79]

Checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

We encountered this issue while working on rerun-io/rerun#5966 (can be reproduced with commit 44412db2519cf2a37b0bd9d951cc4d8630eac037).

rm -rf .pixi
pixi run -e examples clock
# error indicating that rerun-sdk is pulled from pypi as expected

pixi run -e examples py-build  # calls maturin develop, which calls pip — should fix the error

pixi run -e examples clock

-> leads to the same initial error, as if py-build had no effect.

Issue description

It actually turns out that py-build does install a build-from-source rerun-sdk correctly, but this is reverted by the last call to pixi, possibly due to #998?

Interestingly, activating a shell is a possibly work-around:

rm -rf .pixi
pixi run -e examples clock
# error indicating that rerun-sdk is pulled from pypi as expected

pixi shell -e examples
pixi run -e examples py-build

clock  # -> works as expected

Another case where it appears to work as expected is in CI, with an environment set for the setup-pixi action: https://github.com/rerun-io/rerun/blob/36f00a5b0bd3edf06bc0f4ec65ce58f86a32e458/.github/workflows/reusable_build_examples.yml#L66-L86

Expected behavior

Ideally, pixi would make it easy and robust to manually "override" packages in the environments, including via pip specifically for when there is no other way (e.g maturin). This would cover at least two workflow:

• dev environment (run ${STUFF} against a build-from-source rerun)
• CI (run/test ${STUFF} against a CI-built wheel)

Workaround

For our particular case, @jleibs just had a workaround idea that appears to work:

1. we created a "empty" rerun-sdk local package somewhere (that would install a differently named empty python file)
2. we added an explicitly dependency on it in pixi.toml for the examples environment

That way, examples defaults to failing (no module named 'rerun'), and somehow a subsequent pixi run -e examples py-build doesn't end up being overridden.

pyproject.toml:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "rerun-sdk"
version = "0.10.0"

[tool.hatch.build.targets.wheel]
packages = ["empty_rerun_sdk.py"]

[tdejager]

I think I understand the use-case, it begs the question though, is the package that py-buid creates also locked by pixi itself?

[abey79]

I should have clarified: py-build runs maturin build, which creates our rerun-sdk package. This is what all our examples depend on. So without the explicit dependency on the "empty" rerun-sdk package is [pypi-dependencies], pixi locks on the pypi version of it.

[tdejager]

So the use-case is:

• You want to use the latest rerun-sdk from source, and have the examples depend on rerun-sdk as well?

Couldn't you add rerun-sdk as an editable with a pyproject.toml with the maturin backend?

[abey79]

We'd rather keep a "lax" dependency on rerun-sdk on our example to cover the end user use case of running examples from the latest branch (corresponding to the last release). In that case, pulling from pypi is the desired behaviour, and avoid any compilation (or rustc to be installed at all—most of our users are not rust devs).

Edit: also it wouldn't cover the CI use case, where we want to run against a very specific, pre-built wheel.

[tdejager]

Couldn't you just make a seperate environment for running with the latest version?

EDIT:

Also is my read of your use-case correct ):?

[abey79]

Four our use-case (use-cases, really), our ideal requirements are:

A) As a user:

• I can run any example from the latest branch against a pypi version of rerun-sdk
• In particular, no compilation should be needed (rust not installed).
• I can do that with standard python tooling (pip install ...).
• I can do that with any python version I might have around (provided that it's compatible with rerun-sdk and the particular example at hand)

B) As a dev:

• I can run any example in editable mode.
• Running an example should not automatically trigger any rust compilation (most rust changes do not impact the python sdk) to ensure fast iteration cycles (eg when working on said example).
• I can manually trigger a SDK install from source (aka py-build) when needed. (It's ok to have to do that at least once after initial environment creation.)
• It's ok to "force" dev to use pixi.

C) As CI:

• I can run any example against a CI-built rerun-sdk wheel (see this in action in OP's link).
• I can run any (compatible) example against any supported Python version (and, again, against a specific wheel).
• Stuff is cached as much as possible such as to reduce CI $$$

(rerun-sdk currently support 3.8-3.12, some examples have further restrictions)

[abey79]

I believe (A) forces examples to have a lax dependency on rerun-sdk

[tdejager]

Thanks, let me see..

So for a), I think this is already possible right:

• The user clones the repo.
• Navigates to the example, uses any package manager that does not require specif tool sections pdm, pip, uv etc. to install.
• Runs it and uses the rerun-sdk from pypi.

B) I believe we should support

• Do you mean rust compilation for the rerun-sdk or the maturin wheel build? For the wheel build, for editables we cache the pyproject.toml similar as to what maturin does I believe. We should only trigger a rebuild when the metadata changes, at least in edtable mode. Installation should be fast for edita
• I do see this happening a lot more than it's supposed to, but those we should fix on our end.
• If that is working you could add an editable dependency to rerun-sdk that would trigger a rebuild when it should. I believe that should cover B?

C) You are right there is no real good support for it as long as the rerun-sdk is managed in the lock as well.

[abey79]

I think your summary matches our experience. The workaround I mentioned in OP provides us with an escape hatch for (C), and, as a dev, for the odd time you might still need to install an alternative rerun-sdk (e.g. some random wheels from CI for testing purposes, or a quick test against the last release from pypi, etc.).

(Thinking of it, I should have listed (B-extra) which is the ability as a dev to occasionally install any rerun-sdk from anywhere because.... life happens 😄)

[tdejager]

Yeah for B-extra, you would need the ability to override the locked dependency in some way. Maybe checking the INSTALLER would help "break" the environment. However, the question is then, how to "un-break" it :)

[tdejager]

I think, and that's kinda totally unrelated to your issue, but somehow it does come back to it. The reason we need to install the environments when using pypi-dependencies is the fact that we need python to be able to resolve python dependencies because of sdists. Because of this we cannot use the --no-install (that skips installation) flag, as we need to install.

Two ways of going about this:

• Be able to define that you only want wheels in a environment (not useful for you) as an editable is not a wheel. But potentially useful in CI if the wheel is built seperately or in a sperate env at least. That way you could record it as a direct-url dependency.
• Have a sdist build environment specification in pixi. This would be a shared environment that is just used for building wheels. We only need to resolve and install this once, or not a lot of times at least.

If we have that we could reinstate the full functionality of --no-install (if there is a sdist build env), and you could go willy nilly on your environment and "unbreak" it with a pixi install.