Shim messes up with other Python softwares

[ksamuel]

Steps to Reproduce

Was going to backup my machine this morning, and found this. On ubuntu 20.04, install backintime, then rye. Run backintime from the cli:

user@laptop:~ $ 
backintime

Expected Result

Backintime runs normally:

user@laptop:~ $ 
backintime

Back In Time
Version: 1.2.1

Back In Time comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; type `backintime --license' for details.

Actual Result

Backintime can't find Python, despite it being installed:

user@laptop:~ $ 
backintime
error: Requested Python version (cpython@3.11.7) is not installed. Install with `rye fetch`
user@laptop:~ $ 
python3.11 --version
Python 3.11.7

Uninstalling rye solves the problem:

user@laptop:~ $ 
rye self uninstall --yes
Done!

Don't forget to remove the sourcing of $HOME/.rye/env from your shell config.
user@laptop:~ $ 
backintime

Back In Time
Version: 1.2.1

Back In Time comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; type `backintime --license' for details.

Looking at the backintime main script, it's basically bash calling python3 -Es $APP_PATH/backintime.py "$@".

Running python3 by itself produces the same result:

python3 --version
error: Requested Python version (cpython@3.11.7) is not installed. Install with `rye fetch`

I'm only using backintime as an example because this is real software doing this, and the cause of the problem would be opaque to the user.

Version Info

0.23.0

Installing the latest version of rye does not solves the issue, event if it default to 3.12:

user@laptop:~ $ 
curl -sSf https://rye-up.com/get | bash
This script will automatically download and install rye (latest) for you.
######################################################################## 100,0%
Welcome to Rye!

This installer will install rye to /home/user/.rye
This path can be changed by exporting the RYE_HOME environment variable.

Details:
  Rye Version: 0.23.0
  Platform: linux (x86_64)

✔ Continue? · yes
Installed binary to /home/user/.rye/shims/rye
Bootstrapping rye internals
Downloading cpython@3.12.1

Checking checksum
success: Downloaded cpython@3.12.1
Upgrading pip
Installing internal dependencies
Updated self-python installation at /home/user/.rye/self

All done!
user@laptop:~ $ 
user@laptop:~ $ 
backintime
error: Requested Python version (cpython@3.11.7) is not installed. Install with `rye fetch`

There are two ways to see this bug.

1 - Fixing this particular instance of it and close it.
2 - Use it to put in perspective the consequence of using shims.

We discussed before why I tend to distrust shims, and this is a good example of it. NVM has the same problem: it's very hard to figure all the combinations of things that can go wrong. It works for cargo because cargo is not a system script runner. A dev and it's tooling are the only one messing with it. But Python is different: it's used by scripts, apps, shells, and who knows what.

Since one appeal of of rye is to reduce the number of possible failure modes, I renew my proposal to extract the python installation part:

#583

Path problems are already plaguing Python, so having the possibility to explore more basic ways of bootstrapping the language would be nice. I'd love to see a push-button robust solution just to provide Python. No shim, nothing. It just provides a standardized Python with as little magic as possible.

Stacktrace

No response

[mitsuhiko]

Did you enable global shims? If they are not enabled, then rye on the path should transparently pass through and not interfere with your existing python. Can you paste the contents of ~/.rye/config.toml?

[ksamuel]

Apparently I did:

user@laptop:~ $ 
cat ~/.rye/config.toml
[behavior]
global-python = true

[default]
toolchain = "3.11"

Although I was not aware I did so. Which is a problem by itself, as I can imagine I'm not going to be the only one having this problem. In fact, I don't know how I enabled it, would I know it exist or what it does without this interaction with you.

[mitsuhiko]

The installer currently prints the following question that flips this. How can can this be made clearer?

? Determine Rye's python Shim behavior outside of Rye managed projects ›
❯ Make Rye's own Python distribution available
  Transparently pass through to non Rye (system, pyenv, etc.) Python

[ksamuel]

My (very personal) opinion is there no good way to ask this because making it available outside cannot be guaranteed to be safe in any way, yet not making it available outside will be weird, surprising and even frustrating.

I would rather have not the question at all, especially since it will be very difficult to explain the consequences of this without making a wall of text nobody would read.

But since no good default can be offered, I would rather have an alternative. It could be:

• provide a version-suffixed executable for global use.
• provide the py launcher and make the python available with it but not on the PATH. There is a Cannon's rust version available for unix now: https://github.com/brettcannon/python-launcher
• print the absolute path to the installed python so that people can use them explicitly outside.

I know, I'm being kinda insistent on this.

But remember what made the success of Flask: you created the simplest hello world possible, and people instantly got it. No ambiguity. If you are a beginner, it's all there, under your eyes.

That was a brilliant move that also worked for Reitz's requests, and if you can pull it of for rye, it could change everything.

[mitsuhiko]

I do not like that suggestion because it presumes that the future of this is the mess we are in. I want us to move past that mess. :)

I think the main action item here would be to pass through to the underlying python in more cases.

[ksamuel]

I guess given the uv announcement this ticket doesn't matter much.

Let's close it and see how astral handles this.