Should Rye Exist?

[mitsuhiko]

We all know XKCD #927:

This is how I feel about all the Python packaging. And this is why I never wanted to publish rye and kept it for myself. It's also incredibly hacky internally because it was never intended to be shared. However I really like what it does (at least in theory) and I desperately want it to exist.

As it stands today, it solves my problems — sort of. I wish it solved more of my problems but to a large degree what is missing, rye cannot really solve. Rye should not exist, it should just be the out of the box Python experience. The same way as Rust comes with cargo and rustup, my dream solution for Python was that something like rye exists and becomes the standard way to manage Python projects including getting the right interpreters. However that requires that such interpreters are actually distributed as binaries. Likewise for rye to work, the underlying packages still need stuff that does not exist. Here are some things that would make a real rye work:

• A standard for declaring development dependencies
• A standard for how local dependencies should work
• A proper lockfile standard
• A way to fetch metadata from indexes, without having to fetch wheels
• Unzippable binary distributions of all major Python interpreters on all major platforms
• A standard for declaring basic scripts and tools in pyproject.toml

But even without all of this, rye could exist. In a sense it does today, but it exists to solve one developer's personal problems in his very personal way. I really have no desire to contribute more to the proliferation of Python packaging. Today the pypa org on github is what appears to be dozens of competing, largely independent efforts. In addition to that there is conda, there are now even companies appearing that are putting layers around conda as well. It's a pretty big mess.

I just want it solved, and I want to never have to think about Python packaging and project management every again. And until there is a standard tool we all rally around, I don't see that happen. I'm not sure if rye (or something like rye) could be that, but I do put the offer out.

So here is the question: should rye exist?

[amirkarimi]

Would be nice if you could compare it with existing efforts like Poetry. I didn't use Poetry extensively but my first impression was great. Curious to know what's your take on that?

[mitsuhiko]

I'm not a huge fan of making comparisons to existing efforts because this very quickly turns into a fight about which tools is better. What's important for me is the end user experience: I want a world where I download the "one tool" that manages by python projects, python interpreters, globally installed tools, monorepos and more. That is open to any type of build system, that supports the problems a Python developer has on a daily basis.

Maybe Poetry one day can be that. However today Poetry is a system that largely requires it's own proprietary everything, does not at all solve the installation and management of the Python interpreters, and just increases the proliferation of the Python ecosystem. And yes, rye does the same and that is why I am so hesitant about what to do with it. I want the experience that I'm used to from the Rust ecosystem, and I don't want a world where we keep talking about the broken state of Python's packaging forever. I think the delta in the Python ecosystem between where we are and a good outcome, is tighter than ever before, but we haven't reached that goal yet, and I do not think that Poetry is the solution towards that.

If I were to put my money on a project that exists today, that is close, it would be pdm. And maybe rye really should not exist, and pdm should be more like rye. I am however quite confident based on my own experience with poetry, that the future will not be that.

[sdispater]

However today Poetry is a system that largely requires it's own proprietary everything

Maintainer of Poetry here 🙂

I assume this comment refers to PEP 621 so I will reiterate something I said a while back on Twitter:

First, let me start of by saying that the PEPs we will be talking about did not even exist at the time Poetry was created, so we had to come up with our own way of doing things within the loose constraints of PEP 517 and 518.

Now, regarding PEP support: PEP 621 (Storing project metadata in pyproject.toml) will be supported by Poetry in its 2.0 version – which is still a long way – however this won’t happen without tradeoffs since PEP 621 is more limited than what Poetry provides today.

This won’t happen overnight either since the number of people using Poetry is a magnitude higher than other similar projects so we need to come up with a seamless transition plan and a clear migration path. This takes time – which is limited for all maintainers – and caution.

On that note – and if download statistics are any indicator – Poetry is now more used that Pipenv and even pip-tools so we no longer have the liberty to pivot as easily as when the project started.

(Full twitter thread here: https://twitter.com/SDisPater/status/1521932863389712385)

So, yes, PEP 621 support is coming and the high level plan is available here : python-poetry/roadmap#3 (comment)

and I do not think that Poetry is the solution towards that
I am however quite confident based on my own experience with poetry, that the future will not be that.

I am curious to know the reasoning behind these comments because I did not see you address other shortcomings except the "proprietary" format and the management of Python interpreters, the former currently being address and the other easily addressable either in-house or via a plugin.

[flying-sheep]

I think it all comes down to if that feature would be as seamlessly integrated as all other functionality.

E.g. a rustup-managed cargo can use a +toolchain selector, e.g. cargo +nightly test. If a poetry plugin could work as seamlessly as that, Poetry could fill that role.

Hatch already does all that, it just doesn’t have add and doesn’t download python versions for you …

[eli-schwartz]

I assume this comment refers to PEP 621 so I will reiterate something I said a while back on Twitter:

I would not assume this is about PEP 621 as such. That's not really the big problem with poetry's current format.

PEP 621 is a standard, and as such it interoperates with and depends on other standards. For example, PEP 621 depends on PEP 440.

PEP 440 predated poetry, but poetry made a decision to deviate from it. That's really the major distinction of poetry's format.

Regardless of any and all rationales for why poetry made that proprietary format, or why it's asserted to be superior, there's no denying that it is, in fact, a proprietary format rather than leaning on standards.

Also, my understanding is that the use of proprietary non-PEP-440 versioning is the sole reason why this requires poetry 2.0 rather than doing what other build backends did, and introducing support for PEP 621 in a minor point release without impacting any users that haven't migrated.

[sdispater]

PEP 440 predated poetry, but poetry made a decision to deviate from it. That's really the major distinction of poetry's format.

Poetry did – and still does – deviate from PEP 440 by default, but standard operators outlined in PEP 440 are supported by Poetry. You could replace the caret or tilde notation with compatible release clauses and it would work without.

Also, my understanding is that the use of proprietary non-PEP-440 versioning is the sole reason why this requires poetry 2.0 rather than doing what other build backends did, and introducing support for PEP 621 in a minor point release without impacting any users that haven't migrated.

Poetry 2.0 is when PEP 621 will be made the default format, not when the format will be supported (which should be, hopefully, in the 1.5 version). We'll transition slowly towards it, giving users a clear migration path before making the switch in 2.0.

[mrswats]

Would a series of PEPs be a way forward to solve this issues you mention here? Would the effort be worth it?

[cnpryer]

Yea so a re-purposing/refining of py. Does the core team want to maintain a py anyway? Corner pip into the "installer" role (both Python packages and Python itself; support PEP711). Build py out as a separate workflow and ship as part of a "full" tool-chain.

[mitsuhiko]

My hunch is that rather than thinking if something should be py, the more important question is what the something is. The name itself is quite likely to just be the source of a lot of non productive bikeshedding.

[cnpryer]

Agreed. To me that something is along the lines of something like rye. If I had to guess, showcasing ideas like rye, posy, monotrail, etc. is how you test the Cargo-for-Python hypothesis.

I'm using py as a stand-in to translate the ideas towards something more tangible. I didn't mean to sound like the name is all that important for the discussion.

[frostming]

There are PEPs trying to address some of the problems listed above:

• A proper lockfile standard

PEP 665. Rejected due to the issue of sdist.

• A way to fetch metadata from indexes, without having to fetch wheels

PEP 658. Accepted but the implementation progress is slow.

• Unzippable binary distributions of all major Python interpreters on all major platforms

PEP 711. Draft

[flying-sheep]

There’s some discussion about a PEP 665 successor here: https://discuss.python.org/t/how-should-a-lockfile-pep-665-successor-look-like/17690

[cnpryer]

it should just be the out of the box Python experience

Yea I consider the Rust tool-chain a kind of gold-standard when it comes to experience. I understand why it's not that simple to just turn pip into more of a workflow tool, but to me the ideal scenario is one with py as a kind of rustup and pip as the cargo.

The xkcd is a good one, but I still think there's value in creating and sharing your ideas. Garnering interest and forcing more discussion is beneficial for the ecosystem, no?

[sultaniman]

@mitsuhiko anything new and consistent is a win for community, I think however we all need a better packaging, distribution and build system for Python.

[sztomi]

Yes, 100% should exist. You are right that something "official" should exist for Python that fills this niche, but pypa adopted several external projects that were relevant for it, so maybe rye could be such project.

[href]

I don't see myself using rye quite yet, but I appreciate the use of @indygreg's standalone Python builds. Personally I use those too (on macOS), together with some shell scripts to manage my virtual environments. I use poetry as well, but that can change.

But I think rye should exist, and I'll track its development with interest.

[seekaddo]

I believe your summary already addresses the question at hand. The advantages clearly surpass any reasonable concerns. A well-defined roadmap should be created to identify the best practices, thus enabling the community to choose the package manager that best meets their needs. I haven't used poetry before, and the main idea behind rye isn't to instigate a C++ vs Rust debate, but rather to tackle significant issues that everyday developers face. I have been using conda for most scientific work and personal projects, though it has its drawbacks, such as complicating the export process. When using conda, you need to call pip freeze to export the environment, which seems counterintuitive. Furthermore, it doesn't detect all pip-installed packages - only those sourced from PyPI.

I think rye has the potential to provide simplified and transparent standards. If implemented correctly, the community could reap substantial benefits from it.

[phaustin]

Have you tried conda-lock? Works well for our use cases: https://github.com/conda/conda-lock

[Butch78]

I think something similar to cargo & rustup for python should exist! I think setting up environments in python and installation are the biggest barriers to entry. If we can reduce those I think it'll be better for everyone :)

I don't know if you've seen this project:
https://github.com/cnpryer/huak

I've used it a little bit and it seems great, if rye or huak can do all the similar things to cargo I will use them :)

[mwarkentin]

Does PEP-711 (PyBI: a standard format for distributing Python Binaries) help with the python binary issue?

Caveat that it gets accepted and implemented of course.

[glyph]

mostly because the PyBI builds try to retain this framework structure for macOS that's just not needed

It's definitely needed for a whole slew of use-cases; basically anything that calls a platform API on macOS.

[mitsuhiko]

@glyph I would actually be curious for when a framework build is needed. As @indygreg mentioned a lot of macOS APIs work fine in a non framework context.

[glyph]

As @indygreg mentioned a lot of macOS APIs work fine in a non framework context.

Some do, some don't. Very few (beyond UNIX-y and CoreFoundation ones) are documented to do so; there's just a strong assumption throughout the platform frameworks that you exist in a bundle (usually an app), and that your frameworks do as well. So it's very annoying to research specific APIs because Apple won't tell you. You have to build your own examples, and then you get arbitrary undocumented failures about Mach ports which boil down to "did you build as a bundle in an app / framework? no? oh, that's your problem".

But let's assume that all APIs are actually callable from a non-framework build. If you want to distribute software for macOS, you can't do so, because all the code-signing tools break. If you want to distribute the build of Python itself, for example, unless you live entirely in CLI-land and opt out of quarantining (i.e. "files you downloaded from a web browser") entirely, it is not possible to distribute a stand-alone executable that lives outside of a bundle. Individual binary files can be code-signed, but they cannot be notarized, because there's nowhere to attach the notarization ticket. In other words, you have to end-run the entire security apparatus of the platform, and rather than it being a temporary issue for lack of attention, standardizing on non-framework builds takes a step further towards cementing it as the permanent status of Python.

If you try to distribute an app bundle using a non-framework build of Python, then you suddenly run into a bunch of other issues related to the way that codesign tool respects bundle resources and deals with dynamic linking. None of these are totally insurmountable — it is possible to ship a raw dylib in a framework bundle (and in fact Python binary wheels do have this problem as well), but it breaks a bunch of assumptions from tools like py2app that want to link against Python as a framework, and codesign, stapler, and notarytool.

Then there's the issue of callbacks. If you want to include Python as a dependency of a native Xcode project, there's no way for Swift or Objective C tooling to consume "arbitrary directory with a pile of UNIX-ish junk in it"; they expect to see a Framework with Headers, Versions, et. al.

If you ignore all of this, then sure, you can limp along and kinda call a bunch of APIs and kinda muddle through to make something distributable, but you cut off all kinds of possibilities and make natively accessing the platform arbitrarily weird and difficult and broken in edge-cases for no particular reason; we already solved framework builds and they work fine. Getting rid of them would reinforce the idea that Python is only for backend/web stuff because it's just too confusing and broken on client platforms, and the Mac port would regress further towards treating macOS as a slightly-broken version of Linux rather than its own platform.

[glyph]

You have to build your own examples, and then you get arbitrary undocumented failures about Mach ports which boil down to "did you build as a bundle in an app / framework? no? oh, that's your problem".

Case in point: CoreLocation used to be my go-to example of an API that didn't work outside of a bundle context, but it looks like maybe that does work OK now? NSUserNotificationCenter also works now but didn't used to, but it's deprecated now and in a quick test (which could be broken, btw, it's annoying to switch back and forth between framework and non-framework builds and set up the appropriate context) its un-deprecated replacement UNUserNotificationCenter appears to silently do nothing. I can't quickly find a "crash with some message about mach ports" example but there are like a hundred frameworks on the platform and I only have examples for a few of them, so this is unfortunately more like folk wisdom.

[glyph]

Oops, I double checked, and looks like I did in fact use a framework build by accident. CoreLocation no longer spits out an error message on a non-framework build, it just segfaults immediately.

[indygreg]

You can definitely call macOS APIs without being a framework. Framework-less Python builds are evidence of this. My understanding is the framework builds of Python are more to facilitate other apps calling into Python’s APIs. The framework does provide a more consistent filesystem layout / API to target. But it isn’t strictly necessary.

[danielfrg]

For what its worth I moved a small personal project to rye today and its a very good and clean experience. I had a series of scripts and used other tools like Taskfile to have a very similar experience to what rye offers.

It's a big step in the right direction of tools in other languages like volta and the mentioned rust + cargo experience. Python is behind even if you can survive without those when you come back to Python it feels like a struggle.

I guess in an ideal world it should not exist but its not the world we live in. Thanks for putting it out there!

[Abdur-rahmaanJ]

I am 100% pip tools fan but by default i appreciate all tools you suggested so far, from click to your blog engine. On the basis of past programs' design, i am optimistic for rye.

I am not a big fan of lock files though.

Edit: Like i mean using pip-tools behind the scene is too cool!

[cnpryer]

What don't you like about lockfiles?

[Abdur-rahmaanJ]

The time taken to lock + locking os-dependent files.

[rtpg]

This all seems very cool. I do think the XKCD is very misleading, because if you have 14 standards, and then introduce something that is overlaying over existing standards... you have 15 standards, but in theory you are sharing work and that can help proliferate good changes across the stack.

We have already seen how virtualenvs being the "bottom level" of a lot of these projects has meant that virtualenvs "just work" for many definitions of work, and efforts to improve those helps everything! Any tool that is trying hard to use what exists, and inform the greater community of specific missing parts (beyond unactionable stuff like "confusing" or "hard for beginners"... it's hard for practitioners! Maybe we should get that right first) then we'll be making loads of forward progress.

Anyways I really like how this tool should in theory work all over and doesn't involve any black magic. Will try to use it and see how far I can get

[spapas]

Sorry for this but my answer to your question is No.

Before my ostracization please allow me to explain.

First of all, not actually against this specific tool but I'm against all similar tools (i.e conda, pyenv, poetry whatever). But since you specifically actually asked I'm happy to answer this discussion. The python ecosystem is already way too crumbled with tools like this. Having options may seem like a good thing but in my opinion for the Python world it is not a good thing as the the Zen of Python reminds us:

There should be one-- and preferably only one --obvious way to do it.

Now, which is the obvious way? For me it is using the standard tools that python has to offer: pip and venv. All effort should be combined and geard towards improving these tool to support any missing features or workflows, not by introducing a new tool.

Beyond that, please consider users that are new to Python. What should they use? How can they decide? Why burden them with options? I consider myself an experienced python developer (10+ years) and yet when I see all these tools I feel overwhelmed. Why not help these people with how to use Python with the standard tools instead?

Finaly, the fact that native Windows is not fully supported is (at least for me) a very big problem for any Python package.

Ok I'd be happy to be ostracized now.

[flying-sheep]

Can you imagine a viable path that reduces those 70 command line switches to something manageable? I can’t.

A streamlined, simple interface like cargo works because it’s build around one opinionated way of working. Tools like cargo, poetry, pdm, and now rye work around the idea of “projects”. Each project exclusively has access to its declared dependencies. There’s no global environment that scripts can access from anywhere. Every piece of code has a clearly associated environment. (cargo install and pipx exist when you want to simply install some CLI package globally. They works very differently from pip install, as they create an isolated environment for each of those CLIs instead of having a global Python package environment that contains everything including those CLIs)

Pip on the other hand works very manually: It operates from within a global environment or a venv, and doesn’t know the concept of a “project”. One can hack around this by having an IDE offer to create and activate a project-specific venv per project, but that’s not pip+venv by themselves, that’s another tool helping. And then you have the situation that you have to manage the venv manually. You can of course do pip install -e . whenever you modify your dependencies, but the tool envisioned here aims to make dependency management automatic.

Pip can’t go away from this unopinionated way of working because its CLI is built around those concepts. Of course, to get we could always create a new CLI for pip that has a different design, but then we’re essentially building a new tool on top of pip’s internals.

That’s why I’m 100% certain that pip will never grow to be the tool envisioned here: It already is a tool, made for a different use case.

[spapas]

@flying-sheep I understand what you're saying about the complexity and lack of capabilities of pip and it's not far from the truth. However I also think that @gandhis1 has a point. All these python tools (except maybe pip-tools) have the purpose of replacing as opposed to complementing the standard tools. "It's my way or the highway" is not a good ideology for the python community.

I wouldn't mind if there were tools that worked on top of pip/venv to support or make simpler some kind of workflow that can't be supported by pip/venv. However my opinion is still that that the python community should prioritize on improving the standard tools; there should be ways to implement these without increasing the complexity.

[gandhis1]

I think all points made here are good ones. I certainly don't want to take it upon myself to defend pip. But touching on what was just said, I can't help but find it regrettable that there's no meaningful effort to converge into a single line of thinking. rye is not special here - all of its predecessors took the same approach of identifying a flaw in existing packaging tools and attempting to address it. You can argue endlessly about whether a particular tool, rye or otherwise, does it better, but such debate I feel detracts from the real problem.

To this point, I actually think the most useful comment in this entire thread is the one made by @mrswats about whether the PEP process is perhaps a superior avenue to try to improve this reality.

[brianbruggeman]

The Python ecosystem would greatly benefit from a unified tool similar to Rust's rustup and cargo. By consolidating Python's tooling, we can reduce fragmentation and streamline development processes. Several reasons underscore the importance of this goal, and there is evidence to suggest that this can be achieved through refinement of pip or the implementation of PEPs specifically aimed at unification.

1. Reducing fragmentation: A unified tool can help minimize confusion and incompatibility issues that arise from using multiple tools. For instance, JavaScript's ecosystem experienced this fragmentation with package managers like npm and Yarn, which sometimes caused dependency conflicts and slowed down development processes. However, after the introduction of package-lock.json and yarn.lock files, the ecosystem has become more streamlined, and developers can work more efficiently.

2. Consistency in the development experience: A unified tool allows developers to follow a single set of best practices and processes. This consistency simplifies the learning curve for newcomers and encourages adherence to common standards. The Rust community has seen this benefit, as rustup and cargo provide a consistent development experience across platforms and projects, making it easier for developers to collaborate and share their expertise.

3. Community support: A unified tool requires strong community support for development and maintenance. However, this can be achieved by rallying around a common goal, as seen with the success of the Rust community in developing and maintaining rustup and cargo. Additionally, the Python community has a history of coming together to support crucial projects, such as the transition from Python 2 to Python 3, which demonstrated the power of collective effort in improving the ecosystem.

4. Success stories from other programming languages: Besides Rust, other programming languages like Go have also benefited from a unified tooling approach. Go's go command, which handles tasks like package management, compilation, and testing, has significantly contributed to the language's popularity and ease of use. These examples illustrate that a unified tool can indeed enhance the developer experience and streamline the ecosystem.

In conclusion, refining pip or implementing targeted PEPs to create a rustup and cargo equivalent for Python would be a strategic move for the community. This consolidation can lead to reduced fragmentation, a more consistent development experience, and a more collaborative ecosystem. By learning from the successes of other programming languages and leveraging the power of community support, Python can successfully adopt a unified tool that meets the needs of its diverse user base.

[flying-sheep]

To this point, I actually think the most useful comment in this entire thread is the one made by @mrswats about whether the PEP process is perhaps a superior avenue to try to improve this reality.

That’s exactly what we’ve been doing for years now. And it mostly works*, it’s just slow, and when complex problems don’t have someone being super excited about them (e.g. lockfiles), there’s no sufficient momentum to work out all the kinks and PEPs get rejected. I hope we’ll get a PEP 665 successor soon, but I don’t like lockfiles enough to make the effort to write it myself.

*pyproject.toml has been a result of this, and the build-system table has vastly improved things. New projects don’t cargo-cult buggy setup.py files from somewhere. Tools can use a real config file format (TOML) instead of being forced to manually validate deserialized values coming from unstandardized formats like INI.

[Kwpolska]

As the author of a slightly popular (and divisive) blog post entitled How to improve Python packaging, or why fourteen tools are at least twelve too many, my opinion is s/fourteen/fifteen/; s/twelve/thirteen/.

The thing that is the most worrisome is the use of Rust. I think Rust is a great language for lower-level systems programming; for many things, something higher-level than Rust is easier to work with and achieves equally good results. But in this specific case, we’re talking about tooling for Python. Does Python suck so much / is Python such a toy language that the packaging tools (and also code formatting tools) must be written in another language? At that point, why not skip Python entirely and write everything in Rust (or another competent language with tooling largely written in itself, like Java or C#?)

In my opinion, installing and distributing Python should be distinct from installing and distributing packages. For your average Windows user, the user experience of “download and run the latest .msi from python.org” or “click Install in the Microsoft Store” is good enough. Beginners don’t need multiple Python versions and a Python version manager (but they do need packages). Businesses may standardise on “whatever RHEL ships and supports”, or on a single version they can install from custom packages tailored to their needs. Forcing the use of a custom-built Python limits the possible users of Rye and thus leads to more tooling proliferation.

Ideally, there would be two separate tools¹:

• A binary distribution management tool — may be written in Rust or anything that’s easy to distribute and run anywhere
• A package management tool — should be written in Python (and PDM would be a good contender for the one-true tool)

Those tools would be separate, but they would also co-operate. Perhaps with a unified py frontend that can invoke whichever of these is necessary, or perhaps the Package Management Tool might say “this package requires Python 3.11, but you’re on 3.10, do you want to abvort, or do you want upgrade this project to 3.11 with the help of the Binary Distribution Management Tool?” Users who are happy with a single version of Python coming from whichever source they like most can still get a sane and unified package manager.

And even though the Steering Council killed PEP 582, I still think it’s the way to go, and that venvs are too confusing and unnecessary. But seeing the SC chose not to accept this PEP, and seeing that most of the 14 15 packaging tools are associated with the PyPA, I don’t think improvements in the packaging world (seeing as many things you’re missing are standards) are likely to appear soon. So, abandon all hope and use Rust or C# for your next project, I guess?

¹ Not the “two” I had in mind when writing the post’s headline — I accept the existence of a “scientist” tool separate from the “normal user” tool, so there could be three tools in total if the unified package management tool can’t solve all the scientists’ problems.

[ifeeltiredboss]

Does Python suck so much / is Python such a toy language that the packaging tools (and also code formatting tools) must be written in another language?

In terms of performance? Yes - see the comparison in the project you linked.

For your average Windows user, the user experience of “download and run the latest .msi from python.org” or “click Install in the Microsoft Store” is good enough.

Speechless.

Ideally, there would be two separate tools¹:
A binary distribution management tool — may be written in Rust or anything that’s easy to distribute and run anywhereA package management tool — should be written in Python (and PDM would be a good contender for the one-true tool)

I don't understand why do we need to separate it? By the way, have you ever used cargo?

[JacobCoffee]

Does Python suck so much / is Python such a toy language that the packaging tools (and also code formatting tools) must be written in another language?

I understand your point, and while Python is by no means a "toy" language, the performance difference with a language like Rust is quite significant. If utilizing Rust can significantly enhance the tooling ecosystem and ultimately improve the developer experience within Python, it would be unwise to overlook the elephant performant crustacean in the room.

I do wish we had a one-stop shop though to download/install Python and not have to worry about managing my project with insert latest tool here.

[Kwpolska]

In terms of performance? Yes - see the comparison in the project you linked.

Is performance the only thing that matters? If yes, you will be happier writing zero Python, and you will not need a Python management tool. (I certainly am happy not writing any Python and not wrangling with any of the 14+ packaging tools at $DAYJOB, to be completely honest.)

I don't understand why do we need to separate it? By the way, have you ever used cargo?

Very little, to be honest, but I found it fine in my limited use. But doesn’t Rust have a separate rustup tool for managing Rust installs? How does that work, and why is rustup separate from cargo?

A good tool that wants to be successful in solving packaging problems should take the various ways in which people use Python into account and should be usable by those people.

Rust is a relatively young language, used by experienced developers, which compiles into mostly-independent binaries. Python is an old language, which requires an interpreter to be installed to run code (unless the author cared to produce some kind of bundle with pyinstaller or the like). That interpreter ships with most Linux distros, and even if the Linux Pythons are sometimes ugly (especially so on Debian and friends), they are good enough for most users of Python. They do the job, and that’s enough for many people, whether you like it or not. And as I mentioned before, the Python that ships in RHEL’s repos is covered by the expensive RHEL support contract $company bought — the Python that Rye/etc would install is not.

Another important group of users: Python is also used a lot in education, with many schools teaching it to teens (or even younger kids?). A good, user-friendly distribution, that does not require touching the terminal much, is crucial here. Something installable with a few clicks is what those users need (and I do not understand your dissatisfaction with that). Rust does not have this problem, because — as I mentioned — Rust is too advanced for mandatory school computer science classes.

Hence two tools: the packaging tool is for everyone. The Python management tool is for advanced Python users for whom the “simple” install option is not enough or unsatisfactory.

[mitsuhiko]

I don't actually disagree with a lot of this. While I think that there is quite a bit of Python packaging code that might benefit of being written in Rust (requirement, version parser and dependency resolver), I don't think there is a lot of benefit of writing everything in Rust. Even today if you use this, it just uses pip, pip-compile, pip-sync and virtualenv behind the scenes.

There is some sort of balance here, I'm not sure where it falls though.

[indygreg]

Since it was only casually mentioned in a reply, I thought I'd drop a link to https://discuss.python.org/t/announce-pybi-and-posy/23021/1 which announced PyBI (eventually PEP 711) and posy.

Posy on the surface appears very similar in scope / vision to rye.

There are significant parallels between this discussion and the linked PyBI/posy thread.

My $0.02 is that tools like rye/posy python-build-standalone/PyBI are clearly demonstrating end-user benefits versus established solutions. Their presence ultimately enriches the Python ecosystem by at the very least providing valuable feedback on how to enshrine standards via PEPs and/or iteratively improve existing/popular tools/workflows. This experimentation is good and should be encouraged. As long as the presence of these alternative tools isn't externalizing significant hardship on Python package maintainers or the larger Python ecosystem, I have no basis to discourage or criticize any work in this space.

Best of luck with Rye. I hope it meaningfully contributes to improving Python's ergonomics for n>>1 people.