Add shared toolchain directories

[Rx1513]

Problem you are trying to solve

Hello!

As a package maintainer, I want to package different Rust toolchain distributions, such as stable, nightly, specific stable versions, or custom forks (for example, the ESP toolchain). To make this practical, some kind of toolchain multiplexer is needed. While I could write a minimal multiplexer myself, it would lack many useful features.

On the other hand, I could use rustup, which is widely adopted and well supported. However, its main drawback is poor integration with OS package managers. There is no automatic way to make a system-provided Rust toolchain available through rustup. Although it is possible to manually link a system toolchain, there is no mechanism to keep it synchronized with updates from the package manager.

As a user, I want to be able to use both the system-provided toolchain and the official Rust toolchain. For example, I may use one for day-to-day development and the other for feature testing or validation or in some cases I may want to use local distribution infrastructure. Which is possible but with some extra work.

Solution you'd like

To solve these issues, a separate shared directory managed by external tools could be added.

rustup should automatically discover toolchains from the shared directory.

Toolchains in RUSTUP_HOME should take precedence over toolchains from the shared directory. If a local toolchain shadows a shared toolchain with the same name, this should be clearly indicated. For example:

rustup default alt
info: shared toolchain overridden by `RUSTUP_HOME` toolchain
info: default toolchain set to alt

This feature should be opt-in. It should be possible to enable it through user or system configuration, or with an environment variable.

rustup toolchain list should clearly indicate which toolchains belong to which mechanism. For example:

$ rustup toolchain list
shared toolchains:
alt-1.90 (default)
alt-1.92
alt-nightly-1.95 (shadowed)
esp (shadowed)

local:
stable
alt-nightly-1.95
esp (active)
nightly

$ rustup toolchain list -v
shared toolchains (/usr/lib64/rust-toolchains/):
alt-1.90 (default) /usr/lib64/rust-toolchains/alt-1.90/
alt-1.92 /usr/lib64/rust-toolchains/alt-1.92/
alt-nightly-1.95 (shadowed) /usr/lib64/rust-toolchains/alt-nightly-1.95/
esp (shadowed) /usr/lib64/rust-toolchains/esp

local (/home/user/.rustup):
stable /home/user/.rustup/toolchains/stable
alt-nightly-1.95 /home/user/.rustup/toolchains/alt-nightly-1.95
esp (active) /home/user/RPM/BUILD/rust-esp/build/host/stage1
nightly /home/user/.rustup/toolchains/nightly

Notes

Related to #313, #1085, and #2383.

Unresolved problem:
Should this feature provide a mechanism for specifying multiple shared directories, or should it support only a single shared directory?

I have some initial work in progress, so I would like to implement this feature.

[rami3l]

Although it is possible to manually link a system toolchain, there is no mechanism to keep it synchronized with updates from the package manager.

@Rx1513 Hello! My apologies but I'm not quite sure what are you looking for with this proposal.

As I understand it, since it's a link, a linked system toolchain from the package manager is automatically synchronized with updates from the package manager.

rustup should automatically discover toolchains from the shared directory.

Why would batch-assigning a directory of linked toolchains simplify your workflow? Could you please elaborate?

To solve these issues, a separate shared directory managed by external tools could be added.

Should it be the responsibility of the external tools, whatever they may be, to add such links?

PS: We are avoiding adding more layered configurations so I would prefer being more careful with that part of your proposal, and it's vital that we reach an understanding of what exactly is needed in your use case.

PPS: If you are referring to using project-specific sets of tools under a specific name, I have this proposal that may suit your needs: #4636

[Rx1513]

Although it is possible to manually link a system toolchain, there is no mechanism to keep it synchronized with updates from the package manager.

@Rx1513 Hello! My apologies but I'm not quite sure what are you looking for with this proposal.

As I understand it, since it's a link, a linked system toolchain from the package manager is automatically synchronized with updates from the package manager.

The problem is that each link is handled manually.

• If the package is removed, rustup still lists the linked toolchain until the user manually removes the link.
• When the package manager installs a package, user must manually link the distributed toolchain and then select it. It may sound simple but in practice it looks something like this:

apt-get install rust
Look where package places its toolchain
rustup toolchain link <user-specific name> <generally long absolute toolchain path>
rustup default <user-specific name> || rustup override set <user-specific name>
# Done

Two middle steps seems very unnecessary as it requires new useless information for user and some extra steps, especially when compared to official toolchains installation:

rustup default
# Done

What I want is to make usage of system's toolchains is similar to usage of official toolchains:

apt-get install rust
rustup default <distro specific name> || rustup override set <distro specific name>
# Done

or even:

apt-get install rust
# Done

In my opinion the additional lookup directory should create much smoother user experience when dealing with toolchains from other external sources.

Why would batch-assigning a directory of linked toolchains simplify your workflow? Could you please elaborate?

It doesn't require from each user doing manual tedious work. It helps separate responsibilities: external tools manage external directories under administrator control, while the user's home directory remains solely the user's responsibility.

To solve these issues, a separate shared directory managed by external tools could be added.

Should it be the responsibility of the external tools, whatever they may be, to add such links?

As far as I know many Linux distributions discourage when package manager messes with home directory so it's not the way. The only way is if user manually links every external toolchain or if administrator does it for them, which yet again messes with responsibility.

PS: We are avoiding adding more layered configurations so I would prefer being more careful with that part of your proposal, and it's vital that we reach an understanding of what exactly is needed in your use case.

I think this feature keeps the configuration largely unchanged. It only adds an additional lookup directory for externally managed toolchains, while the existing per-user links continue to work unchanged.

PPS: If you are referring to using project-specific sets of tools under a specific name, I have this proposal that may suit your needs: #4636

I don't think this is related to the problems I'm addressing here, but thanks for help.

[rami3l]

@Rx1513 Thanks for your detailed reply! As I roughly understand it, you are referring to adding support for a new toolchain directory for automatic toolchain discovery (but read-only)?

When the package manager installs a package, user must manually link the distributed toolchain and then select it.

IRRC rustup treats every folder under the toolchain directory as a valid toolchain, so I am curious about how we could probably make it work with your case.

What is your specific distro/external tool that installs externally-managed Rust toolchains in a way that is compatible with your new directory proposed here? For Homebrew (what I'm using) for example, the system Rust toolchain is installed alongside other packages. Is rustup supposed to distinguish between those?

I'm not opposed to adding a new folder in the current rustup installation layout of some sort that you can possibly link a local directory to, IMHO the most important thing here is still what tool you are trying to make rustup work with.

I think this feature keeps the configuration largely unchanged. It only adds an additional lookup directory for externally managed toolchains, while the existing per-user links continue to work unchanged.

Not quite; the alt-nightly-1.95 (shadowed) in your previous example might indicate that alt-nightly-1.95 has different semantics under different situations, that is exactly the kind of extra complexity we would like to avoid.