Make Cargo aware of standard library dependencies

[Ericson2314]

Currently, all packages implicitly depend on libstd. This makes Cargo unsuitable for packages that
need a custom-built libstd, or otherwise depend on crates with the same names as libstd and the
crates behind the facade. The proposed fixes also open the door to a future where libstd can be
Cargoized.

I've been busy with school until very recently, so apologies in advance if it turns out I am not taking into account some recent change.

Rendered

[posix4e]

Wow neat

[tshepang]

s/were/where

[Ericson2314]

Thanks!

[llogiq]

A funny 'coin'cidence is that the Java folks at Oracle are currently testing a similar concept (and they needed 9 major versions to get there). 👍

[Valloric]

The specific name doesn't concern me too much, but do make it boolean-positive instead of negative. So instead of having no-implicit-deps and setting it to false by default, make it use-implicit-deps and make it true by default.

Whatever name is bikeshedded should be boolean-positive.

[Valloric]

Similar here; instead of --no-sysroot, make it --use-sysroot=false (with the default being true).

[Ericson2314]

@Valloric in case you don't get a notification from the commit message, I did what you suggested with the exception that the field is just called implicit-deps. I did that as I didn't want to make anyone think of Rust's use, because then the fields sounds more like #![no_std].

[Valloric]

@Ericson2314 That's fine; like I said, I don't personally care much about the name (you'll get more than enough bikeshedding from others), only that a double negative is avoided.

[llogiq]

That should probably read "...is to be downloaded...".

[llogiq]

Also the text states that "std should be on crates.io" – the rust distribution should still contain all of std anyway, right? Otherwise people who download rust to try it during their commute are going to have a hard time. 😄 (Case in point: I recently had to download a doc snapshot from servo so I could continue to hack on rust-clippy when offline, because the standard docs no longer contain the rustc… crates).

[alexcrichton]

Thanks for the RFC @Ericson2314! I'm always quite eager to improve the cross-compilation experience of Rust, and I think that this will help out embedded targets and such quite a bit!

At a high level, could you clarify the motivation for this RFC a bit more in the text? After reading it it's not clear to me what the RFC is trying to achieve. For example:

• Is the point of this RFC to enable easy cross compilation of the standard library?
• Is this enabling swapping in an alternate standard library?
• Is this just disabling the ability to extern crate std if a crate opts-out?

I think that refining the motivation will help me understand more of the detailed design as well, but I'll make some specific comments below about this.

However, it also passes in --sysroot /path/to/some/libdir, the directory being were libstd is.

I'm curious where you came up with this? Cargo doesn't pass --sysroot at all.

This need not imply Rust's no_std, as one might want to use their own build of libstd by default.

This worries me a bit, I don't think we want the ability to swap in an alternate standard library in a super easy fashion. If you're depending on your own "standard library", then I think it should be just that, a dependency in Cargo.toml which goes through the normal crates.io + Cargo process. In general I think it's pretty brittle to swap out the std that an upstream crate is using, praying the APIs match up and the compile still succeeds.

This hotfix is enough to allow us bare-metal devs to use Cargo for our own projects, but doesn't suffice for creating an ecosystem of packages that depend on crates behind the facade but not libstd itself.

Can you clarify why you can't use Cargo for these kinds of projects today? If you use #![no_std] and don't use extern crate std, you can basically do anything with Cargo you can do with the compiler. For example I have written my own kernel which is built with Cargo and depends on rlibc from crates.io, and it all works out just fine (no standard library exists).

But users can use config files to extend the namespace so their own modded libstds can be used instead. Even if they don't want to change libstd but just cross-compile it, this is frankly the easiest way as Cargo will seemliest cross compile both their project and it's transitive dependencies.

I also unfortunately don't fully understand what's going on here. Can you elaborate why you want to swap out a separate libstd than the one the compiler may already find elsewhere? If crates can be tracked in Cargo, then I definitely think they should, so I don't follow the motivation to not use #![no_std] and instead try to name a crate std somewhere.

In this way we can put packages on crates.io that depend on the crates behind the facade.

I don't quite understand this comment because this can already be done? I can upload a crate which says #![no_std] and doesn't depend on the standard library (only libcore). Can you elaborate on how this is different and what you're thinking here?

---

Overall I think this RFC feels like it's "bolting on" features after the fact instead of creating a well-integrated and smooth system with Cargo. I believe this is also (depending on the motivation) highly inter-related with #1005, and it may wish to be considered here. I think some refinement of the motivation, however, will help guide this RFC, though!

[Ericson2314]

Ok, I talked to @alexcrichton for a long while on IRC, and then rewrite the RFC. In short, I massively elaborated the motivation, and cut phase 1. I'm respond to some his points, which will also offer some summery of what we talked about, though go to the IRC logs if you're really curious.

At a high level, could you clarify the motivation for this RFC a bit more in the text? After reading it it's not clear to me what the RFC is trying to achieve. For example:

• Is the point of this RFC to enable easy cross compilation of the standard library?
• Is this enabling swapping in an alternate standard library?
• Is this just disabling the ability to extern crate std if a crate opts-out?

I think that refining the motivation will help me understand more of the detailed design as well, but I'll make some specific comments below about this.

All three of those are valid motivation for this.

I'm curious where you came up with this? Cargo doesn't pass --sysroot at all.

I extrapolated incorrectly from his comments in the issue I opened long ago. It's fixed in the current RFC. The larger picture is the same either way however: Cargo allows rustc to look in the sysroot for libs.

This worries me a bit, I don't think we want the ability to swap in an alternate standard library in a super easy fashion.

I'm OK with having implicit-deps = false imply #![no_std], I just wanted to point out that technically the two features are orthogonal. The RFC now as some examples of how they can be used orthogonally, but again I don't care very much about this bit.

If you're depending on your own "standard library", then I think it should be just that, a dependency in Cargo.toml which goes through the normal crates.io + Cargo process.

I agree wholeheartedly, but this is currently not possible if crate names + target platform overlap with anything that is in the sysroot.

In general I think it's pretty brittle to swap out the std that an upstream crate is using, praying the APIs match up and the compile still succeeds.

This this can only be accomplished with a Cargo config override, which I consider sufficiently hacky. It's possible phase 1 would have allowed this to happen more accidentally, but that's outta here.

Can you clarify why you can't use Cargo for these kinds of projects today? If you use #![no_std] and don't use extern crate std, you can basically do anything with Cargo you can do with the compiler. For example I have written my own kernel which is built with Cargo and depends on rlibc from crates.io, and it all works out just fine (no standard library exists).

I wanted libcore to be cross compiled automatically, but if on some machines the host and target match (i.e. no cross compile), the build will fail. See the expanded motivation for more details.

I don't quite understand this comment because this can already be done? I can upload a crate which says #![no_std] and doesn't depend on the standard library (only libcore). Can you elaborate on how this is different and what you're thinking here?

I meant do this in a such that all deps can be (cross-)compiled automatically. See rewritten RFC and below.

Overall I think this RFC feels like it's "bolting on" features after the fact instead of creating a well-integrated and smooth system with Cargo. I believe this is also (depending on the motivation) highly inter-related with #1005, and it may wish to be considered here. I think some refinement of the motivation, however, will help guide this RFC, though!

I think I was able to explain my case on IRC. Basically @alexcrichton and I (and presumably you, dear reader, too) would like easy no-brainer cross compiling. If we don't make packages state which of libstd or its backing crates packages they need, future Cargo needs to download/build everything to play it safe, and needs to do that for ever.

By treating these things as normal deps, Cargo can do something adhoc like querying a Mozilla build farm, but can seamlessly transition to just building them like any other package if they get Cargoized. Then all that ad hoc functionality can be thrown out the window.

Finally, if you use a unstable compiler, you can cross compile your own libstd today, and it is nice to do so. While it would be great to standardize libcore, using it effectively requires a fair amount of lang items, and if those are standardized, perhaps all of libcore's implementation could to[1]. Thus, for the time being, freestanding development is probably going to require a compiler that is capable of building libcore, and it just so happens that those people benefit from that the most, so we might as well give them a way to cross compile their whole project---libcore up---today.

[1] @alexcrichton pointed out that with standardization there might be less overlap with the lang needed items to use libcore and the lang items it uses in its implementation, quite possible.

[alexcrichton]

I think it's important to spell out here that it's far from standard practice to dump libraries in the sysroot, and the only stable library in the sysroot today is libstd. We have been very hesitant to stabilize any more than precisely one library for many of these reasons.

[Ericson2314]

I'm a bit confused what you'd like me to elaborate on. I already wanted to emphasize we just do this in the case of libstd and its dependencies---in other words that we are so close---just 1 library away!---from not linking with the sysroot at all.

[alexcrichton]

Hm re-reading I'm not quite sure what I was thinking... It may have been from the aspect that "and its deps" isn't so relevant in stable Rust today as libstd is the only library that can be implicitly linked to.

[Ericson2314]

Ah OK. I'll clarify the situation for stable Rust.

[alexcrichton]

This isn't quite true, for example liblibc exists in both the sysroot and on crates.io. Cargo will pass --extern libc=... which overrides everything (including the sysroot).

[Ericson2314]

Hmm, I'll need to update my code so I can see why the build is failing, or whether it does today.

[alexcrichton]

s/x86_64-gnu-linux/x86_64-unknown-linux-gnu/

[Ericson2314]

Oops, thanks!

[alexcrichton]

Could you clarify what this means for "a package name it cannot resolve"? For example Cargo does not attempt to resolve the name "std" in any way today, so I'm not sure where this sort of resolution failure will start from.

[alexcrichton]

Reading a little more, my interpretation is that you're proposing that a crate explicitly declares that it depends on core and std (if the boolean field above is specified), is that right? If so, can you go into some more detail about what the syntax for doing so might be?

[Ericson2314]

Basically, I want it so

implicit-deps = false;

[dependencies]
core = "*" # Or some more appropriate version specifier
alloc = "*"
# ...other crates behind the facade....
std = "*"

and

implicit-deps = true;

and

# implicit-deps is true by default

all mean the same thing.

[alexcrichton]

Hm ok, your first snippet has a bit of a different interpretation because it means the dependencies like std come from crates.io, which probably isn't going to happen any time soon. Put another way there's no way to express a dependency on std in Cargo.toml today because we distribute it in binary form instead of on crates.io.

I think the reason I'm somewhat uneasy to add implicit-deps is that if you specify implicit-deps there's no way for you to actually link to the standard library. Did you have something in mind for doing that?

[Ericson2314]

Mmm, this RFC changes the way cargo works so that depended-on crates not on crates.io are looked for in the syroot instead, precisely so we can continue distributing those crates the same way for the time being. This allows:

• A clear migration path to a future where those crates are defined on crates.io
• A finer way to distinguish what functionality libraries intended for kernel use need
• Unstable rust users to locally define those crates so cargo cross-compiles them with the rest of their program.

[alexcrichton]

Ah I see what this is saying now, although unfortunately I feel like that's a little too much magic going on under the hood. Cargo understands the "source" for any particular package, and it needs to understand if that source is crates.io or the sysroot ahead of time. Along those lines I think that this needs to have some new source syntax, such as:

[dependencies]
std = { rustc-sysroot = true }

[Ericson2314]

While yes, that does make Cargo's life easier, does this information really belong in the package metadata? A package just cares what version std is, not how cargo obtained it. Also, if we switch to deploying these crates via crates.io before the end of 1.0, we wouldn't want packages to break because they mandated that std must come from the sysroot.

That said, I'd still rather have that than the status quo. IIRC local packages (with cargo config) override no matter the version so this doesn't prohibit my third bullet point in my previous post.

[alexcrichton]

Looking at this from another angle, the only implicitly available crate that is available in stable Rust is std. This means that we have quite a bit of freedom when considering the other crates distributed with Rust itself. For example this field could in theory just be implicit-std = false which passes a flag to the compiler disabling the implicit usage of std, and then the compiler will implicitly deny access to all other crates by default (e.g. even libcore).

Just a note that we have very few constraints today (just the name "std"), and we can do whatever we like with the other deps.

[Ericson2314]

I'm confused, I think the semantics you are describing is exactly what I proposed. implicit-deps controls access std and it's dependencies---whatever those may be.

I went with -deps and not -std because of unstable Rust. But if we want to gear the name around stable Rust, sure.

[alexcrichton]

Oh yes we're definitely thinking of the same thing, I was just wondering if the name implicit-std was better. For example the name implicit-deps seems kinda scary that any crate could be an implicit dependency, when in fact there is only one crate in the stable world that can be an implicit dependency -- libstd.

I agree there are more crates that can be implicitly depended upon, but none of them are stable today, so we may not need to consider them. I think I was just somewhat startled at how this may imply that implicit dependencies are allowed from anywhere (when it's in fact just the sysroot)

[Ericson2314]

I see. Somebody beginning Rust, or beginning just even unstablized Rust, has no idea what dependencies would normally be implicit. One the other hand implicit-std sounds like one is just getting access to std, since Cargo normally doesn't allow one to extern transitive dependencies lest a direct dependency changes its dependencies. A name like implicit-std-and-its-deps would be the most clear, but my is it wordy.

Another option is to change it so that implicit-std = true only gives access to std itself, and dependencies on the crates behind the facade must always be explicit. This would also help on the off-chance that we want to version unstable std independently from its backing crates. Unfortunately, it also breaks backwards compatibility with existing packages, but only those using unstable Rust.

[eternaleye]

Another, more concise phrasing might be implicit-stdlibs = true - the 's' punches well above its weight class.

[Ericson2314]

Not bad! I tried to think of a short name to capture all that, but came up short.

[alexcrichton]

Ah one point I forgot about previously, which is probably pretty relevant to this, is: the compiler can only link against libraries it previously built. This means that we would need a build cache per-revision of the compiler, which unfortunately makes this much more infeasible :(

[Ericson2314]

Hmm, we over at NixOS maintain a build cache for Haskell and it works. If you only focus the slower release channels, and prioritize popular packages, it can still be useful.

The idea of a stable ABI scares me, but if/when it happens, that problem goes away too.

[alexcrichton]

Ah yeah we could definitely pre-cache builds of std for each official release of the compiler, for example, but it may want to be mentioned here as a potential downside. For example all custom builds of the compiler (e.g. nightlies) will not have access to pre-built archives.

[Ericson2314]

I still don't see the downside. Today, the compiler and library are built together, so we are already building and storing std for each prebuilt compiler. Whether or not std is downloaded with a prebuilt compiler or separately from a crates.io build cache, the build time and storage requirements are the same.

[llogiq]

Downloading std separately will stop those who have installed rust to look at it during their commute (case in point: me, a few weeks ago). Not everyone has a fast internet connection everywhere, so there may be other hidden downsides.

[Ericson2314]

Mmm, once std is downloaded once, it doesn't in principle need to be downloaded again. Assuming a local build cache shared between projects is implemented at this point in the future, the compiler's install script could set it up and pre-populate it with std. That way nobody forgets std on their commute :).

[nrc]

rust-lang/cargo#4959 is a partial (and more minimal) alternative to this RFC. I propose we close this RFC in favour of that proposal.

@rfcbot fcp close

[rfcbot]

Team member @nrc has proposed to close this. The next step is review by the rest of the tagged teams:

• @alexcrichton
• @aturon
• @carols10cents
• @fitzgen
• @japaric
• @jonathandturner
• @joshtriplett
• @killercup
• @matklad
• @michaelwoerister
• @nrc
• @withoutboats
• @wycats

No concerns currently listed.

Once a majority of reviewers approve (and none object), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[jonathandturner]

Go ahead and make me a noop vote

[mark-i-m]

I think we should keep this RFC open simply as a tradition 😛

[killercup]

@rfcbot reviewed

[japaric]

@rfcbot reviewed

[fitzgen]

@rfcbot reviewed

[Ericson2314]

I do agree we should have a more minimal first step, but I don't think it should be (just) rust-lang/cargo#4959. The most important part of this is the interface for specifying stdlilb deps, so we can cleanly bypass all the crufty sysroot and boostrapping implementation details needed today. Much of the discussion thread of this PR also mistakenly focused on uninteresting implementation minutia.

I'm happy to rewrite this or open a new one to that end.

[matklad]

+1 to the idea that sysroot should die and that stdlib crates should not be much different from crates from crates.io. We still, after several years, would like std to be present in cargo metadata, for example, to simplify some stuff in IntelliJ :)

[Ericson2314]

@matklad Yeah that's a great point. When I started preaching death to the sysroot, it was a pretty academic question of avoiding the complexity of needless distinctions. But now with all the new tooling work, be it incremental compilation, IDE support, etc, there's real tangible benefits visible to just about every regular user (far beyond the niche of embedded and OS devs).

[nrc]

@Ericson2314 I'd like to not lose track of the other issues. I don't think a new RFC is the best way to do that though. Perhaps opening an issue or two on the Cargo repo would be better? Where there are UI things, I'd be keen to experiment on top of the sysroot stuff as a plugin if possible, otherwise as an unstable feature, then RFC later. For other things it would be good to identify small chunks of work which could be isolated and have their own RFCs or implementation.

[Ericson2314]

@nrc unstable feature -> RFC is fine with me. The stdlib dep is pretty orthogonal to @japaric's sysroot stuff, however.

[Ericson2314]

@nrc OK made 2 (and found 1). Those basically correspond to what I implemented in my Cargo PR way back when, and can all be handled "decently independently".

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[rfcbot]

The final comment period is now complete.

[Centril]

Closing the RFC per complete FCP #1133 (comment) with a disposition to close.
Thanks @Ericson2314 for the RFC!