rust_toolchain now generates a Rust sysroot

[UebelAndre]

Currently, rust_toolchain does not support the use of toolchain components that are not already a part of a valid sysroot structure. This means that if I have a rustc artifact and a rust-std(Rust standard library) artifact, I cannot use them if they were not downloaded into the same repository rule and follow a specific directory structure. This makes toolchain management hard and leads to unnecessary data being downloaded if users want to have toolchains for multiple target platforms as before any 1 toolchain could be used, all rust-std artifacts (or any other target specific assets) would all need to be downloaded regardless of whether or not that target platform is being built for. This PR aims to solve for allowing users to use individual Rust static assets without needing to do additional maintenance to maintain asset bundles specific to all targets or waste time fetching components that will not be used.

[hlopko]

Adding @krasimirgg who is doing something similar on our end :)

[krasimirgg]

We'd like to do some more extensive internal testing of this PR.
I will aim to share our findings in a few business days.

[krasimirgg]

We hit a tricky issue with this PR internally. This PR causes some unstable clients that use extern crate rustc_driver to fail to build under a configuration with remote caching. An example is Kythe: https://osscs.corp.google.com/kythe/kythe/+/master:kythe/rust/extractor/src/lib.rs;l=16?q=rustc_driver&ss=kythe%2Fkythe:kythe%2Frust%2F,
but I've got a smaller example here.

#![feature(rustc_private)]
extern crate rustc_driver;
fn main() {}

Compiling something like this in linux that refers to the librustc_driver-$HASH.so distributed with the toolchain causes rustc to invoke the linker with a -L + -l combination e.g.:
-L<stuff>/.cache/bazel/_bazel_krasimir/c736e0c317c9884eaf6f814e37ef83da/external/rust_linux_x86_64/lib/rustlib/x86_64-unknown-linux-gnu/lib -lrustc_driver-cbd31b920a2b2b82
The linker is happy, as it finds the librustc_driver-$HASH.so in the -L dir.

Now, suppose that instead librustc_driver-$HASH.so happened to be a symlink to a file with a different name in a different directory, e.g., librustc_driver-$HASH.so -> /tmp/dir/FILE. Importantly the symlink it points to doesn't follow the lib<name>.so naming convention (something like this happens in our configuration with remote caching). In a case like this, the linker sees something like -L/tmp/dir -lFILE, which doesn't work since the linker can't find libFILE.so under /tmp/dir. Note that this by itself is a pre-existing issue, it didn't work before this PR. With Bazel, I can repro this by manually copying librustc_driver-$HASH.so from the bazel cache sysroot to /tmp/dir/FILE and making the librustc_driver-$HASH.so in the bazel cache sysroot to be a symlink to that FILE. We'll investigate this further and possibly bring this to the attention to upstream rustc (e.g., for rustc to specify the location rustc_driver literally).

Now this PR seems to trigger this pre-existing issue in remote caching builds in certain configurations where the librustc_driver-$HASH.so is provided via the data attribute of some sh_binary. Prior to this PR, the paths resolve to a temporary directory where the librustc_driver-$HASH.so is extracted to with exactly this filename. After this PR, the paths resolve to a remote cache directory where the librustc_driver-$HASH.so exists under an unrelated name (hash of the contents). I tried but unfortunately couldn't get to reproduce the problem using the RBE rules_rust buildbots.

@hlopko had an idea that may help us out: In this PR we always generate symlinks for sysroot files. Instead, could this PR be enchanted so that if the inputs are already in a directory tree that resembles sysroot, we use them directly without creating symlinks?

Alternatively as a bit of a workaround, wdyt about adding a new attribute, say copy_sysroot_artifacts_pattern that allows us to specify a pattern for sysroot artifacts that should be copied instead of just symlinked?

[krasimirgg]

We hit a tricky issue with this PR internally, caused by a misfortunate combination of factors. This PR causes some unstable clients that use extern crate rustc_driver to fail to build under a configuration with remote caching. An example is Kythe: https://osscs.corp.google.com/kythe/kythe/+/master:kythe/rust/extractor/src/lib.rs;l=16?q=rustc_driver&ss=kythe%2Fkythe:kythe%2Frust%2F,
but I've got a smaller example here.

So "-L" "/usr/local/google/home/krasimir/.cache/bazel/_bazel_krasimir/c736e0c317c9884eaf6f814e37ef83da/external/rust_linux_x86_64/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-lrustc_driver-cbd31b920a2b2b82"

[UebelAndre]

@hlopko had an idea that may help us out: In this PR we always generate symlinks for sysroot files. Instead, could this PR be enchanted so that if the inputs are already in a directory tree that resembles sysroot, we use them directly without creating symlinks?

I don't think I like the forked behavior. I'd rather the toolchains work the same way all the time and not need to do any work to detect a directory structure. Creating symlinks should be very fast but alternatively a directory could be created where files are copied into place if symlinks are a concern. WDYT?

[krasimirgg]

Creating symlinks should be very fast but alternatively a directory could be created where files are copied into place if symlinks are a concern. WDYT?

Sounds roughly along the lines of the symlink pattern approach from my previous comment:

Alternatively as a bit of a workaround, wdyt about adding a new attribute, say copy_sysroot_artifacts_pattern that allows us to specify a pattern for sysroot artifacts that should be copied instead of just symlinked?

Are you thinking about something like this, or something where we always copy everything instead of using symlinks ever? I personally don't have a sense if copying all the toolchain artifacts (some of which are quite large files) like that may become prohibitively expensive in some situations; @hlopko may have an opinion on this.

[UebelAndre]

Are you thinking about something like this, or something where we always copy everything instead of using symlinks ever? I personally don't have a sense if copying all the toolchain artifacts (some of which are quite large files) like that may become prohibitively expensive in some situations; @hlopko may have an opinion on this.

I'm thinking that we would always copy the same files into the sysroot. My expectation is that instead of having an extra attribute for determining what to copy (or symlink), that would be determined by the attributes themselves. Meaning if you don't want something in your sysroot, don't include it in the toolchain (behavior may vary with specific attributes but this would generally be the rule).

[UebelAndre]

@krasimirgg Also, thank you so much for the quick replies! Much appreciated 🙏

[krasimirgg]

... My expectation is that instead of having an extra attribute for determining what to copy (or symlink), that would be determined by the attributes themselves. Meaning if you don't want something in your sysroot, don't include it in the toolchain

The subtlety is in the distinction between the [what to copy] and [what to symlink] cases. If we go with the approach to copy everything coming from all attributes, we don't have the rustc_driver problem above since we're not dealing with symlinks.

If we go the approach to use symlinks (for some of the stuff coming from the attributes), we would still want librustc_driver-$HASH.so to exist, but as a real file, not as a symlink, for cases like that example: since it uses #![feature(rustc_private)] extern crate rustc_driver; (part of the dev_components bundle), it refers to librustc_driver-$HASH.so from the sysroot. If we don't include librustc_driver-$HASH.so in the sysroot, building the binary will just fail with a cannot find crate rustc_driver error. We want to have it in the sysroot, just ensure it's a real file, since the way rustc passes it to the linker breaks if it's a symlink to a random other file [in some configurations].

[UebelAndre]

The subtlety is in the distinction between the [what to copy] and [what to symlink] cases. If we go with the approach to copy everything coming from all attributes, we don't have the rustc_driver problem above since we're not dealing with symlinks.

If we go the approach to use symlinks (for some of the stuff coming from the attributes), we would still want librustc_driver-$HASH.so to exist, but as a real file, not as a symlink, for cases like that example: since it uses #![feature(rustc_private)] extern crate rustc_driver; (part of the dev_components bundle), it refers to librustc_driver-$HASH.so from the sysroot. If we don't include librustc_driver-$HASH.so in the sysroot, building the binary will just fail with a cannot find crate rustc_driver error. We want to have it in the sysroot, just ensure it's a real file, since the way rustc passes it to the linker breaks if it's a symlink to a random other file [in some configurations].

I guess, it doesn't matter to me if symlinks or copies are used, I don't want the behavior to be configurable outside of what is passed to what attribute (e.g. rust_stdlib_filegroup). Though, I'm confused at how a situation like this occurs. I won't be able to test on linux till later today but I'm surprised the solution to a problem like this is not "include the missing file in the sysroot" which I'd expect to be done by expanding the globs here.

edit:

I don't want the behavior to be configurable outside of what is passed to what attribute (e.g. rust_stdlib_filegroup)

Expanding on this a bit, I feel there's already lots of work needed for setting up external crates and thing the configuration tax of using Rust in Bazel is already at capacity. If possible, keeping the toolchain behavior as consistent as possible for users would be better for everyone over all IMO.

[krasimirgg]

Update: the rustc_driver issue was triggered by the additional --sysroot in the older version of this PR. It appears that rustc changes how it canonizes filepaths with this flag.

The current version of this PR looks good (using symlinks for everything, no need to copy anything, no need for a pattern-knob workaround) IMO.

[UebelAndre]

Update: the rustc_driver issue was triggered by the additional --sysroot in the older version of this PR. It appears that rustc changes how it canonizes filepaths with this flag.

The current version of this PR looks good (using symlinks for everything, no need to copy anything, no need for a pattern-knob workaround) IMO.

Oh, I actually did not test the removal of --sysroot correctly yesterday in that I didn't test with the right toolchains. The --sysroot flag still seems largely unneeded but does cause issues in rust_doc_test where the Rust standard lib files cannot be found.

[UebelAndre]

The last commit fixes the issue for me but does rely on the --sysroot flag for rustdoc...

[UebelAndre]

Though, I'm confused at how a situation like this occurs. I won't be able to test on linux till later today but I'm surprised the solution to a problem like this is not "include the missing file in the sysroot" which I'd expect to be done by expanding the globs here.

@krasimirgg do you know why the issues with rustc_driver are not fixed by updating the glob pattern for rustc_lib?

[krasimirgg]

The last commit fixes the issue for me but does rely on the --sysroot flag for rustdoc...

Using the --sysroot flag for rustdoc sounds good to me (just using it with rustc seems problematic).

do you know why the issues with rustc_driver are not fixed by updating the glob pattern for rustc_lib?

I don't understand the mechanism that causes the issues with --sysroot while using the rustc_driver in our remote build configurations. I'll double check, but I believe the [equivalent of the] glob patterns we use are correct.
I suspect it's something to do with symlink canonization in rustc (e.g., there's some non-trivial canonization here), but that's pure speculation. I'll poke around this more and share what I discover.

[UebelAndre]

@hlopko @krasimirgg If there are no outstanding concerns, could I get a final review from each of you? 🙏 😅

[krasimirgg]

Looks good to me. I just noticed a small nit.
Please wait for @hlopko's review/ approval.

[UebelAndre]

Looks good to me. I just noticed a small nit. Please wait for @hlopko's review/ approval.

@krasimirgg Will do.

I don't understand the mechanism that causes the issues with --sysroot while using the rustc_driver in our remote build configurations. I'll double check, but I believe the [equivalent of the] glob patterns we use are correct.
I suspect it's something to do with symlink canonization in rustc (e.g., there's some non-trivial canonization here), but that's pure speculation. I'll poke around this more and share what I discover.

Separate from the reviews, I'd still love to learn what you find here. Should I make a new issue for this so it's easier to track?

[hlopko]

Only minor things, in general looks good to me, thank you!!

[hlopko]

Thank you! :)