Vendor libtest's dependencies in the rust-src component

[Gankra]

This is the Rust side of rust-lang/wg-cargo-std-aware#23

Note that this won't produce a useful result for cargo -Zbuild-std if there are multiple versions of a crate vendored, but will otherwise produce a valid vendor dir.

See rust-lang/cargo#8834 for the other half of this change.

[rust-highfive]

r? @Mark-Simulacrum

(rust_highfive has picked a reviewer for you, use r? to override)

[alexcrichton]

I'm actually somewhat surprised this works and it may be a bug in Cargo, because if lock files exist in locations other than the workspace root they should be ignored. In that sense it shouldn't matter where the lock file is for vendoring.

I also tried running this locally and it just vendored all of rustc's deps, so I'm curious how this is working locally for you now...

[Gankra]

Hmm, maybe I'm triggering bugs in vendor on windows mingw?

Did you get more in your vendor than

addr2line
adler
cc
cfg-if
compiler_builtins
dlmalloc
fortanix-sgx-abi
getopts
gimli
hashbrown
hermit-abi
libc
miniz_oxide
object
rustc-demangle
unicode-width
wasi
(and the rustc-std-workspace crates)

[alexcrichton]

Yeah when I run cargo vendor in library/test (with or without the top-level Cargo.lock in this dir) it vendors the entirety of the crates rust-lang/rust depends on (e.g. a hundred or so crates)

[Gankra]

Note that this runs vendor inside the rust-src directory, so it shouldn't even be able to see rustc? And extra junk in the Cargo.lock doesn't affect the vendor, right? I believe that's why copying the lock is necessary too -- there isn't an actual workspace inside rust-src.

[alexcrichton]

Oh dear I apologize, that I agree should work perfectly and makes sense as a result, ignore me!

[matklad]

Oh, this is perfect for IDEs, thanks! I wonder if we can leave Cargo.lock in the rust-src component as well? This will (maybe, I am not sure?) help intellij rust and rust-analyzer to just run cargo metadata inside of the rust-src component.

cc @vlad20012

[Gankra]

Note that there is a Cargo.lock in the root of the rust-src component, but there's no workspace or .cargo/config connecting everything.

[matklad]

Riiight, I was thinking about Cargo.lock inside /library, so that we can look only at the library (and not on the rest of the compiller), but that's obviously useless without corrresponding Cargo.toml

Anyway, this is off-topic for this PR :)

[Gankra]

@alexcrichton So who needs to sign off on this? Some team?

[alexcrichton]

I'm not sure about sign-offs, and I'm a little uncomfortable to r+ myself, but I'd say from the Cargo team perspective this looks good to me (and a rustbuild perspective), but @Mark-Simulacrum should probably sign-off too.

[Mark-Simulacrum]

I'm not 100% sure we'll be able to write into the src directory for the lockfile -- IIRC, the dist builders have that marked as readonly. But I'm okay with checking in CI for that, since it might not be quite right. Ultimately if that doesn't work we can just copy the whole library directory or something along those lines.

@bors r+ rollup=iffy

[bors]

📌 Commit dd68d0b has been approved by Mark-Simulacrum

[Dylan-DPC-zz]

failed in rollup (#78885)

@bors r-

[Gankra]

hmm, I would have liked a chance to land on my own to be sure. As-is, I can't really make sense of why the specific breakage CI had
might happen.

[Mark-Simulacrum]

Hm, let's actually try a @bors try -- I suspect the dist failure is not Windows specific, though I could of course be wrong :)

[Keruspe]

This introduces a regression wrt distributions packaging. The dist step now requires network access, which it commonly cannot have during such a step of the packaging. Is there a way to make this cargo vendor call "offline"?

[Mark-Simulacrum]

Please file a new issue -- I suspect the answer is yes, though it may require some work; presumably if you are running dist offline (which usually builds code implicitly you have already downloaded the relevant pieces here?), so it should be possible to avoid further network traffic.

[Gankra]

vendor has an --offline flag for exactly this purpose.

[Keruspe]

Will open an issue after investigating, but I guess it might be a cargo vendor issue, not reading net.offline from .cargo/config

[Gankra]

The implementation is a bit hacky, hiding the workspace and copying in the lockfile. So perhaps some other things need to be pulled in in a similar manner. That said, I would hope that --offline Just Works here.

[eddyb]

The implementation is a bit hacky, hiding the workspace and copying in the lockfile. So perhaps some other things need to be pulled in in a similar manner. That said, I would hope that --offline Just Works here.

Not sure where to post this but AFAICT this is fixable by generating a workspace Cargo.toml for the rust-src dist/vendor step, with a [patch.crates-io] section that contains:

rust/Cargo.toml

Lines 129 to 133 in 7941b3f

	# See comments in `library/rustc-std-workspace-core/README.md` for what's going on
	# here
	rustc-std-workspace-core = { path = 'library/rustc-std-workspace-core' }
	rustc-std-workspace-alloc = { path = 'library/rustc-std-workspace-alloc' }
	rustc-std-workspace-std = { path = 'library/rustc-std-workspace-std' }

This could potentially even allow Cargo to stop hardcoding them - if we did ship that Cargo.toml, at least.

(cc @ehuss @Mark-Simulacrum)