Add support for loading rustc private crates

[xldenis]

This PR presents a solution to the problem of makingrustc_private crates visible to rust-analyzer.
Currently developers add dependencies to those crates behind a cfg(NOT_A_TARGET) target to prevent cargo from building them.
This solution is unsatisfactory since it requires modifying Cargo.toml and causes problems for collaboration or CI.

The proposed solution suggested by @matklad is to allow users to give RA a path where the rustc sources could be found and then load that like a normal workspace.

This PR implements this solution by adding a rustcSource configuration item and adding the cargo metadata to the crate graph if it is provided.

---

I have no idea how this should be tested, or if this code is generally tested at all. I've locally run the extension with these changes and it correctly loads the relevant crates on a rustc_private project of mine.

[xldenis]

unfortunately, this match arm becomes more complex since it must load rustc_workspace (if provided) and cargo. I tried to clean it up a bit by pulling out the logic around adding a pkg, tgt crate root to a separate function, but there is still quite a bit of repitition.

[matklad]

Yeah, I'll look into fixing this after the PR is meged. I bet the code could be refactored somewhat more radically.

[xldenis]

yea, don't feel confident enough with the codebase to go ahead and do that, but I can definitely see some possible refactorings.

[matklad]

Changes look good to me, modulo dbg!. It would help a bit if you could revert package-lock.json changes, I'll land them separately.

We have one integration test for project loading, but for rustc crates, I think it's better to stick with manual testing.

[matklad]

cc @RalfJung I think you wanted something like this for miri, no? I vaguely recall that I promised you to look into something, but I don't remember what it was 😅

[RalfJung]

Wow I think that was a long time ago.^^ I haven't been doing a lot of Miri stuff lately so I didn't really run into RA problems with Miri. But there are a lot of {unknown} when I hover variable names, and I assume this would help with that?

[matklad]

If they come from rustc_private crates, then yes, this would help

[RalfJung]

So there is no rustup package that contains the sources? I guess this is sufficiently niche to not really make that worth it.

[eddyb]

We have rust-src and rustc-dev, maybe it's about time we add rustc-src? cc @Mark-Simulacrum @pietroalbini

[pietroalbini]

Can we just stick the source in rustc-dev?

[xldenis]

I believe the sources are shipped in rustc-dev no?

λ(ada) syn-specs → git spec-lang-2 * ls ~/.rustup/toolchains/nightly-2020-10-20-x86_64-apple-darwin/lib/rustlib/rustc-src/rust/compiler
rustc                 rustc_error_codes     rustc_lint            rustc_resolve
rustc_apfloat         rustc_errors          rustc_llvm            rustc_save_analysis
rustc_arena           rustc_expand          rustc_macros          rustc_serialize
rustc_ast             rustc_feature         rustc_metadata        rustc_session
rustc_ast_lowering    rustc_fs_util         rustc_middle          rustc_span
rustc_ast_passes      rustc_graphviz        rustc_mir             rustc_symbol_mangling
rustc_ast_pretty      rustc_hir             rustc_mir_build       rustc_target
rustc_attr            rustc_hir_pretty      rustc_parse           rustc_trait_selection
rustc_builtin_macros  rustc_incremental     rustc_parse_format    rustc_traits
rustc_codegen_llvm    rustc_index           rustc_passes          rustc_ty
rustc_codegen_ssa     rustc_infer           rustc_plugin_impl     rustc_typeck
rustc_data_structures rustc_interface       rustc_privacy
rustc_driver          rustc_lexer           rustc_query_system

However, there's no Cargo.toml in the root so the approach I implemented here wouldn't work. If we included a Cargo.toml in the root, then cargo metadata would work and we could use this PR seamlessly.

[matklad]

bors r+

Thanks @xldenis

[bors]

Build succeeded:

• Rust (ubuntu-latest)
• TypeScript (windows-latest)
• Rust (windows-latest)
• TypeScript (ubuntu-latest)
• Rust (macos-latest)

[RalfJung]

However, there's no Cargo.toml in the root so the approach I implemented here wouldn't work. If we included a Cargo.toml in the root, then cargo metadata would work and we could use this PR seamlessly.

The root Cargo.toml should be in rust-src.

[xldenis]

There's a Cargo.lock in rustc-src/rust is that what you mean?

[RalfJung]

Oh right possibly I only added Cargo.lock (xargo needs that) and not Cargo.toml.