Rework rustc output file tracking.

[ehuss]

This is some cleanup around how Cargo computes the outputs from rustc, and changes how cargo clean -p works. This also includes some minor observable differences detailed below.

clean -p changes

Previously cargo clean -p would build the unit graph and attempt to guess what all the filename hashes are. This has several drawbacks. It incorrectly guesses the hashes in many cases (such as different features). It also tends to cause bugs because it passes every permutation of every setting into Cargo's internals, which may not be prepared to accept unsupported combinations like "test a build-script".

This new implementation uses a strategy of querying rustc for the filename prefix/suffix, and then deletes the files using globs.

cargo clean -p should now be more complete in deleting a package's artifacts, for example:

• Now deletes incremental files.
• Deletes dep-info files (both rustc and cargo).
• Handles changes in profiles, features, (anything in the hash).
• Covers a few more files (for example, dSYM in some cases, etc.). Should delete almost all files for most targets.

Internal changes

There are a bunch of internal changes to make Cargo do a better job of tracking the outputs from rustc, and to make the code easier to understand:

• The list of output files is now solely computed in TargetInfo. The files which are uplifted are solely computed in CompilationFiles::uplift_to. Previously both responsibilities were awkwardly spread in both locations.
• Whether or not a file should have a hyphen or underscore is now determined in one place (FileType::should_replace_hyphens).
• Added CrateType to replace LibKind, to avoid usage of strings, and to use the same structure for all of the target kinds.
• Added FileFlavor::Rmeta to be more explicit as to which output files are ".rmeta". (Previously the Linkable{rmeta} flag was specific for pipelining, and rmeta was false for things like cargo check, which was a bit hard to deal with.)
• Removed hyphen/underscore renaming in rustc. This was mostly unused, because it did not consider hashes in the filename, so it only applied to binaries without hashes, which is essentially just wasm32 and msvc. This hyphen/underscore translation still happens during "uplift".
• Changed it so that Metadata is always computed for every unit. The logic of whether or not something should use it is moved to should_use_metadata. I didn't realize that multiple units were sharing the same fingerprint directory (when they don't have a hash), which caused some bugs (like bad output caching).

Behavioral changes

Cargo now does more complete tracking of the files generated by rustc (and the linker). This means that some files will now be uplifted that previously weren't. It also means they will show up in the artifact JSON notifications. The newly tracked files are:

• .exp export files for Windows MSVC dynamic libraries. I don't know if these are actually useful (nobody has asked for them AFAIK). Presumably the linker creates them for some reason. Note that lld doesn't generate these files, this is only link.exe.
• Proc-macros on Windows track import/export files.
• All targets (like tests, etc.) that generate separate debug files (pdb/dSYM) are tracked.
• Added .map files for wasm32-unknown-emscripten.

Some other subtle changes:

• A binary example with a hyphen on Windows MSVC will now show up as examples/foo_bar.exe and examples/foo-bar.exe. Previously Cargo would just rename it to contain the hyphen. This is a consequence of simplifying the code, and I was reluctant to add a special case for this very narrow situation.
• Example libs now follow the same rules for hyphen/underscore translation as normal libs (they will now use underscores).
• Fingerprint changes:
  • Fingerprint files no longer have a hash in them. Their parent directory already contained the hash.
  • The fingerprint filename now uses hyphens instead of converting to underscores.
  • The fingerprint directory is now separated even if the unit doesn't use Metadata, to fix a caching bug.
• macOS: dSYM is uplifted for all dynamic libraries (dylib/cdylib/proc-macro) and for build-script-build (in case someone wants to debug a build script?).

Notes

• I suspect that the implementation of clean -p may change again in the future. If and when Cargo implements some kind of on-disk database that tracks artifacts (for the purpose of garbage collection), then cargo clean -p can be rewritten to use that mechanism if appropriate.
• The build_script_build incremental directory isn't deleted because Cargo doesn't know which one belongs to which package. I'm uncertain if that's reasonably fixable. The only option I've thought of is to place each package's incremental output in a separate directory.
• Should Cargo use globset to handle non-utf-8 filenames? I suspect that Cargo's support for non-utf-8 names is pretty poor, so I'm uncertain how important that is.

Closes #8149
Closes #6937
Closes #5788
Closes #5375
Closes #3530

[rust-highfive]

r? @Eh2406

(rust_highfive has picked a reviewer for you, use r? to override)

[alexcrichton]

Thanks for pushing on this @ehuss! I only sort of lightly skimmed the internal refactorings. It all sounds fine at a high-level and everything internally looked "correct enough" that it should either be covered by tests or we'll figure it out on nightly pretty soon anyway.

I'm left with really only one minor comment, but otherwise this I think is a great improvement to cargo clean -p, so r=me when ready

[alexcrichton]

FWIW the get_many here starts a new download session each time, so maybe these could all be collected from the opts.spec.iter() loop and one get_many could be used?

[ehuss]

👍

[ehuss]

I pushed an update that reverted the "always compute Metadata" change. It doesn't work for targets like MSVC in the following scenario:

1. cargo build --bin foo --features f1 <- generates deps/foo.exe
2. cargo build --bin foo --features f2 <- Overwrites deps/foo.exe
3. cargo build --bin foo --features f1 <- If there is a separate fingerprint, this is "fresh", and doesn't regenerate the exe when it should.

In the case where there is no metadata hash in the filename, the units must reuse the same fingerprint file.

I changed the fix for the compiler output cache by changing the output filename to use the same format as fingerprints, like output-bin-foo instead of just output.

[alexcrichton]

@bors: r+

[bors]

📌 Commit 7438770 has been approved by alexcrichton

[bors]

⌛ Testing commit 7438770 with merge c719a05...

[bors]

☀️ Test successful - checks-azure
Approved by: alexcrichton
Pushing c719a05 to master...