Flatten all -L search paths into a single list of files

[Kobzol]

This is a pre-emptive optimization for the new Cargo build dir layout.

Before, Cargo usually used to pass rustc a single -L path to a directory containing potentially a large number of .rlib or other dependency paths. Rustc is optimized for this, and does a preprocessing step, where it preloads all files from this directory into a sorted vector, in which it then performs binary search to look for files having a given prefix and postfix.

With the new build dir layout, this changes, and Cargo will pass potentially many (even hundreds, if your crate has many dependencies) -L directories, where each directory will usually have only 1-2 files. This is not ideal for the current rustc search implementation, because the old preprocessing isn't really worth it when the -L directory typically only has a single file, and because rustc will do a lot of work per each -L directory in the lookup phase, including some quadratic (O(n^2)) work.

With the large-workspace stress test from rustc-perf, where the final binary has ~1000 total and ~500 direct dependencies:

• With the old build dir layout, rustc does 4274 calls to FilesIndex::query
• With the new build dir layout, it does 1938314 calls to FilesIndex::query. Not ideal :)

This PR modifies the logic so that instead of going through all -L directories one by one when we are looking for a specific dependency, we pre-gather all files from all passed -L directories at the start, and sort them all together. Then, in the lookup phase, instead of performing <number of -L dirs> * 4 (.rmeta, .rlib, etc.) searched, we only perform the 4 searches across all files. This gets rid of the O(n^2) behavior.

I thought about pre-filtering the files further, e.g. based on their kind, but that is not worth it in usual situations, where ~all of the deps will be .rlibs or .rmetas anyway. What might work is pre-filtering based on their actual names (we strip known prefixes from them and then search based on the names alone, in a hashmap or something?), but that can be a follow-up.

Below are perf. runs with rustc/cargo using the new build dir layout:

• Without this PR (this is what we would likely see in rustc-perf once we merge the Cargo submodule update that enables the new build dir layout by default)
• With this PR
• Diff

Note that this is not intended to be merged as-is, it is not cleaned up and it is not very relevant until Cargo switches to the new build dir layout anyway. But I wanted to get a vibe-check if you think that something like this might make sense, e.g. from @petrochenkov. The main change is in locator.rs, along with the search_paths2 function.

[rustbot]

r? @mati865

rustbot has assigned @mati865.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

• Owners of files modified in this PR: compiler
• compiler expanded to 75 candidates
• Random selection from 21 candidates

[Kobzol]

Sorry, meant to open this as a draft, but missclicked.

r? @ghost

[rust-log-analyzer]

The job pr-check-2 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)