foreign_modules query hash table lookups

[rylev]

When compiling a large monolithic crate we're seeing huge times in the foreign_modules query due to repeated iteration over foreign modules (in order to find a module by its id). This implements hash table lookups so that which massively reduces time spent in that query in this particular case. We'll need to see if the overhead of creating the hash table has a negative impact on performance in more normal compilation scenarios.

I'm working with @wesleywiser on this.

[rust-highfive]

r? @petrochenkov

(rust_highfive has picked a reviewer for you, use r? to override)

[Mark-Simulacrum]

@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion

[bors]

⌛ Trying commit 69dc981 with merge d0a0399ed1260c2db19783ec7bc6f5bd5b750138...

[bors]

☀️ Try build successful - checks-actions
Build commit: d0a0399ed1260c2db19783ec7bc6f5bd5b750138 (d0a0399ed1260c2db19783ec7bc6f5bd5b750138)

[rust-timer]

Queued d0a0399ed1260c2db19783ec7bc6f5bd5b750138 with parent 2a71e45, future comparison URL.

[rust-timer]

Finished benchmarking try commit (d0a0399ed1260c2db19783ec7bc6f5bd5b750138): comparison url.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. Please note that if the perf results are neutral, you should likely undo the rollup=never given below by specifying rollup- to bors.

Importantly, though, if the results of this run are non-neutral do not roll this PR up -- it will mask other regressions or improvements in the roll up.

@bors rollup=never

[Mark-Simulacrum]

@rylev judging by those results, perf doesn't have a test case similar to the one you are benchmarking on. Would you be up for adding that? Feel free to ping me on #t-compiler/performance in Zulip and I can help guide you on how to do so.

(I guess the alternative is that this has no effect on compile times, but seems like that's likely not true?)

[wesleywiser]

r? @wesleywiser

Waiting on rust-lang/rustc-perf#789 so we can do a perf run showing the improvement.

[rylev]

@wesleywiser rust-lang/rustc-perf#789 has been merged so we should be able to get perf numbers on this now.

[wesleywiser]

Awesome!

@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion

[wesleywiser]

Excellent!

@bors r+

[bors]

📌 Commit 81444b2 has been approved by wesleywiser

[bors]

⌛ Testing commit 81444b2 with merge 437651c85d101488f3d14d4e581b5d85ae725cda...

[bors]

💔 Test failed - checks-actions

[wesleywiser]

@bors retry

[bors]

⌛ Testing commit 81444b2 with merge d4264061164e7615024245a27e33126f4b2421ea...

[bors]

💔 Test failed - checks-actions

[wesleywiser]

@bors retry

[bors]

⌛ Testing commit 81444b2 with merge 7b5a9e9...

[bors]

☀️ Test successful - checks-actions
Approved by: wesleywiser
Pushing 7b5a9e9 to master...

[Mark-Simulacrum]

As expected, a 95% win on instruction counts for the newly added externs stress test (neutral otherwise); essentially 1-1 matched by the 90% wall time reduction for the same crate.

[rylev]

Now that this is on nightly I tested this on the code base which prompted me to investigate this issue. On current stable (1.47.0), the crate compiles in 2min31s and on nightly (f2bbdd0 2020-11-04) it takes 58s - a more than 60% speed up 🎉!