rustdoc: Don't load all extern crates unconditionally #83738
Conversation
jyn514
added
T-rustdoc
This comment has been minimized.
This comment has been minimized.
rust-highfive
added
the
S-waiting-on-review
jyn514
force-pushed
the
only-load-some-crates
branch
from
e10e1dc
to
4b0178b
Compare
This comment has been minimized.
This comment has been minimized.
jyn514
force-pushed
the
only-load-some-crates
branch
from
4b0178b
to
36d6800
Compare
This comment has been minimized.
This comment has been minimized.
jyn514
force-pushed
the
only-load-some-crates
branch
from
36d6800
to
3d0961e
Compare
| Err(_) => continue, | ||
| }; | ||
| self.resolver.borrow_mut().access(|resolver| { | ||
| let _ = resolver.resolve_str_path_error( |
There was a problem hiding this comment.
If you save the resolution results for later instead of throwing them away, then you'll no longer need the resolver later on.
That's basically the idea behind fixing the resolver cloning/freezing issue.
Yes, could you create the issue? |
petrochenkov
added
S-waiting-on-author
jyn514
force-pushed
the
only-load-some-crates
branch
from
3d0961e
to
cc69c65
Compare
Sure thing, I opened #83761. I left a question about the approach, but it's not related to this PR. |
jyn514
added
S-waiting-on-review
|
@bors r+ |
|
📌 Commit cc69c65 has been approved by |
bors
added
S-waiting-on-bors
…rochenkov rustdoc: Don't load all extern crates unconditionally Instead, only load the crates that are linked to with intra-doc links. This doesn't help very much with any of rustdoc's fundamental issues with freezing the resolver, but it at least fixes a stable-to-stable regression, and makes the crate loading model somewhat more consistent with rustc's. I tested and it unfortunately does not help at all with rust-lang#82496. Closes rust-lang#68427. Let me know if you want me to open a separate issue for not freezing the resolver. r? `@petrochenkov` cc `@eddyb` `@ollie27`
…rochenkov rustdoc: Don't load all extern crates unconditionally Instead, only load the crates that are linked to with intra-doc links. This doesn't help very much with any of rustdoc's fundamental issues with freezing the resolver, but it at least fixes a stable-to-stable regression, and makes the crate loading model somewhat more consistent with rustc's. I tested and it unfortunately does not help at all with rust-lang#82496. Closes rust-lang#68427. Let me know if you want me to open a separate issue for not freezing the resolver. r? ``@petrochenkov`` cc ``@eddyb`` ``@ollie27``
|
Ok, I added that. Not sure how to test it worked from linux. I also had to add |
|
@bors r+ |
|
📌 Commit e4244e3 has been approved by |
bors
added
S-waiting-on-bors
|
☀️ Test successful - checks-actions |
|
Somehow this works good for cargo-doc, but not for other: |
|
@klensy this is a bug fix - it doesn't really matter what the performance is as long as it's not a giant regression. Cargo is probably lower because it has a lot of |
|
@jyn514 No, i mean, look at |
|
I don't understand what you're trying to tell me. On the "detailed self profile results" page I see no change in time, which is what I'd expect from this PR. If there's an inconsistency between that and instructions that sounds like an issue with perf.rlo, I think there's some ongoing work to measure instructions for self-profile. |
|
Those runs suggest that there is some difference in the documented code for cargo and clap-rs, and this particular PR sees a larger improvement in terms of less queries being run on the cargo benchmark (compared to the other ones you've cited). That's not really a problem, it's pretty intentional that our benchmarks differ in what they look like to the compiler (otherwise there'd be no point in having two different ones) -- I'm trying to understand what problem or note you're trying to make :) |
…crate-load, r=jyn514 Revert "Don't load all extern crates unconditionally" Fixes rust-lang#84738. This reverts rust-lang#83738. For the "smart" load of external crates, we need to be able to access their items in order to check their doc comments, which seems, if not impossible, quite complicated using only the AST. For some context, I first tried to extend the `IntraLinkCrateLoader` visitor by adding `visit_foreign_item`. Unfortunately, it never enters into this call, so definitely not the right place... I then added `visit_use_tree` to then check all the imports outside with something like this: ```rust let mut loader = crate::passes::collect_intra_doc_links::IntraLinkCrateLoader::new(resolver); ast::visit::walk_crate(&mut loader, krate); let mut items = Vec::new(); for import in &loader.imports_to_check { if let Some(item) = krate.items.iter().find(|i| i.id == *import) { items.push(item); } } for item in items { ast::visit::walk_item(&mut item); for attr in &item.attrs { loader.check_attribute(attr); } } ``` This was, of course, a failure. We find the items without problems, but we still can't go into the external crate to check its items' attributes. Finally, `@jyn514` suggested to look into the [`CrateLoader`](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_metadata/creader/struct.CrateLoader.html), but it only seems to provide metadata (I went through [`CStore`](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_metadata/creader/struct.CStore.html) and [`CrateMetadata`](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_metadata/rmeta/decoder/struct.CrateMetadata.html)). I think we are too limited here (with AST only) to be able to determine the crates we actually need to import, but it's very likely that I missed something. Maybe `@petrochenkov` or `@Aaron1011` have an idea? So until we find a way to make it work completely, we need to revert it to fix the ICE. Once merged, we'll need to re-open rust-lang#68427. r? `@jyn514`
Reland rust-lang#83738: "rustdoc: Don't load all extern crates unconditionally" I hopefully found all the bugs 🤞 time for a take two. See the last commit for details on what went wrong before. r? `@petrochenkov` (but feel free to reassign to Guillaume if you don't have time.) Closes rust-lang#68427. Includes a fix for rust-lang#84738.
…arth Rollup of 11 pull requests Successful merges: - rust-lang#87832 (Fix debugger stepping behavior with `match` expressions) - rust-lang#88123 (Make spans for tuple patterns in E0023 more precise) - rust-lang#88215 (Reland rust-lang#83738: "rustdoc: Don't load all extern crates unconditionally") - rust-lang#88216 (Don't stabilize creation of TryReserveError instances) - rust-lang#88270 (Handle type ascription type ops in NLL HRTB diagnostics) - rust-lang#88289 (Fixes for LLVM change 0f45c16) - rust-lang#88320 (type_implements_trait consider obligation failure on overflow) - rust-lang#88332 (Add argument types tait tests) - rust-lang#88340 (Add `c_size_t` and `c_ssize_t` to `std::os::raw`.) - rust-lang#88346 (Revert "Add type of a let tait test impl trait straight in let") - rust-lang#88348 (Add field types tait tests) Failed merges: r? `@ghost` `@rustbot` modify labels: rollup
…trochenkov rustdoc: Go back to loading all external crates unconditionally This *continues* to cause regressions. This code will be unnecessary once access to the resolver happens fully before creating the tyctxt (rust-lang#83761), so load all crates unconditionally for now. To minimize churn, this leaves in the code for loading crates selectively. "Fixes" rust-lang#84738. Previously: rust-lang#83738, rust-lang#85749, rust-lang#88215 r? `@petrochenkov` cc `@camelid` (this should fix the "index out of bounds" error you had while looking up `crate_name`).
…ochenkov rustdoc: Go back to loading all external crates unconditionally This *continues* to cause regressions. This code will be unnecessary once access to the resolver happens fully before creating the tyctxt (rust-lang#83761), so load all crates unconditionally for now. To minimize churn, this leaves in the code for loading crates selectively. "Fixes" rust-lang#84738. Previously: rust-lang#83738, rust-lang#85749, rust-lang#88215 r? `@petrochenkov` cc `@camelid` (this should fix the "index out of bounds" error you had while looking up `crate_name`).
Instead, only load the crates that are linked to with intra-doc links.
This doesn't help very much with any of rustdoc's fundamental issues
with freezing the resolver, but it at least fixes a stable-to-stable
regression, and makes the crate loading model somewhat more consistent
with rustc's. I tested and it unfortunately does not help at all with #82496.
Closes #68427. Opened #83761 to track the more fundamental issues.
r? @petrochenkov cc @eddyb @ollie27