Store tokens alongside more AST expressions #70091
Conversation
This fails to bootstrap, since we completely ignore `#[cfg]` and other attributes/macros
This comment has been minimized.
This comment has been minimized.
rust-highfive
added
the
S-waiting-on-review
|
The job Click to expand the log.I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact |
|
@bors try |
[WIP] Store tokens alongside more AST expressions See #43081 (comment) This PR calls `collect_tokens` during the parsing of more AST nodes, and stores the captured tokens in the parsed AST structs. These tokens are then used in `nt_to_tokenstream` to avoid needing to stringify AST nodes. Since this implementation completely ignores attributes, it will probably explode when given any kind of complicated input. This PR is intended mainly to estimate the performance impact of collecting and storing more tokens - a correct implementation will most likely have to do more work than this. I've only implemented token collecting for a few types of expressions - the current expression parsing implementation makes it difficult to get the proper tokens for every expression. Nevertheless, this is able to bootstrap libstd, and generate better error messages for a simple proc-macro example: (https://github.com/Aaron1011/for-await-test) ```rust #![feature(stmt_expr_attributes, proc_macro_hygiene)] use futures::stream::Stream; use futures_async_stream::for_await; async fn collect(stream: impl Stream<Item = i32>) -> Vec<i32> { let mut vec = Vec::new(); #[for_await] for value in stream.foo() { vec.push(value); } vec } fn main() { println!("Hello, world!"); } ``` on the latest nightly: ``` error[E0599]: no method named `foo` found for type parameter `impl Stream<Item = i32>` in the current scope --> src/main.rs:7:5 | 7 | #[for_await] | ^^^^^^^^^^^^ method not found in `impl Stream<Item = i32>` error: aborting due to previous error ``` with this PR: ``` error[E0599]: no method named `foo` found for type parameter `impl Stream<Item = i32>` in the current scope --> src/main.rs:8:25 | 8 | for value in stream.foo() { | ^^^ method not found in `impl Stream<Item = i32>` ```
petrochenkov
added
S-waiting-on-perf
|
☀️ Try build successful - checks-azure |
|
@rust-timer build e58cfd9 |
|
Queued e58cfd9 with parent 6724d58, future comparison URL. |
|
Finished benchmarking try commit e58cfd9, comparison URL. |
|
Those results are pretty disappointing. However, the worst regressions seem to be on 'weird' crates - e.g. |
|
@petrochenkov: What do you think about modifying |
petrochenkov
added
S-waiting-on-review
So, I looked at the uses of So, we need to collect tokens in two cases:
In other cases the collection shouldn't be required. |
|
@Aaron1011 |
petrochenkov
added
S-waiting-on-author
@petrochenkov: How does this interact with custom inner attributes? We need to already be collecting tokens by the time we parse them. |
petrochenkov
added
S-waiting-on-review
|
@Aaron1011 Btw, you slicing idea is one of the original reasons for |
Some history of |
Good catch. However, the main question is - what we are aiming for here, a practical improvement, or a proper holistic solution?
A proper solution is a big design task, which I'm certainly not ready to drive now. In the meantime, we can improve the situation for common cases in ways that don't regress compile times. This largely means ignoring inner macro attributes. |
petrochenkov
added
S-waiting-on-author
|
(To clarify, I'm not against making improvements to treatment of inner attributes in a separate PR with a separate perf run, as one more incremental improvement.) |
Dylan-DPC-zz
changed the title
|
r? @oli-obk |
|
This is waiting for author |
|
Could close due to inactivity as well, it's been ~1.5 months. |
…r=petrochenkov Store tokens inside `ast::Expr` This is a smaller version of rust-lang#70091. We now store captured tokens inside `ast::Expr`, which allows us to avoid some reparsing in `nt_to_tokenstream`. To try to mitigate the performance impact, we only collect tokens when we've seen an outer attribute. This makes progress towards solving rust-lang#43081. There are still many things left to do: * Collect tokens for other AST items. * Come up with a way to handle inner attributes (we need to be collecting tokens by the time we encounter them) * Avoid re-parsing when a `#[cfg]` attr is used. However, this is enough to fix spans for a simple example, which I've included as a test case.
See #43081 (comment)
This PR calls
collect_tokensduring the parsing of more AST nodes, and stores the captured tokens in the parsed AST structs. These tokens are then used innt_to_tokenstreamto avoid needing to stringify AST nodes.Since this implementation completely ignores attributes, it will probably explode when given any kind of complicated input. This PR is intended mainly to estimate the performance impact of collecting and storing more tokens - a correct implementation will most likely have to do more work than this.
I've only implemented token collecting for a few types of expressions - the current expression parsing implementation makes it difficult to get the proper tokens for every expression.
Nevertheless, this is able to bootstrap libstd, and generate better error messages for a simple proc-macro example: (https://github.com/Aaron1011/for-await-test)
on the latest nightly:
with this PR: