Start removing `Nonterminal` #114647

nnethercote · 2023-08-09T05:57:23Z

nnethercote · 2023-08-09T06:03:17Z

This is a successor to #96724, where I started doing this with NtExpr and then gave up because it was difficult. It is a necessary predecessor for fixing #67062.

The code is draft quality. So far I have removed NtVis and NtTy, both of which are easier than NtExpr. The pretty printing of vis and ty captures is now much worse. #114571 would fix that, by improving pretty printing of token streams.

It's unclear to me how to best stage this work, whether to do it in pieces, or all in a single huge PR.

cc @petrochenkov

nnethercote · 2023-08-10T07:03:35Z

The latest update removes NtPat.

nnethercote · 2023-08-12T22:37:38Z

The latest update removes NtItem. Doing this made me realize that removing Token::Interpolated/Nonterminal will remove this restriction, because a test about that restriction started passing.

petrochenkov · 2023-08-13T01:42:25Z

The PR is marked as draft, so marking is as waiting on author as well (I'll be busy for the next few days, and won't be able to review anyway).

Once this is ready I'll ask to split all the preparational commits + one of the smaller nonterminals (NtVis) to a separate PR.
@rustbot author

nnethercote · 2023-08-13T06:13:20Z

#114571 should be considered ahead of this one. It improves token stream pretty printing greatly, which helps minimize changes to tests in this PR.

nnethercote · 2023-08-13T11:30:15Z

The latest update removes NtMeta.

nnethercote · 2023-08-14T01:23:34Z

The latest update removes NtPath.

nnethercote · 2023-08-14T02:09:37Z

And now NtBlock.

bors · 2023-08-15T04:14:34Z

☔ The latest upstream changes (presumably #114803) made this pull request unmergeable. Please resolve the merge conflicts.

nnethercote · 2023-08-16T22:56:37Z

The latest updates remove NtExpr, NtLiteral, NtIdent, NtLifetime, Nonterminal, and Token::Interpolated. The commit removing NtExpr and NtLiteral is rough, with lots of njn: comments and a couple of test failures. But it all basically works.

I haven't yet added handling of invisible delimiters from proc macros, which will be required to fix #67062.

@petrochenkov: it might be worth taking a look now, to see what you think of the overall approach.

nnethercote · 2023-08-16T23:25:44Z

That test failure above on the bail! call is a known one from the NtExpr/NtLiteral removal.

I have opened #114915 for all the preliminary cleanup commits.

This will make things nicer as more variants are added to `ParseNtResult`.

Token sequences delimited with these invisible delimiters will eventually replace `Token::Interpolated` for declarative macro expansions. There are no uses yet, but this commit does modify `nonterminal_may_begin_with` so that every `Interpolated` test is now mirrored with a `InvisibleSource::MetaVar` test. This means that `Nt*` cases can now be removed from that function one at a time without any other changes required.

We now use invisible delimiters for expanded `vis` fragments, instead of `Token::Interpolated`. This slightly changes some pretty-printed output in tests, but not in any significant way.

Notes about tests: - tests/ui/parser/macro/trait-object-macro-matcher.rs: the syntax error is duplicated, because it occurs now when parsing the decl macro input, and also when parsing the expanded decl macro. But this won't show up for normal users due to error de-duplication. - The output of stringify! is uglier, due to `print_tts` producing uglier output than AST pretty-printing.

This requires adding `stream_pretty_printing_compatibility_hack`, which will replace `nt_pretty_printing_compatibility_hack` once `NtStmt` is removed. Notable test changes: - `tests/ui/macros/nonterminal-matching.rs` now passes. Removal of `Token::Interpolated` will remove the "captured metavariables except for `:tt`, `:ident` and `:lifetime` cannot be compared to other tokens" restriction. - `tests/ui/proc-macro/expand-to-derive.rs`: the diff looks large but the only difference is the insertion of a single invisible-delimited group. I hope this doesn't cause any new `ProceduralMasquerade`-type problems.

This means `nt_pretty_printing_compatibility_hack` can also be removed. The handling of statements in `transcribe` is slightly different to other nonterminal kinds, due to the lack of `from_ast` implementation for empty statements.

Note: there was an existing code path involving `Interpolated` in `MetaItem::from_tokens` that was dead. This commit transfers that to the new form, but puts an `unreachable!` call inside it.

[xt] move from_token Getting a test failure here: ``` Building tool error_index_generator (stage1 -> stage2, x86_64-unknown-linux-gnu) Compiling cfg-if v1.0.0 ... Compiling mdbook v0.4.31 error: internal compiler error: the following error was constructed but not emitted error: unexpected token: keyword `self` --> /home/njn/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mdbook-0.4.31/src/book/summary.rs:275:31 | 275 | bail!(self.parse_error("Suffix chapters cannot be followed by a list")); | ^^^^ thread 'rustc' panicked at compiler/rustc_errors/src/diagnostic_builder.rs:775:21: error was constructed but not emitted ```

Remnants of `Token::Interpolated` still exist in the form of the new `token::IdentMv` and `token::LifetimeMv` tokens. I did them like that because there's a lot of code that assumes an interpolated ident/lifetime fits in a single token, and changing all that code to work with invisible delimiters would have been a pain. (Maybe it could be done in a follow-up.) Fully kills off the "captured metavariables except for `:tt`, `:ident` and `:lifetime` cannot be compared to other tokens" restriction.

nnethercote · 2023-08-22T02:11:19Z

The last commit partially implements removal of invisible delimiter skipping. This is enough to make the examples in #67062 work, but the problems mentioned above occur. The code in that commit is very rough.

Dylan-DPC · 2024-02-12T18:35:18Z

@nnethercote any updates on this? this has ended up bitrotting heavily

…ors,cjgillot macro_rules: Preserve all metavariable spans in a global side table This PR preserves spans of `tt` metavariables used to pass tokens to declarative macros. Such metavariable spans can then be used in span combination operations like `Span::to` to improve all kinds of diagnostics. Spans of non-`tt` metavariables are currently kept in nonterminal tokens, but the long term plan is remove all nonterminal tokens from rustc parser and rely on the proc macro model with invisible delimiters (rust-lang#114647, rust-lang#67062). In particular, `NtIdent` nonterminal (corresponding to `ident` metavariables) becomes easy to remove when this PR lands (rust-lang#119412 does it). The metavariable spans are kept in a global side table keyed by `Span`s of original tokens. The alternative to the side table is keeping them in `SpanData` instead, but the performance regressions would be large because any spans from tokens passed to declarative macros would stop being inline and would work through span interner instead, and the penalty would be paid even if we never use the metavar span for the given original span. (But also see the comment on `fn maybe_use_metavar_location` describing the map collision issues with the side table approach.) There are also other alternatives - keeping the metavar span in `Token` or `TokenTree`, but associating it with `Span` itsel is the most natural choice because metavar spans are used in span combining operations, and those operations are not necessarily tied to tokens.

nnethercote · 2024-03-01T21:32:00Z

There are still some good ideas in here that I'd like to pursue, so I'll keep it open.

nnethercote · 2024-04-18T23:28:52Z

My next attempt at this is now in #124141.

The extra span is now recorded in the new `TokenKind::InterpolatedIdent` and `TokenKind::InterpolatedLifetime`. These both consist of a single token, and so there's no operator precedence problems with inserting them directly into the token stream. The other way to do this would be to wrap the ident/lifetime in invisible delimiters, but there's a lot of code that assumes an interpolated ident/lifetime fits in a single token, and changing all that code to work with invisible delimiters would have been a pain. (Maybe it could be done in a follow-up.) This change might not seem like much of a win, but it's a first step toward the much bigger and long-desired removal of `Nonterminal` and `TokenKind::Interpolated`. That change is big and complex enough that it's worth doing this piece separately. (Indeed, this commit is based on part of a late commit in rust-lang#114647, a prior attempt at that big and complex change.)

…=<try> Remove `NtIdent` and `NtLifetime` This is one part of the bigger "remove `Nonterminal` and `TokenKind::Interpolated`" change drafted in rust-lang#114647. More details in the individual commit messages. r? `@petrochenkov`

The extra span is now recorded in the new `TokenKind::InterpolatedIdent` and `TokenKind::InterpolatedLifetime`. These both consist of a single token, and so there's no operator precedence problems with inserting them directly into the token stream. The other way to do this would be to wrap the ident/lifetime in invisible delimiters, but there's a lot of code that assumes an interpolated ident/lifetime fits in a single token, and changing all that code to work with invisible delimiters would have been a pain. (Maybe it could be done in a follow-up.) This change might not seem like much of a win, but it's a first step toward the much bigger and long-desired removal of `Nonterminal` and `TokenKind::Interpolated`. That change is big and complex enough that it's worth doing this piece separately. (Indeed, this commit is based on part of a late commit in rust-lang#114647, a prior attempt at that big and complex change.)

The extra span is now recorded in the new `TokenKind::NtIdent` and `TokenKind::NtLifetime`. These both consist of a single token, and so there's no operator precedence problems with inserting them directly into the token stream. The other way to do this would be to wrap the ident/lifetime in invisible delimiters, but there's a lot of code that assumes an interpolated ident/lifetime fits in a single token, and changing all that code to work with invisible delimiters would have been a pain. (Maybe it could be done in a follow-up.) This change might not seem like much of a win, but it's a first step toward the much bigger and long-desired removal of `Nonterminal` and `TokenKind::Interpolated`. That change is big and complex enough that it's worth doing this piece separately. (Indeed, this commit is based on part of a late commit in rust-lang#114647, a prior attempt at that big and complex change.)

…=petrochenkov Remove `NtIdent` and `NtLifetime` This is one part of the bigger "remove `Nonterminal` and `TokenKind::Interpolated`" change drafted in rust-lang#114647. More details in the individual commit messages. r? `@petrochenkov`

rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Aug 9, 2023

nnethercote marked this pull request as draft August 9, 2023 06:03

This comment has been minimized.

Sign in to view

petrochenkov self-assigned this Aug 9, 2023

nnethercote force-pushed the start-rm-Nonterminal branch 3 times, most recently from 86999c2 to 1034bc0 Compare August 10, 2023 07:02

nnethercote force-pushed the start-rm-Nonterminal branch 3 times, most recently from 390563a to be30bb6 Compare August 12, 2023 22:35

rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 13, 2023

nnethercote force-pushed the start-rm-Nonterminal branch from be30bb6 to 3db455d Compare August 13, 2023 11:25

nnethercote force-pushed the start-rm-Nonterminal branch from 3db455d to a93445e Compare August 14, 2023 01:23

nnethercote force-pushed the start-rm-Nonterminal branch from d5df8c4 to 67f4e3c Compare August 16, 2023 22:53

This comment has been minimized.

Sign in to view

nnethercote mentioned this pull request Aug 16, 2023

Nonterminal-related cleanups #114915

Merged

nnethercote mentioned this pull request Aug 16, 2023

Improve print_tts #114571

Merged

nnethercote added 14 commits August 22, 2023 11:02

Streamline NamedMatch.

fcce5dd

This will make things nicer as more variants are added to `ParseNtResult`.

Remove NtVis.

8da7e52

We now use invisible delimiters for expanded `vis` fragments, instead of `Token::Interpolated`. This slightly changes some pretty-printed output in tests, but not in any significant way.

Remove NtPat.

cab935c

Remove NtStmt.

06ff446

This means `nt_pretty_printing_compatibility_hack` can also be removed. The handling of statements in `transcribe` is slightly different to other nonterminal kinds, due to the lack of `from_ast` implementation for empty statements.

Remove NtMeta.

b6e6ee0

Note: there was an existing code path involving `Interpolated` in `MetaItem::from_tokens` that was dead. This commit transfers that to the new form, but puts an `unreachable!` call inside it.

Remove NtPath.

467c76e

Remove NtBlock.

ad011f2

Impl Copy for Token and TokenKind.

8fc2809

[xtk-ui 1 failure] Don't skip invisible delimiters.

848d2d2

nnethercote force-pushed the start-rm-Nonterminal branch from d2664fa to 848d2d2 Compare August 22, 2023 02:10

petrochenkov mentioned this pull request Jan 8, 2024

macro_rules: Preserve all metavariable spans in a global side table #119673

Merged

nnethercote closed this Apr 18, 2024

nnethercote mentioned this pull request Apr 22, 2024

Remove NtIdent and NtLifetime #124256

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Start removing `Nonterminal` #114647

Start removing `Nonterminal` #114647

nnethercote commented Aug 9, 2023

nnethercote commented Aug 9, 2023 •

edited

This comment has been minimized.

nnethercote commented Aug 10, 2023

nnethercote commented Aug 12, 2023

petrochenkov commented Aug 13, 2023

nnethercote commented Aug 13, 2023

nnethercote commented Aug 13, 2023

nnethercote commented Aug 14, 2023

nnethercote commented Aug 14, 2023

bors commented Aug 15, 2023

nnethercote commented Aug 16, 2023

This comment has been minimized.

nnethercote commented Aug 16, 2023

nnethercote commented Aug 22, 2023 •

edited

Dylan-DPC commented Feb 12, 2024

nnethercote commented Mar 1, 2024

nnethercote commented Apr 18, 2024

Start removing Nonterminal #114647

Start removing Nonterminal #114647

Conversation

nnethercote commented Aug 9, 2023

nnethercote commented Aug 9, 2023 • edited

This comment has been minimized.

nnethercote commented Aug 10, 2023

nnethercote commented Aug 12, 2023

petrochenkov commented Aug 13, 2023

nnethercote commented Aug 13, 2023

nnethercote commented Aug 13, 2023

nnethercote commented Aug 14, 2023

nnethercote commented Aug 14, 2023

bors commented Aug 15, 2023

nnethercote commented Aug 16, 2023

This comment has been minimized.

nnethercote commented Aug 16, 2023

nnethercote commented Aug 22, 2023 • edited

Dylan-DPC commented Feb 12, 2024

nnethercote commented Mar 1, 2024

nnethercote commented Apr 18, 2024

Start removing `Nonterminal` #114647

Start removing `Nonterminal` #114647

nnethercote commented Aug 9, 2023 •

edited

nnethercote commented Aug 22, 2023 •

edited