Start removing `Nonterminal` #114647

This has no effect right now because invisible delimiters aren't produced by `TokenCursor`, but it will make more sense once invisible delimiter sources are introduced.

It's not used meaningfully yet, but will be needed to get rid of interpolated tokens.

Invisible delimiters pretty-print as empty strings, and changing that can break some proc macros. But error messages saying "expected identifer, found ``" are bad. So this commit adds support for invisible open delimiters in `TokenDescription` so they print as "invisible open delimiter" in error messages, instead of "``". It's not used meaningfully yet, but will be needed to get rid of interpolated tokens.

By using `token_descr`, as is done for many other errors, we can get slightly better descriptions in error messages, e.g. "macro expansion ignores token `let` and any following" becomes "macro expansion ignores keyword `let` and any tokens following". This will be more important once invisible delimiters start being mentioned in error messages.

This will make things nicer as more variants are added to `ParseNtResult`.

Token sequences delimited with these invisible delimiters will eventually replace `Token::Interpolated` for declarative macro expansions. There are no uses yet, but this commit does modify `nonterminal_may_begin_with` so that every `Interpolated` test is now mirrored with a `InvisibleSource::MetaVar` test. This means that `Nt*` cases can now be removed from that function one at a time without any other changes required.

We now use invisible delimiters for expanded `vis` fragments, instead of `Token::Interpolated`. This slightly changes some pretty-printed output in tests, but not in any significant way.

Notes about tests: - tests/ui/parser/macro/trait-object-macro-matcher.rs: the syntax error is duplicated, because it occurs now when parsing the decl macro input, and also when parsing the expanded decl macro. But this won't show up for normal users due to error de-duplication. - The output of stringify! is uglier, due to `print_tts` producing uglier output than AST pretty-printing.

This requires adding `stream_pretty_printing_compatibility_hack`, which will replace `nt_pretty_printing_compatibility_hack` once `NtStmt` is removed. Notable test changes: - `tests/ui/macros/nonterminal-matching.rs` now passes. Removal of `Token::Interpolated` will remove the "captured metavariables except for `:tt`, `:ident` and `:lifetime` cannot be compared to other tokens" restriction. - `tests/ui/proc-macro/expand-to-derive.rs`: the diff looks large but the only difference is the insertion of a single invisible-delimited group. I hope this doesn't cause any new `ProceduralMasquerade`-type problems.

This means `nt_pretty_printing_compatibility_hack` can also be removed. The handling of statements in `transcribe` is slightly different to other nonterminal kinds, due to the lack of `from_ast` implementation for empty statements.

Note: there was an existing code path involving `Interpolated` in `MetaItem::from_tokens` that was dead. This commit transfers that to the new form, but puts an `unreachable!` call inside it.

[xt] move from_token Getting a test failure here: ``` Building tool error_index_generator (stage1 -> stage2, x86_64-unknown-linux-gnu) Compiling cfg-if v1.0.0 ... Compiling mdbook v0.4.31 error: internal compiler error: the following error was constructed but not emitted error: unexpected token: keyword `self` --> /home/njn/.cargo/registry/src/index.crates.io-6f17d22bba15001f/mdbook-0.4.31/src/book/summary.rs:275:31 | 275 | bail!(self.parse_error("Suffix chapters cannot be followed by a list")); | ^^^^ thread 'rustc' panicked at compiler/rustc_errors/src/diagnostic_builder.rs:775:21: error was constructed but not emitted ```

Remnants of `Token::Interpolated` still exist in the form of the new `token::IdentMv` and `token::LifetimeMv` tokens. I did them like that because there's a lot of code that assumes an interpolated ident/lifetime fits in a single token, and changing all that code to work with invisible delimiters would have been a pain. (Maybe it could be done in a follow-up.) Fully kills off the "captured metavariables except for `:tt`, `:ident` and `:lifetime` cannot be compared to other tokens" restriction.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Start removing `Nonterminal` #114647

Start removing `Nonterminal` #114647

Commits on Aug 22, 2023

Start removing Nonterminal #114647

Start removing Nonterminal #114647

Commits on Aug 22, 2023

Start removing `Nonterminal` #114647

Start removing `Nonterminal` #114647