rustfmt: Check that a token can begin a nonterminal kind before parsing it as a macro arg

[compiler-errors]

r? @ytmimi and/or @calebcartwright
cc @fmease

I'm putting this on r-l/rust since it should fix the nightly rustfmt version. If you don't care about having this regression until the next rustfmt->rust sync, then I can move that PR over to r-l/rustfmt.

---

Any idea why the formatting would have changed [from #119099]?

Copied over explanation:

This has to do with the weirdness of the way that parse_macro_arg works. Unlike parsing nonterminal args in a macro-by-example, it eagerly tries, for example, to parse a type without checking that the beginning token may begin a type:

https://github.com/rust-lang/rustfmt/blob/bf967319e258acb9b1648a952bba52665eceaf52/src/parse/macros/mod.rs#L54

Contrast this to the nonterminal parsing code, which first checks that the nonterminal may begin with a given token:

rust/compiler/rustc_parse/src/parser/nonterminal.rs

Line 47 in ef71f10

NonterminalKind::Ty => token.can_begin_type(),

In #119099, @fmease implemented a change so that const Tr would be parsed as dyn const Tr (a trait object to a const trait) in edition 2015.

This is okay for the purposes of macros, because he explicitly made sure that const did not get added to the list of tokens that may begin a :ty nonterminal kind: #119099 (comment)

However, since rustfmt is not so careful about eagerly parsing macro args before checking that they're legal in macro position, this changed the way that the string of tokens is being parsed into macro args.

[rustbot]

Some changes occurred in src/tools/rustfmt

cc @rust-lang/rustfmt

[ytmimi]

Could we also add a test case for rust-lang/rustfmt#6035

[compiler-errors]

I did have a test but I lost it during rebasing 💀

[ytmimi]

I'm personally happy to apply the fix here in r-l/rust given that this is a very rare circumstance and it'll address the nightly issue sooner. @calebcartwright any objections?

Might be better to hold off on merging until Caleb has had a chance to look this over.

[calebcartwright]

Probably the worst part about making the change here in-tree is having to do a bit more manual formatting 😄

Thanks for the quick fix, I'm fine for this to be made in tree

r=me once the minor formatting issue is fixed

[calebcartwright]

and more generally, I think it's fair to say that we've got a decent chunk of code in rustfmt that's likely duplicative of something within rustc and doesn't necessarily need to exist within rustfmt.

I suspect this is mostly a relic from the days when rustfmt consumed the internals from the auto publish crates and the rustfmt team felt the rustc APIs were immutable, but if there's ever any opportunity to drop more code from rustfmt and defer something to a rustc module, I am 100% in favor

[compiler-errors]

@bors r=calebcartwright,ytmimi

[bors]

📌 Commit 5297433 has been approved by calebcartwright,ytmimi

It is now in the queue for this repository.