proc_macro types Display impls don’t respect the input layout

[hadronized]

Hey there,

I’m not sure whether it’s the expected behavior of the Display implementations for token types in proc_macro, but the Display implementations lose positional information. I made a blog post about it and release a crate to fix that problem, proc-macro-faithful-display.

I posted that on Discord and @eddyb advised me to open an issue here. So here it is. Not sure whether:

• There’s an actual problem, because perhaps the Display implementors have never meant to provide positional correctness.
• If there’s a problem:
  • Should we assume the current behavior of Display correct and instead provide another type to perform the faithful display (like the .display() function for &Path).
  • Or instead just fix Display.

\o/

[eddyb]

IMO we should, at least, provide some functionality, on Span or elsewhere, to get at that information that is being lost by looking at only the tokens.

cc @alexcrichton @dtolnay

[hadronized]

If you’re interested, here’s my implementation for reference.

[alexcrichton]

This seems fine to me to add an an unstable basis to the proc_macro crate, and it also seems fine by me to improve the default output of Display wherever possible! We just need to be sure to not regress anything as usual!

[eddyb]

I'm a bit worried changing fmt::Display might break various tools that don't expect arbitrary whitespace and/or comments between tokens.

[alexcrichton]

In discussions in the past @dtolnay and I have felt that any changes to Display which produce still-valid-Rust-code is fine to make. Just like Debug in the standard library we don't guarantee an exact output, only that it conforms to a set of guidelines, and for TokenStream it's that it always produces valid Rust code.

In that sense if we update displayed output and parsers break we'd consider it bugs in the parsers rather than the compiler, although we would of course work with the crates if there's enough fallout.

[eddyb]

This is an alternative, I suppose: #55780.