Wrong tokenization of doc comments in proc_macro

[dtolnay]

In #49545 I believe we want /// doc to become doc = " doc", not "/// doc".

You can see this handled correctly by macro_rules:

macro_rules! tokens {
    (#[doc = $tt:tt]) => {
        println!("{:?}", $tt);
    }
}

fn main() {
    tokens! {
        /// doc
    }
}

" doc"

The incorrect behavior in proc_macro:

#!/bin/sh

cargo new --lib repro_macro
cargo new --lib repro

echo >repro_macro/src/lib.rs '
#![feature(proc_macro)]

extern crate proc_macro;
use proc_macro::{TokenStream, TokenNode};

#[proc_macro]
pub fn repro(_input: TokenStream) -> TokenStream {
    let mut tts = "/// doc\n".parse::<TokenStream>().unwrap().into_iter();
    println!("{}", tts.next().unwrap());
    match tts.next().unwrap().kind {
        TokenNode::Group(_, tts) => {
            let mut tts = tts.into_iter();
            println!("{}", tts.next().unwrap());
            println!("{}", tts.next().unwrap());
            println!("{}", tts.next().unwrap());
        }
        _ => unimplemented!(),
    }
    TokenStream::empty()
}
'

echo >>repro_macro/Cargo.toml '
[lib]
proc-macro = true
'

echo >repro/src/lib.rs '
#![feature(proc_macro)]

extern crate repro_macro;
repro_macro::repro!();
'

echo >>repro/Cargo.toml '
repro_macro = { path = "../repro_macro" }
'

cargo build --manifest-path repro/Cargo.toml

#
doc
=
"/// doc"

@alexcrichton

[alexcrichton]

One interesting consequence of something like this (which makes sense in retrospect) is that I think syn can no longer have an is_sugared_doc_comment field, right? In that syn can't actually differentiate based on the token stream between literal comments like /// and those with #[doc]. I don't think that's too much of an issue though?

In any case I've added a fix for this to #49597

[dtolnay]

Yes, in fact I thought is_sugared_doc was already useless -- dtolnay/syn#389. We can remove it.