Adding #[meta] to an item in a TokenStream

[ratijas]

There is an example in the quote! macro documentation which states that documentation can be added only with #[doc = #msg] syntax. While this is true, I can't seem to be able to add the documentation to an arbitrary item represented as an input TokenStream in some cases.

My use case is building a custom proc macro (proc_macro_attribute) which only adds documentation to an item in question. Like this:

#[proc_macro_attribute]
pub fn add_doc(args: TokenStream, input: TokenStream) -> TokenStream {
    let msg = "Generate custom docs...";
    // quote! accepts only proc_macro2
    let input = proc_macro2::TokenStream::from(input);

    quote!(
        #[doc = #msg]
        #input
    ).into()
}

However it works for only for top-level module items: structs, functions, types — they all work fine. But on nested impl-methods I get this weird error message:

error: expected one of `async`, `const`, `crate`, `default`, `extern`, `fn`, `pub`, `type`, or `unsafe`, found `pub fn capacity(&self) -> i32 { 0 }`
  --> doc_comment_test/src/lib.rs:11:5
   |
11 |     #[add_doc(...)]
   |     ^^^^^^^^^^^^^^^

As you could guess, original method under macro looked like this:

impl MyWrapper {
    #[add_doc(...)]
    pub fn capacity(&self) -> i32 { 0 }
}

Also, note that I do not want to parse input stream into anything meaningful, I only want to proxy-pass it further with all original tokens' spans etc.

[ratijas]

Pretty-printed output proc_macro2::TokenStream, spans manually removed

TokenStream [
    Punct {
        ch: '#',
        spacing: Alone,
    },
    Group {
        delimiter: Bracket,
        stream: TokenStream [
            Ident {
                ident: "doc",
            },
            Punct {
                ch: '=',
                spacing: Alone,
            },
            Literal { lit: Lit { kind: Str, symbol: "Generate custom docs...", suffix: None }, .. },
        ],
    },
    Ident {
        ident: "pub",
    },
    Ident {
        ident: "fn",
    },
    Ident {
        ident: "capacity",
    },
    Group {
        delimiter: Parenthesis,
        stream: TokenStream [
            Punct {
                ch: '&',
                spacing: Alone,
            },
            Ident {
                ident: "self",
            },
        ],
    },
    Punct {
        ch: '-',
        spacing: Joint,
    },
    Punct {
        ch: '>',
        spacing: Alone,
    },
    Ident {
        ident: "i32",
    },
    Group {
        delimiter: Brace,
        stream: TokenStream [
            Literal { lit: Lit { kind: Integer, symbol: "0", suffix: None }, .. },
        ],
    },
]

After converting back .into() compiler's proc_macro::TokenStream, stream remains equivalent to the one above. No extra groups are produced etc.

Additional test cases:

Pure proxy-pass

Without adding doc attribute, pure proxy-passing input to quote! works fine for any item:

    quote!(
        #input
    ).into()

Proxy-pass with other meta attributes

Surprisingly, works. Proc macro code as above; function code here:

impl MyWrapper {
    #[allow(unused)]
    pub fn capacity(&self) -> i32 { 0 }
}

TokenStream up to "pub fn"

TokenStream [
    Punct {
        ch: '#',
        spacing: Alone,
    },
    Group {
        delimiter: Bracket,
        stream: TokenStream [
            Ident {
                ident: "allow",
            },
            Group {
                delimiter: Parenthesis,
                stream: TokenStream [
                    Ident {
                        ident: "attributesunused",
                    },
                ],
            },
        ],
    },
    Ident {
        ident: "pub",
    },
    Ident {
        ident: "fn",
    },
    ..
]

Looks pretty-much the same as the first example, but magically works.

Adding other meta attributes

Tried to add other attribute than doc. Failed on #[allow(unused_imports)] as well.

Still can't see the difference between generated (from quote) and supplied (from input stream) attributes in the output TokenStream.

[dtolnay]

This is a compiler bug that was fixed in 1.43.0.

[ratijas]

@dtolnay
Good to hear that! In the meantime, any workarounds for crates targeting older compilers? (Other than to_string round trips, that is)

[dtolnay]

Yes, let input = input.into_iter().collect::<TokenStream>() at the top of the macro should fix it.

[ratijas]

Perfecto! Thanks a million, that was extremely helpful of you.