[experiment] Expand macros in inert key-value attributes #67121

petrochenkov · 2019-12-07T15:18:35Z

Example:

#[doc = include_str!("my_doc.md")]
struct S;

This is a general-purpose feature that is supposed to supersede #[doc(include = "my_doc.md")].

The grammar of key-value attributes turns from

PATH `=` UNSUFFIXED_LITERAL | EXPR_NONTERMINAL

to

PATH `=` EXPR

, but the restriction requiring EXPR to be an unsuffixed literal after expansion is still kept.
The expression goes through macro expansion as any other expression would do.

From the implementation point of view the PR still uses the "expansion in nonterminals" hack that currently powers #[doc = $expr], but my plan is to keep an actual ast::Expr in attributes to avoid the hack and expand in a natural way in the future.

r? @ghost

petrochenkov · 2019-12-07T15:22:35Z

cc @rust-lang/lang @rust-lang/rustdoc

I also wanted to write some more general post about macro expansion points in attributes on internals, but not sure if I'll be able to do it today.

Centril · 2019-12-07T19:18:36Z

This is just an experiment for now, but I'll leave some thoughts for now...

PATH `=` EXPR

I don't agree with the biasing towards expressions here. The right-hand-side of = could just as well be a type (in particular), constraint, or e.g. a pattern. I think the interpretation of RHS should be driven by the attribute.

I would be fine with PATH = TOKEN_TREE or if more is needed, then (#57321 (comment)):

We can make key-value attributes accept almost arbitrary token streams with single special token , (comma) acting as a terminator (#[foo = TOKEN_STREAM, bar = TOKEN_STREAM]).

Also, you noted in #57367 (comment) that "it needs eager expansion" (cc rust-lang/rfcs#2320). So what's up with that?

I think this probably needs to end up with an RFC which you noted yourself at the time.

joshtriplett · 2019-12-07T19:29:39Z

I've wanted this for a while, for things like #[path = concat!(env!(”OUT_DIR"), "/generated.rs")].

abonander · 2019-12-08T02:13:22Z

Also, you noted in #57367 (comment) that "it needs eager expansion" (cc rust-lang/rfcs#2320). So what's up with that?

@Centril I've been out of touch in the macro scene for a while but I think that means that the expansion order of macros would have to be inverted because right now any proc macro attribute will just see include_str!("my_doc.md") for its input and we actually want it to see the result of the expansion of that instead.

I don't think changing the expansion order statically is the right approach, I think we should just allow proc macros the option of getting their input expanded, either with a method like TokenStream::expand() or an argument to #[proc_macro_attribute] that causes the frontend to macro-expand the attribute's input before passing it in.

I think the latter is much less of an API commitment and easier to define the semantics for, because the former would have to allow expanding arbitrary code as well as the macro input or else users will be too confused.

petrochenkov · 2019-12-08T09:07:27Z

Sorry, I forget to tell this, but the important detail is that this is for inert attributes only.
E.g. doc is certainly not a proc macro.
(All key value attributes are inert at the moment and we can keep it so depending on design decisions.)

As a result, this is not eager expansion, and attributes cannot implement the input processing logic themselves and need to delegate to the compiler's expansion infra.
(I still need to write that internals post describing the details.)

luser · 2019-12-08T15:21:50Z

This is a general-purpose feature that is supposed to supersede #[doc(include = "my_doc.md")].

Neat! If this works this would also allow writing proc macros to do even more interesting things. For example, I've wanted to be able to include the contents of README.md in the crate-level documentation since I usually wind up writing the same high-level docs + examples there, but it'd be nice to be able to do that while also leaving out things like CI badges.

petrochenkov · 2019-12-08T18:14:49Z

Internals post - https://internals.rust-lang.org/t/macro-expansion-points-in-attributes/11455.

bors · 2019-12-21T14:24:12Z

☔ The latest upstream changes (presumably #66994) made this pull request unmergeable. Please resolve the merge conflicts.

petrochenkov · 2019-12-28T09:16:09Z

Closing for now, will reopen once the experiment described in #55414 (comment) is done.

Accept arbitrary expressions in key-value attributes at parse time Continuation of rust-lang#77271. We now support arbitrary expressions in values of key-value attributes at parse time. ``` #[my_attr = EXPR] ``` Previously only unsuffixed literals and interpolated expressions (`$expr`) were accepted. There are two immediate motivational cases for this: - External doc strings (`#[doc = include_str!("my_doc.md")]`, eliminating the need in rust-lang#44732) and expanding macros in this position in general. Currently such macro expansions are supported in this position in interpolated `$expr`s (the `#[doc = $doc]` idiom). - Paths (`#[namespace = foo::bar] extern "C++" { ... }`) like proposed in rust-lang#76734. If the attribute in question survives expansion, then the value is still restricted to unsuffixed literals by a semantic check. This restriction doesn't prevent the use cases listed above, so this PR keeps it in place for now. Closes rust-lang#52607. Previous attempt - rust-lang#67121. Some more detailed write up on internals - https://internals.rust-lang.org/t/macro-expansion-points-in-attributes/11455. Tracking issue - rust-lang#78835.

# Stabilization report ## Summary This stabilizes using macro expansion in key-value attributes, like so: ```rust #[doc = include_str!("my_doc.md")] struct S; #[path = concat!(env!("OUT_DIR"), "/generated.rs")] mod m; ``` See the changes to the reference for details on what macros are allowed; see Petrochenkov's excellent blog post [on internals](https://internals.rust-lang.org/t/macro-expansion-points-in-attributes/11455) for alternatives that were considered and rejected ("why accept no more and no less?") This has been available on nightly since 1.50 with no major issues. ## Notes ### Accepted syntax The parser accepts arbitrary Rust expressions in this position, but any expression other than a macro invocation will ultimately lead to an error because it is not expected by the built-in expression forms (e.g., `#[doc]`). Note that decorators and the like may be able to observe other expression forms. ### Expansion ordering Expansion of macro expressions in "inert" attributes occurs after decorators have executed, analogously to macro expressions appearing in the function body or other parts of decorator input. There is currently no way for decorators to accept macros in key-value position if macro expansion must be performed before the decorator executes (if the macro can simply be copied into the output for later expansion, that can work). ## Test cases - https://github.com/rust-lang/rust/blob/master/src/test/ui/attributes/key-value-expansion-on-mac.rs - https://github.com/rust-lang/rust/blob/master/src/test/rustdoc/external-doc.rs The feature has also been dogfooded extensively in the compiler and standard library: - rust-lang#83329 - rust-lang#83230 - rust-lang#82641 - rust-lang#80534 ## Implementation history - Initial proposal: rust-lang#55414 (comment) - Experiment to see how much code it would break: rust-lang#67121 - Preliminary work to restrict expansion that would conflict with this feature: rust-lang#77271 - Initial implementation: rust-lang#78837 - Fix for an ICE: rust-lang#80563 ## Unresolved Questions ~~rust-lang#83366 (comment) listed some concerns, but they have been resolved as of this final report.~~ ## Additional Information There are two workarounds that have a similar effect for `#[doc]` attributes on nightly. One is to emulate this behavior by using a limited version of this feature that was stabilized for historical reasons: ```rust macro_rules! forward_inner_docs { ($e:expr => $i:item) => { #[doc = $e] $i }; } forward_inner_docs!(include_str!("lib.rs") => struct S {}); ``` This also works for other attributes (like `#[path = concat!(...)]`). The other is to use `doc(include)`: ```rust #![feature(external_doc)] #[doc(include = "lib.rs")] struct S {} ``` The first works, but is non-trivial for people to discover, and difficult to read and maintain. The second is a strange special-case for a particular use of the macro. This generalizes it to work for any use case, not just including files. I plan to remove `doc(include)` when this is stabilized. The `forward_inner_docs` workaround will still compile without warnings, but I expect it to be used less once it's no longer necessary.

…=petrochenkov Stabilize extended_key_value_attributes Closes rust-lang#44732. Closes rust-lang#78835. Closes rust-lang#82768 (by making it irrelevant). # Stabilization report ## Summary This stabilizes using macro expansion in key-value attributes, like so: ```rust #[doc = include_str!("my_doc.md")] struct S; #[path = concat!(env!("OUT_DIR"), "/generated.rs")] mod m; ``` See Petrochenkov's excellent blog post [on internals](https://internals.rust-lang.org/t/macro-expansion-points-in-attributes/11455) for alternatives that were considered and rejected ("why accept no more and no less?") This has been available on nightly since 1.50 with no major issues. ## Notes ### Accepted syntax The parser accepts arbitrary Rust expressions in this position, but any expression other than a macro invocation will ultimately lead to an error because it is not expected by the built-in expression forms (e.g., `#[doc]`). Note that decorators and the like may be able to observe other expression forms. ### Expansion ordering Expansion of macro expressions in "inert" attributes occurs after decorators have executed, analogously to macro expressions appearing in the function body or other parts of decorator input. There is currently no way for decorators to accept macros in key-value position if macro expansion must be performed before the decorator executes (if the macro can simply be copied into the output for later expansion, that can work). ## Test cases - https://github.com/rust-lang/rust/blob/master/src/test/ui/attributes/key-value-expansion-on-mac.rs - https://github.com/rust-lang/rust/blob/master/src/test/rustdoc/external-doc.rs The feature has also been dogfooded extensively in the compiler and standard library: - rust-lang#83329 - rust-lang#83230 - rust-lang#82641 - rust-lang#80534 ## Implementation history - Initial proposal: rust-lang#55414 (comment) - Experiment to see how much code it would break: rust-lang#67121 - Preliminary work to restrict expansion that would conflict with this feature: rust-lang#77271 - Initial implementation: rust-lang#78837 - Fix for an ICE: rust-lang#80563 ## Unresolved Questions ~~rust-lang#83366 (comment) listed some concerns, but they have been resolved as of this final report.~~ ## Additional Information There are two workarounds that have a similar effect for `#[doc]` attributes on nightly. One is to emulate this behavior by using a limited version of this feature that was stabilized for historical reasons: ```rust macro_rules! forward_inner_docs { ($e:expr => $i:item) => { #[doc = $e] $i }; } forward_inner_docs!(include_str!("lib.rs") => struct S {}); ``` This also works for other attributes (like `#[path = concat!(...)]`). The other is to use `doc(include)`: ```rust #![feature(external_doc)] #[doc(include = "lib.rs")] struct S {} ``` The first works, but is non-trivial for people to discover, and difficult to read and maintain. The second is a strange special-case for a particular use of the macro. This generalizes it to work for any use case, not just including files. I plan to remove `doc(include)` when this is stabilized (rust-lang#82539). The `forward_inner_docs` workaround will still compile without warnings, but I expect it to be used less once it's no longer necessary.

[experiment] Expand macros in key-value attributes

f9fd803

petrochenkov mentioned this pull request Dec 7, 2019

Tracking issue: rfc 1990 - add external doc attribute to rustc #44732

Closed

3 tasks

jonas-schievink added A-attributes Area: Attributes (`#[…]`, `#![…]`) A-macros Area: All kinds of macros (custom derive, macro_rules!, proc macros, ..) T-lang Relevant to the language team, which will review and decide on the PR/issue. labels Dec 7, 2019

petrochenkov added S-waiting-on-team Status: Awaiting decision from the relevant subteam (see the T-<team> label). T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. labels Dec 7, 2019

petrochenkov changed the title ~~[experiment] Expand macros in key-value attributes~~ [experiment] Expand macros in inert key-value attributes Dec 8, 2019

Centril added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Dec 8, 2019

petrochenkov closed this Dec 28, 2019

mystor mentioned this pull request Jan 9, 2020

docs: consider adding a table of contents to readmes tokio-rs/tracing#516

Open

This was referenced Jan 9, 2020

Creating a doc attribute from stringify! or concat! in a macro works in some situations but not others #52607

Closed

Macro ICE #68058

Closed

petrochenkov mentioned this pull request Nov 7, 2020

Accept arbitrary expressions in key-value attributes at parse time #78837

Merged

jyn514 mentioned this pull request Mar 22, 2021

Stabilize extended_key_value_attributes #83366

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[experiment] Expand macros in inert key-value attributes #67121

[experiment] Expand macros in inert key-value attributes #67121

petrochenkov commented Dec 7, 2019 •

edited

Loading

petrochenkov commented Dec 7, 2019

Centril commented Dec 7, 2019

joshtriplett commented Dec 7, 2019

abonander commented Dec 8, 2019

petrochenkov commented Dec 8, 2019 •

edited

Loading

luser commented Dec 8, 2019

petrochenkov commented Dec 8, 2019

bors commented Dec 21, 2019

petrochenkov commented Dec 28, 2019

[experiment] Expand macros in inert key-value attributes #67121

[experiment] Expand macros in inert key-value attributes #67121

Conversation

petrochenkov commented Dec 7, 2019 • edited Loading

petrochenkov commented Dec 7, 2019

Centril commented Dec 7, 2019

joshtriplett commented Dec 7, 2019

abonander commented Dec 8, 2019

petrochenkov commented Dec 8, 2019 • edited Loading

luser commented Dec 8, 2019

petrochenkov commented Dec 8, 2019

bors commented Dec 21, 2019

petrochenkov commented Dec 28, 2019

petrochenkov commented Dec 7, 2019 •

edited

Loading

petrochenkov commented Dec 8, 2019 •

edited

Loading