Eager expansion of macros

[nrc]

This RFC proposes eager expansion of macros. The primary motivation is providing a mechanism for creating new identifiers in macros (e.g., by concatenating identifiers). It is not currently practical to do this because macros cannot be used in identifier position.

[kennytm]

Is it a "macro 2.0"-only feature or will it be backported to macro_rules!?

[glaebhoerl]

I think it's kind of cute that $! is also the operator for strict (eager) function application in Haskell :)

[nrc]

@kennytm primarily a macro 2.0 feature. If it is not too much implementation work and doesn't break backwards compatibility (which I don't think it does), then we could backport.

[comex]

Can anyone explain the advantage of not always eagerly expanding macros?

Alternately, why only allow this inside macro definitions? concat_idents is not the only use case for having a macro expand to something other than a full item/expression/etc. (And the limit could be trivially circumvented by putting the entire file inside a macro definition, if necessary!)

[kennytm]

(Just asking about the semantics, not that this feature is important :: )

Would this be valid?

macro a {
    (0) => { 1, 2, 3 };
    (1) => { [$!a(0)] };
}
fn main() {
    let m = a!(1);
}

Would this be valid?

macro b {
     (0) => { , };
     (1) => { [1 $!b(0) 2 $!b(0) 3] };
}
fn main() {
    let m = b!(1);
}

[eddyb]

The two other syntax choices I've seen for this feature are $*concat!(a, b) and concat!!(a, b), btw.

[Manishearth]

Alternative: macro-level $let which lets you do things like $$let $x = concat_idents(...); foo!($x). This does the same thing as eager expansion, but is more readable.

[eddyb]

Binding is orthogonal to eager expansion, though, e.g. you might want to use $let for a handful of (unexpanded) tokens to reuse them a few times.

[Ericson2314]

This ultimately generalizes to quasi-quoting, just saying.

[chris-morgan]

Ooh, doubling the !, I like that comparatively. $!foo instead of foo! felt icky. But doubling the ! feels like it means “seriously, do it! Now!!” which is what’s happening.

[glaebhoerl]

As long as this can only be used inside macros, $!foo is also good at signifying that (by starting with $).

[nrc]

Getting into the syntax bikeshed - I like $! because it suggests something halfway between a macro expansion and a macro variable. !! could work though.

[nrc]

@kennytm I think both uses would be valid. I should add a note to the RFC about recursive use. I guess we would bail when we detect a cycle of expansion, rather than a cycle of names.

[nrc]

@comex I don't think there are technical reasons for not eagerly expanding all macros. But regular expansion just feels more natural. There are some subtle differences with hygiene, I wonder if they might cause issues if all macros were done that way.

To allow macros in any position macro expansion would have to happen on tokens, rather than on the AST. That would prohibit modularisation for macros and make hygiene much more complex, if not impossible.

[durka]

So from @kennytm's examples, eager expansion drops the requirement that macros expand to a full expr/item/type/etc.

[comex]

@nrc Hmm... I reread the RFC again and read your blog post about scopes... still a bit confused. The limitations you mention seem like they would also apply to the contents of a macro definition. Does eager expansion happen on tokens? I guess you sidestep the issue by not allowing names defined within a token list that gets eagerly expanded to affect macro resolution from that same eager expansion... (Which also means that performing the same process on the entire file first would be mostly useless, as no non-builtin macros could be in scope yet.) What happens if you try, though?

macro bar { () => { macro baz { ... } } }
macro foo { () => { $!bar() } }

What scopes does baz have? Is there any way to invoke it?

Is there a way to 'escape' expansions, so that in this example $!test() would not be expanded as part of bar, but rather be part of the inner macro definition?

macro bar { ($mname:ident) => { macro $mname { $!test() } } }

(Or if it's always part of the inner macro definition, then what if the word macro is itself introduced by an expansion?)

If eager expansions are nested, does the whole thing look up macros based on the innermost non-eager expansion's scope?

Getting into pedantry a bit...

Is it possible for an eager expansion to have unbalanced braces/parens? e.g. what happens if you write $!include("parens.rs") where parens.rs contains the text ) (. If it works, then if you write

$!foo($!include("parens.rs") $!bar())

will bar be expanded before foo even though it isn't actually an argument to foo?

(on second thought, allowing that would kind of break the world anyway... since then you could exit the scope of the macro definition you're in)

[eddyb]

Is there a way to 'escape' expansions

I would like to see \<token> being a way to write an opaque entity that expands to <token>.

This is pretty much the only way I see forward for being able to define macros from macros (modulo some name resolution magic around $x), having sequences separated by + or *, etc.
However, I'm not entirely sure how it interacts with all the other plans around macros.

[kennytm]

@comex The question would be whether eager expansion should be done before lexing or after it. If expansion is done after lexing, ) ( would be invalid since it is not quite a valid TokenTree array. I think it should be done after lexing.

[kennytm]

If [1 $!b(1) 2 $!b(1) 3] is valid, considering b could be a procedural macro, the TTMacroExpander in libsyntax would need to change from returning MacResult to a TokenStream, since , is not an expression, item, pattern, statement or type.

Or, since this is "macro 2.0", we would assume #1566 is implemented already, and all syntax extensions (including those in libstd) are migrated to the libmacro API, which produces TokenStream.

[alexreg]

Has any consensus been reached on this? For me, this would be an extremely desirable feature that would go a long way towards uncrippling the current macro system. Macros 2.0 will be great, but a long way off, of course.

[eddyb]

@alexreg The prerequisites will be reached for macros 1.1 (cc @alexcrichton and @nrc) so eager expansion should be doable then, although it might remain unstable for a while.

[alexreg]

Thanks for the update. That’s good to hear. Any timeframe for that?

On 18 Jul 2016, at 15:03, Eduard-Mihai Burtescu notifications@github.com wrote:

@alexreg https://github.com/alexreg The prerequisites will be reached for macros 1.1 (cc @alexcrichton https://github.com/alexcrichton and @nrc https://github.com/nrc) so eager expansion should be doable then, although it might remain unstable for a while.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub #1628 (comment), or mute the thread https://github.com/notifications/unsubscribe-auth/AAEF3IRJZaGS0ZBGgiNqJfwSn5tG9fFRks5qW4e6gaJpZM4IlH72.

[nikomatsakis]

Hear ye, hear ye! This RFC is now entering final comment period with an inclination to accept.

Summary of discussion follows:

• There was some discussion of alternative syntax. This RFC proposes $!foo(a), with the intention of invoking how $foo will substitute foo eagerly, but some alternatives are:
  • foo!!(a)
  • $*foo!(a)
  • $$let $x = foo(a); $x -- a more involved binding mechanism
• Some questions about how this interacts with scoping (that don't seem to have been fully answered?)

@rust-lang/lang members, please check off your name to signal agreement. Leave a comment with concerns or objections. Others, please leave comments. Thanks!

• @nikomatsakis
• @nrc
• @aturon
• @eddyb
• @pnkfelix (on vacation)

[nrc]

re syntax, I still prefer my original proposal, but could also live with foo!!. I'm not keen on the other proposals.

re scoping,

What scopes does baz have? Is there any way to invoke it?

This is a hygiene question somewhat orothogonal to this RFC and the answer hasn't been completely nailed down anywhere yet. My current understanding is that, as written, baz is only nameable inside the decl of bar, however, one could rewrite this (by passing the name from outside) or use some hygiene-manipulating library macros to make baz nameable from outside.

Is there a way to 'escape' expansions

There is not. For nested macros, there is no proposal for escaping variables, there should be and when that happens, the same mechanism should apply to eager expansions too.

Is it possible for an eager expansion to have unbalanced braces/parens?

No, never, macros operate on token trees and they do not admit unbalanced delimiters.

[Ericson2314]

No, never, macros operate on token trees and they do not admit unbalanced delimiters.

Can we formalize this? Perhaps we could say say something like "an eager macro invocation expands to a single token tree which is merged with the node containing the invocation", à la unquote-splicing. edit upon more thought, it's probably just regular unquote.

[eddyb]

At that point "eager" starts being meaningless, I'd almost call it expand! or bind! or some other verb.
I do like the power it offers and reuse of synthesized identifiers is somewhat common.

[Ericson2314]

@jseyfried if we mock up a syntax to generalize this eager to token-level quasi_quote, sounds good to me.

(The restriction to macro definitions is just a narrow case of unquote needing to occur within quasi-quote.)

[aturon]

@nrc: given the relative lack of bandwidth around macros, I think we should consider postponing this RFC (until someone has the time to take up the mantle). What do you think?

[jseyfried]

I think we should postpone until proc-macros have the API to eagerly expand token streams and then see if a third-party eager! or the like is sufficient.

[nikomatsakis]

@rfcbot fcp postpone

I agree with @jseyfried and @aturon that it makes sense to wait here. I remain pretty conflicted about this proposal in any case, as has been previously discussed on thread. @nrc -- feel free to object, however. =)

[rfcbot]

Team member @nikomatsakis has proposed to postpone this. The next step is review by the rest of the tagged teams:

• @aturon
• @eddyb
• @nikomatsakis
• @nrc
• @pnkfelix
• @withoutboats

No concerns currently listed.

Once these reviewers reach consensus, this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[retep998]

So much for my dream of one day being able to concatenate idents in declarative macros...

[nrc]

I agree we can postpone, I still think this is the best solution though. I think that if someone wanted to implement this (behind a flag, obvs) then that should be fine too and might offer some insight into the details (plus of course usage experience).

[nrc]

Also, why does GitHub not offer me a sadface emoji?

[aturon]

I've checked the review box for @pnkfelix, who is on PTO for an extended period.

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[rfcbot]

The final comment period is now complete.

[aturon]

Closing as postponed. We'll definitely be revisiting this topic!

[alexreg]

Hey @nrc. Per our discussion on Discord, could we get this reopened, with a view to merging it soon? I put my hand up to implement this, of course, and feel I'm in a good position to do so given my experience discussing this over many months, and my previous work with the macro system. I'll also draft a separate RFC for some new libstd macros at the same time (to do with hygiene manipulation).

[withoutboats]

My read of this thread is that the lang team is divided on the design here; its not implementation bandwidth that led it to be postponed. (I myself am totally undecided!)

[alexreg]

@withoutboats Okay, that certainly wasn't clear from the above, but I'm glad that's made plain now, thanks. So, it sounds like more discussion is needed then. I had some conservations with @nrc lately... a few points seemed to come up in discussion:

• The principal current motivation for eager expansion is macros-in-ident-position (within macro definitions at least).
• The main alternative is to allow normal macros in ident position, but the concern here is ugly/unreadable syntax. The worst case to me seems to be something like generate_fn_name!(macro_arg1, marco_arg2, ...)(arg1, arg2, ...). Choices include a) accepting this syntax everywhere and biting the bullet, b) only allowing macros-in-ident-position within other macro definitions, c) requiring some slight syntax modification for the ident-position case, e.g. (generate_fn_name!(macro_arg1, marco_arg2, ...))(arg1, arg2, ...), which may increase readability.
• Eager expansion apparently has the advantage of not requiring special-casing for the ident position – this would make it easier to implement perhaps, though it's not immediately clear where else one might want to use it. According to @nrc, it is also has the advantage of "compositionality", though I think he needs to expand on it more.

Shall we reopen so we can at least discuss it more? Maybe address some of the above points, and your present concerns, would be the best thing.

[retep998]

b) only allowing macros-in-ident-position within other macro definitions,
c) requiring some slight syntax modification for the ident-position case, e.g. (generate_fn_name!(macro_arg1, marco_arg2, ...))(arg1, arg2, ...), which may increase readability.

Both of these options are perfectly acceptable to me.

[vi]

Is easer expansion a good idea not for only working around some missing feature (ident-position), but also as a tool to push DRY further, when repetitions happen inside a foreign macro call?

[alexreg]

Both of these options are perfectly acceptable to me.

Good to know. Personally I'm also relatively happy with either. Would like to get some other @rust-lang/lang members' feedback on these points too though.

[alexreg]

@vi Not sure what you mean...

[vi]

@alexreg, I mean implementing macros to work within other non-cooperating macros.