Hygiene opt-out for idents in expansion of declarative macros

[alexreg]

This PR adds basic support for hygiene opt-out for idents during macro expansion. This has been previously discussed in #40847 and #39412.

The syntax involves prefixing idents with the # (i.e. pound/hash) character to indicate that the syntax context of the ident should be that of the call site rather than the definition site.

The following now compiles and runs as expected:

#![feature(decl_macro)]

macro foo($mod_name:ident) {
    pub mod $mod_name {
        pub const #BAR: u32 = 123;
    }
}

foo!(foo_mod);

fn main() {
    println!("{}", foo_mod::BAR);
}

as does:

#![feature(decl_macro)]

macro foo() {
    pub mod #foo_mod {
        pub const #BAR: u32 = 123;
    }
}

foo!();

fn main() {
    println!("{}", foo_mod::BAR);
}

but the following does not:

#![feature(decl_macro)]

macro foo($mod_name:ident) {
    pub mod $mode_name {
        pub const BAR: u32 = 123;
    }
}

foo!(foo_mod);

fn main() {
    println!("{}", foo_mod::BAR);
}

nor does:

#![feature(decl_macro)]

macro foo() {
    pub mod #foo_mod {
        pub const BAR: u32 = 123;
    }
}

foo!();

fn main() {
    println!("{}", foo_mod::BAR);
}

Questions:

• Is the above the behaviour we want?
• Should we implement syntax context "lifting" using functionality suggested by RFC? rust-lang/rfcs#2320 – this related to the 2nd question of @petrochenkov here
• Hygiene opt-out doesn't make any sense for macro_rules macros, of course, but should we explicitly disallow the # syntax in such places? (Currently that is not done.)

CC @jseyfried

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @petrochenkov (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

[petrochenkov]

Could you add all the examples from the PR descriptions as tests?
The example from #39412 (comment) needs to be added as a test as well.
It would also be good to test that #foo is accepted in position expecting ident matcher (macro m(arg: ident) { ... }; m!(#foo)) and probably make sure that identifier prefixes $ and # are composable with each other ($#foo - use a macro argument foo while referring to it unhygienically).

[alexreg]

@petrochenkov Yep, sounds fair. run-pass and compile-fail tests, I presume? Are you okay with the examples 3 and 4 failing?

Also, thoughts on my above comment about use of the syntax in legacy macro_rules?

[alexreg]

Oh, and I think the syntax #$foo makes slightly more sense BTW.

[alexreg]

A further question:

• Should we do hygiene opt-out for lifetimes too? (This should cover all ident-like tokens that have hygiene, if I'm not mistaken.)

[alexreg]

Okay, my present thoughts are that 3 and 4 should indeed fail. To make them work would mean having the macro parser understand items, which would mean binding it more tightly to the normal parser, and really make a mess of things.

[pierzchalski]

@alexreg I'm confused as to what it would mean for them to work, even if it would involve reworking the parser. If I understand correctly, in examples 3 and 4 foo!() expands into:

// `main` scope
//      vvvvvvv
pub mod foo_mod {
    // `foo!` scope
    //        vvv
    pub const BAR: u32 = 123;
}

In both cases BAR is out of scope in main, right? I'm curious what alternative semantics there are here.

[alexreg]

@pierzchalski Well, the alternative semantics would be BAR inheriting its syntax context from foo_mod (since its part of the item whose name is foo_mod). They are complicated semantics though, I admit.

i.e. We have

// `main` scope
//      vvvvvvv
pub mod foo_mod {
    // `main` scope
    //        vvv
    pub const BAR: u32 = 123;
}

(The same end result in terms of hygiene as for macro_rules! now, but requiring the # ident prefix in example 4.)

[pierzchalski]

@alexreg Ah, I see! Yeah, making scopes semantics-aware sounds like a lot of work. It also means you'd need a way to 'un-lift' any identifiers you expected to use privately:

// `main` scope
//      vvvvvvv
pub mod foo_mod {
    // `main` scope
    //     vvv
    pub fn bar() {
        // I want this to be in `foo_mod!` scope
        baz();
    }

    // Perhaps I want this private to `foo_mod!` but
    // `pub` because it's used by... other modules generated by `foo_mod!`,
    // or something else silly.
    pub fn baz() { ... }
}

[petrochenkov]

@alexreg

Also, thoughts on my above comment about use of the syntax in legacy macro_rules?

If it can be easily prohibited, then let's prohibit it for a start, otherwise any use of the # syntax can be feature gated so it's not available in macro_rules on stable.
(Also, what if a macro_rules is defined inside of macro and uses # inside it? That's an interesting question!)

[petrochenkov]

Oh, and I think the syntax #$foo makes slightly more sense BTW.

Why?
I see # is an innate property of the identifier (Ident = {Symbol, Hygiene} ), while $ is "just some syntax" to use that identifier in a certain way (MacroParam = {Ident}).

[petrochenkov]

Should we do hygiene opt-out for lifetimes too?

Yes, ideally we should.

[petrochenkov]

Okay, my present thoughts are that 3 and 4 should indeed fail.

What is "3 and 4"?

[pierzchalski]

@petrochenkov The 3rd and 4th code snippets in the starting post (I was also briefly lost on that).

[petrochenkov]

The 3rd and 4th code snippets in the starting post

They should fail then, BAR being inaccessible is kinda the point of macro 2.0 hygiene.

[petrochenkov]

Should we do hygiene opt-out for lifetimes too?

cc https://internals.rust-lang.org/t/pre-rfc-splitting-liftime-into-two-tokens/6716

[alexreg]

@pierzchalski Exactly. The difficulties outweigh the small benefits and added complexity I think.

[alexreg]

@petrochenkov Sounds good to me. If we can solve the spaces problem mentioned there, it will definitely simplify a lot of code. For now I may hack around it.

[alexreg]

If it can be easily prohibited, then let's prohibit it for a start, otherwise any use of the # syntax can be feature gated so it's not available in macro_rules on stable.
(Also, what if a macro_rules is defined inside of macro and uses # inside it? That's an interesting question!)

Sure. I'll have a go at this – I don't think it's too hard.

[alexreg]

Why?
I see # is an innate property of the identifier (Ident = {Symbol, Hygiene}), while $ is "just some syntax" to use that identifier in a certain way (MacroParam = {Ident}).

I think of it in terms of composability. For a metavar foo, #foo doesn't make sense in general, but taking $foo then prefixing it with # to yield #$foo does make sense. i.e. The metavar ident needs to be substituted for the actual used ident (macro argument, if you will), before we can talk about applying hygiene.

[alexreg]

Okay, tests added (both run-pass & compile-fail) – everything passing locally.

r? @jseyfried

CC @nikomatsakis too, in case @jseyfried is too busy still (I've noticed a bit of activity from him though).

[nikomatsakis]

r? @nikomatsakis

[rust-highfive]

The job x86_64-gnu-llvm-3.9 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[00:56:42] ...........test [run-pass] run-pass/mir_heavy_promoted.rs has been running for over 60 seconds
[00:57:11] .........................................................................................
[00:57:30] ..................................................................................ii................
[00:58:27] ..............................................i....................................................i
[00:58:32] .ii............test [run-pass] run-pass/saturating-float-casts.rs has been running for over 60 seconds
[00:59:31] .......iiiiiii......................................................................................
[00:59:53] ....................................................................................................
[01:00:13] ....................................................................................................
[01:00:32] .................................................................................
---
[01:03:35] ---- [compile-fail] compile-fail/feature-gate-macro-hygiene-optout.rs stdout ----
[01:03:35]  
[01:03:35] error: compile-fail test compiled successfully!
[01:03:35] status: exit code: 0
[01:03:35] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/compile-fail/feature-gate-macro-hygiene-optout.rs" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-Zui-testing" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail/feature-gate-macro-hygiene-optout.stage2-x86_64-unknown-linux-gnu" "-Crpath" "-O" "-Zunstable-options" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail/feature-gate-macro-hygiene-optout.stage2-x86_64-unknown-linux-gnu.aux" "-A" "unused"
[01:03:35] ------------------------------------------
[01:03:35] 
[01:03:35] ------------------------------------------
[01:03:35] stderr:
[01:03:35] stderr:
[01:03:35] ------------------------------------------
[01:03:35] 
[01:03:35] ------------------------------------------
[01:03:35] 
[01:03:35] thread '[compile-fail] compile-fail/feature-gate-macro-hygiene-optout.rs' panicked at 'explicit panic', tools/compiletest/src/runtest.rs:3033:9

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

[rust-highfive]

The job x86_64-gnu-llvm-3.9 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[00:51:23] .....................test [run-pass] run-pass/mir_heavy_promoted.rs has been running for over 60 seconds
[00:51:44] ...............................................................................
[00:52:03] ..................................................................................ii................
[00:52:52] ..............................................i....................................................i
[00:53:05] .ii...............................test [run-pass] run-pass/saturating-float-casts.rs has been running for over 60 seconds
[00:53:53] .......iiiiiii......................................................................................
[00:54:14] ....................................................................................................
[00:54:32] ....................................................................................................
[00:54:50] .................................................................................
---
[00:55:01] ....................................................................................................
[00:55:07] ....................................................................................................
[00:55:14] ....................................................................................................
[00:55:20] ....................i............................................................ii.iii.............
[00:55:26] ..............................................................................................F.....
[00:55:39] ....................................................................................................
[00:55:44] ..................i.................................................................................
[00:55:51] ....................................................................................................
[00:55:59] ....................................................................................................

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

[alexreg]

How are we looking now, @petrochenkov / @jseyfried? :-)

[joshtriplett]

We discussed this in the language team meeting, and we agree that this needs an actual RFC to specify this for review. This is not the kind of thing we should just do without an RFC.

[alexreg]

@joshtriplett Okay. What I'll do then is get @petrochenkov & @jseyfried's feedback now, and then start writing up an RFC including what's been done (and learnt) in this PR, plus future plans. Maybe we can leave this open in the meanwhile, then once that's accepted, we can hopefully get this merged with few changes?

[pietroalbini]

@alexreg I'd prefer for this PR to be closed in the meantime, otherwise triage would have to check this PR every week. Then when you're ready you can open it again.

[alexreg]

@pietroalbini As long as it can be reopened easily, sure. Hopefully @petrochenkov & @jseyfried will see my above comments anyway, and leave some feedback before I get to writing the RFC.

[petrochenkov]

Overriding fold_ident is not necessary, overridden new_span applies to identifiers as well.

[alexreg]

Ah, I see. Why is it necessary in the Folder implementation for Marker though?

[petrochenkov]

It isn't necessary there either, it's just something I forgot to remove in #49154.

[alexreg]

Okay. :-)

[petrochenkov]

@alexreg

Just to confirm, we only want this feature gate to apply to the macro! (macros 2.0) syntax, right?

I'm not even sure there's a need in a separate feature gate beyond feature(decl_macro), but I guess it won't hurt.
In #47992 (comment) I suggested how to treat # in the most forward-compatible way.

[petrochenkov]

@alexreg

Hopefully @petrochenkov ... leave some feedback before I get to writing the RFC.

Feedback: the implementation looks good to me, but if lang team thinks it needs an RFC, then it needs an RFC.

[pietroalbini]

Ok, closing this until an RFC is approved. @alexreg when you're ready to work on this again just click the reopen button ;)

[petrochenkov]

@pietroalbini
FYI, only organization members can click the reopen button (unless PR is closed by author).

[alexreg]

@pietroalbini It turns out I don't have the rights to reopen my PRs (as I thought), but I'll ping you or someone else when it's time, sure!

[alexreg]

@petrochenkov There's no easy way to get whether we're in legacy mode from within the quoted::parse fn, is there? I have to add a new parameter, right? (Just reworking a few things according to your latest feedback.)

[alexreg]

@petrochenkov Also, should the errors be emitted when doing the actual parsing of the macro or the expansion?

[petrochenkov]

@alexreg

There's no easy way to get whether we're in legacy mode from within the quoted::parse fn, is there?

Don't know, looks like no.

Also, should the errors be emitted when doing the actual parsing of the macro or the expansion?

Ideally, if some error will be reported at any expansion, then it should be reported at macro definition.
But I'm not even sure we are looking at macro's contents on definition beyond building a basic token tree, so if it's not reasonably doable, then everything can be reported during expansion.

[alexreg]

@petrochenkov Makes sense, thanks. I think we can bail within the quoted::parse method for the errors mentioned in your comment here.

[alexreg]

@petrochenkov Incidentally, do you want to disallow $#ident with an explicit error message about that syntax being unsupported, or just the error message for "unexpected # symbol" or such...?

I'm not sure what the error message should be when an interpolated token follows # either. Any ideas?

[petrochenkov]

Something like "Hygiene opt-out is not supported for macro parameters" or "... on the left side of the macro".
"Hygiene opt-out is not supported for interpolated tokens".

[alexreg]

Hmm, what are interpolated tokens? Something to do with proc macros I seem to recall, but I don’t really know. The term might confuse users, no?

[alexreg]

"Hygiene opt-out is not supported for macro parameters"

The problem with this is, we're allowing #$ident but not $#ident.

[alexreg]

@petrochenkov Pushed new commits anyway, in case you want to have a look. May make another, depending on your answers to the above two queries, but otherwise I'll focus on the RFC now.