Tracking issue: declarative macros 2.0

[nrc]

Tracking issue for declarative macros 2.0 (aka macro aka decl_macro aka macros-by-example).

RFC: https://github.com/rust-lang/rfcs/blob/master/text/1584-macros.md

RFC PR: rust-lang/rfcs#1584

cc @rust-lang/compiler

---

Tasks

• Complete the draft hygiene prototype and RFC.
  • Hygiene for items, explicit imports, lexical scopes, and module scopes.
  • Hygiene for globs and trait methods (w.r.t. extension trait candidates in scope for a method call).
  • Hygiene for type directed name resolutions (i.e. methods, associated types, and fields).
  • Support macro in blocks as well as modules.
  • Implement inter-crate hygiene, except for nested macros.
  • Implement "hygiene bending" for when users want a name from macro def to "escape".
  • Implement inter-crate hygiene for nested macros (pending macro def encoding).
  • Make private_in_public hygienic.
  • Make unsafe and lints hygienic (if appropriate).
• Add variant ast::ItemKind::MacroDef for macro_rules! items and macro items (PR #40220).
• Encode macros in the crate metadata using TokenStream, not String.
• Fix #30476 (private_in_public details).
• Fix span issues #30506 and #39450 (PR #40597).
• Land macro behind a feature gate (PR #40847).
• Future-proof matchers (e.g. $e:expr) by employing a simpler, more general grammar.
• Allow fragments (e.g. $e where $e:expr) to be parsed in more contexts (c.f. #26361).
• Decide whether we want macro invocations in identifier positions and/or eager expansion.
• Remove $:meta matcher (#49629)

[CryZe]

@nrc Typo in Issue Name: "issuse"

[jseyfried]

Tasks

(dtolnay edit: moved the checklist up to the OP)

[nrc]

We need to RFC a whole bunch of stuff here. In particular, I would like to propose some new syntax for declaring macros and we should RFC the changes to matchers.

[tikue]

Can the hygiene RFC mention pattern hygiene? This in particular scares me:

// Unsuspecting user's code
#[allow(non_camel_case_types)]
struct i(i64);

macro_rules! ignorant_macro {
    () => {
        let i = 0;
        println!("{}", i);
    };
}

fn main() {
    // oh no!
    ignorant_macro!();
}

[jseyfried]

@tikue I'm not sure patterns need special treatment with respect to hygiene.

For example, on the hygiene prototype,

#[allow(non_camel_case_types)]
struct i(i64);

macro ignorant_macro() {
    let i = 0; // ERROR: let bindings cannot shadow tuple structs
    println!("{}", i); 
}

fn main() {
    ignorant_macro!(); // NOTE: in this macro invocation
}

This makes sense to me since let i = 0; is shadowing a tuple struct.
In particular, if let i = 0; were removed then the following use of i would resolve to the tuple struct, no matter where ignorant_macro is used.

Note the symmetry to this example:

#[allow(non_camel_case_types)]
struct i(i64);

fn ignorant_fn() {
    let i = 0; // ERROR: let bindings cannot shadow tuple structs
    println!("{}", i); 
}

If the tuple struct i isn't in scope at macro ignorant_macro() { ... }, then the let i = 0; does not shadow it and there is no error. For example, the following compiles on the hygiene prototype:

mod foo {
    pub macro ignorant_macro() {
        let i = 0;
        println!("{}", i); // Without `let i = 0;`, there would be no `i` in scope here.
    }
}

// Unsuspecting user's code
#[allow(non_camel_case_types)]
struct i(i64);

fn main() {
    foo::ignorant_macro!();
}

[alexreg]

Has there been any progress on Macros 2.0 lately?

[mark-i-m]

Note for those who haven't seen yet: macros 2.0 is apparently slated to be stable later this year, according to the proposed roadmap (rust-lang/rfcs#2314)...

On the one hand, that's pretty exciting 🎉!

On the other hand, I was really surprised that the feature is so close to being done and so little is known about it by the broader community... The RFC is really vague. The unstable book only has a link to this issue. This issue(s) in the issue tracker mostly have detailed technical discussions. And I can't find that much info anywhere about what's changed or how stuff works.

I don't mean to complain, and I really truly appreciate all the hard work by those implementing, but I would also appreciate more transparency on this.

[alexreg]

@marcbowes Yeah, I'm kind of worried too. It seems like there's quite a lot of work left for stabilisation this year. I offered to work on opt-out hygiene for identifiers myself, but have received no response yet... Some greater transparency would be nice, as you say.

[petrochenkov]

Macros 2.0 are not even in the RFC stage yet - the @jseyfried's RFC linked in the issue was never submitted, there's only very high level RFC 1584.
The implementation is purely experimental and mostly unspecified, and large part of it (not related to hygiene) is reused from macro_rules! without fixing problems that macros 2.0 were supposed to fix.

Some greater transparency would be nice

The problem is that no work happen right now, so there's nothing to reveal :(
@nrc is busy, @jseyfried is busy, I worked on macros 2.0 a bit and would really like to dig into them more, but I'm busy too, unfortunately.
We need someone to adopt the feature, support and extend it, and gain expert-level knowledge of it, otherwise we will end up breaking hygiene and syntax details left and right after stabilization.

[alexreg]

@petrochenkov Ah, fair enough. I mean, that's a shame, but it makes sense at least. I guess this is an open call to anyone who might be able to take ownership of this feature, since no one comes to mind? I could still have a go at a little sub-feature, but I'm certainly in no position to take ownership of this.

[petrochenkov]

@alexreg

I could still have a go at a little sub-feature

Yes, please do!
Hygiene opt-out is one of the primary missing parts and implementing it would be useful in any case.

IIRC, two questions will need to be decided on during implementation:

• Syntax for "unhygienic" identifiers (@jseyfried tentatively suggested #ident in #40847)
• What exactly hygienic context the identifier introduced with #ident in a macro m will have - context of m's invocation? context after expanding all macros ("no hygiene")? something else?

[alexreg]

Syntax for "unhygienic" identifiers (@jseyfried tentatively suggested #ident in #40847)

Yeah, this was the plan. :-)

What exactly hygienic context the identifier introduced with #ident in a macro m will have - context of m's invocation? context after expanding all macros ("no hygiene")? something else?

What's your inclination? I'm leaning towards the context of m's invocation, but curious to hear your thoughts...

[alexreg]

@pierzchalski Yeah, that's not a bad idea at all. Thoughts, @jseyfried / @petrochenkov?

[pierzchalski]

@alexreg Actually, I just realised this is the use-case I was looking for for call_from in the proc macro RFC I put up.

[alexreg]

@pierzchalski I'll give that RFC a read tomorrow. Anyway, I certainly won't be including this lift macro or similar into my RFC; only the basic hygiene opt-out syntax. If the lift macro could go in another crate eventually, that would be ideal.

Incidentally, you seem to have a good knowledge of macro expansion and hygiene. Could I tempt you to contribute to rust-lang/rustc-guide#15? :-)

[pierzchalski]

@alexreg Unfortunately my understanding of expansion and hygiene is a bit abstract - you'll notice the reference-level explanation in the RFC was rather thin! But if the RFC is accepted and I end up implementing it then I'd definitely like to document what I discover along the way.

[dtolnay]

I filed #49629 to consider dropping support for #[$m:meta] in favor of #[$($meta:tt)*] now that attributes are allowed to contain an arbitrary token stream.

[isiahmeadows]

Stylistic question: why is it macro foo() and not macro foo!(), like the way it's called?

[mark-i-m]

I believe the question mark isn't considered part of the name.

[isiahmeadows]

@mark-i-m You mean exclamation point, not question mark, right?

Also, in the docs, it usually refers to macros including the exclamation point as if it were part of the name, such as println! or vec!. As a concrete example, here's one page in the book that uses it exclusively.

[mark-i-m]

Oh, yes, I meant exclamation mark. I might be wrong, but I believe the compiler itself doesn't count the exclamation mark as party of the ident.

[isiahmeadows]

@mark-i-m I would expect the compiler not to, but I'm speaking of the language itself, not the implementation. I'm suggesting matching what people think, not what computers process.

[mark-i-m]

That said, I don't see any reason why we cannot do something different from the internal representation.

If we did want to make ! part of the name, we would also want to do it in macro imports in the 2018 edition.

[isiahmeadows]

@mark-i-m If you change how you do it in imports, you could even integrate them into use, to avoid the need to deal with separate syntax altogether:

use mod::foo::some_macro!;
use mod::foo::{some_fn, some_macro!};

Even as recently as a couple days ago, I had to google how to import a macro from a peer module, and this would make it a million times simpler. It also wouldn't require any special attribute or whatever - it'd just work.

Similarly, exporting macros could be simply pub macro foo!() or similar. Alternatively, because ! could serve as a delimiter, you could remove macro altogether and just do pub fn foo!($a: ident) { ... }, or you could reserve macro foo! for macros with multiple syntax rules (like vec!), and let fn foo! be for ones with only a single value (like how try! could be written if this RFC gets implemented.

[mark-i-m]

@isiahmeadows you can do use foo::bar::some_macro in the 2018 edition. It is part of the module/macro system changes being stabilized.

Long term IIUC, the plan is to get rid of the attributes altogether (currently, you still need them to export macros).

I strongly prefer not to conflate functions and macros. They are very different.

[dhardy]

use mod::foo::{some_fn, some_macro!};

The ! suffix is not technically necessary here, but since this may be the only thing pointing out that it is indeed a macro, seems like a good idea (if feasible and not too late — probably is too late).

But of course if foo and foo! are two distinct identifiers, they should not both be allowed to exist in the same scope.

[mark-i-m]

Macros live in another namespace. You can have a macro foo and a function foo and a type foo all coexisting.

[Ekleog]

This makes me wonder. If I have a crate mycrate defining function foo and procedural macro foo, then if I do use mycrate::foo, which one is imported into scope? both? (I'm starting from the idea that some day in the hopefully near future the -derive crates and the not--derive crates will be merge-able)

Adding the ! as something to remove the ambiguity would likely help (in addition to being more explicit about what happens, which is always a good thing).

Assuming it's still time to do it, though… otherwise, add the ! suffix as a possibility and a warning to push using it?

[Nemo157]

@Ekleog yes, both (along with potentially a module foo and type foo as well).

[SimonSapin]

If I have a crate mycrate defining function foo and procedural macro foo

You cannot do that today:

error: `proc-macro` crate types cannot export any items other than functions tagged with `#[proc_macro_derive]` currently
 --> src/lib.rs:1:1
  |
1 | pub fn some_public_function() {}
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to previous error

However in an hypothetical future Rust where that restriction is lifted somehow, or if a non-proc-macro crate re-exports a proc macro and also a function of the same name, then yes use will import both.

[Nokel81]

Should small proposals for Macros2.0 be brought up here or in a separate issue?