Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Macro naming and modularisation #1561

Merged
merged 3 commits into from Aug 22, 2016

Conversation

Projects
None yet
@nrc
Copy link
Member

commented Mar 29, 2016

This RFC proposes making macros a first-class citizen in the Rust module system.
Both macros by example (macro_rules macros) and procedural macros (aka syntax
extensions) would use the same naming and modularisation scheme as other items
in Rust.

rendered

Macro naming and modularisation
This RFC proposes making macros a first-class citizen in the Rust module system.
Both macros by example (`macro_rules` macros) and procedural macros (aka syntax
extensions) would use the same naming and modularisation scheme as other items
in Rust.

@nrc nrc self-assigned this Mar 29, 2016

@nrc nrc added the T-lang label Mar 29, 2016

@brendanzab

This comment has been minimized.

Copy link
Member

commented Mar 30, 2016

This would be very useful for library developers, especially if we could reexport them as well! For example I would love this for my approx crate. Not sure about any subtle edge cases in this proposal though.

very different schemes for naming macros depending on whether a macro is defined
by example or procedurally. That would be inconsistent and annoying. However, I
hope we can make the new macro system appealing enough and close enough to the
existing system that migration is both desirable and easy.

This comment has been minimized.

Copy link
@oli-obk

oli-obk Mar 30, 2016

Contributor

We could simply lint on #[macro_use] and #[macro_export]. This would never show up in dependencies, but only in the crate that's currently compiled

and do not shadow or overlap items with the same name in the type or value
namespaces.

E.g., `use foo::bar::baz;` imports the macro `baz` from the module `::foo::bar`.

This comment has been minimized.

Copy link
@oli-obk

oli-obk Mar 30, 2016

Contributor

isn't that problematic, because it also imports module/function baz from module ::foo::bar?

This comment has been minimized.

Copy link
@nrc

nrc Mar 30, 2016

Author Member

We have the same issue today with baz importing both a type and value if they exist in the specified module. I don't think that causes problems, and it makes any other solution here less desirable.

This comment has been minimized.

Copy link
@ticki

ticki Mar 31, 2016

Contributor

Why not use the syntax, use foo::bar::baz!;?

This comment has been minimized.

Copy link
@brendanzab

brendanzab Mar 31, 2016

Member

Yeah, I would prefer paths to use bangs in the identifiers for macros. Eg. use approx::{relative_eq!, assert_ulps_eq!};

This comment has been minimized.

Copy link
@P1start

P1start Apr 2, 2016

Contributor

rustdoc also displays macro names with !s (e.g., std::println!). Obviously this could be changed, but I think it’s sort of become an informal standard to write macro names with the bang, even now when macro names are just identifiers and not paths.

Also, types and constants have (unenforced, but still generally followed) capitalisation schemes which help distinguish between them in imports, while macros are named in the same manner as functions (if you don’t count the !).

This comment has been minimized.

Copy link
@eternaleye

eternaleye Apr 22, 2016

I think you misunderstand me.

extern crate has_macro;
use has_macro::useful! as invokable!;
use has_macro::useful as callable;

#[cfg(macro)]
fn my_macro(...) -> ... {
    let x = callable(...);
    invokable!(x)
}

This should link has_macro to the compiler when compiling my crate, and to my crate when compiling the user's source, where they invoke my macro.

This comment has been minimized.

Copy link
@eddyb

eddyb Apr 22, 2016

Member

Those imports also need #[cfg(macro)]. Also, I expect macro functions are not be present in the function namespace in the current crate, because it's not in the same crate, really, it's a separate plugin crate.

This comment has been minimized.

Copy link
@eternaleye

eternaleye Apr 22, 2016

That'd ba a shame, because it'd be quite useful IMO.

This comment has been minimized.

Copy link
@eddyb

eddyb Apr 22, 2016

Member

I don't think the #[cfg(macro)] stuff has settled - I believe in its current form it can't even work.

This comment has been minimized.

Copy link
@paulstansifer

paulstansifer May 24, 2016

I don't believe that there's any need for #[cfg(macro)]s to be usable as functions at all. I don't believe that the equivalent feature is present in Racket, probably because functions that spit out syntax aren't useful except where syntax is expected. So expanding to code that invokes procedural macros will probably happen all the time (even recursive invocation of the macro currently being defined!), but there's probably little benefit to calling them as if they were functions.

I would like that macros follow the same rules for privacy as other Rust items,
i.e., they are private by default and may be marked as `pub` to make them
public. This is not as straightforward as it sounds as it requires parsing `pub
macro_rules! foo` as a macro definition, etc. I leave this for a separate RFC.

This comment has been minimized.

Copy link
@oli-obk

oli-obk Mar 30, 2016

Contributor

This would need to be solved before stabilization, as it's not backwards compatible to make every macro_rules definition private by default.

This comment has been minimized.

Copy link
@nrc

nrc Mar 30, 2016

Author Member

For macros by example, privacy would only apply to macros 2.0, which would probably use macro!, none of this RFC would apply to macro_rules! macros which would remain unchanged. (I should probably not use macro_rules in the example here, that seems confusing).

This comment has been minimized.

Copy link
@glaebhoerl

glaebhoerl Apr 1, 2016

Contributor

There are two other mentions of macro_rules in the RFC; same thing applies?

This comment has been minimized.

Copy link
@nrc

nrc Apr 5, 2016

Author Member

RFC updated

@camlorn

This comment has been minimized.

Copy link

commented Apr 2, 2016

A question: does this mean I can put a macro in an impl block?
At the moment, we have MyType::new, the equivalent of a constructor. For data structures that wish to be initialized with lists of items, it might also be helpful to do MyType::new!, i.e. Vec::new![1, 2, 3, 4, 5] as opposed to the gloval vec! macro.
If not, it should be noted that this would solve the most common case for which I could see a C++-style varargs or initializer list being useful while avoiding the pollution problem that we have currently.

@nrc

This comment has been minimized.

Copy link
Member Author

commented Apr 2, 2016

@camlorn No. Name resolution happens before type information is collected, so that wouldn't necessarily follow from this RFC. This RFC does not preclude doing that in the future, but I don't see a nice way to implement it in the general case.


Some day, I hope that procedural macros may be defined in the same crate in
which they are used. I leave the details of this for later, however, I don't
think this affects the design of naming - it should all Just Work.

This comment has been minimized.

Copy link
@xeno-by

xeno-by Apr 8, 2016

Could you provide some hints on how you plan to implement this?

This comment has been minimized.

Copy link
@nrc

nrc Apr 20, 2016

Author Member

Procedural macros and any items marked with #[cfg(macro)] would be split off and compiled into one crate, everything else compiled into another. The first crate would be linked in to the compiler while it compiles the second (or equivalently, called using IPC).

This comment has been minimized.

Copy link
@xeno-by

xeno-by Apr 21, 2016

So macros will only be able to call functions marked with #[cfg(macro)]?

This comment has been minimized.

Copy link
@eddyb

eddyb Apr 21, 2016

Member

That seems strange, #[cfg]-gating can only remove items with #[cfg(P)] where the cfg predicate P doesn't hold.

@solson

This comment has been minimized.

Copy link
Member

commented Apr 10, 2016

@nrc The Rendered link in the original post links to the original commit (https://github.com/nrc/rfcs/blob/573bd83aaac0896842f136262458c3b3bf671cd7/text/0000-macro-naming.md) rather than the most up-to-date version (https://github.com/nrc/rfcs/blob/macro-naming/text/0000-macro-naming.md).

@withoutboats

This comment has been minimized.

Copy link
Contributor

commented Apr 10, 2016

In the original blog series, it was suggested that macros imported using this system would be imported using a new symbol. I don't see that in this RFC, so I take it that this will change how macro_rules! macros can be imported. Are there any backwards compatibility issues connected with this change?

@nrc

This comment has been minimized.

Copy link
Member Author

commented Apr 20, 2016

In the original blog series, it was suggested that macros imported using this system would be imported using a new symbol. I don't see that in this RFC

This is #1584.

macro! foo { ... }
```

This comment has been minimized.

Copy link
@paulstansifer

paulstansifer May 24, 2016

I don't think this is a good idea, because it won't play nice with macro-defining macros. For example, the following sort of thing should (and currently does) work:

  macro_rules! macro_defining { 
    ($name:ident) => { macro_rules! $name { () => (println!("Yay!");) } }
  }

  macro_defining!( say_yay );

  say_yay!();

...but it only works if the macros are expanded in the order they are written, and there's no reasonable static analysis that can tell us that fact.

@paulstansifer

This comment has been minimized.

Copy link

commented May 25, 2016

Regarding the issue of ! in use statements: since macro! isn't nailed down yet, we can achieve total consistency:

macro! {
  my_macro!(...) => {...}
}

and

use some_mod::my_macro!;

I think that this would be perfect if it weren't for attribute-style macros. Even so, I think that I prefer to use ! as if it were really part of the name for non-attribute-style macros:

use some_mod::{my_normal_macro!, my_attribute_macro, my_ordinary_function};

I feel like this is what I would try if I didn't know what the actual rule was.

@solson

This comment has been minimized.

Copy link
Member

commented May 25, 2016

@paulstansifer There has been a lot of discussion on that topic, but I guess I'll point to #1561 (comment) in particular.

It doesn't seem worthwhile to use ! here, and it's more consistent to keep use syntax uniformly unadorned.

@eddyb

This comment has been minimized.

Copy link
Member

commented Jul 17, 2016

I was under the impression definitions and uses would be orthogonal even if we had two systems for each, but @nrc pointed out on IRC that is not the case as currently proposed.

To which I make the counter-suggestion: allow the usage of #[macro_use] or use, regardless of how the macro was defined or even exported, at least across crates (as long as the macro is visible at the top-level of the crate from which the macro is being imported, in thr case of #[macro_use]).
At the same time, this provides 2 benefits:

  • allow crate authors to update without breaking their old users
  • allow crate users to upgrade without changing old dependencies
@nikomatsakis

This comment has been minimized.

Copy link
Contributor

commented Aug 8, 2016

Hear ye, hear ye! This RFC is now entering final comment period. In the most recent @rust-lang/lang meeting, @nrc and I (the only two in attendance) felt inclined to accept. In particular, we're definitely in favor of the high-level aims of this RFC (controlling macros via ordinary use statements).

The major points of interest in the thread so far have been:

  • whether to place macros in their own namespace (the RFC does)
  • whether to import macros using a trailing ! (use foo::bar vs use foo::bar!). We two felt that use foo::bar was better, since the ! is more properly considered a "use macro operator" and not part of the macro name. As evidence for this position, consider that one can write higher-order macros that operate like macro_rules! foo { ($x:ident, $y:tt) => { $x!($y) } } (invoked like foo!(bar, baz), which is roughly equivalent to bar!(baz)). See this example on play. There is some good discussion here.
  • @paulstansifer's contention that not rely on macro-ordering will break macro-defining-macros; this is worth discussing more (I believe that not to be true, but there are some differences that emerge versus systems like Racket, which have a more mutable view of the universe).

(There may be some implementation details to be hashed out.)

@nikomatsakis

This comment has been minimized.

Copy link
Contributor

commented Aug 8, 2016

@paulstansifer

I don't think this is a good idea, because it won't play nice with macro-defining macros

This is definitely an important point, thanks for raising it. In fact, I think the system as we've envisioned it plays great with macro-defining macros, but there are some patterns in Racket that will not work. Let's start with your example:

macro_rules! macro_defining { 
    ($name:ident) => { macro_rules! $name { () => (println!("Yay!");) } }
}
macro_defining!( say_yay );
say_yay!();

This interacts with the overall name resolution algorithm defined in #1560. The idea is roughly that expansion and name resolution is a process that occurs in rounds. In this case, in the first round, the path say_yay! will be "unresolved" -- because there is no definition for it. However, macro_defining! will resolve and expand. In the next round, then, say_yay will resolve. Errors are only reported if we have expanded all macros and still have paths that cannot be resolved at the end of the process.

One danger here is that expanding a macro may cause a path that resolved in the past to resolve differently. Mostly this isn't possible, because expanding a macro can only add items, and duplicate items cause errors. However, one case where it can cause trouble is around globs and shadowing -- if a path relies on item X that is later shadowed by another item X produced by a macro, then it resolves differently. To counter this, we make it an error (in #1560) for a macro to expand to an item X that shadows an item in an outer scope, at least if that item was used in a macro expansion path. (See @jseyfried's comment for more details.)

Personally I find the idea of item ordering being significant in Rust to be very unappealing. It is strikingly different from how most things in Rust work, and it particularly doesn't mix well with use statements in my mind. (Perhaps the ordering could be some sort of DAG derived from the use statements, but my efforts to find such a system that works out well mostly failed.)

But this does make it more difficult to have "comunicating" macros, which I understand in Racket at least often rely on expansion order. An example that I was given by @samth where this could break Racket patterns of macro interop was something about pattern matching. I think the idea was that a class! macro might generate some metadata that is later used by a match! macro (in Racket, I mean), and that this introduces an ordering dependency. However, he also added that in practice macros like match! would generally try to "reschedule" themselves to run late in the expansion process so that the relative ordering of a class declaration and a match wouldn't be important. (I don't remember more details than that.)

I think a potential example of who this could come into play in Rust might be that, e.g., a macro might want to access data from a type definition. (Perhaps one would be expected e.g. to annotate the type definition in one way, and this would enable uses elsewhere that "communicate" with that annotation to extract data about the types of the fields, or something.) This is not well supported in the current system, which is targeted at more "self-contained" macros.

I'm not sure what's a "general" fix here but I think a lot will depend on the details. For example, if the "uses" that wish to communicate with the definition are inside of fns or other code blocks, I expect we can do things to ensure that all module-levels macros and items are expanded first (given that names defined in fns can't be used from outside that fn).

I can also imagine patterns where a module is decorated (#[foo] mod { ... }) -- this would then allow code in the module to reference data generated by the #[foo] macro. This works just by default because the code that will reference what #[foo] generates is ultimately governed by foo and results from the expansion of foo itself.

Finally, we might develop somewhat more ad-hoc solutions, similar to how Racket macros might "reschedule" themselves to make themselves order independent, where a macro can ask to be deferred from executing if it finds that other macros it might depend on are not yet expanded.

In short, I'd rather tackle this in a declarative way than relying on the "AST ordering", which I don't think is a natural concept in Rust.

(As one last point, I'd sort of like to get rid of explicit mod declarations, at least for private modules, and instead infer their presence from the file-system -- but if we did so, it would interact very poorly with having a defined AST ordering.)

@camlorn

This comment has been minimized.

Copy link

commented Aug 9, 2016

Can we get a macro_namespaced attribute added to current macros which would allow use of this feature now?

As I read it, this is applying to a hypothetical future in which we rewrite macros by example to use a new macro! keyword. But as far as I know, there isn't much of a timeline.

Specifically:

  • A macro with macro_namespaced requires explicit importing and reference per this RFC to be used.
  • All macros that choose not to use it can be used as they are now.
  • All macros that choose not to use it are treated as being in the root of their crate and may be used in the new style as well as the old.

I would like to see this feature before the macro redesign is finished, and this seems like a good compromise that allows for code transition. The disadvantage is that I believe the old macro_rules stuff is on its way out, and obviously this adds something else to it.

The name is off the top of my head. I'm not attached to it.

@withoutboats

This comment has been minimized.

Copy link
Contributor

commented Aug 10, 2016

Maybe I missed it, but the RFC doesn't seem say anything about redefining / shadowing rules for macros in regard to other macros. Currently, the macro rules for this work very differently from the rules for functions. Currently, you can redefine macros whenever you want, and whichever definition is most recent in the source at the expansion site is the definition that is used.

If macros use the same import system as other symbols, I would expect their shadowing behavior to be the same. That is, you can't provide two definitions in the same module or import and define, but you can shadow prelude macros and you can shadow macros by providing a definition inside the body of a function.

(As one last point, I'd sort of like to get rid of explicit mod declarations, at least for private modules, and instead infer their presence from the file-system -- but if we did so, it would interact very poorly with having a defined AST ordering.)

(This is totally offtopic but this seems like a very good idea in the long term which would make Rust's module story more grokkable to users coming from other languages which work this way)

Also this PR needs the final-comment-period label. :-)

@ubsan

This comment has been minimized.

Copy link
Contributor

commented Aug 11, 2016

Agreed with @withoutboats about shadowing.

use foo::bar is much better than use foo::bar!

placing macros into their own namespace is also a good idea; it'd be weird to have macros be in the same namespace as types and values.

@arielb1

This comment has been minimized.

Copy link
Contributor

commented Aug 11, 2016

(As one last point, I'd sort of like to get rid of explicit mod declarations, at least for private modules, and instead infer their presence from the file-system -- but if we did so, it would interact very poorly with having a defined AST ordering.)

But I like explicit mod declarations. The "every crate is 1 file" model is so liberating.

@nikomatsakis

This comment has been minimized.

Copy link
Contributor

commented Aug 11, 2016

Can we get a macro_namespaced attribute added to current macros which would allow use of this feature now?

That's a good question. I would hope we could have some way to use this with old-school macros, whether that require some sort of opt-in, or maybe we can make it work by default without breakage.

@nikomatsakis

This comment has been minimized.

Copy link
Contributor

commented Aug 11, 2016

If macros use the same import system as other symbols, I would expect their shadowing behavior to be the same.

I'm a bit confused by this comment -- are you saying that this is why we can't easily use these new rules for existing macros? Or are you saying that you believe the newer rules in this RFC would cause macros to shadow in different ways than other names that are brought into scope via use? (Or are you referring to the errors that can result if you generate an item via a macro?)

EDIT: Re-reading, I see that you're just saying that the shadowing rules are not explicit enough in the RFC text. I think the answer is that they would follow the same rules as any other items, because we will literally be resolved macro paths using the same lookup rules we use for any other path.

@nrc nrc removed the I-nominated label Aug 11, 2016

@nikomatsakis

This comment has been minimized.

Copy link
Contributor

commented Aug 12, 2016

We discussed this in the @rust-lang/lang meeting and decided to leave it open for FCP a little longer. For one thing, we wanted to incorporate various bits of the discussion in to the RFC text. Nonetheless, I think we're all feeling generally positive about this direction.

@paulstansifer

This comment has been minimized.

Copy link

commented Aug 15, 2016

@nikomatsakis

This comment has been minimized.

Copy link
Contributor

commented Aug 22, 2016

Huzzah! The @rust-lang/lang team has decided to accept this RFC.

@nikomatsakis

This comment has been minimized.

Copy link
Contributor

commented Aug 22, 2016

Tracking issue: rust-lang/rust#35896

If you'd like to keep following the development of this feature, please subscribe to that issue, thanks! :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.