New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: Procedural macros 1.1 #1681

Merged
merged 8 commits into from Aug 22, 2016

Conversation

Projects
None yet
@alexcrichton
Member

alexcrichton commented Jul 18, 2016

Extract a very small sliver of today's procedural macro system in the compiler,
just enough to get basic features like custom derive working, to have an
eventually stable API. Ensure that these features will not pose a maintenance
burden on the compiler but also don't try to provide enough features for the
"perfect macro system" at the same time. Overall, this should be considered an
incremental step towards an official "plugins 2.0".

Rendered

RFC: Macros 1.1
Extract a very small sliver of today's procedural macro system in the compiler,
just enough to get basic features like custom derive working, to have an
eventually stable API. Ensure that these features will not pose a maintenance
burden on the compiler but also don't try to provide enough features for the
"perfect macro system" at the same time. Overall, this should be considered an
incremental step towards an official "plugins 2.0".
@aturon

This comment has been minimized.

Show comment
Hide comment
@aturon

aturon Jul 18, 2016

Member

Suggestion: this might want to be titled Plugins 1.1, as it doesn't really impact the "macro" system (if you think of that as macro_rules, which I think many do).

Member

aturon commented Jul 18, 2016

Suggestion: this might want to be titled Plugins 1.1, as it doesn't really impact the "macro" system (if you think of that as macro_rules, which I think many do).

@aturon

This comment has been minimized.

Show comment
Hide comment
@aturon
Member

aturon commented Jul 18, 2016

@eddyb

This comment has been minimized.

Show comment
Hide comment
@eddyb

eddyb Jul 18, 2016

Member

@aturon I believe this is because @nrc wanted to avoid "plugins" for "procedural macros".

Member

eddyb commented Jul 18, 2016

@aturon I believe this is because @nrc wanted to avoid "plugins" for "procedural macros".

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Jul 18, 2016

Member

@aturon yeah it's true that this only really affects what we call plugins today, but I believe that @nrc wanted to start avoiding the term "plugin" as everything will eventually be a macro in the "macros 2.0" world.

Member

alexcrichton commented Jul 18, 2016

@aturon yeah it's true that this only really affects what we call plugins today, but I believe that @nrc wanted to start avoiding the term "plugin" as everything will eventually be a macro in the "macros 2.0" world.

* Sharing an API of the entire compiler
* Frozen interface between the compiler and macros
### `librustc_macro`

This comment has been minimized.

@eddyb

eddyb Jul 18, 2016

Member

This name is unfortunate, couldn't we use libmacro?

@eddyb

eddyb Jul 18, 2016

Member

This name is unfortunate, couldn't we use libmacro?

This comment has been minimized.

@alexcrichton

alexcrichton Jul 18, 2016

Member

This is currently done because macro is a keyword, and this doesn't propose changing that just yet. I was thinking that we could perhaps wholesale rename the crate one day as well if macro becomes an available name.

@alexcrichton

alexcrichton Jul 18, 2016

Member

This is currently done because macro is a keyword, and this doesn't propose changing that just yet. I was thinking that we could perhaps wholesale rename the crate one day as well if macro becomes an available name.

This comment has been minimized.

@MovingtoMars

MovingtoMars Jul 19, 2016

How about libmacros? This follows the approach that Go takes, ie the strings package.

@MovingtoMars

MovingtoMars Jul 19, 2016

How about libmacros? This follows the approach that Go takes, ie the strings package.

Like the rlib and dylib crate types, the `rustc-macro` crate
type is intended to be an intermediate product. What it *actually* produces is
not specified, but if a `-L` path is provided to it then the compiler will

This comment has been minimized.

@sfackler

sfackler Jul 18, 2016

Member

This feels a bit of an abuse of "link" if we switch plugins to IPC executables - maybe a separate flag would be a good idea?

@sfackler

sfackler Jul 18, 2016

Member

This feels a bit of an abuse of "link" if we switch plugins to IPC executables - maybe a separate flag would be a good idea?

This comment has been minimized.

@alexcrichton

alexcrichton Jul 18, 2016

Member

I believe the intention is that --extern works to load macros (it's what Cargo will pass), so this was just a natural extension of that. We could perhaps add something like -L macro=..., but at this point it doesn't seem strictly necessary as the compiler can always distinguish whether an artifact is a macro or an rlib.

@alexcrichton

alexcrichton Jul 18, 2016

Member

I believe the intention is that --extern works to load macros (it's what Cargo will pass), so this was just a natural extension of that. We could perhaps add something like -L macro=..., but at this point it doesn't seem strictly necessary as the compiler can always distinguish whether an artifact is a macro or an rlib.

This comment has been minimized.

@eddyb

eddyb Jul 18, 2016

Member

I believe -L for rustc means "library search path" not "link search path".

@eddyb

eddyb Jul 18, 2016

Member

I believe -L for rustc means "library search path" not "link search path".

Show outdated Hide outdated text/0000-macros-1.1.md
@aturon

This comment has been minimized.

Show comment
Hide comment
@aturon

aturon Jul 18, 2016

Member

Thanks @alexcrichton for the awesome writeup! We've been working for a while on some way enable usage of Serde and similar libraries on stable in the near term, without massive stabilization. For example, there's a code generation proposal (which sits entirely on the side from the macro/plugin system), as well as attempts to find a very small increment toward "macros 2.0". But this is the first proposal that can be implemented immediately, has a clear path toward speedy stabilization, and imposes virtually no maintenance burden. I'm incredibly excited to be so close to a solution for the biggest cause of nightly usage today.

One thing that wasn't totally obvious to me, given that the macros 2.0 plan is a bit out of cache: what, if any, parts of what you propose to stabilize here would end up being deprecated once macros 2.0 has fully landed?

(To be clear, I have zero problem stabilizing the small slice of functionality you've outlined here in the near term even if in the long term we plan to deprecate it.)

Member

aturon commented Jul 18, 2016

Thanks @alexcrichton for the awesome writeup! We've been working for a while on some way enable usage of Serde and similar libraries on stable in the near term, without massive stabilization. For example, there's a code generation proposal (which sits entirely on the side from the macro/plugin system), as well as attempts to find a very small increment toward "macros 2.0". But this is the first proposal that can be implemented immediately, has a clear path toward speedy stabilization, and imposes virtually no maintenance burden. I'm incredibly excited to be so close to a solution for the biggest cause of nightly usage today.

One thing that wasn't totally obvious to me, given that the macros 2.0 plan is a bit out of cache: what, if any, parts of what you propose to stabilize here would end up being deprecated once macros 2.0 has fully landed?

(To be clear, I have zero problem stabilizing the small slice of functionality you've outlined here in the near term even if in the long term we plan to deprecate it.)

Show outdated Hide outdated text/0000-macros-1.1.md
Show outdated Hide outdated text/0000-macros-1.1.md

@nrc nrc changed the title from RFC: Macros 1.1 to RFC: Procedural macros 1.1 Jul 18, 2016

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Jul 18, 2016

Member

@aturon

what, if any, parts of what you propose to stabilize here would end up being deprecated once macros 2.0 has fully landed?

This is a good question! Of the pieces here, what I would imagine is:

  • If we decide to rename anything, we'd accumulate deprecations for old names.
  • The behavior of "go to a string, parse, then go back to a token stream" will likely be deprecated in favor of directly manipulating the token stream.
  • The definition form of custom #[derive] will likely change and the proposed form here would be deprecated. We likely eventually want one that's imported via use (somehow).

Other than that though I imagine these pieces to survive stabilization:

  • The concept of a "macro" crate type
  • The concept of a "macro crate" provided in the standard distribution, providing all details necessary to talk to the compiler.
  • The ability to inform Cargo of crates which are macro crates
  • Cargo's behavior of compiling macro crates for the host and passing them to the compiler for the next unit of compilation.
  • Transformers like #[derive] operating over token streams and allowing customization via attributes that are stripped during the expansion process
Member

alexcrichton commented Jul 18, 2016

@aturon

what, if any, parts of what you propose to stabilize here would end up being deprecated once macros 2.0 has fully landed?

This is a good question! Of the pieces here, what I would imagine is:

  • If we decide to rename anything, we'd accumulate deprecations for old names.
  • The behavior of "go to a string, parse, then go back to a token stream" will likely be deprecated in favor of directly manipulating the token stream.
  • The definition form of custom #[derive] will likely change and the proposed form here would be deprecated. We likely eventually want one that's imported via use (somehow).

Other than that though I imagine these pieces to survive stabilization:

  • The concept of a "macro" crate type
  • The concept of a "macro crate" provided in the standard distribution, providing all details necessary to talk to the compiler.
  • The ability to inform Cargo of crates which are macro crates
  • Cargo's behavior of compiling macro crates for the host and passing them to the compiler for the next unit of compilation.
  • Transformers like #[derive] operating over token streams and allowing customization via attributes that are stripped during the expansion process
@ubsan

This comment has been minimized.

Show comment
Hide comment
@ubsan

ubsan Jul 18, 2016

Contributor

First, I don't think this RFC has gone through enough design, and fast-tracking it just to get people off of nightly is not a good idea, IMO. However, I am in favor of eventually (soon-ish) getting this sliver.

Some of my specific issues:

First: librustc_macro is a terrible stable name! My vote's in for libmacro. It's not a rustc-specific thing, it's an all-of-Rust thing!

Second: Context seems like an unnecessary implementation detail, and, if they're used the same way as in the original proposal "Procedural macros should report errors, warnings, etc. via the MacroContext [Context]."... why. We already have an existing error reporting mechanism, with Results. This new way seems very un-Rustic.

Third: I don't like how macros are defined. I'd rather they be defined with something like macro instead, especially as we eventually are going to allow proc macros and normal stuff in the same library.

Contributor

ubsan commented Jul 18, 2016

First, I don't think this RFC has gone through enough design, and fast-tracking it just to get people off of nightly is not a good idea, IMO. However, I am in favor of eventually (soon-ish) getting this sliver.

Some of my specific issues:

First: librustc_macro is a terrible stable name! My vote's in for libmacro. It's not a rustc-specific thing, it's an all-of-Rust thing!

Second: Context seems like an unnecessary implementation detail, and, if they're used the same way as in the original proposal "Procedural macros should report errors, warnings, etc. via the MacroContext [Context]."... why. We already have an existing error reporting mechanism, with Results. This new way seems very un-Rustic.

Third: I don't like how macros are defined. I'd rather they be defined with something like macro instead, especially as we eventually are going to allow proc macros and normal stuff in the same library.

@eddyb

This comment has been minimized.

Show comment
Hide comment
@eddyb

eddyb Jul 18, 2016

Member

@ubsan I don't like Context either, but Result won't cut it, macros 2.0 will eventually want a full-blown diagnostics API, where you can emit multiple warnings and non-fatal errors.
If the necessary information is placed in TLS, the API would work without a Context type.

Member

eddyb commented Jul 18, 2016

@ubsan I don't like Context either, but Result won't cut it, macros 2.0 will eventually want a full-blown diagnostics API, where you can emit multiple warnings and non-fatal errors.
If the necessary information is placed in TLS, the API would work without a Context type.

@sfackler

This comment has been minimized.

Show comment
Hide comment
@sfackler

sfackler Jul 18, 2016

Member

@ubsan Result is an all-or-nothing thing. In the compilation process, it is very common to be creating many diagnostics during one compilation pass. A compilation error is not necessarily fatal - you (within reason) want to continue as long as possible to avoid forcing the user to go through a recompilation cycle for every error when many could all be reported at once. In addition, warnings, notes, etc are certainly not fatal.

Member

sfackler commented Jul 18, 2016

@ubsan Result is an all-or-nothing thing. In the compilation process, it is very common to be creating many diagnostics during one compilation pass. A compilation error is not necessarily fatal - you (within reason) want to continue as long as possible to avoid forcing the user to go through a recompilation cycle for every error when many could all be reported at once. In addition, warnings, notes, etc are certainly not fatal.

@arielb1

This comment has been minimized.

Show comment
Hide comment
@arielb1

arielb1 Jul 18, 2016

Contributor

I think librust_macro could be a fine name.

On the integration level: plugins imply some form of host ABI. Stable plugins mean a stable HOSTRUSTC. I think we can just leave HOSTRUSTC as an implementation-defined thing, but we must be careful not to change its definition on existing platforms.

Contributor

arielb1 commented Jul 18, 2016

I think librust_macro could be a fine name.

On the integration level: plugins imply some form of host ABI. Stable plugins mean a stable HOSTRUSTC. I think we can just leave HOSTRUSTC as an implementation-defined thing, but we must be careful not to change its definition on existing platforms.

@ubsan

This comment has been minimized.

Show comment
Hide comment
@ubsan

ubsan Jul 18, 2016

Contributor

@sfackler makes sense. I still don't think Context is the best, though.

Contributor

ubsan commented Jul 18, 2016

@sfackler makes sense. I still don't think Context is the best, though.

@sfackler

This comment has been minimized.

Show comment
Hide comment
@sfackler

sfackler Jul 18, 2016

Member

Do you have any suggestions for a better replacement?

Member

sfackler commented Jul 18, 2016

Do you have any suggestions for a better replacement?

@ubsan

This comment has been minimized.

Show comment
Hide comment
@ubsan

ubsan Jul 18, 2016

Contributor

@sfackler No, but I think @eddyb does :P

Contributor

ubsan commented Jul 18, 2016

@sfackler No, but I think @eddyb does :P

@arielb1

This comment has been minimized.

Show comment
Hide comment
@arielb1

arielb1 Jul 18, 2016

Contributor

On naming: I see librust_macro as an ordinary plugin library, and rustc_macro_derive as an exposed proc macro that (unstable) compiler plugins should eventually be able to reproduce (BTW, what is the nomenclature wrt. proc macro vs. plugin vs. compiler extension?)

rustc_macro_derive is a bad name, but we still don't have modular macros. I think pseudo-modular #[WHATEVER_CRATE_NAME_WE_PICK_derive] would work.

As for the API: maybe have #[CRATE_derive_v0] for the Context-free version?

Contributor

arielb1 commented Jul 18, 2016

On naming: I see librust_macro as an ordinary plugin library, and rustc_macro_derive as an exposed proc macro that (unstable) compiler plugins should eventually be able to reproduce (BTW, what is the nomenclature wrt. proc macro vs. plugin vs. compiler extension?)

rustc_macro_derive is a bad name, but we still don't have modular macros. I think pseudo-modular #[WHATEVER_CRATE_NAME_WE_PICK_derive] would work.

As for the API: maybe have #[CRATE_derive_v0] for the Context-free version?

@sfackler

This comment has been minimized.

Show comment
Hide comment
@sfackler

sfackler Jul 18, 2016

Member

How does shoving Context into TLS hide implementation details?

Member

sfackler commented Jul 18, 2016

How does shoving Context into TLS hide implementation details?

@eddyb

This comment has been minimized.

Show comment
Hide comment
@eddyb

eddyb Jul 18, 2016

Member

@sfackler The TLS slot wouldn't be exposed, the APIs would just not take any context.

Member

eddyb commented Jul 18, 2016

@sfackler The TLS slot wouldn't be exposed, the APIs would just not take any context.

@sfackler

This comment has been minimized.

Show comment
Hide comment
@sfackler

sfackler Jul 18, 2016

Member

@ubsan's concern with Context seemed to be that it was an implementation detail, and I'm confused as to how moving all of the methods on Context to free functions hides any details of the implementation. It seems totally isomorphic in that regard.

Member

sfackler commented Jul 18, 2016

@ubsan's concern with Context seemed to be that it was an implementation detail, and I'm confused as to how moving all of the methods on Context to free functions hides any details of the implementation. It seems totally isomorphic in that regard.

@arielb1

This comment has been minimized.

Show comment
Hide comment
@arielb1

arielb1 Jul 18, 2016

Contributor

Yeah. A Context parameter is easy to ignore and hard to add later.

Contributor

arielb1 commented Jul 18, 2016

Yeah. A Context parameter is easy to ignore and hard to add later.

@nikomatsakis

This comment has been minimized.

Show comment
Hide comment
@nikomatsakis

nikomatsakis Aug 18, 2016

Contributor

@Ericson2314 Regarding namespacing, I see your point about predicting the future being considered difficult; that said, I think the API we are talking about here feels pretty minimal, and we could deprecate it easily enough and find alternative names if it came to that. Put another way, just because something will take a while to implement, doesn't necessarily mean that the big picture design is going to change that much. That said, I could easily see myself eating crow on this point.

A compromise I could imagine would be having the methods be attached to a trait, so we can deprecate the old trait and add a new one.

Contributor

nikomatsakis commented Aug 18, 2016

@Ericson2314 Regarding namespacing, I see your point about predicting the future being considered difficult; that said, I think the API we are talking about here feels pretty minimal, and we could deprecate it easily enough and find alternative names if it came to that. Put another way, just because something will take a while to implement, doesn't necessarily mean that the big picture design is going to change that much. That said, I could easily see myself eating crow on this point.

A compromise I could imagine would be having the methods be attached to a trait, so we can deprecate the old trait and add a new one.

@nrc

This comment has been minimized.

Show comment
Hide comment
@nrc

nrc Aug 18, 2016

Member

I'm in favour of this. I'm not sure how I feel about the namespacing issue, I'd like to see how this looks when implemented.

Member

nrc commented Aug 18, 2016

I'm in favour of this. I'm not sure how I feel about the namespacing issue, I'd like to see how this looks when implemented.

@nikomatsakis

This comment has been minimized.

Show comment
Hide comment
@nikomatsakis

nikomatsakis Aug 22, 2016

Contributor

Huzzah! The @rust-lang/lang team has decided to accept this RFC. We decided to ask @alexcrichton to move some of the detailed matters (e.g., prefixes, etc) into unresolved questions to be addressed during stabilization. Thanks everyone!

Contributor

nikomatsakis commented Aug 22, 2016

Huzzah! The @rust-lang/lang team has decided to accept this RFC. We decided to ask @alexcrichton to move some of the detailed matters (e.g., prefixes, etc) into unresolved questions to be addressed during stabilization. Thanks everyone!

@nikomatsakis nikomatsakis referenced this pull request Aug 22, 2016

Closed

Tracking issue for "Macros 1.1" (RFC #1681) #35900

50 of 53 tasks complete

@nikomatsakis nikomatsakis merged commit ac22573 into rust-lang:master Aug 22, 2016

@nikomatsakis

This comment has been minimized.

Show comment
Hide comment
@nikomatsakis

nikomatsakis Aug 22, 2016

Contributor

Tracking issue: rust-lang/rust#35900

If you'd like to keep following the development of this feature, please subscribe to that issue, thanks! :)

Contributor

nikomatsakis commented Aug 22, 2016

Tracking issue: rust-lang/rust#35900

If you'd like to keep following the development of this feature, please subscribe to that issue, thanks! :)

nikomatsakis added a commit that referenced this pull request Aug 22, 2016

@llogiq

This comment has been minimized.

Show comment
Hide comment
@llogiq

llogiq Aug 22, 2016

Contributor

I still miss my comment about the inferior traceability of code through macros 1.1 as opposed to the current systems. Could we at least add it to the drawbacks section?

Contributor

llogiq commented Aug 22, 2016

I still miss my comment about the inferior traceability of code through macros 1.1 as opposed to the current systems. Could we at least add it to the drawbacks section?

@nikomatsakis

This comment has been minimized.

Show comment
Hide comment
@nikomatsakis

nikomatsakis Aug 22, 2016

Contributor

@llogiq I pushed some text about spans

Contributor

nikomatsakis commented Aug 22, 2016

@llogiq I pushed some text about spans

@llogiq

This comment has been minimized.

Show comment
Hide comment
@llogiq

llogiq Aug 22, 2016

Contributor

Ok then. I'll keep an eye on the implementation. 😉

Contributor

llogiq commented Aug 22, 2016

Ok then. I'll keep an eye on the implementation. 😉

untitaker added a commit to untitaker/rfcs that referenced this pull request Sep 11, 2016

nrc added a commit that referenced this pull request Sep 11, 2016

hauleth added a commit to rust-num/num that referenced this pull request Sep 18, 2016

Fix `num-macros` `FromPrimitive` implementation
Current solution follow syntax proposed in rust-lang/rfcs#1681.

Tracking issue rust-lang/rust#35900

Close #227

@hauleth hauleth referenced this pull request Sep 18, 2016

Merged

Fix `num-macros` `FromPrimitive` implementation #228

0 of 7 tasks complete

hauleth added a commit to rust-num/num that referenced this pull request Sep 18, 2016

Fix `num-macros` `FromPrimitive` implementation
Current solution follow syntax proposed in rust-lang/rfcs#1681.

Tracking issue rust-lang/rust#35900

Close #227

homu added a commit to rust-num/num that referenced this pull request Sep 30, 2016

phaazon pushed a commit to phaazon/rfcs that referenced this pull request Feb 10, 2017

phaazon pushed a commit to phaazon/rfcs that referenced this pull request Feb 10, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment