Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue for "Macros 1.1" (RFC #1681) #35900

Closed
nikomatsakis opened this issue Aug 22, 2016 · 268 comments
Closed

Tracking issue for "Macros 1.1" (RFC #1681) #35900

nikomatsakis opened this issue Aug 22, 2016 · 268 comments

Comments

@nikomatsakis
Copy link
Contributor

@nikomatsakis nikomatsakis commented Aug 22, 2016

Tracking issue for rust-lang/rfcs#1681.

cc @alexcrichton

Stabilization TODO

Litmus tests:

Features:

  • Crate name, currently is proc_macro
  • Crate type, currently proc-macro
  • The #[proc_macro_derive(Foo)] attribute
  • Loading proc-macro crates with -L and #[macro_use] to load them
  • shadowing is an error
  • no hygiene
  • passing a token stream for the entire struct and receiving it all back
  • Cargo manifest attribute, currently proc-macro = true

Known bugs:

Implementation TODO

  • - Create a rustc_macro crate
    • - Have librustc_macro link to libsyntax. Depend on librustc_macro in librustc_driver
    • - Tag rustc_macro as unstable with our standard header.
    • - Only tag rustc_macro with #![crate_type = "rlib"], do not produce a dylib.
    • - Implement the API of rustc_macro using libsyntax's TokenStream internally
    • - tag rustc_macro with a TokenStream lang item so the compiler knows about it.
  • - Add rustc_macro_derive attribute
    • - validate it's of the exact form foo(bar), no other arguments/formats
    • - verify it's only applied to functions
    • - verify it's only applied to functions in the root module
    • - verify the signature with the TokenStream lang item added above
    • - encode all function symbols with rustc_macro_derive into metadata along with the derive mode they're used for.
  • - Add a rustc-macro crate type for other crates
    • - wire it up to produce a dylib
    • - ensure the dylib gets metadata
    • - ensure rustc-macro crates cannot be linked as dylibs
    • - ensure there are no reachable items other than those tagged with #[rustc_macro_derive]
    • - Add cfg(rustc_macro) as an unstable cfg directive, set it for the rustc-macro crate type
    • - ensure that rustc-macro crates link dynamically to libsytnax
  • - Fill in #[macro_use] support for rustc-macro crates
    • - extend library loader to find rustc-macro crates separately from dylib/rlib tracked today when loding crates
    • - Parse metadata for rustc-macro crates to learn about symbol/derive mode pairings
    • - dlopen the dylib
    • - generate an error if any derive mode would shadow any other.
  • - Add cargo integeration
    • - recognize rustc-macro similar to plugin = true
    • - pass --crate-type=rustc-macro when depending on it
    • - plumb the same host/target logic for rustc-macro crates as is present for plugin crates (e.g. always compile rustc-macro crates for hosts)
  • - Tests
    • - smoke test loading a rustc-macro, dummy #[derive] trait
    • - name conflicts are an error
    • - compiling for the wrong architecture is an error
    • - cannot emit crate type rustc-macro and anything else (e.g. rustc-macro+dylib) is an error
    • - span information not being horrendous
    • - removing attributes from a struct, as well as fields
    • - adding impls next to a struct
    • - cross compilation looks for the host rustc-macro crate type
    • - don't load vanilla dylibs as rustc-macro crate types
    • - can't have public exports beyond macro derive functions in a rustc-macro crate
    • - derive macros must have required signature
    • - load two macro crates in one compilation
@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Aug 22, 2016

I've updated the issue description with a checklist of what needs to be done. It's likely not exhaustive but it should get us 90% of the way there hopefully

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Aug 22, 2016

Ok, I'm going to look into this today and see how far I get.

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Aug 24, 2016

cc #35957, an implementation

@nrc
Copy link
Member

@nrc nrc commented Aug 29, 2016

From #35957: we should bikeshed the name of the librustc_macro crate some more. In particular, this is intended to be a long-lived crate which will have essentials for all macro authors in it, so limiting to rustc_macro (which in my mind, at least) is just about the 1.1 idea seems bad. I'd previously wanted libmacro for this, but since macro is a reserved word (and we might want it as a keyword in the future) that is impossible. @cgswords and I have been using libproc_macro, and I think that is not a bad name, although I am not 100% happy with it.

@eternaleye
Copy link
Contributor

@eternaleye eternaleye commented Aug 29, 2016

@nrc: Okay, my immediate thoughts on naming:

  • extern crate macros; - short and sweet, but might be read as containing macros, rather than support code for writing them
  • extern crate macro_runtime; - exactly what it says on the tin
  • extern crate metarust; - write Rust, about Rust, for operating on Rust
  • extern crate bikeshed; - with procedural macros, you can have whatever color of Rust you want!
  • extern crate macrame; - sounds like "macro make[r]"; possibly better left for a future "nice" API over the raw library.
@aturon
Copy link
Member

@aturon aturon commented Aug 30, 2016

@nrc It seems like an important aspect of this question is the naming of our various macro styles over all -- in particular, if we go with libproc_macro, we're making a hard commitment to the "procedural macro" terminology. I don't have a strong opinion here, but I'm not sure if we've openly explored the space of terminology.

To be clear, are you thinking that today's macro_rules would simply be "macros", i.e. the default thing we mean by macros, while you have to qualify "procedural macros"? That seems like a pretty reasonable plan to me. And in that world, I'd argue that libproc_macro is better than libmacros.

@nrc
Copy link
Member

@nrc nrc commented Aug 30, 2016

My thinking here is that all macros are "macros", and when we need to make a distinction based on the implementation (the usage should be exactly the same for all kinds) we use "procedural macros" vs "macros by example". I would like to banish "syntax extension" and "compiler plugin" entirely (and one day re-use the latter for actual plugins).

But, yes, I strongly want to get behind the "procedural macro" terminology.

@aturon
Copy link
Member

@aturon aturon commented Aug 30, 2016

@nrc Makes good sense to me! While "macros by example" is a bit unwieldy, it's also a very evocative/intuitive term. My one worry about "procedural macro" is that "procedure" is not a thing in Rust. I wonder if there's a way of making the connection more direct. "Function macro" isn't quite right, but maybe gives you a sense of what I mean?

@nrc
Copy link
Member

@nrc nrc commented Aug 30, 2016

Yeah, it's not quite perfect, but given that it is a well-known/used term outside of Rust I think it is better than coining our own term. "Programmatic macro" is possible, but I prefer "procedural".

@eternaleye
Copy link
Contributor

@eternaleye eternaleye commented Aug 30, 2016

Perl's term for the closest equivalent it has to "procedural macros" is "source filters", which (especially with the shift from AST to tokens) is a pretty fitting description.

@mystor
Copy link
Contributor

@mystor mystor commented Aug 30, 2016

Perhaps 'Syntax transformers' or 'programmatic macros' would work well as names? I don't have a problem with procedural macros however.

@jimmycuadra
Copy link
Contributor

@jimmycuadra jimmycuadra commented Aug 31, 2016

"Procedural" is already what people call this, and I think it's clearly understood what it means, especially in contrast to "macros by example." I wouldn't worry about trying to find a different name.

@nikomatsakis
Copy link
Contributor Author

@nikomatsakis nikomatsakis commented Aug 31, 2016

I like the term "procedural macro" for regular use (or maybe "custom macro"). I particularly like using the word macro so that it's clear that they can (eventually...) be used in the same way as "regular macros". For this reason, I don't like "source filters" (I also expect a filter to just drop data, not transform it, though I know the term is used for both).

I am fine with either libproc_macro or libmacros. I sort of prefer the latter just because I don't love having _ in crate names when it can be readily avoided. =)

@nikomatsakis
Copy link
Contributor Author

@nikomatsakis nikomatsakis commented Aug 31, 2016

One question: do we ever expect to have "support routines" that could be used from non-procedural macros? I know of no such plans, but if we did, and we wanted them in the same crate, then libmacros would be a better name. =)

(I am thinking a bit about e.g., @dherman's comment from the eager-expansion RFC.)

@eternaleye
Copy link
Contributor

@eternaleye eternaleye commented Aug 31, 2016

@nikomatsakis: A related but subtly different question is a use case I brought up in rust-lang/rfcs#1561 (comment) - will we want procedural macros' implementation functions to be able to call other procedural macros' implementation functions?

I can easily see wanting to allow one custom derive to call another, and that would essentially make procedural macro definitions themselves capable of being used as such "support routines"

But yes, I think @dherman's gensym example is pretty compelling. Of course, if the answer to my question above is "yes", gensym is both a macro (which could be used by macros-by-example) and a support function (which could be used by procedural macros).

@dtolnay
Copy link
Member

@dtolnay dtolnay commented Sep 1, 2016

I have a cargo PR rust-lang/cargo#3064 which should check all the "cargo integration" boxes in the checklist.

@Ericson2314
Copy link
Contributor

@Ericson2314 Ericson2314 commented Sep 1, 2016

I left a comment on the cargo PR, but I think we want a different type of dependency, not just a different type of package. Firstly, I think this just is just better aestheticly and ergnomicly, but that's just my opinion. But I have two concrete reasons too.

  • In a future with public and private deps, it's important to know that its to know that public deps of procedural deps and regular public deps need not coincide. E.g.

  • In a future with inline procedural macros/quasi-quoting, any library can be used at compile time.

  • will we want procedural macros' implementation functions to be able to call other procedural macros' implementation functions?

    As this shows the same crate could be a regular dep of another crate of procedural macros, or a macro dep of another crate.

@dtolnay
Copy link
Member

@dtolnay dtolnay commented Sep 2, 2016

Does serde work?

Yes https://github.com/serde-rs/serde/releases/tag/v0.8.6

(except for container attributes in some cases #36211)

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Sep 2, 2016

Awesome, thanks for the updates @dtolnay!

@White-Oak
Copy link
Contributor

@White-Oak White-Oak commented Sep 2, 2016

Are there docs on these macros? I suppose, the only example of using them is in serde, am I right?

nikomatsakis added a commit to nikomatsakis/rust that referenced this issue Jan 6, 2017
This commit stabilizes the `proc_macro` and `proc_macro_lib` features in the
compiler to stabilize the "Macros 1.1" feature of the language. Many more
details can be found on the tracking issue, rust-lang#35900.

Closes rust-lang#35900
nikomatsakis added a commit to nikomatsakis/rust that referenced this issue Jan 6, 2017
These are some bare-bones documentation for custom derive, needed
to stabilize "macros 1.1",
rust-lang#35900

The book chapter is based off of a blog post by @cbreeden,
https://cbreeden.github.io/Macros11/

Normally, we have a policy of not mentioning external crates in
documentation. However, given that syn/quote are basically neccesary
for properly using macros 1.1, I feel that not including them here
would make the documentation very bad. So the rules should be bent
in this instance.
nikomatsakis added a commit to nikomatsakis/rust that referenced this issue Jan 6, 2017
This commit stabilizes the `proc_macro` and `proc_macro_lib` features in the
compiler to stabilize the "Macros 1.1" feature of the language. Many more
details can be found on the tracking issue, rust-lang#35900.

Closes rust-lang#35900
nikomatsakis added a commit to nikomatsakis/rust that referenced this issue Jan 6, 2017
These are some bare-bones documentation for custom derive, needed
to stabilize "macros 1.1",
rust-lang#35900

The book chapter is based off of a blog post by @cbreeden,
https://cbreeden.github.io/Macros11/

Normally, we have a policy of not mentioning external crates in
documentation. However, given that syn/quote are basically neccesary
for properly using macros 1.1, I feel that not including them here
would make the documentation very bad. So the rules should be bent
in this instance.
nikomatsakis added a commit to nikomatsakis/rust that referenced this issue Jan 6, 2017
This commit stabilizes the `proc_macro` and `proc_macro_lib` features in the
compiler to stabilize the "Macros 1.1" feature of the language. Many more
details can be found on the tracking issue, rust-lang#35900.

Closes rust-lang#35900
nikomatsakis added a commit to nikomatsakis/rust that referenced this issue Jan 6, 2017
These are some bare-bones documentation for custom derive, needed
to stabilize "macros 1.1",
rust-lang#35900

The book chapter is based off of a blog post by @cbreeden,
https://cbreeden.github.io/Macros11/

Normally, we have a policy of not mentioning external crates in
documentation. However, given that syn/quote are basically neccesary
for properly using macros 1.1, I feel that not including them here
would make the documentation very bad. So the rules should be bent
in this instance.
@tomaka
Copy link
Contributor

@tomaka tomaka commented Jan 13, 2017

If the user writes #[derive(Foo)] #[foo_def = "definition.json"] struct MyStruct; the macro handler has no way to know what "the current directory" is and thus can't find definition.json.

This is by-design and thus wouldn't be easy to fix, and I guess it's too late to fix this anyway.

@eddyb
Copy link
Member

@eddyb eddyb commented Jan 13, 2017

You can go Span -> FileMap -> filename -> directory. All that's missing is accessing the information through proc_macro.

@tomaka
Copy link
Contributor

@tomaka tomaka commented Jan 13, 2017

You would also need to tell the compiler to add a dependency to definition.json so that the build is dirty if it gets modified.

@dtolnay
Copy link
Member

@dtolnay dtolnay commented Jan 13, 2017

The proc macro can use env::var("CARGO_MANIFEST_DIR") to get the directory containing Cargo.toml of the crate containing the macro invocation. Presumably foo_def is relative to that. See dtolnay/syn#70 (comment).

@est31
Copy link
Contributor

@est31 est31 commented Jan 13, 2017

@tomaka that can be done by mutating FileMap, e.g. this is how the include_str! macro does it.

@tomaka
Copy link
Contributor

@tomaka tomaka commented Jan 13, 2017

Presumably foo_def is relative to that.

I think it's not very intuitive to have to put the path relative to the Cargo.toml.

@tomaka
Copy link
Contributor

@tomaka tomaka commented Jan 13, 2017

that can be done by mutating FileMap, e.g. this is how the include_str! macro does it.

Yeah I know it can be done, it just can't be done with the current procedural macros API.

The parameter is an Item. It would be acceptable to add a method to retrieve a span from that Item, but adding a method to Item to ask the compiler to add a dependency would be a hack IMO.

@SimonSapin
Copy link
Contributor

@SimonSapin SimonSapin commented Jan 13, 2017

You can go Span -> FileMap -> filename -> directory.

Are these APIs (in particular FileMap) on a path to be stabilized?

@eddyb
Copy link
Member

@eddyb eddyb commented Jan 14, 2017

They don't have to be, in fact I wouldn't want to stabilize any of the internals. We can, instead, stabilize APIs that extract information about a Span (e.g. line, column, filename).

@alexreg
Copy link
Contributor

@alexreg alexreg commented Feb 2, 2017

I just got this error in a crate of mine. What's going on?

error: Cannot use `#![feature(proc_macro)]` and `#![feature(custom_attribute)] at the same time
@abonander
Copy link
Contributor

@abonander abonander commented Feb 2, 2017

@alexreg If you're using #[derive], it's stable now. You don't need #![feature(proc_macro)].

@jseyfried
Copy link
Contributor

@jseyfried jseyfried commented Feb 2, 2017

@alexreg
proc_macro_derives (macros 1.1) are now stable -- you can just remove #![feature(proc_macro)].

#[proc_macro_attribute] recently landed behind the #![feature(proc_macro)] feature gate; these are incompatible with #![feature(custom_attribute)]. #![feature(custom_attribute)] will be deprecated once the replacement lands (rust-lang/rfcs#1755).

@abonander
Copy link
Contributor

@abonander abonander commented Feb 2, 2017

@jseyfried I think we should change the tracking issue on proc_macro since this is closed and doesn't contain relevant information.

@alexreg
Copy link
Contributor

@alexreg alexreg commented Feb 2, 2017

Thanks guys. That makes sense.

@jseyfried
Copy link
Contributor

@jseyfried jseyfried commented Feb 2, 2017

@abonander Yeah, #![feature(proc_macro)] should definitely point to #38356 now -- I should have verified that when reviewing #38842. Could you submit a PR?

phansch added a commit to phansch/rust-clippy that referenced this issue Mar 26, 2018
`empty_line_after_outer_attribute` produced a false positive warning when
deriving `Copy` and/or `Clone` for an item.

It looks like the second point in [this comment][that_comment] is related,
as the attribute that causes the false positive has a path of
`rustc_copy_clone_marker`.

Fixes rust-lang#2475

[that_comment]: rust-lang/rust#35900 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.