Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Procedural macros #1566

Merged
merged 3 commits into from Dec 13, 2016
Merged

Procedural macros #1566

merged 3 commits into from Dec 13, 2016

Conversation

@nrc
Copy link
Member

nrc commented Apr 1, 2016

This RFC proposes an evolution of Rust's procedural macro system (aka syntax
extensions, aka compiler plugins). This RFC specifies syntax for the definition
of procedural macros, a high-level view of their implementation in the compiler,
and outlines how they interact with the compilation process.

At the highest level, macros are defined by implementing functions marked with
a #[macro] attribute. Macros operate on a list of tokens provided by the
compiler and return a list of tokens that the macro use is replaced by. We
provide low-level facilities for operating on these tokens. Higher level
facilities (e.g., for parsing tokens to an AST) should exist as library crates.

This RFC proposes an evolution of Rust's procedural macro system (aka syntax
extensions, aka compiler plugins). This RFC specifies syntax for the definition
of procedural macros, a high-level view of their implementation in the compiler,
and outlines how they interact with the compilation process.

At the highest level, macros are defined by implementing functions marked with
a `#[macro]` attribute. Macros operate on a list of tokens provided by the
compiler and return a list of tokens that the macro use is replaced by. We
provide low-level facilities for operating on these tokens. Higher level
facilities (e.g., for parsing tokens to an AST) should exist as library crates.
When a `#[cfg(macro)]` crate is `extern crate`ed, it's items (even public ones)
are not available to the importing crate; only macros declared in that crate.
The crate is dynamically linked with the compiler at compile-time, rather
than with the importing crate at runtime.

This comment has been minimized.

@oli-obk

oli-obk Apr 1, 2016 Contributor

This should come in hand with a warning/lint about adding public items in a #[cfg(macro)] crate


Which delimiter was used should be available to the macro implementation via the
`MacroContext`. I believe this is maximally flexible - the macro implementation
can throw an error if it doesn't like the delimiters used.

This comment has been minimized.

@oli-obk

oli-obk Apr 1, 2016 Contributor

I prefer this over hiding it from the macro implementor. I personally think that things like vec!{1, 2, 3} and vec!(5, 6, 7) should be forbidden or at least linted against (for backwards compat).

@jimmycuadra
Copy link

jimmycuadra commented Apr 1, 2016

Rendered

(I am SOOOOO excited about this. :D)

## Tokens

Procedural macros will primarily operate on tokens. There are two main benefits
to this principal: flexibility and future proofing. By operating on tokens, code

This comment has been minimized.

@steveklabnik

steveklabnik Apr 1, 2016 Member

nit: principle

@steveklabnik
Copy link
Member

steveklabnik commented Apr 1, 2016

I am also psyched to see movement on this 😄


```
#[macro]
pub fn foo(TokenStream, &mut MacroContext) -> TokenStream;

This comment has been minimized.

@seanmonstar

seanmonstar Apr 1, 2016 Contributor

Alternative: use the macro keyword (it was reserved in 1.0). It would make it feel more part of the language, in my opinion.

pub macro foo(TokenStream, &mut MacroContext) -> TokenStream;

This comment has been minimized.

@pczarn

pczarn Apr 4, 2016

What happens to conflicting names of function/attribute-like macros?

#[macro]
pub fn foo(TokenStream, &mut MacroContext) -> TokenStream;
#[macro_attribute]
pub fn foo(Option<TokenStream>, TokenStream, &mut MacroContext) -> TokenStream;

One possibility is allowing the above, and another is providing access to all data through MacroContext:

#[macro]
#[macro_attribute]
pub fn foo(context: &mut MacroContext) -> TokenStream {
    let tokens = context.token_stream();
    if let Attribute(params) = context.macro_kind() {
        // ...
    } else {
        // ...
    }
}

This comment has been minimized.

@nrc

nrc Apr 5, 2016 Author Member

Using macro is appealing, but I'm not sure what we would do for attribute-like macros or macros with an extra ident. Should we depend just on the signature? That feels a bit fragile to me, but might work.

# Detailed design
[design]: #detailed-design

There are two kinds of procedural macro: function-like and macro-like. These two

This comment has been minimized.

@sfackler

sfackler Apr 1, 2016 Member

"macro-like and attribute-like"?

This comment has been minimized.

@pczarn

pczarn Apr 4, 2016

also, "of procedural macros"

I think we could first design the syntax, interfaces, etc. and later evolve into
a process-separated model (if desired). However, if this is considered an
essential feature of macro reform, then we might want to consider the interfaces
more thoroughly with this in mind.

This comment has been minimized.

@sfackler

sfackler Apr 1, 2016 Member

I would very much like to move towards this kind of setup, but I think that the interface proposed here should work just fine in that world so we shouldn't necessarily block on figuring this out right now.

This comment has been minimized.

@eddyb

eddyb Apr 2, 2016 Member

We might have to do clever things like use shared memory alongside message-based IPC to get the most out of such a model, but I believe it would be worth it.

Dynamic linking for plugin systems works in practice but just making it safe to unload said plugins is an entire area of design Rust hasn't even touched yet (although lifetimes would play a very important role).

I would rather have an IPC solution that lets me:

  • download a single binary and have a working cross-compiler without even system dependencies (static linking against musl on linux)
  • safely unload plugins after expansion without introducing complexity into the language
  • use a forking zygote to share most of the plugins across compilations (via cargo or RLS)
    • EDIT: This technique can also be used to (relatively) cheaply erase any per-thread/process state in between expansions of the same macro, if we want to completely deny that (it would make order dependence much harder, I think) - since we control the compilation of the macro crate, we could just ban statics altogether, but that doesn't account for them messing with pthreads on their own
  • implement my own libmacro in an alternative Rust compiler
  • more exciting: implement my own libmacro for a Rust tool which is not a full compiler
}
pub enum TokenKind {
Sequence(Delimiter, Vec<TokenTree>),

This comment has been minimized.

@eddyb

eddyb Apr 2, 2016 Member

Shouldn't this be Sequence(Delimiter, TokenStream)?

This comment has been minimized.

@pczarn

pczarn Apr 4, 2016

What's the point of having delimited sequences? If users are expected to use external libraries to parse AST, surely they can use these libraries to find and match delimiters.

This comment has been minimized.

@nrc

nrc Apr 5, 2016 Author Member

We have to tokenise into sequences to parse patterns for macros, so I think the compiler must do it. It is then convenient for macro authors to have that info. Since we can't be more flexible by not doing it, I don't think there is an advantage of keeping that info private. Furthermore, it seems like a win for the macro author to see what the compiler sees in terms of why it is being passed the data it is.

Finally, using un-delimited sequences is useful for macros to affect precedence without introducing scopes.

This comment has been minimized.

@eddyb

eddyb Apr 5, 2016 Member

IMO the biggest win is making capturing a sequence O(1), assuming a representation which can reuse TokenStreams without copying all the tokens.
Such an example would be a macro taking a large function body but not inspecting it by itself.

// Not actually sure if we need this or if semicolons can be treated like
// other punctuation.
Semicolon, // `;`
Eof,

This comment has been minimized.

@eddyb

eddyb Apr 2, 2016 Member

Do we really need EOF, instead of relying on the TokenStream ending?

This comment has been minimized.

@nrc

nrc Apr 5, 2016 Author Member

Good question. I'm not sure. Given that a TokenSteam could be internal, it might be useful to know if a stream is ending because of a close delimiter or an EOF. On the other hand, I can't think of a use case for that and it might be better for macro authors not to know about file boundaries.

This comment has been minimized.

@Ericson2314

Ericson2314 Apr 5, 2016 Contributor

it might be better for macro authors not to know about file boundaries.

Definitely!

pub enum StringKind {
Regular,
// usize is for the count of `#`s.
Raw(usize),

This comment has been minimized.

@eddyb

eddyb Apr 2, 2016 Member

Maybe use struct variants? Also, it seems string / byte-string is orthogonal to "rawness".

// Word is defined by Unicode Standard Annex 31 -
// [Unicode Identifier and Pattern Syntax](http://unicode.org/reports/tr31/)
Word(InternedString),
Punctuation(char),

This comment has been minimized.

@eddyb

eddyb Apr 2, 2016 Member

What about <<, >>, ->, <-, and all the compound assign operators?
For example, the following compiles:

macro_rules! just_one_token { ($x:tt) => {} }
just_one_token!(->);
just_one_token!(<<);
just_one_token!(+=);
just_one_token!(>>=);

This comment has been minimized.

@nrc

nrc Apr 5, 2016 Author Member

Hmm, I guess it is better to take multiple chars and maintain backwards compatibility here. I had wanted to avoid macros having to split tokens (e.g., << into < < in generics). Combining them where necessary feels less annoying. I'll add some discussion to the RFC.

This comment has been minimized.

@eddyb

eddyb Apr 5, 2016 Member

Could be useful to have both a plain iterator and a Parser-like API for consuming punctuation tokens (which would split << into < < for the procedural macro) but that requires the lookahead buffer.

Dollar, // `$`
// Not actually sure if we need this or if semicolons can be treated like
// other punctuation.
Semicolon, // `;`

This comment has been minimized.

@eddyb

eddyb Apr 2, 2016 Member

Can't we use Punctuation for all 3 of these?

This comment has been minimized.

@nrc

nrc Apr 5, 2016 Author Member

I'm not really sure, there does seem to be some advantage to making these special. However, I think it is probably a convenience rather than essential. I think I would start implementing and maybe change this later.

pub enum CommentKind {
Regular,
InnerDoc,
OuterDoc,

This comment has been minimized.

@eddyb

eddyb Apr 2, 2016 Member

Don't we expand doc comments into attributes nowadays? I recall being able to use them with macros taking attributes.

This comment has been minimized.

@pczarn

pczarn Apr 4, 2016

Yes, doc comments become doc attributes in macro input.

This comment has been minimized.

@nrc

nrc Apr 5, 2016 Author Member

Yeah. It is gross and hackey and I have no idea why we even have doc comment attributes. I would like to kill them in the surface syntax, but that seems a bit unlikely given the discussion on that RFC :-(

I would prefer to have explicit doc comments. Is there a reason to treat them like attributes (in macros I mean, I realise that the compiler/rustdoc wants them to be attributes).

This comment has been minimized.

@eddyb

eddyb Apr 5, 2016 Member

You can use them in macro_rules macros with just generic attribute pass-through.

Currently, procedural macros are dynamically linked with the compiler. This
prevents the compiler being statically linked, which is sometimes desirable. An
alternative architecture would have procedural macros compiled as independent
programs and have them communicate with the compiler via IPC.

This comment has been minimized.

@eddyb

eddyb Apr 2, 2016 Member

I think we can start by running each plugin's registrar and all expansion in a thread dedicated to that plugin.
This will avoid accidental dependence on thread-local state (which we have some of, most notably the string interner).

If we have a &mut context, we should be able to temporarily "transfer ownership" to the expanding thread, without the context even implementing Sync, just Send.

This comment has been minimized.

@DemiMarie

DemiMarie Jun 7, 2016

Another advantage of the IPC solution (which OCaml has chosen for its own syntax extensions) is that crashes in the plugin don't crash the compiler. Furthermore, should it become desirable, the plugin could be run in a process with reduced OS-level privileges.

Both procedural macros and constant evaluation are mechanisms for running Rust
code at compile time. Currently, and under the proposed design, they are
considered completely separate features. There might be some benefit in letting
them interact.

This comment has been minimized.

@eddyb

eddyb Apr 2, 2016 Member

The main problem with mixing macros and constants is that if you somehow run impure procedural macro code for each monomorphization of a generic type, you generally cannot enforce determinism so safe code can break coherence and possibly even type safety (not that unsafe should let you do such things either).

OTOH, I would welcome a TokenStream -> Result<TokenStream, ...> method on MacroContext for compiling just a constant and outputting its evaluated form (if available - ADT "trees" with primitive literals as leaves should work), but I don't have a specific usecase in mind right now.

This comment has been minimized.

@9il

9il Jan 6, 2018

Hi, I am looking for D's static if analog in Rust. It is required to implement fast full featured multidimensional arrays in Rust ndarray cargo package is slower then D's ndslice. Rust is going to have constant templates parameters. Mixing macros and constants allows to implement macro like static_if!(ctfe_cond, codeBlock1, codeBlock2) if I am not wrong, plus it will make Rust the most generic system programming language ever.

// The content of the comment can be found from the span.
Comment(CommentKind),
// The Span is the span of the string itself, without delimiters.
String(Span, StringKind),

This comment has been minimized.

@eddyb

eddyb Apr 2, 2016 Member

What if the plugin wants to create a string literal which is unrelated to anything found in the source?

This comment has been minimized.

@eddyb

eddyb Apr 2, 2016 Member

Also, seems to be missing numeric and character literals.

This comment has been minimized.

@nrc

nrc Apr 5, 2016 Author Member

Hmm, yeah, I guess we need to be able to do that. Shame, would be nice to avoid allocating a String in the common case.

Can we get away with treating numeric literals as Words?

This comment has been minimized.

@eddyb

eddyb Apr 5, 2016 Member

I would be against allocating a String myself. Having a common Symbol type for both string literals and "words" (not unlike the current implementation?) seems optimal.

I agree with treating numeric literals as something else (and making, e.g. i64::parse easy to use - except that doesn't handle extra _, does it?), as long as you have a solution for literal suffixes which fits the current semantics.

// Word is defined by Unicode Standard Annex 31 -
// [Unicode Identifier and Pattern Syntax](http://unicode.org/reports/tr31/)
Word(InternedString),

This comment has been minimized.

@eddyb

eddyb Apr 2, 2016 Member

Exposing InternedString makes me uneasy - the fact that we intern these is an implementation detail, although it does affect whether the API for accessing the contents involves MacroContext.

We could get away with either an interned index with the interner in TLS, or RC+SSO or some other strange combination, if we don't want to involve the MacroContext (which is the real decision to be made).

Still, this could probably use a less specific name, such as Symbol or even Word.

This comment has been minimized.

@nrc

nrc Apr 5, 2016 Author Member

I assumed we would use the MacroContext to access the string. The MacroContext should be ubiquitous, so I don't think that is a problem. Might be best to change the name from InternedString to Symbol or something. We could then use where there are string literals too without prejudicing the implementation.

This comment has been minimized.

@eddyb

eddyb Apr 5, 2016 Member

Sounds good to me, as long as it doesn't get in your way.

pub struct TokenTree {
pub kind: TokenKind,
pub span: Span,
pub hygiene: HygieneObject,

This comment has been minimized.

@eddyb

eddyb Apr 2, 2016 Member

Might want to consider combining the span and the hygiene information - couldn't the expansion traces be deduced from hygiene scopes?

This comment has been minimized.

@Zoxc

Zoxc Apr 2, 2016

How will we create TokenTrees?

This comment has been minimized.

@eddyb

eddyb Apr 2, 2016 Member

I'd start with better quasi-quoting and see what else we need from there.
One worry I have about Sequence is that it might be expensive to convert into the optimized representation.
Given that the average sequence has 8 tokens, it would be a waste to keep an actual tree in memory, when a lot of cases could fit in the same space the pointers take right now.

This comment has been minimized.

@nrc

nrc Apr 5, 2016 Author Member

Hmm, it might be a good idea to combine Spans and HygieneObject here. There have different roles, but are kind of two sides of the same coin. Spans are informative and transparent. HygieneObjects are meant to be normative and opaque. I can imagine that macro authors might want to change them independently, but the common case definitely would be to operate on both at the same time.

This comment has been minimized.

@nrc

nrc Apr 5, 2016 Author Member

I'll cover creating token trees in an upcoming RFC

This comment has been minimized.

@eddyb

eddyb Apr 6, 2016 Member

How does Origin sound as a name for the combined span & hygiene info?

pub struct TokenStream(Vec<TokenTree>);
// A borrowed TokenStream
pub struct TokenSlice<'a>(&'a [TokenTree]);

This comment has been minimized.

@eddyb

eddyb Apr 2, 2016 Member

These two types might be better served by Cow<'a, [TokenTree]> - or if an optimized representation such as "reference-counted ropes of slices" is chosen, no lifetime might be needed at all.

That said, having a lifetime in both the stream type and the context type expands the design space significantly, and the following is probably really close to optimal:

pub struct TokenStream<'a>(Vec<&'a [CompactToken]>);
pub struct MacroContext<'a> {
    tokens: &'a TypedArena<Vec<CompactToken>>,
    ...
}

We must be careful to only provide APIs which can be implemented efficiently regardless of representation, such as forward/reverse iterators (but not random access iterators).

@Zoxc
Copy link

Zoxc commented Apr 2, 2016

I am missing Space, Tab and Newline from the tokens. Without them, it's hard to tell the difference between + = and +=. They would also allow you to embed languages which are newline sensitive, like assembly.

@thepowersgang
Copy link
Contributor

thepowersgang commented Apr 2, 2016

They're not needed - += should be a distinct token to + =

@eddyb
Copy link
Member

eddyb commented Apr 2, 2016

@Zoxc spaces are intentionally omitted, they complicate everything. We skip those tokens in the lexer (Reader::real_token) in the current parser.

@eddyb
Copy link
Member

eddyb commented Apr 3, 2016

@nrc I've been talking to @mrmonday about his pnet crate, which gives you the tools to define arbitrary packets - network is the main target, but I suspect IPC or serialization would also be valid usecases; I hear dropbox is using it for something.

From what I understand, it would be real helpful if given an arbitrary type path (e.g. the type of a field), a procedural macro could query the MacroContext and get some information about the type definition (a struct or enum, AFAICT).

My reading of #1560 suggests this should be possible, and if nothing else, having a macro next to the type definition would also work, i.e. given field: path::to::Type, pnet could query path::to::Type__pnet_info (previously generated by pnet) or at least defer to it for decisions.

@Zoxc
Copy link

Zoxc commented Apr 4, 2016

Are you supposed to be able to generate Spans dynamically for token you generate? For example include! should make a new Span for the tokens in the external file. That means there is at least hope of inferring whitespace/newlines from Spans. It would make the fields of Comment and String seem redundant, since those could easily be inferred from the Span.

I wonder if it would be a good idea to include a Unknown token, which could contain an AST or newer tokens not supported by the current version of libmacro. Inline assembly seems to be a good candidate to put inside such a token, since asm! cannot currently return tokens.

@eddyb
Copy link
Member

eddyb commented Apr 4, 2016

@Zoxc Well, one solution to make matches over tokens, from stable crates, non-exhaustive is to have a dummy variant which is unstable - InlineAsm is another good example of an unstable variant.
Not doing anything like this would force libmacro into a fixed set of tokens which cannot be extended.

@pnkfelix
Copy link
Member

pnkfelix commented Nov 3, 2016

@rfcbot resolved multi-char-operators

@nrc
Copy link
Member Author

nrc commented Nov 28, 2016

ping for approval - @aturon @withoutboats #1566 (comment)

I plan to update the RFC with the recent discussion about multi-char-operators.

@aturon
Copy link
Member

aturon commented Nov 29, 2016

@nrc I've re-read the RFC and the thread. Like most everyone else, I'm broadly in favor of the overall direction here (operating on token streams, easing the declaration system). My sense is that there's a lot of room for iteration on the details, but I'm taking this RFC as largely about setting the overall direction of exploration. In particular, I imagine that libproc_macro itself is going to take significant iteration that will feed back into the design elements here.

I'm also in agreement with the various points of scope cutting (e.g., making proc_macro work at crate granularity, leaving out foo! bar (..) macros). There's going to be a lot of work to put this new system together, so anywhere we can punt extensions to the future, we should.

👍 from me!

@withoutboats
Copy link
Contributor

withoutboats commented Nov 30, 2016

👍 from me. I don't think this is impacted by the conversation on #1584, at least not in any way that should block the RFC from being accepted.

@rfcbot
Copy link

rfcbot commented Nov 30, 2016

🔔 This is now entering its final comment period, as per the review above. 🔔

@aturon aturon merged commit 1c2a50d into rust-lang:master Dec 13, 2016
@aturon
Copy link
Member

aturon commented Dec 13, 2016

The RFC bot has gotten stuck, but almost two weeks have elapsed since FCP, and positive consensus around this feature remains. I'm merging the RFC! Thanks @nrc!

Tracking issue

two kinds exist today, and other than naming (see
[RFC 1561](https://github.com/rust-lang/rfcs/pull/1561)) the syntax for using
these macros remains unchanged. If the macro is called `foo`, then a function-
like macro is used with syntax `foo!(...)`, and an attribute-like macro with

This comment has been minimized.

@est31

est31 Dec 14, 2016 Contributor

So this means you can't call procedural macros with [] syntax anymore, like vec! for example does?

This comment has been minimized.

@Connicpu

Connicpu Dec 21, 2016

No, it's talking about function-like (foo!(...), foo![...], foo!{...}) vs attribute macros (#[my_macro] ...)

bors added a commit to rust-lang/rust that referenced this pull request Jan 17, 2017
Implement `#[proc_macro_attribute]`

This implements `#[proc_macro_attribute]` as described in rust-lang/rfcs#1566

The following major (hopefully non-breaking) changes are included:

* Refactor `proc_macro::TokenStream` to use `syntax::tokenstream::TokenStream`.
    * `proc_macro::tokenstream::TokenStream` no longer emits newlines between items, this can be trivially restored if desired
    * `proc_macro::TokenStream::from_str` does not try to parse an item anymore, moved to `impl MultiItemModifier for CustomDerive` with more informative error message

* Implement `#[proc_macro_attribute]`, which expects functions of the kind `fn(TokenStream, TokenStream) -> TokenStream`
    * Reactivated `#![feature(proc_macro)]` and gated `#[proc_macro_attribute]` under it
    * `#![feature(proc_macro)]` and `#![feature(custom_attribute)]` are mutually exclusive
    * adding `#![feature(proc_macro)]` makes the expansion pass assume that any attributes that are not built-in, or introduced by existing syntax extensions, are proc-macro attributes

* Fix `feature_gate::find_lang_feature_issue()` to not use `unwrap()`

    * This change wasn't necessary for this PR, but it helped debugging a problem where I was using the wrong feature string.

* Move "completed feature gate checking" pass to after "name resolution" pass

    * This was necessary for proper feature-gating of `#[proc_macro_attribute]` invocations when the `proc_macro` feature flag isn't set.

Prototype/Litmus Test: [Implementation](https://github.com/abonander/anterofit/blob/proc_macro/service-attr/src/lib.rs#L13) -- [Usage](https://github.com/abonander/anterofit/blob/proc_macro/service-attr/examples/post_service.rs#L35)
bors added a commit to rust-lang/rust that referenced this pull request Jan 18, 2017
Implement `#[proc_macro_attribute]`

This implements `#[proc_macro_attribute]` as described in rust-lang/rfcs#1566

The following major (hopefully non-breaking) changes are included:

* Refactor `proc_macro::TokenStream` to use `syntax::tokenstream::TokenStream`.
    * `proc_macro::tokenstream::TokenStream` no longer emits newlines between items, this can be trivially restored if desired
    * `proc_macro::TokenStream::from_str` does not try to parse an item anymore, moved to `impl MultiItemModifier for CustomDerive` with more informative error message

* Implement `#[proc_macro_attribute]`, which expects functions of the kind `fn(TokenStream, TokenStream) -> TokenStream`
    * Reactivated `#![feature(proc_macro)]` and gated `#[proc_macro_attribute]` under it
    * `#![feature(proc_macro)]` and `#![feature(custom_attribute)]` are mutually exclusive
    * adding `#![feature(proc_macro)]` makes the expansion pass assume that any attributes that are not built-in, or introduced by existing syntax extensions, are proc-macro attributes

* Fix `feature_gate::find_lang_feature_issue()` to not use `unwrap()`

    * This change wasn't necessary for this PR, but it helped debugging a problem where I was using the wrong feature string.

* Move "completed feature gate checking" pass to after "name resolution" pass

    * This was necessary for proper feature-gating of `#[proc_macro_attribute]` invocations when the `proc_macro` feature flag isn't set.

Prototype/Litmus Test: [Implementation](https://github.com/abonander/anterofit/blob/proc_macro/service-attr/src/lib.rs#L13) -- [Usage](https://github.com/abonander/anterofit/blob/proc_macro/service-attr/examples/post_service.rs#L35)
alexcrichton added a commit to alexcrichton/rust that referenced this pull request Jan 19, 2017
…seyfried

Implement `#[proc_macro_attribute]`

This implements `#[proc_macro_attribute]` as described in rust-lang/rfcs#1566

The following major (hopefully non-breaking) changes are included:

* Refactor `proc_macro::TokenStream` to use `syntax::tokenstream::TokenStream`.
    * `proc_macro::tokenstream::TokenStream` no longer emits newlines between items, this can be trivially restored if desired
    * `proc_macro::TokenStream::from_str` does not try to parse an item anymore, moved to `impl MultiItemModifier for CustomDerive` with more informative error message

* Implement `#[proc_macro_attribute]`, which expects functions of the kind `fn(TokenStream, TokenStream) -> TokenStream`
    * Reactivated `#![feature(proc_macro)]` and gated `#[proc_macro_attribute]` under it
    * `#![feature(proc_macro)]` and `#![feature(custom_attribute)]` are mutually exclusive
    * adding `#![feature(proc_macro)]` makes the expansion pass assume that any attributes that are not built-in, or introduced by existing syntax extensions, are proc-macro attributes

* Fix `feature_gate::find_lang_feature_issue()` to not use `unwrap()`

    * This change wasn't necessary for this PR, but it helped debugging a problem where I was using the wrong feature string.

* Move "completed feature gate checking" pass to after "name resolution" pass

    * This was necessary for proper feature-gating of `#[proc_macro_attribute]` invocations when the `proc_macro` feature flag isn't set.

Prototype/Litmus Test: [Implementation](https://github.com/abonander/anterofit/blob/proc_macro/service-attr/src/lib.rs#L13) -- [Usage](https://github.com/abonander/anterofit/blob/proc_macro/service-attr/examples/post_service.rs#L35)
bors added a commit to rust-lang/rust that referenced this pull request Jan 20, 2017
Implement `#[proc_macro_attribute]`

This implements `#[proc_macro_attribute]` as described in rust-lang/rfcs#1566

The following major (hopefully non-breaking) changes are included:

* Refactor `proc_macro::TokenStream` to use `syntax::tokenstream::TokenStream`.
    * `proc_macro::tokenstream::TokenStream` no longer emits newlines between items, this can be trivially restored if desired
    * `proc_macro::TokenStream::from_str` does not try to parse an item anymore, moved to `impl MultiItemModifier for CustomDerive` with more informative error message

* Implement `#[proc_macro_attribute]`, which expects functions of the kind `fn(TokenStream, TokenStream) -> TokenStream`
    * Reactivated `#![feature(proc_macro)]` and gated `#[proc_macro_attribute]` under it
    * `#![feature(proc_macro)]` and `#![feature(custom_attribute)]` are mutually exclusive
    * adding `#![feature(proc_macro)]` makes the expansion pass assume that any attributes that are not built-in, or introduced by existing syntax extensions, are proc-macro attributes

* Fix `feature_gate::find_lang_feature_issue()` to not use `unwrap()`

    * This change wasn't necessary for this PR, but it helped debugging a problem where I was using the wrong feature string.

* Move "completed feature gate checking" pass to after "name resolution" pass

    * This was necessary for proper feature-gating of `#[proc_macro_attribute]` invocations when the `proc_macro` feature flag isn't set.

Prototype/Litmus Test: [Implementation](https://github.com/abonander/anterofit/blob/proc_macro/service-attr/src/lib.rs#L13) -- [Usage](https://github.com/abonander/anterofit/blob/proc_macro/service-attr/examples/post_service.rs#L35)
bors added a commit to rust-lang/rust that referenced this pull request Jan 20, 2017
Implement `#[proc_macro_attribute]`

This implements `#[proc_macro_attribute]` as described in rust-lang/rfcs#1566

The following major (hopefully non-breaking) changes are included:

* Refactor `proc_macro::TokenStream` to use `syntax::tokenstream::TokenStream`.
    * `proc_macro::tokenstream::TokenStream` no longer emits newlines between items, this can be trivially restored if desired
    * `proc_macro::TokenStream::from_str` does not try to parse an item anymore, moved to `impl MultiItemModifier for CustomDerive` with more informative error message

* Implement `#[proc_macro_attribute]`, which expects functions of the kind `fn(TokenStream, TokenStream) -> TokenStream`
    * Reactivated `#![feature(proc_macro)]` and gated `#[proc_macro_attribute]` under it
    * `#![feature(proc_macro)]` and `#![feature(custom_attribute)]` are mutually exclusive
    * adding `#![feature(proc_macro)]` makes the expansion pass assume that any attributes that are not built-in, or introduced by existing syntax extensions, are proc-macro attributes

* Fix `feature_gate::find_lang_feature_issue()` to not use `unwrap()`

    * This change wasn't necessary for this PR, but it helped debugging a problem where I was using the wrong feature string.

* Move "completed feature gate checking" pass to after "name resolution" pass

    * This was necessary for proper feature-gating of `#[proc_macro_attribute]` invocations when the `proc_macro` feature flag isn't set.

Prototype/Litmus Test: [Implementation](https://github.com/abonander/anterofit/blob/proc_macro/service-attr/src/lib.rs#L13) -- [Usage](https://github.com/abonander/anterofit/blob/proc_macro/service-attr/examples/post_service.rs#L35)
alexcrichton added a commit to alexcrichton/rust that referenced this pull request Jan 20, 2017
…seyfried

Implement `#[proc_macro_attribute]`

This implements `#[proc_macro_attribute]` as described in rust-lang/rfcs#1566

The following major (hopefully non-breaking) changes are included:

* Refactor `proc_macro::TokenStream` to use `syntax::tokenstream::TokenStream`.
    * `proc_macro::tokenstream::TokenStream` no longer emits newlines between items, this can be trivially restored if desired
    * `proc_macro::TokenStream::from_str` does not try to parse an item anymore, moved to `impl MultiItemModifier for CustomDerive` with more informative error message

* Implement `#[proc_macro_attribute]`, which expects functions of the kind `fn(TokenStream, TokenStream) -> TokenStream`
    * Reactivated `#![feature(proc_macro)]` and gated `#[proc_macro_attribute]` under it
    * `#![feature(proc_macro)]` and `#![feature(custom_attribute)]` are mutually exclusive
    * adding `#![feature(proc_macro)]` makes the expansion pass assume that any attributes that are not built-in, or introduced by existing syntax extensions, are proc-macro attributes

* Fix `feature_gate::find_lang_feature_issue()` to not use `unwrap()`

    * This change wasn't necessary for this PR, but it helped debugging a problem where I was using the wrong feature string.

* Move "completed feature gate checking" pass to after "name resolution" pass

    * This was necessary for proper feature-gating of `#[proc_macro_attribute]` invocations when the `proc_macro` feature flag isn't set.

Prototype/Litmus Test: [Implementation](https://github.com/abonander/anterofit/blob/proc_macro/service-attr/src/lib.rs#L13) -- [Usage](https://github.com/abonander/anterofit/blob/proc_macro/service-attr/examples/post_service.rs#L35)
@chriskrycho chriskrycho mentioned this pull request Mar 29, 2017
18 of 48 tasks complete
bors added a commit to rust-lang/rust that referenced this pull request May 16, 2018
Review proc macro API 1.2

cc #38356

Summary of applied changes:
- Documentation for proc macro API 1.2 is expanded.
- Renamed APIs: `Term` -> `Ident`, `TokenTree::Term` -> `TokenTree::Ident`, `Op` -> `Punct`, `TokenTree::Op` -> `TokenTree::Punct`, `Op::op` -> `Punct::as_char`.
- Removed APIs: `Ident::as_str`, use `Display` impl for `Ident` instead.
- New APIs (not stabilized in 1.2): `Ident::new_raw` for creating a raw identifier (I'm not sure `new_x` it's a very idiomatic name though).
- Runtime changes:
    - `Punct::new` now ensures that the input `char` is a valid punctuation character in Rust.
    - `Ident::new` ensures that the input `str` is a valid identifier in Rust.
    - Lifetimes in proc macros are now represented as two joint tokens - `Punct('\'', Spacing::Joint)` and `Ident("lifetime_name_without_quote")` similarly to multi-character operators.
- Stabilized APIs: None yet.

A bit of motivation for renaming (although it was already stated in the review comments):
- With my compiler frontend glasses on `Ident` is the single most appropriate name for this thing, *especially* if we are doing input validation on construction. `TokenTree::Ident` effectively wraps `token::Ident` or `ast::Ident + is_raw`, its meaning is "identifier" and it's already named `ident` in declarative macros.
- Regarding `Punct`, the motivation is that `Op` is actively misleading. The thing doesn't mean an operator, it's neither a subset of operators (there is non-operator punctuation in the language), nor superset (operators can be multicharacter while this thing is always a single character). So I named it `Punct` (first proposed in [the original RFC](rust-lang/rfcs#1566), then [by @SimonSapin](#38356 (comment))) , together with input validation it's now a subset of ASCII punctuation character category (`u8::is_ascii_punctuation`).
@Ekleog
Copy link

Ekleog commented Aug 14, 2018

So, I've been looking a bit, and can't figure out: is there a way to call a function in the crate defining the procedural macro as part of the output of said procedural macro? The equivalent of $crate.

@eddyb
Copy link
Member

eddyb commented Aug 16, 2018

@Ekleog No, the proc macro crate "lives in a different world" than the code it's used in. Imagine cross-compilation: the proc macro is compiled for, and runs on, the host, whereas the crate using it is compiled for the cross-compilation target.

What people do is have two crates, like serde_derive and serde, and the former generates code using the latter.

@Ekleog
Copy link

Ekleog commented Aug 17, 2018

Oh sorry I must have forgotten I had posted here, when I search for whether I already asked this question or not a few days ago! The discussion appears to have happened on rust-lang/rust#38356 (comment), sorry for the noise.

Basically, my issue with your answer is, there is no way for the proc macro to generate code that refers to crates that are not the crate they're being expanded into, just like serde also has an issue open with.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

You can’t perform that action at this time.