Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue for RFC 1566: Procedural macros #38356

Closed
aturon opened this issue Dec 13, 2016 · 184 comments
Closed

Tracking issue for RFC 1566: Procedural macros #38356

aturon opened this issue Dec 13, 2016 · 184 comments

Comments

@aturon
Copy link
Member

@aturon aturon commented Dec 13, 2016

Current Status

This issue has been closed in favor of more fine-grained tracking issues

~~~Updated Description~~~

Next steps:

Possible Stabilization Showstoppers

Original Description

RFC.

This RFC proposes an evolution of Rust's procedural macro system (aka syntax
extensions, aka compiler plugins). This RFC specifies syntax for the definition
of procedural macros, a high-level view of their implementation in the compiler,
and outlines how they interact with the compilation process.

At the highest level, macros are defined by implementing functions marked with
a #[macro] attribute. Macros operate on a list of tokens provided by the
compiler and return a list of tokens that the macro use is replaced by. We
provide low-level facilities for operating on these tokens. Higher level
facilities (e.g., for parsing tokens to an AST) should exist as library crates.

Roadmap: #38356 (comment).


Tasks

  • Implement #[proc_macro_attribute] (PR #38842).
  • Implement #[proc_macro] (PR #40129).
  • Identify and collect uses of proc_macro_derives in the InvocationCollector (PR #39391).
  • Support macro-expanded proc_macro_derive imports.
    • For example:
#[derive(Trait, OtherTrait)] struct S; // Both these derives should resolve
macro_rules! m { () => {
    #[macro_use(Trait)] extern crate derives;
    use derives::OtherTrait; // this kind of import is gated behind `#![feature(proc_macro)]`
} }
m!();
  • Expand items before expanding applied proc_macro_derives (PR #48465).
  • Implement warnings for unused #[macro_use] imports (PR #39060).
  • Refactor the parser to consume token trees (PR #39118).
  • Clean up TokenStream in preparation for further refactoring (PR #39173).
  • Remove TokenTree::Sequence (PR #39419).
  • Use TokenStreams instead of Vec<TokenTree> in tokenstream::TokenTree's Delimited variant (PR #40202).
  • Use Paths and TokenStreams in ast::Attributes (PR #40346).
    • Support nontrivial paths in attribute/derive macros (e.g. #[foo::bar], #[derive(foo::Bar)]).
  • Include hygiene information with all tokens, not just identifiers (PR #40597).
  • Implement a minimal API for proc_macro::TokenStream as outlined in the RFC (PR #40939).
    • Include source TokenStreams for interpolated AST fragments in Token::Interpolated tokens.
    • Include a TokenStream quoter proc_macro::quote! behind the proc_macro feature gate.
  • Provide a way for proc_macro authors to create expansions that use items in a predetermined crate foo without requiring the macro user to include extern crate foo; at the crate root (PR #40939).
    • Improve ergonomics.
  • Include source TokenStreams for items in the AST.
  • Stability check (proc-)macros (issue #34079).
  • Allow proc macro to initialize a private field with a def_site value (issue #47311). (PR #48082)
  • Inconsistency between accessing field of braced struct vs tuple struct in proc macro (issue #47312). (PR #48083)
  • Make std available to proc macro root in phase 1 (issue #47314).
  • Improve error from invalid syntax inside proc_macro::quote! (issue #47315).
  • Inconsistency between Display and IntoIterator for a TokenStream containing a module (issue #47627).
  • #[cfg_attr] makes .to_string() and TokenStream disagree (issue #48644).
  • Wishlist for libproc_macro (checklist in #47786).
@aturon
Copy link
Member Author

@aturon aturon commented Dec 13, 2016

@abonander
Copy link
Contributor

@abonander abonander commented Jan 1, 2017

I'd love for #[proc_macro_attribute] to be implemented soon. I already have a prototype and test usage that I banged out before realizing there's no compiler support yet 😒 :

Prototype: https://github.com/abonander/anterofit/blob/proc_macro/macros/src/lib.rs
Example/Test: https://github.com/abonander/anterofit/blob/proc_macro/examples/post_service_proc_macro.rs

@jseyfried
Copy link
Contributor

@jseyfried jseyfried commented Jan 23, 2017

Tasks

(dtolnay edit: moved the checklist up to the OP)

cc @nrc @petrochenkov @durka @Ralith

@abonander
Copy link
Contributor

@abonander abonander commented Jan 24, 2017

@jseyfried I ran into an issue where if a legacy macro and an attribute with the same name are imported into the same scope, trying to use the attribute throws an error that macros cannot be used as attributes. Can we make this work so that both can be in the same scope and can be used as intended?

@jseyfried
Copy link
Contributor

@jseyfried jseyfried commented Jan 24, 2017

@abonander All macros (bang, attribute, and derive) share the same namespace, so we can't use two different macros with the same name in the same scope. However, we could improve that error message -- could you open an issue?

@SimonSapin
Copy link
Contributor

@SimonSapin SimonSapin commented Feb 1, 2017

Sorry I’m late to the party. I’m happy with the direction to expose tokens rather than an AST, but I have some concerns about the specific TokenStream API proposed in the RFC:

pub enum TokenKind {
    Sequence(Delimiter, TokenStream),

    // The content of the comment can be found from the span.
    Comment(CommentKind),

    // `text` is the string contents, not including delimiters. It would be nice
    // to avoid an allocation in the common case that the string is in the
    // source code. We might be able to use `&'codemap str` or something.
    // `raw_markers` is for the count of `#`s if the string is a raw string. If
    // the string is not raw, then it will be `None`.
    String { text: Symbol, raw_markers: Option<usize>, kind: StringKind },

    // char literal, span includes the `'` delimiters.
    Char(char),

    // These tokens are treated specially since they are used for macro
    // expansion or delimiting items.
    Exclamation,  // `!`
    Dollar,       // `$`
    // Not actually sure if we need this or if semicolons can be treated like
    // other punctuation.
    Semicolon,    // `;`
    Eof,          // Do we need this?

    // Word is defined by Unicode Standard Annex 31 -
    // [Unicode Identifier and Pattern Syntax](http://unicode.org/reports/tr31/)
    Word(Symbol),
    Punctuation(char),
}

pub enum StringKind {
    Regular,
    Byte,
}

It’s not clear if this API was intended as a complete plan that was accepted when the RFC was merged, or just an example to be worked out later.

Generally, this seems far from the "normal" Rust syntax accepted by the compiler outside of macros. While some macros will want to parse some ad-hoc domain-specific language, others will want to parse "actual Rust" syntax and make sense of it.

  1. (Minor) I don’t think Eof is necessary. An Iterator will presumably be used to, well, iterate over a TokenStream and Iterator::next already returns None to signal the end of iteration.

  2. (Minor) I don’t think Exclamation, Dollar, or Semicolon are necessary. Matching on Punctuation('!') for example is not more difficult.

  3. (Minor) As others have mentioned in the RFC PR, we might want to omit comments that are not doc-comments. (Any use case that wants to preserve comment likely wants to preserve whitespace too.)

  4. As far as I can tell, what to do with multi-character operators (that probably should be a single token each) is still an open question. A possible solution is discussed in PR comments, but it looks like that didn’t make it into RFC text.

  5. Number literals are missing. Are macros supposed to parse [Punct('1'), Punct('_'), Punct('2'), Punct('3'), Punct('4'), Punct('.'), Punct('5'), Punct('e'), Punct('6')] by themselves to evaluate a literal? They can’t even use str::parse::<f32> to do that, since the syntax it accepts is not the same as Rust literal syntax (which can have _ in the middle, for example).

    I imagine that there’s a stability concern here. Can we introduce new numeric types like u128 / i128 (and possibly in the future f128, u256, …) and their literals, without breaking changes to the tokens API? One way to make this possible might be:

    struct IntegerLiteral { negative: bool, decimal_digits: String, type_suffix: Option<String> }
    impl TryInto<u32> IntegerLiteral { type Err = OutOfRange; /* … */ }
    // Other impls for integer types supported in this compiler version
    
    // Something similarly for floats

    Or maybe something else. But I don’t think "pretend numbers don’t exist" is a good way to do it.

  6. // Word is defined by Unicode Standard Annex 31 -

    This definition needs to be more precise than that. UAX 31 specifies a couple different variations of identifier syntax, and none of them is called "word". But choosing which exact variation we want is why non-ASCII identifiers are feature-gated at the moment.

    Instead, I think this should be defined as "whatever the current compiler accepts as an identifier or keyword" (which can change per #28979). Maybe with a pub fn is_identifier(&str) -> bool public API in libmacro.

  7. Unicode strings and byte string literals share a single token variant, which I think is wrong as the memory representations of their values have different types (str vs [u8]). It’s also not clear if the text: Symbol component is intended to be a literal slice of the source code or the value after resolving backslash escapes. I think it should definitely be the latter. (For comparison, Char(char) has to be the latter since \u{A0} takes more than one char to represent literally.)

@jseyfried jseyfried mentioned this issue Feb 2, 2017
50 of 53 tasks complete
abonander added a commit to abonander/rust that referenced this issue Feb 2, 2017
frewsxcv added a commit to frewsxcv/rust that referenced this issue Feb 4, 2017
…, r=jseyfried

Change tracking issue for `proc_macro` feature to rust-lang#38356

r? @jseyfried
frewsxcv added a commit to frewsxcv/rust that referenced this issue Feb 4, 2017
…, r=jseyfried

Change tracking issue for `proc_macro` feature to rust-lang#38356

r? @jseyfried
frewsxcv added a commit to frewsxcv/rust that referenced this issue Feb 5, 2017
…, r=jseyfried

Change tracking issue for `proc_macro` feature to rust-lang#38356

r? @jseyfried
frewsxcv added a commit to frewsxcv/rust that referenced this issue Feb 5, 2017
…, r=jseyfried

Change tracking issue for `proc_macro` feature to rust-lang#38356

r? @jseyfried
frewsxcv added a commit to frewsxcv/rust that referenced this issue Feb 5, 2017
…, r=jseyfried

Change tracking issue for `proc_macro` feature to rust-lang#38356

r? @jseyfried
@SimonSapin SimonSapin mentioned this issue Feb 19, 2017
18 of 38 tasks complete
@porky11
Copy link

@porky11 porky11 commented Feb 21, 2017

another way to write high level macros would be using lisp like macros, but this would need a s-expression representation for the whole rust ast.

@petrochenkov
Copy link
Contributor

@petrochenkov petrochenkov commented Aug 21, 2018

With #![feature(proc_macro)] stabilized, what's left of this issue?

Ideally, fresh issues need be filed for all individual remaining issues and not stabilized features, and then this issue can be closed (in the same way like it was done for #44660).

@mark-i-m
Copy link
Member

@mark-i-m mark-i-m commented Aug 21, 2018

Oh, I thought Spans were only for error reporting, as early blog posts mentioned a Hygiene (or similarly named) struct that played this role. I had assumed that these had vanished, and was apparently wrong. Thanks! :)

IIUC, Spans are the primary way of tracking hygiene context.

@alexreg
Copy link
Contributor

@alexreg alexreg commented Aug 21, 2018

@mark-i-m Kind of. They include both source code location information (for user-facing messages/diagnostics) and syntax context (i.e. hygiene information).

@sgrif
Copy link
Contributor

@sgrif sgrif commented Sep 7, 2018

Given the amount of discussion/traffic this issue gets, does it make sense to move proc_macro_diagnostic to its own tracking issue? I'd also like to figure out what the blockers are for the stabilization of that feature, and see if we can push it through. Diesel uses it, and it's been great so far. The lack of this feature on stable is leading to the community creating funky workarounds, like syn's newest version using compile_error! in its place.

@SergioBenitez
Copy link
Contributor

@SergioBenitez SergioBenitez commented Sep 11, 2018

@sgrif I've opened up such an issue: #54140.

@xd009642
Copy link
Contributor

@xd009642 xd009642 commented Sep 17, 2018

So I'm interesting in helping to stabilise the methods in Span and the LineColumn struct. I'll look at the open issues in the first comment, but if anyone wants to point a newbie to the compiler in a particular direction I'd appreciate it 👍

@jjpe
Copy link

@jjpe jjpe commented Sep 28, 2018

The proc_macro_gen feature gate pointed me here, but in the check list at the top I see nothing that obviously refers to a (proc_)macro that generates other macro definitions. Has this been accounted for (other than in rustc's feature gate)?

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Sep 28, 2018

@jjpe it's probably best at this time that we spin off dedicated tracking issues for those feature gates, this issue was largely dedicated to the initial wave of stabilizations

@jjpe
Copy link

@jjpe jjpe commented Sep 30, 2018

@alexcrichton I'm perfectly fine with that, it's more that in the process of looking at the proc_macro_gen feature, it referred me here only to find as good as no mention of it. That's a bit odd to me, so I decided to mention it, at least.

@xieyuheng What would CodeString/Code even look like i.e. what would its semantics be?
Keep in mind that TokenStream is nothing more than the source code verbatim except as a series of token values rather than bothersome text.

@SimonSapin
Copy link
Contributor

@SimonSapin SimonSapin commented Sep 30, 2018

The extended API for TokenStream (with TokenTree) is stable in 1.29. Importing function-like proc macros will be stable in 1.30.

@CodeSandwich
Copy link

@CodeSandwich CodeSandwich commented Sep 30, 2018

It seems that since rustc 1.30.0-nightly (63d51e89a 2018-09-28) it is no longer possible to traverse code inside module in separate file. If you process mod module;, you get TokenStream containing mod module;, WYSIWYG.

I understand, that this is necessary in order to preserver spans, hygiene and consistency with processed code. Is there or will be any way to work with content of modules in separate files? Its lack may be confusing for end users when trivial refactoring of module organization causes change in macro behavior.

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Oct 1, 2018

Ok this issue is massive and has gotten to the point that I don't think it's too useful to keep open any more and track APIs. To that end I've opened #54728 which splits this issue into a number of more fine-grained tracking issues:

  • #54722 - the quote! macro
  • #54723 - Ident::new_raw
  • #54724 - Span::def_site
  • #54140 - proc_macro and diagnostics
  • #54725 - inspecting a proc_macro::Span
  • #54726 - allowing custom inner attributes
  • #54727 - a sort of catch-all for "hygiene 2.0" reports, may want to be sharded itself.

At this point I'm going to close this, but if I've forgotten to split off any other tracking issues please let me know! I can certainly open up some more follow-ups

pietroalbini added a commit to pietroalbini/rust that referenced this issue Oct 2, 2018
…omatsakis

Renumber `proc_macro` tracking issues

Lots of issue links in the compiler still point to rust-lang#38356 which is a bit of a monster issue that isn't serving much purpose any more. I've split the issue into a number of more fine-grained tracking issues to track stabilizations.
pietroalbini added a commit to pietroalbini/rust that referenced this issue Oct 2, 2018
…omatsakis

Renumber `proc_macro` tracking issues

Lots of issue links in the compiler still point to rust-lang#38356 which is a bit of a monster issue that isn't serving much purpose any more. I've split the issue into a number of more fine-grained tracking issues to track stabilizations.
@XX
Copy link

@XX XX commented Nov 23, 2018

@alexcrichton What about of sub-attributes for attribute-like macro?
#38356 (comment)
Is there a issue for this?

@petrochenkov
Copy link
Contributor

@petrochenkov petrochenkov commented Nov 23, 2018

@XX
Such sub-attributes don't necessarily have to be a language feature?
In case of derive registration of custom attributes is needed because derive macros cannot remove attributes from their input (the input is immutable).
Attribute macros can remove #[other_attribute]s from its input, so they will never get to name resolution and never report the "unresolved attribute" error.

(Beside that classic legacy unstable custom attributes mentioned in #38356 (comment) are now compatible with proc macros.)

@XX
Copy link

@XX XX commented Nov 23, 2018

@petrochenkov Yes, thanks for the clarification.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.