Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue for RFC 1566: Procedural macros #38356

Closed
24 of 31 tasks
aturon opened this issue Dec 13, 2016 · 184 comments
Closed
24 of 31 tasks

Tracking issue for RFC 1566: Procedural macros #38356

aturon opened this issue Dec 13, 2016 · 184 comments
Labels
A-macros Area: All kinds of macros (custom derive, macro_rules!, proc macros, ..) A-macros-1.2 Issues which affect macros 1.2 A-macros-2.0 Area: declarative macros 2.0, https://github.com/rust-lang/rust/issues/39412 A-plugin Area: compiler plugins, doc.rust-lang.org/nightly/unstable-book/language-features/plugin.html B-RFC-approved Approved by a merged RFC but not yet implemented. B-unstable Implemented in the nightly compiler and unstable. C-tracking-issue Category: A tracking issue for an RFC or an unstable feature. finished-final-comment-period The final comment period is finished for this PR / Issue. T-lang Relevant to the language team, which will review and decide on the PR/issue. T-libs-api Relevant to the library API team, which will review and decide on the PR/issue.

Comments

@aturon
Copy link
Member

aturon commented Dec 13, 2016

Current Status

This issue has been closed in favor of more fine-grained tracking issues

~~~Updated Description~~~

Next steps:

Possible Stabilization Showstoppers

Original Description

RFC.

This RFC proposes an evolution of Rust's procedural macro system (aka syntax
extensions, aka compiler plugins). This RFC specifies syntax for the definition
of procedural macros, a high-level view of their implementation in the compiler,
and outlines how they interact with the compilation process.

At the highest level, macros are defined by implementing functions marked with
a #[macro] attribute. Macros operate on a list of tokens provided by the
compiler and return a list of tokens that the macro use is replaced by. We
provide low-level facilities for operating on these tokens. Higher level
facilities (e.g., for parsing tokens to an AST) should exist as library crates.

Roadmap: #38356 (comment).


Tasks

#[derive(Trait, OtherTrait)] struct S; // Both these derives should resolve
macro_rules! m { () => {
    #[macro_use(Trait)] extern crate derives;
    use derives::OtherTrait; // this kind of import is gated behind `#![feature(proc_macro)]`
} }
m!();
@aturon aturon added A-macros Area: All kinds of macros (custom derive, macro_rules!, proc macros, ..) A-plugin Area: compiler plugins, doc.rust-lang.org/nightly/unstable-book/language-features/plugin.html B-RFC-approved Approved by a merged RFC but not yet implemented. labels Dec 13, 2016
@aturon
Copy link
Member Author

aturon commented Dec 13, 2016

cc @nrc @jseyfried

@abonander
Copy link
Contributor

abonander commented Jan 1, 2017

I'd love for #[proc_macro_attribute] to be implemented soon. I already have a prototype and test usage that I banged out before realizing there's no compiler support yet 😒 :

Prototype: https://github.com/abonander/anterofit/blob/proc_macro/macros/src/lib.rs
Example/Test: https://github.com/abonander/anterofit/blob/proc_macro/examples/post_service_proc_macro.rs

@jseyfried
Copy link
Contributor

jseyfried commented Jan 23, 2017

Tasks

(dtolnay edit: moved the checklist up to the OP)

cc @nrc @petrochenkov @durka @Ralith

@abonander
Copy link
Contributor

@jseyfried I ran into an issue where if a legacy macro and an attribute with the same name are imported into the same scope, trying to use the attribute throws an error that macros cannot be used as attributes. Can we make this work so that both can be in the same scope and can be used as intended?

@jseyfried
Copy link
Contributor

jseyfried commented Jan 24, 2017

@abonander All macros (bang, attribute, and derive) share the same namespace, so we can't use two different macros with the same name in the same scope. However, we could improve that error message -- could you open an issue?

@SimonSapin
Copy link
Contributor

Sorry I’m late to the party. I’m happy with the direction to expose tokens rather than an AST, but I have some concerns about the specific TokenStream API proposed in the RFC:

pub enum TokenKind {
    Sequence(Delimiter, TokenStream),

    // The content of the comment can be found from the span.
    Comment(CommentKind),

    // `text` is the string contents, not including delimiters. It would be nice
    // to avoid an allocation in the common case that the string is in the
    // source code. We might be able to use `&'codemap str` or something.
    // `raw_markers` is for the count of `#`s if the string is a raw string. If
    // the string is not raw, then it will be `None`.
    String { text: Symbol, raw_markers: Option<usize>, kind: StringKind },

    // char literal, span includes the `'` delimiters.
    Char(char),

    // These tokens are treated specially since they are used for macro
    // expansion or delimiting items.
    Exclamation,  // `!`
    Dollar,       // `$`
    // Not actually sure if we need this or if semicolons can be treated like
    // other punctuation.
    Semicolon,    // `;`
    Eof,          // Do we need this?

    // Word is defined by Unicode Standard Annex 31 -
    // [Unicode Identifier and Pattern Syntax](http://unicode.org/reports/tr31/)
    Word(Symbol),
    Punctuation(char),
}

pub enum StringKind {
    Regular,
    Byte,
}

It’s not clear if this API was intended as a complete plan that was accepted when the RFC was merged, or just an example to be worked out later.

Generally, this seems far from the "normal" Rust syntax accepted by the compiler outside of macros. While some macros will want to parse some ad-hoc domain-specific language, others will want to parse "actual Rust" syntax and make sense of it.

  1. (Minor) I don’t think Eof is necessary. An Iterator will presumably be used to, well, iterate over a TokenStream and Iterator::next already returns None to signal the end of iteration.

  2. (Minor) I don’t think Exclamation, Dollar, or Semicolon are necessary. Matching on Punctuation('!') for example is not more difficult.

  3. (Minor) As others have mentioned in the RFC PR, we might want to omit comments that are not doc-comments. (Any use case that wants to preserve comment likely wants to preserve whitespace too.)

  4. As far as I can tell, what to do with multi-character operators (that probably should be a single token each) is still an open question. A possible solution is discussed in PR comments, but it looks like that didn’t make it into RFC text.

  5. Number literals are missing. Are macros supposed to parse [Punct('1'), Punct('_'), Punct('2'), Punct('3'), Punct('4'), Punct('.'), Punct('5'), Punct('e'), Punct('6')] by themselves to evaluate a literal? They can’t even use str::parse::<f32> to do that, since the syntax it accepts is not the same as Rust literal syntax (which can have _ in the middle, for example).

    I imagine that there’s a stability concern here. Can we introduce new numeric types like u128 / i128 (and possibly in the future f128, u256, …) and their literals, without breaking changes to the tokens API? One way to make this possible might be:

    struct IntegerLiteral { negative: bool, decimal_digits: String, type_suffix: Option<String> }
    impl TryInto<u32> IntegerLiteral { type Err = OutOfRange; /* … */ }
    // Other impls for integer types supported in this compiler version
    
    // Something similarly for floats

    Or maybe something else. But I don’t think "pretend numbers don’t exist" is a good way to do it.

  6. // Word is defined by Unicode Standard Annex 31 -

    This definition needs to be more precise than that. UAX 31 specifies a couple different variations of identifier syntax, and none of them is called "word". But choosing which exact variation we want is why non-ASCII identifiers are feature-gated at the moment.

    Instead, I think this should be defined as "whatever the current compiler accepts as an identifier or keyword" (which can change per Tracking issue for non-ASCII identifiers (feature "non_ascii_idents") #28979). Maybe with a pub fn is_identifier(&str) -> bool public API in libmacro.

  7. Unicode strings and byte string literals share a single token variant, which I think is wrong as the memory representations of their values have different types (str vs [u8]). It’s also not clear if the text: Symbol component is intended to be a literal slice of the source code or the value after resolving backslash escapes. I think it should definitely be the latter. (For comparison, Char(char) has to be the latter since \u{A0} takes more than one char to represent literally.)

abonander added a commit to abonander/rust that referenced this issue Feb 2, 2017
frewsxcv added a commit to frewsxcv/rust that referenced this issue Feb 4, 2017
…, r=jseyfried

Change tracking issue for `proc_macro` feature to rust-lang#38356

r? @jseyfried
frewsxcv added a commit to frewsxcv/rust that referenced this issue Feb 4, 2017
…, r=jseyfried

Change tracking issue for `proc_macro` feature to rust-lang#38356

r? @jseyfried
frewsxcv added a commit to frewsxcv/rust that referenced this issue Feb 5, 2017
…, r=jseyfried

Change tracking issue for `proc_macro` feature to rust-lang#38356

r? @jseyfried
frewsxcv added a commit to frewsxcv/rust that referenced this issue Feb 5, 2017
…, r=jseyfried

Change tracking issue for `proc_macro` feature to rust-lang#38356

r? @jseyfried
frewsxcv added a commit to frewsxcv/rust that referenced this issue Feb 5, 2017
…, r=jseyfried

Change tracking issue for `proc_macro` feature to rust-lang#38356

r? @jseyfried
@porky11
Copy link

porky11 commented Feb 21, 2017

another way to write high level macros would be using lisp like macros, but this would need a s-expression representation for the whole rust ast.

@petrochenkov
Copy link
Contributor

petrochenkov commented Aug 21, 2018

With #![feature(proc_macro)] stabilized, what's left of this issue?

Ideally, fresh issues need be filed for all individual remaining issues and not stabilized features, and then this issue can be closed (in the same way like it was done for #44660).

@mark-i-m
Copy link
Member

Oh, I thought Spans were only for error reporting, as early blog posts mentioned a Hygiene (or similarly named) struct that played this role. I had assumed that these had vanished, and was apparently wrong. Thanks! :)

IIUC, Spans are the primary way of tracking hygiene context.

@alexreg
Copy link
Contributor

alexreg commented Aug 21, 2018

@mark-i-m Kind of. They include both source code location information (for user-facing messages/diagnostics) and syntax context (i.e. hygiene information).

@sgrif
Copy link
Contributor

sgrif commented Sep 7, 2018

Given the amount of discussion/traffic this issue gets, does it make sense to move proc_macro_diagnostic to its own tracking issue? I'd also like to figure out what the blockers are for the stabilization of that feature, and see if we can push it through. Diesel uses it, and it's been great so far. The lack of this feature on stable is leading to the community creating funky workarounds, like syn's newest version using compile_error! in its place.

@SergioBenitez
Copy link
Contributor

@sgrif I've opened up such an issue: #54140.

@xd009642
Copy link
Contributor

So I'm interesting in helping to stabilise the methods in Span and the LineColumn struct. I'll look at the open issues in the first comment, but if anyone wants to point a newbie to the compiler in a particular direction I'd appreciate it 👍

@jjpe
Copy link

jjpe commented Sep 28, 2018

The proc_macro_gen feature gate pointed me here, but in the check list at the top I see nothing that obviously refers to a (proc_)macro that generates other macro definitions. Has this been accounted for (other than in rustc's feature gate)?

@alexcrichton
Copy link
Member

@jjpe it's probably best at this time that we spin off dedicated tracking issues for those feature gates, this issue was largely dedicated to the initial wave of stabilizations

@jjpe
Copy link

jjpe commented Sep 30, 2018

@alexcrichton I'm perfectly fine with that, it's more that in the process of looking at the proc_macro_gen feature, it referred me here only to find as good as no mention of it. That's a bit odd to me, so I decided to mention it, at least.

@xieyuheng What would CodeString/Code even look like i.e. what would its semantics be?
Keep in mind that TokenStream is nothing more than the source code verbatim except as a series of token values rather than bothersome text.

@SimonSapin
Copy link
Contributor

The extended API for TokenStream (with TokenTree) is stable in 1.29. Importing function-like proc macros will be stable in 1.30.

@CodeSandwich
Copy link

CodeSandwich commented Sep 30, 2018

It seems that since rustc 1.30.0-nightly (63d51e89a 2018-09-28) it is no longer possible to traverse code inside module in separate file. If you process mod module;, you get TokenStream containing mod module;, WYSIWYG.

I understand, that this is necessary in order to preserver spans, hygiene and consistency with processed code. Is there or will be any way to work with content of modules in separate files? Its lack may be confusing for end users when trivial refactoring of module organization causes change in macro behavior.

@alexcrichton
Copy link
Member

Ok this issue is massive and has gotten to the point that I don't think it's too useful to keep open any more and track APIs. To that end I've opened #54728 which splits this issue into a number of more fine-grained tracking issues:

At this point I'm going to close this, but if I've forgotten to split off any other tracking issues please let me know! I can certainly open up some more follow-ups

pietroalbini added a commit to pietroalbini/rust that referenced this issue Oct 2, 2018
…omatsakis

Renumber `proc_macro` tracking issues

Lots of issue links in the compiler still point to rust-lang#38356 which is a bit of a monster issue that isn't serving much purpose any more. I've split the issue into a number of more fine-grained tracking issues to track stabilizations.
pietroalbini added a commit to pietroalbini/rust that referenced this issue Oct 2, 2018
…omatsakis

Renumber `proc_macro` tracking issues

Lots of issue links in the compiler still point to rust-lang#38356 which is a bit of a monster issue that isn't serving much purpose any more. I've split the issue into a number of more fine-grained tracking issues to track stabilizations.
@XX
Copy link

XX commented Nov 23, 2018

@alexcrichton What about of sub-attributes for attribute-like macro?
#38356 (comment)
Is there a issue for this?

@petrochenkov
Copy link
Contributor

petrochenkov commented Nov 23, 2018

@XX
Such sub-attributes don't necessarily have to be a language feature?
In case of derive registration of custom attributes is needed because derive macros cannot remove attributes from their input (the input is immutable).
Attribute macros can remove #[other_attribute]s from its input, so they will never get to name resolution and never report the "unresolved attribute" error.

(Beside that classic legacy unstable custom attributes mentioned in #38356 (comment) are now compatible with proc macros.)

@XX
Copy link

XX commented Nov 23, 2018

@petrochenkov Yes, thanks for the clarification.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-macros Area: All kinds of macros (custom derive, macro_rules!, proc macros, ..) A-macros-1.2 Issues which affect macros 1.2 A-macros-2.0 Area: declarative macros 2.0, https://github.com/rust-lang/rust/issues/39412 A-plugin Area: compiler plugins, doc.rust-lang.org/nightly/unstable-book/language-features/plugin.html B-RFC-approved Approved by a merged RFC but not yet implemented. B-unstable Implemented in the nightly compiler and unstable. C-tracking-issue Category: A tracking issue for an RFC or an unstable feature. finished-final-comment-period The final comment period is finished for this PR / Issue. T-lang Relevant to the language team, which will review and decide on the PR/issue. T-libs-api Relevant to the library API team, which will review and decide on the PR/issue.
Projects
None yet
Development

No branches or pull requests