New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue for RFC 1566: Procedural macros #38356

Open
aturon opened this Issue Dec 13, 2016 · 175 comments

Comments

@aturon
Member

aturon commented Dec 13, 2016

Updated Description

Next steps:

Possible Stabilization Showstoppers

Original Description

RFC.

This RFC proposes an evolution of Rust's procedural macro system (aka syntax
extensions, aka compiler plugins). This RFC specifies syntax for the definition
of procedural macros, a high-level view of their implementation in the compiler,
and outlines how they interact with the compilation process.

At the highest level, macros are defined by implementing functions marked with
a #[macro] attribute. Macros operate on a list of tokens provided by the
compiler and return a list of tokens that the macro use is replaced by. We
provide low-level facilities for operating on these tokens. Higher level
facilities (e.g., for parsing tokens to an AST) should exist as library crates.

Roadmap: #38356 (comment).


Tasks

  • Implement #[proc_macro_attribute] (PR #38842).
  • Implement #[proc_macro] (PR #40129).
  • Identify and collect uses of proc_macro_derives in the InvocationCollector (PR #39391).
  • Support macro-expanded proc_macro_derive imports.
    • For example:
#[derive(Trait, OtherTrait)] struct S; // Both these derives should resolve
macro_rules! m { () => {
    #[macro_use(Trait)] extern crate derives;
    use derives::OtherTrait; // this kind of import is gated behind `#![feature(proc_macro)]`
} }
m!();
  • Expand items before expanding applied proc_macro_derives (PR #48465).
  • Implement warnings for unused #[macro_use] imports (PR #39060).
  • Refactor the parser to consume token trees (PR #39118).
  • Clean up TokenStream in preparation for further refactoring (PR #39173).
  • Remove TokenTree::Sequence (PR #39419).
  • Use TokenStreams instead of Vec<TokenTree> in tokenstream::TokenTree's Delimited variant (PR #40202).
  • Use Paths and TokenStreams in ast::Attributes (PR #40346).
    • Support nontrivial paths in attribute/derive macros (e.g. #[foo::bar], #[derive(foo::Bar)]).
  • Include hygiene information with all tokens, not just identifiers (PR #40597).
  • Implement a minimal API for proc_macro::TokenStream as outlined in the RFC (PR #40939).
    • Include source TokenStreams for interpolated AST fragments in Token::Interpolated tokens.
    • Include a TokenStream quoter proc_macro::quote! behind the proc_macro feature gate.
  • Provide a way for proc_macro authors to create expansions that use items in a predetermined crate foo without requiring the macro user to include extern crate foo; at the crate root (PR #40939).
    • Improve ergonomics.
  • Include source TokenStreams for items in the AST.
  • Stability check (proc-)macros (issue #34079).
  • Allow proc macro to initialize a private field with a def_site value (issue #47311). (PR #48082)
  • Inconsistency between accessing field of braced struct vs tuple struct in proc macro (issue #47312). (PR #48083)
  • Make std available to proc macro root in phase 1 (issue #47314).
  • Improve error from invalid syntax inside proc_macro::quote! (issue #47315).
  • Inconsistency between Display and IntoIterator for a TokenStream containing a module (issue #47627).
  • #[cfg_attr] makes .to_string() and TokenStream disagree (issue #48644).
  • Wishlist for libproc_macro (checklist in #47786).
@aturon

This comment has been minimized.

Show comment
Hide comment
@aturon
Member

aturon commented Dec 13, 2016

@steveklabnik steveklabnik referenced this issue Dec 23, 2016

Open

Compile with stable Rust #19

6 of 14 tasks complete

@lfairy lfairy referenced this issue Dec 30, 2016

Open

Stable support #17

@abonander

This comment has been minimized.

Show comment
Hide comment
@abonander

abonander Jan 1, 2017

Contributor

I'd love for #[proc_macro_attribute] to be implemented soon. I already have a prototype and test usage that I banged out before realizing there's no compiler support yet 😒 :

Prototype: https://github.com/abonander/anterofit/blob/proc_macro/macros/src/lib.rs
Example/Test: https://github.com/abonander/anterofit/blob/proc_macro/examples/post_service_proc_macro.rs

Contributor

abonander commented Jan 1, 2017

I'd love for #[proc_macro_attribute] to be implemented soon. I already have a prototype and test usage that I banged out before realizing there's no compiler support yet 😒 :

Prototype: https://github.com/abonander/anterofit/blob/proc_macro/macros/src/lib.rs
Example/Test: https://github.com/abonander/anterofit/blob/proc_macro/examples/post_service_proc_macro.rs

@jseyfried

This comment has been minimized.

Show comment
Hide comment
@jseyfried

jseyfried Jan 23, 2017

Contributor

Tasks

(dtolnay edit: moved the checklist up to the OP)

cc @nrc @petrochenkov @durka @Ralith

Contributor

jseyfried commented Jan 23, 2017

Tasks

(dtolnay edit: moved the checklist up to the OP)

cc @nrc @petrochenkov @durka @Ralith

@abonander

This comment has been minimized.

Show comment
Hide comment
@abonander

abonander Jan 24, 2017

Contributor

@jseyfried I ran into an issue where if a legacy macro and an attribute with the same name are imported into the same scope, trying to use the attribute throws an error that macros cannot be used as attributes. Can we make this work so that both can be in the same scope and can be used as intended?

Contributor

abonander commented Jan 24, 2017

@jseyfried I ran into an issue where if a legacy macro and an attribute with the same name are imported into the same scope, trying to use the attribute throws an error that macros cannot be used as attributes. Can we make this work so that both can be in the same scope and can be used as intended?

@jseyfried

This comment has been minimized.

Show comment
Hide comment
@jseyfried

jseyfried Jan 24, 2017

Contributor

@abonander All macros (bang, attribute, and derive) share the same namespace, so we can't use two different macros with the same name in the same scope. However, we could improve that error message -- could you open an issue?

Contributor

jseyfried commented Jan 24, 2017

@abonander All macros (bang, attribute, and derive) share the same namespace, so we can't use two different macros with the same name in the same scope. However, we could improve that error message -- could you open an issue?

@SimonSapin

This comment has been minimized.

Show comment
Hide comment
@SimonSapin

SimonSapin Feb 1, 2017

Contributor

Sorry I’m late to the party. I’m happy with the direction to expose tokens rather than an AST, but I have some concerns about the specific TokenStream API proposed in the RFC:

pub enum TokenKind {
    Sequence(Delimiter, TokenStream),

    // The content of the comment can be found from the span.
    Comment(CommentKind),

    // `text` is the string contents, not including delimiters. It would be nice
    // to avoid an allocation in the common case that the string is in the
    // source code. We might be able to use `&'codemap str` or something.
    // `raw_markers` is for the count of `#`s if the string is a raw string. If
    // the string is not raw, then it will be `None`.
    String { text: Symbol, raw_markers: Option<usize>, kind: StringKind },

    // char literal, span includes the `'` delimiters.
    Char(char),

    // These tokens are treated specially since they are used for macro
    // expansion or delimiting items.
    Exclamation,  // `!`
    Dollar,       // `$`
    // Not actually sure if we need this or if semicolons can be treated like
    // other punctuation.
    Semicolon,    // `;`
    Eof,          // Do we need this?

    // Word is defined by Unicode Standard Annex 31 -
    // [Unicode Identifier and Pattern Syntax](http://unicode.org/reports/tr31/)
    Word(Symbol),
    Punctuation(char),
}

pub enum StringKind {
    Regular,
    Byte,
}

It’s not clear if this API was intended as a complete plan that was accepted when the RFC was merged, or just an example to be worked out later.

Generally, this seems far from the "normal" Rust syntax accepted by the compiler outside of macros. While some macros will want to parse some ad-hoc domain-specific language, others will want to parse "actual Rust" syntax and make sense of it.

  1. (Minor) I don’t think Eof is necessary. An Iterator will presumably be used to, well, iterate over a TokenStream and Iterator::next already returns None to signal the end of iteration.

  2. (Minor) I don’t think Exclamation, Dollar, or Semicolon are necessary. Matching on Punctuation('!') for example is not more difficult.

  3. (Minor) As others have mentioned in the RFC PR, we might want to omit comments that are not doc-comments. (Any use case that wants to preserve comment likely wants to preserve whitespace too.)

  4. As far as I can tell, what to do with multi-character operators (that probably should be a single token each) is still an open question. A possible solution is discussed in PR comments, but it looks like that didn’t make it into RFC text.

  5. Number literals are missing. Are macros supposed to parse [Punct('1'), Punct('_'), Punct('2'), Punct('3'), Punct('4'), Punct('.'), Punct('5'), Punct('e'), Punct('6')] by themselves to evaluate a literal? They can’t even use str::parse::<f32> to do that, since the syntax it accepts is not the same as Rust literal syntax (which can have _ in the middle, for example).

    I imagine that there’s a stability concern here. Can we introduce new numeric types like u128 / i128 (and possibly in the future f128, u256, …) and their literals, without breaking changes to the tokens API? One way to make this possible might be:

    struct IntegerLiteral { negative: bool, decimal_digits: String, type_suffix: Option<String> }
    impl TryInto<u32> IntegerLiteral { type Err = OutOfRange; /* … */ }
    // Other impls for integer types supported in this compiler version
    
    // Something similarly for floats

    Or maybe something else. But I don’t think "pretend numbers don’t exist" is a good way to do it.

  6. // Word is defined by Unicode Standard Annex 31 -

    This definition needs to be more precise than that. UAX 31 specifies a couple different variations of identifier syntax, and none of them is called "word". But choosing which exact variation we want is why non-ASCII identifiers are feature-gated at the moment.

    Instead, I think this should be defined as "whatever the current compiler accepts as an identifier or keyword" (which can change per #28979). Maybe with a pub fn is_identifier(&str) -> bool public API in libmacro.

  7. Unicode strings and byte string literals share a single token variant, which I think is wrong as the memory representations of their values have different types (str vs [u8]). It’s also not clear if the text: Symbol component is intended to be a literal slice of the source code or the value after resolving backslash escapes. I think it should definitely be the latter. (For comparison, Char(char) has to be the latter since \u{A0} takes more than one char to represent literally.)

Contributor

SimonSapin commented Feb 1, 2017

Sorry I’m late to the party. I’m happy with the direction to expose tokens rather than an AST, but I have some concerns about the specific TokenStream API proposed in the RFC:

pub enum TokenKind {
    Sequence(Delimiter, TokenStream),

    // The content of the comment can be found from the span.
    Comment(CommentKind),

    // `text` is the string contents, not including delimiters. It would be nice
    // to avoid an allocation in the common case that the string is in the
    // source code. We might be able to use `&'codemap str` or something.
    // `raw_markers` is for the count of `#`s if the string is a raw string. If
    // the string is not raw, then it will be `None`.
    String { text: Symbol, raw_markers: Option<usize>, kind: StringKind },

    // char literal, span includes the `'` delimiters.
    Char(char),

    // These tokens are treated specially since they are used for macro
    // expansion or delimiting items.
    Exclamation,  // `!`
    Dollar,       // `$`
    // Not actually sure if we need this or if semicolons can be treated like
    // other punctuation.
    Semicolon,    // `;`
    Eof,          // Do we need this?

    // Word is defined by Unicode Standard Annex 31 -
    // [Unicode Identifier and Pattern Syntax](http://unicode.org/reports/tr31/)
    Word(Symbol),
    Punctuation(char),
}

pub enum StringKind {
    Regular,
    Byte,
}

It’s not clear if this API was intended as a complete plan that was accepted when the RFC was merged, or just an example to be worked out later.

Generally, this seems far from the "normal" Rust syntax accepted by the compiler outside of macros. While some macros will want to parse some ad-hoc domain-specific language, others will want to parse "actual Rust" syntax and make sense of it.

  1. (Minor) I don’t think Eof is necessary. An Iterator will presumably be used to, well, iterate over a TokenStream and Iterator::next already returns None to signal the end of iteration.

  2. (Minor) I don’t think Exclamation, Dollar, or Semicolon are necessary. Matching on Punctuation('!') for example is not more difficult.

  3. (Minor) As others have mentioned in the RFC PR, we might want to omit comments that are not doc-comments. (Any use case that wants to preserve comment likely wants to preserve whitespace too.)

  4. As far as I can tell, what to do with multi-character operators (that probably should be a single token each) is still an open question. A possible solution is discussed in PR comments, but it looks like that didn’t make it into RFC text.

  5. Number literals are missing. Are macros supposed to parse [Punct('1'), Punct('_'), Punct('2'), Punct('3'), Punct('4'), Punct('.'), Punct('5'), Punct('e'), Punct('6')] by themselves to evaluate a literal? They can’t even use str::parse::<f32> to do that, since the syntax it accepts is not the same as Rust literal syntax (which can have _ in the middle, for example).

    I imagine that there’s a stability concern here. Can we introduce new numeric types like u128 / i128 (and possibly in the future f128, u256, …) and their literals, without breaking changes to the tokens API? One way to make this possible might be:

    struct IntegerLiteral { negative: bool, decimal_digits: String, type_suffix: Option<String> }
    impl TryInto<u32> IntegerLiteral { type Err = OutOfRange; /* … */ }
    // Other impls for integer types supported in this compiler version
    
    // Something similarly for floats

    Or maybe something else. But I don’t think "pretend numbers don’t exist" is a good way to do it.

  6. // Word is defined by Unicode Standard Annex 31 -

    This definition needs to be more precise than that. UAX 31 specifies a couple different variations of identifier syntax, and none of them is called "word". But choosing which exact variation we want is why non-ASCII identifiers are feature-gated at the moment.

    Instead, I think this should be defined as "whatever the current compiler accepts as an identifier or keyword" (which can change per #28979). Maybe with a pub fn is_identifier(&str) -> bool public API in libmacro.

  7. Unicode strings and byte string literals share a single token variant, which I think is wrong as the memory representations of their values have different types (str vs [u8]). It’s also not clear if the text: Symbol component is intended to be a literal slice of the source code or the value after resolving backslash escapes. I think it should definitely be the latter. (For comparison, Char(char) has to be the latter since \u{A0} takes more than one char to represent literally.)

@jseyfried jseyfried referenced this issue Feb 2, 2017

Closed

Tracking issue for "Macros 1.1" (RFC #1681) #35900

50 of 53 tasks complete

abonander added a commit to abonander/rust that referenced this issue Feb 2, 2017

frewsxcv added a commit to frewsxcv/rust that referenced this issue Feb 4, 2017

Rollup merge of #39454 - abonander:proc_macro_tracking_issue, r=jseyf…
…ried

Change tracking issue for `proc_macro` feature to #38356

r? @jseyfried

frewsxcv added a commit to frewsxcv/rust that referenced this issue Feb 4, 2017

Rollup merge of #39454 - abonander:proc_macro_tracking_issue, r=jseyf…
…ried

Change tracking issue for `proc_macro` feature to #38356

r? @jseyfried

frewsxcv added a commit to frewsxcv/rust that referenced this issue Feb 5, 2017

Rollup merge of #39454 - abonander:proc_macro_tracking_issue, r=jseyf…
…ried

Change tracking issue for `proc_macro` feature to #38356

r? @jseyfried

frewsxcv added a commit to frewsxcv/rust that referenced this issue Feb 5, 2017

Rollup merge of #39454 - abonander:proc_macro_tracking_issue, r=jseyf…
…ried

Change tracking issue for `proc_macro` feature to #38356

r? @jseyfried

frewsxcv added a commit to frewsxcv/rust that referenced this issue Feb 5, 2017

Rollup merge of #39454 - abonander:proc_macro_tracking_issue, r=jseyf…
…ried

Change tracking issue for `proc_macro` feature to #38356

r? @jseyfried

@SimonSapin SimonSapin referenced this issue Feb 19, 2017

Open

Tracking issue for 1.0.0 tracking issues #39954

8 of 28 tasks complete
@porky11

This comment has been minimized.

Show comment
Hide comment
@porky11

porky11 Feb 21, 2017

another way to write high level macros would be using lisp like macros, but this would need a s-expression representation for the whole rust ast.

porky11 commented Feb 21, 2017

another way to write high level macros would be using lisp like macros, but this would need a s-expression representation for the whole rust ast.

@brson brson added the B-unstable label Mar 1, 2017

frewsxcv added a commit to frewsxcv/rust that referenced this issue Mar 2, 2017

Rollup merge of #40129 - abonander:proc_macro_bang, r=jseyfried
Implement function-like procedural macros ( `#[proc_macro]`)

Adds the `#[proc_macro]` attribute, which expects bare functions of the kind `fn(TokenStream) -> TokenStream`, which can be invoked like `my_macro!()`.

cc rust-lang/rfcs#1913, #38356

r? @jseyfried
cc @nrc

frewsxcv added a commit to frewsxcv/rust that referenced this issue Mar 2, 2017

Rollup merge of #40129 - abonander:proc_macro_bang, r=jseyfried
Implement function-like procedural macros ( `#[proc_macro]`)

Adds the `#[proc_macro]` attribute, which expects bare functions of the kind `fn(TokenStream) -> TokenStream`, which can be invoked like `my_macro!()`.

cc rust-lang/rfcs#1913, #38356

r? @jseyfried
cc @nrc

frewsxcv added a commit to frewsxcv/rust that referenced this issue Mar 2, 2017

Rollup merge of #40129 - abonander:proc_macro_bang, r=jseyfried
Implement function-like procedural macros ( `#[proc_macro]`)

Adds the `#[proc_macro]` attribute, which expects bare functions of the kind `fn(TokenStream) -> TokenStream`, which can be invoked like `my_macro!()`.

cc rust-lang/rfcs#1913, #38356

r? @jseyfried
cc @nrc
@jan-hudec

This comment has been minimized.

Show comment
Hide comment
@jan-hudec

jan-hudec Mar 19, 2017

@SimonSapin,

As others have mentioned in the RFC PR, we might want to omit comments that are not doc-comments. (Any use case that wants to preserve comment likely wants to preserve whitespace too.)

Please don't. I have a use-case where I want to use (though not preserve—they will be written into a separate compilation product instead) comments in the syntax.

Specifically, I want to create translation macros that would load translations of a string from a separate source file(s) and I would like to generate a list of strings to be translated as by-product in debug build. And there needs to be a way to include comments to be emitted into that list (rust-locale/rust-locale#19). So it makes sense to use comment syntax and the macro needs to see them.

I agree with the other points in that post.

jan-hudec commented Mar 19, 2017

@SimonSapin,

As others have mentioned in the RFC PR, we might want to omit comments that are not doc-comments. (Any use case that wants to preserve comment likely wants to preserve whitespace too.)

Please don't. I have a use-case where I want to use (though not preserve—they will be written into a separate compilation product instead) comments in the syntax.

Specifically, I want to create translation macros that would load translations of a string from a separate source file(s) and I would like to generate a list of strings to be translated as by-product in debug build. And there needs to be a way to include comments to be emitted into that list (rust-locale/rust-locale#19). So it makes sense to use comment syntax and the macro needs to see them.

I agree with the other points in that post.

@TeXitoi

This comment has been minimized.

Show comment
Hide comment
@TeXitoi
Contributor

TeXitoi commented Jul 17, 2018

@TeXitoi

This comment has been minimized.

Show comment
Hide comment
@TeXitoi

TeXitoi Jul 17, 2018

Contributor

@realcr your new compilation problem is not related to this issue, please stay on topic.

Contributor

TeXitoi commented Jul 17, 2018

@realcr your new compilation problem is not related to this issue, please stay on topic.

@realcr

This comment has been minimized.

Show comment
Hide comment
@realcr

realcr Jul 17, 2018

@TeXitoi : Feel free to edit or remove anything if you think it is not relevant. I try my best to help you out, it is difficult for me to know what is on topic and what not. The error message "(see issue #38356)" is what brought me here.

realcr commented Jul 17, 2018

@TeXitoi : Feel free to edit or remove anything if you think it is not relevant. I try my best to help you out, it is difficult for me to know what is on topic and what not. The error message "(see issue #38356)" is what brought me here.

@aturon aturon added this to the Rust 2018 Preview 2 milestone Jul 17, 2018

@Pzixel

This comment has been minimized.

Show comment
Hide comment
@Pzixel

Pzixel Jul 19, 2018

I tried to upgrade my compiler version and I've got this error. My code

#![no_std]
#![feature(proc_macro)]
#![feature(proc_macro_gen)]
#![feature(custom_attribute)]
#![feature(alloc)]

#[macro_use(eth_abi)]
extern crate pwasm_abi_derive;

and error that says that I didn't use #![feature(proc_macro)], but I did!

error[E0658]: attribute procedural macros are experimental (see issue #38356)
  --> src\lib.rs:67:5
   |
8  | #[macro_use(eth_abi)]
   |             ------- procedural macro imported here
...
67 |     #[eth_abi(TokenEndpoint, TokenClient)]
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = help: add #![feature(proc_macro)] to the crate attributes to enable

Pzixel commented Jul 19, 2018

I tried to upgrade my compiler version and I've got this error. My code

#![no_std]
#![feature(proc_macro)]
#![feature(proc_macro_gen)]
#![feature(custom_attribute)]
#![feature(alloc)]

#[macro_use(eth_abi)]
extern crate pwasm_abi_derive;

and error that says that I didn't use #![feature(proc_macro)], but I did!

error[E0658]: attribute procedural macros are experimental (see issue #38356)
  --> src\lib.rs:67:5
   |
8  | #[macro_use(eth_abi)]
   |             ------- procedural macro imported here
...
67 |     #[eth_abi(TokenEndpoint, TokenClient)]
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = help: add #![feature(proc_macro)] to the crate attributes to enable

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Jul 19, 2018

Member

@Pzixel you'll want to switch #![feature(proc_macro)] to #![feature(use_extern_macros)] and that should do the trick

Member

alexcrichton commented Jul 19, 2018

@Pzixel you'll want to switch #![feature(proc_macro)] to #![feature(use_extern_macros)] and that should do the trick

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Jul 19, 2018

Member

I think you'll also need to use the module system to import procedural macros (and make sure you have an up-to-date nightly compiler)

Member

alexcrichton commented Jul 19, 2018

I think you'll also need to use the module system to import procedural macros (and make sure you have an up-to-date nightly compiler)

@Pzixel

This comment has been minimized.

Show comment
Hide comment
@Pzixel

Pzixel Jul 19, 2018

@alexcrichton yes, I just figured it out, thanks to the article. However, it still not work:

error[E0433]: failed to resolve. Use of undeclared type or module `Vec`
  --> src\lib.rs:66:5
   |
66 |     #[eth_abi(TokenEndpoint, TokenClient)]
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Use of undeclared type or module `Vec`

error[E0412]: cannot find type `Vec` in this scope
  --> src\lib.rs:66:5
   |
66 |     #[eth_abi(TokenEndpoint, TokenClient)]
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ not found in this scope

Import rules for macros have changed too? Or I can't figure out why it started complain here.

Pzixel commented Jul 19, 2018

@alexcrichton yes, I just figured it out, thanks to the article. However, it still not work:

error[E0433]: failed to resolve. Use of undeclared type or module `Vec`
  --> src\lib.rs:66:5
   |
66 |     #[eth_abi(TokenEndpoint, TokenClient)]
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Use of undeclared type or module `Vec`

error[E0412]: cannot find type `Vec` in this scope
  --> src\lib.rs:66:5
   |
66 |     #[eth_abi(TokenEndpoint, TokenClient)]
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ not found in this scope

Import rules for macros have changed too? Or I can't figure out why it started complain here.

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Jul 19, 2018

Member

@Pzixel that may be a bug in the procedural macro or the compiler, can you file a dedicated issue for that?

Member

alexcrichton commented Jul 19, 2018

@Pzixel that may be a bug in the procedural macro or the compiler, can you file a dedicated issue for that?

@Pzixel

This comment has been minimized.

Show comment
Hide comment
@Pzixel

Pzixel Jul 19, 2018

Well, I think I should rewrite my code first to make it at least looking working :) Right now it doesn't, you guys changed a lot with this feature. BRB when done. If it doesn't disappear I create an issue.

Pzixel commented Jul 19, 2018

Well, I think I should rewrite my code first to make it at least looking working :) Right now it doesn't, you guys changed a lot with this feature. BRB when done. If it doesn't disappear I create an issue.

@LukasKalbertodt LukasKalbertodt referenced this issue Jul 20, 2018

Open

Tests #15

4 of 6 tasks complete
@petrochenkov

This comment has been minimized.

Show comment
Hide comment
@petrochenkov

petrochenkov Jul 22, 2018

Contributor

@alexcrichton
Do you know what "preprocessing" is applied to tokens before they are passed to a procedural macro?
I know that derives has at least $crate eliminated and cfg expanded before they get input tokens (plus roundtrips through strings are not entirely lossless, but that's fixable).

We should make sure none of this happens with newly stabilized procedural macros and they get input tokens precisely (modulo bugs).

Contributor

petrochenkov commented Jul 22, 2018

@alexcrichton
Do you know what "preprocessing" is applied to tokens before they are passed to a procedural macro?
I know that derives has at least $crate eliminated and cfg expanded before they get input tokens (plus roundtrips through strings are not entirely lossless, but that's fixable).

We should make sure none of this happens with newly stabilized procedural macros and they get input tokens precisely (modulo bugs).

@Pzixel

This comment has been minimized.

Show comment
Hide comment
@Pzixel

Pzixel Jul 22, 2018

@alexcrichton sorry, cargo expand doesn't work on this crate for some reason. Cannot confirm if problem is on my side or on compiler one. So I'l continue blaming myself until this possibility is not completely excluded.

Pzixel commented Jul 22, 2018

@alexcrichton sorry, cargo expand doesn't work on this crate for some reason. Cannot confirm if problem is on my side or on compiler one. So I'l continue blaming myself until this possibility is not completely excluded.

@alexcrichton

This comment has been minimized.

Show comment
Hide comment
@alexcrichton

alexcrichton Jul 23, 2018

Member

@petrochenkov the expansion of proc-macros has been pretty well vetted to date so I'm much less worried about that than the name resolution pieces. I'm not personally aware of preprocessing, but I do know that there's an expansion order where derives run last, cfgs run first, and otherwise it's mostly iterative.

I agree though that it's good to audit for!

Member

alexcrichton commented Jul 23, 2018

@petrochenkov the expansion of proc-macros has been pretty well vetted to date so I'm much less worried about that than the name resolution pieces. I'm not personally aware of preprocessing, but I do know that there's an expansion order where derives run last, cfgs run first, and otherwise it's mostly iterative.

I agree though that it's good to audit for!

@Ekleog

This comment has been minimized.

Show comment
Hide comment
@Ekleog

Ekleog Aug 14, 2018

Maybe I missed something obvious. But, is there no way to call a function, use a type, etc. from the proc_macro-defining crate from a procedural macro expansion? (or any other crate known from the proc_macro crate, FWIW)

There are workarounds, but AFAIU they won't work if the crate is renamed by the user of the procedural macro.

Ekleog commented Aug 14, 2018

Maybe I missed something obvious. But, is there no way to call a function, use a type, etc. from the proc_macro-defining crate from a procedural macro expansion? (or any other crate known from the proc_macro crate, FWIW)

There are workarounds, but AFAIU they won't work if the crate is renamed by the user of the procedural macro.

@SimonSapin

This comment has been minimized.

Show comment
Hide comment
@SimonSapin

SimonSapin Aug 14, 2018

Contributor

@Ekleog proc-macro crates are typically only compiled for the purpose of building crates that use them. They are not runtime dependencies. You can think of the entire proc-macro crate as a sort of compiler plugin, not a "normal" library.

This is especially visible when cross-compiling: like build scripts, proc-macros are compile for the host platform, not the target one.

Contributor

SimonSapin commented Aug 14, 2018

@Ekleog proc-macro crates are typically only compiled for the purpose of building crates that use them. They are not runtime dependencies. You can think of the entire proc-macro crate as a sort of compiler plugin, not a "normal" library.

This is especially visible when cross-compiling: like build scripts, proc-macros are compile for the host platform, not the target one.

@Ekleog

This comment has been minimized.

Show comment
Hide comment
@Ekleog

Ekleog Aug 14, 2018

@SimonSapin I agree with you, but it may be really useful to delegate part of the work to a function provided by a crate, even if it is not the proc-macro crate (eg. in my link above, the X-derive crate attempts to use a function from the X crate). Because otherwise that would mean that all code generated by proc-macros have to be either self-contain or make assumptions on the state of the call site.

Then, I can understand that rustc is not ready yet for this kind of feature (because I guess it'd work best without the need to split proc-macro from non-proc-macro crates, which appears to be somewhere on the long-term roadmap). But I wonder, if the current interface using only TokenStreams is stabilized, will it be possible to retrofit this feature later? Wouldn't it need something like a PathToBeResolvedFromTopOfGeneratingProcMacroCrate token type? (which would be a breaking change if added later on, afaiu)

Ekleog commented Aug 14, 2018

@SimonSapin I agree with you, but it may be really useful to delegate part of the work to a function provided by a crate, even if it is not the proc-macro crate (eg. in my link above, the X-derive crate attempts to use a function from the X crate). Because otherwise that would mean that all code generated by proc-macros have to be either self-contain or make assumptions on the state of the call site.

Then, I can understand that rustc is not ready yet for this kind of feature (because I guess it'd work best without the need to split proc-macro from non-proc-macro crates, which appears to be somewhere on the long-term roadmap). But I wonder, if the current interface using only TokenStreams is stabilized, will it be possible to retrofit this feature later? Wouldn't it need something like a PathToBeResolvedFromTopOfGeneratingProcMacroCrate token type? (which would be a breaking change if added later on, afaiu)

@SimonSapin

This comment has been minimized.

Show comment
Hide comment
@SimonSapin

SimonSapin Aug 14, 2018

Contributor

Maybe it’ll be possible to make things more flexible eventually, but that seems fairly far off.

In the meantime, if you’re a library author, consider having a foo-proc-macros or foo-derive crate for procedural macros, and a "normal" foo library that contains run-time code but also re-exports the procedural macros. That way the user-facing API can be kept to a single crate. This is what serde does (in some configs) https://github.com/serde-rs/serde/blob/v1.0.71/serde/src/lib.rs#L304

Contributor

SimonSapin commented Aug 14, 2018

Maybe it’ll be possible to make things more flexible eventually, but that seems fairly far off.

In the meantime, if you’re a library author, consider having a foo-proc-macros or foo-derive crate for procedural macros, and a "normal" foo library that contains run-time code but also re-exports the procedural macros. That way the user-facing API can be kept to a single crate. This is what serde does (in some configs) https://github.com/serde-rs/serde/blob/v1.0.71/serde/src/lib.rs#L304

@mjbshaw

This comment has been minimized.

Show comment
Hide comment
@mjbshaw

mjbshaw Aug 15, 2018

It's worth pointing out though that this workaround still doesn't address the issue of the user renaming the root crate (e.g. serde's tracking issue).

mjbshaw commented Aug 15, 2018

It's worth pointing out though that this workaround still doesn't address the issue of the user renaming the root crate (e.g. serde's tracking issue).

@CAD97

This comment has been minimized.

Show comment
Hide comment
@CAD97
Contributor

CAD97 commented Aug 21, 2018

@jan-hudec

This comment has been minimized.

Show comment
Hide comment
@jan-hudec

This comment has been minimized.

Show comment
Hide comment
@jan-hudec

jan-hudec Aug 21, 2018

@Ekleog, the TokenStream is a stream of TokenTrees and each TokenTree has associated Span, which carries the scoping information. Except there is currently no way to create a Span for any other scope than “call site” (or empty). Basically coming up with a reasonably ergonomic way of creating Spans referring to specific crate is what is needed.

jan-hudec commented Aug 21, 2018

@Ekleog, the TokenStream is a stream of TokenTrees and each TokenTree has associated Span, which carries the scoping information. Except there is currently no way to create a Span for any other scope than “call site” (or empty). Basically coming up with a reasonably ergonomic way of creating Spans referring to specific crate is what is needed.

@CAD97

This comment has been minimized.

Show comment
Hide comment
@CAD97

CAD97 Aug 21, 2018

Contributor

The reason I asked is that the checkbox isn't ticked. Would be nice to tick it off then!

With #![feature(proc_macro)] stabilized, what's left of this issue?

Contributor

CAD97 commented Aug 21, 2018

The reason I asked is that the checkbox isn't ticked. Would be nice to tick it off then!

With #![feature(proc_macro)] stabilized, what's left of this issue?

@Ekleog

This comment has been minimized.

Show comment
Hide comment
@Ekleog

Ekleog Aug 21, 2018

@jan-hudec Oh, I thought Spans were only for error reporting, as early blog posts mentioned a Hygiene (or similarly named) struct that played this role. I had assumed that these had vanished, and was apparently wrong. Thanks! :)

Ekleog commented Aug 21, 2018

@jan-hudec Oh, I thought Spans were only for error reporting, as early blog posts mentioned a Hygiene (or similarly named) struct that played this role. I had assumed that these had vanished, and was apparently wrong. Thanks! :)

@petrochenkov

This comment has been minimized.

Show comment
Hide comment
@petrochenkov

petrochenkov Aug 21, 2018

Contributor

With #![feature(proc_macro)] stabilized, what's left of this issue?

Ideally, fresh issues need be filed for all individual remaining issues and not stabilized features, and then this issue can be closed (in the same way like it was done for #44660).

Contributor

petrochenkov commented Aug 21, 2018

With #![feature(proc_macro)] stabilized, what's left of this issue?

Ideally, fresh issues need be filed for all individual remaining issues and not stabilized features, and then this issue can be closed (in the same way like it was done for #44660).

@mark-i-m

This comment has been minimized.

Show comment
Hide comment
@mark-i-m

mark-i-m Aug 21, 2018

Contributor

Oh, I thought Spans were only for error reporting, as early blog posts mentioned a Hygiene (or similarly named) struct that played this role. I had assumed that these had vanished, and was apparently wrong. Thanks! :)

IIUC, Spans are the primary way of tracking hygiene context.

Contributor

mark-i-m commented Aug 21, 2018

Oh, I thought Spans were only for error reporting, as early blog posts mentioned a Hygiene (or similarly named) struct that played this role. I had assumed that these had vanished, and was apparently wrong. Thanks! :)

IIUC, Spans are the primary way of tracking hygiene context.

@alexreg

This comment has been minimized.

Show comment
Hide comment
@alexreg

alexreg Aug 21, 2018

Contributor

@mark-i-m Kind of. They include both source code location information (for user-facing messages/diagnostics) and syntax context (i.e. hygiene information).

Contributor

alexreg commented Aug 21, 2018

@mark-i-m Kind of. They include both source code location information (for user-facing messages/diagnostics) and syntax context (i.e. hygiene information).

@sgrif

This comment has been minimized.

Show comment
Hide comment
@sgrif

sgrif Sep 7, 2018

Contributor

Given the amount of discussion/traffic this issue gets, does it make sense to move proc_macro_diagnostic to its own tracking issue? I'd also like to figure out what the blockers are for the stabilization of that feature, and see if we can push it through. Diesel uses it, and it's been great so far. The lack of this feature on stable is leading to the community creating funky workarounds, like syn's newest version using compile_error! in its place.

Contributor

sgrif commented Sep 7, 2018

Given the amount of discussion/traffic this issue gets, does it make sense to move proc_macro_diagnostic to its own tracking issue? I'd also like to figure out what the blockers are for the stabilization of that feature, and see if we can push it through. Diesel uses it, and it's been great so far. The lack of this feature on stable is leading to the community creating funky workarounds, like syn's newest version using compile_error! in its place.

@SergioBenitez

This comment has been minimized.

Show comment
Hide comment
@SergioBenitez

SergioBenitez Sep 11, 2018

Contributor

@sgrif I've opened up such an issue: #54140.

Contributor

SergioBenitez commented Sep 11, 2018

@sgrif I've opened up such an issue: #54140.

@xd009642

This comment has been minimized.

Show comment
Hide comment
@xd009642

xd009642 Sep 17, 2018

So I'm interesting in helping to stabilise the methods in Span and the LineColumn struct. I'll look at the open issues in the first comment, but if anyone wants to point a newbie to the compiler in a particular direction I'd appreciate it 👍

xd009642 commented Sep 17, 2018

So I'm interesting in helping to stabilise the methods in Span and the LineColumn struct. I'll look at the open issues in the first comment, but if anyone wants to point a newbie to the compiler in a particular direction I'd appreciate it 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment