New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial implementation of declarative macros 2.0 #40847

Merged
merged 13 commits into from May 26, 2017

Conversation

Projects
None yet
@jseyfried
Contributor

jseyfried commented Mar 27, 2017

Implement declarative macros 2.0 (rust-lang/rfcs#1584) behind #![feature(decl_macro)].
Differences from macro_rules! include:

  • new syntax: macro m(..) { .. } instead of macro_rules! m { (..) => { .. } }
  • declarative macros are items:
// crate A:
pub mod foo {
    m!(); // use before definition; declaration order is irrelevant
    pub macro m() {} // `pub`, `pub(super)`, etc. work
}
fn main() {
    foo::m!(); // named like other items
    { use foo::m as n; n!(); } // imported like other items
}
pub use foo::m; // re-exported like other items

// crate B:
extern crate A; // no need for `#[macro_use]`
A::foo::m!(); A::m!();
  • Racket-like hygiene for items, imports, methods, fields, type parameters, privacy, etc.
    • Intuitively, names in a macro definition are resolved in the macro definition's scope, not the scope in which the macro is used.
    • This explaination of hygiene for Racket applies here (except for the "Breaking Hygiene" section). I wrote a similar explanation for Rust.
    • Generally speaking, if fn f() { <body> } resolves, pub macro m() { <body> } ... m!() also resolves, even if m!() is in a separate crate.
    • ::foo::bar in a macro behaves like $crate::foo::bar in a macro_rules!, except it can access everything visible from the macro (thus more permissive).
    • See src/test/{run-pass, compile-fail}/hygiene for examples. Small example:
mod foo {
    fn f() { println!("hello world"); }
    pub macro m() { f(); }
}
fn main() { foo::m!(); }

Limitations:

  • This does not address planned changes to matchers (expr,ty, etc.), c.f. #26361.
  • Lints (including stability and deprecation) and unsafe are not hygienic.
    • adding hygiene here will be mostly or entirely backwards compatible
  • Nested macro definitions (a macro inside another macro) don't always work correctly when invoked from external crates.
    • pending improvements in how we encode macro definitions in crate metadata
  • There is no way to "escape" hygiene without using a procedural macro.

r? @nrc

@jseyfried jseyfried force-pushed the jseyfried:decl_macro branch 5 times, most recently from 0b2df87 to d82defb Mar 27, 2017

@jseyfried jseyfried referenced this pull request Mar 27, 2017

Open

Tracking issue: declarative macros 2.0 #39412

9 of 19 tasks complete
@jseyfried

This comment has been minimized.

Contributor

jseyfried commented Mar 27, 2017

@petrochenkov

This comment has been minimized.

Contributor

petrochenkov commented Mar 27, 2017

(I'm going to review this too in a few days.)

token::Ident(ident) if ident.name == keywords::Macro.name() => {
self.bump();
let ident = self.parse_ident()?;
let tokens = if self.token == token::OpenDelim(token::Brace) {

This comment has been minimized.

@petrochenkov

petrochenkov Mar 28, 2017

Contributor

It's usually better to use self.check(&token::OpenDelim(token::Brace)) to populate expected tokens.

This comment has been minimized.

@jseyfried

jseyfried Mar 28, 2017

Contributor

Good point, will do.

TokenTree::Delimited(_, ref delimited) => delimited.stream(),
_ => unreachable!(),
}
} else if self.token == token::OpenDelim(token::Paren) {

This comment has been minimized.

@petrochenkov

petrochenkov Mar 28, 2017

Contributor

self.check(...) as well

}
} else if self.token == token::OpenDelim(token::Paren) {
let args = self.parse_token_tree();
let body = if self.token == token::OpenDelim(token::Brace) {

This comment has been minimized.

@petrochenkov

petrochenkov Mar 28, 2017

Contributor

And here, plus self.unexpected() can be used for the else clause.

@@ -783,7 +783,7 @@ impl<'a, 'b:'a> ImportResolver<'a, 'b> {
};
if binding.vis == ty::Visibility::Public &&
(binding.is_import() || binding.is_extern_crate()) {
(binding.is_import() || binding.is_macro_def()) {

This comment has been minimized.

@petrochenkov

petrochenkov Mar 28, 2017

Contributor

Not sure what this change does.

This comment has been minimized.

@jseyfried

jseyfried Mar 28, 2017

Contributor

Removing binding.is_extern_crate() is a no-op (it's implied by binding.is_import()), adding binding.is_macro_def() allows the binding to be accessed from external crates.

@@ -1107,7 +1107,7 @@ pub struct Resolver<'a> {
prelude: Option<Module<'a>>,
trait_item_map: FxHashMap<(DefId, Name, Namespace), (Def, bool /* has self */)>,
has_self: FxHashSet<DefId>,

This comment has been minimized.

@petrochenkov

petrochenkov Mar 28, 2017

Contributor

Could you leave a comment that this is used only for better diagnostics, not for the resolution algorithm.
(Maybe group all resolver fields of this kind together.)

This comment has been minimized.

@jseyfried

jseyfried Mar 28, 2017

Contributor

Will do.

@petrochenkov

This comment has been minimized.

Contributor

petrochenkov commented Mar 28, 2017

The RFC draft contains this exception from hygiene rules - "except when matching against names in patterns".
What is this about? Could you give an example?
(I haven't read the few last commits yet, maybe they already contain the answer.)

@jseyfried

This comment has been minimized.

Contributor

jseyfried commented Mar 28, 2017

@petrochenkov I mean something like this:

macro n($i:ident) {
    m!($i, x); // The resolution of `x` depends on whether `$i` is `foo` or `bar`
}

// Here, we are "matching against names in patterns" with `foo` and `bar`:
macro m {
    (foo, $j:ident) => { let $j = 0; $j },
    (bar, $j:ident) => { $j },
}
scope
}
pub fn glob_adjust(&mut self, expansion: Mark, mut glob_ctxt: SyntaxContext)

This comment has been minimized.

@petrochenkov

petrochenkov Mar 28, 2017

Contributor

Could you leave some, maybe minimal, documentation for the new public interface of SyntaxContext? What is the intent of these adjustments, where are they used, how do they correspond to Rust code.

This comment has been minimized.

@jseyfried

jseyfried Mar 28, 2017

Contributor

Yeah, added comments.

@@ -530,12 +530,15 @@ pub enum SyntaxExtension {
/// An attribute-like procedural macro that derives a builtin trait.
BuiltinDerive(BuiltinDeriveFn),
DeclMacro(Box<TTMacroExpander>, Option<Span>),

This comment has been minimized.

@petrochenkov

petrochenkov Mar 28, 2017

Contributor

Doc comment here as well, please.

This comment has been minimized.

@jseyfried

jseyfried Mar 28, 2017

Contributor

Done.

@jseyfried jseyfried force-pushed the jseyfried:decl_macro branch 2 times, most recently from fdfd619 to eaa9bb4 Mar 29, 2017

@jseyfried

This comment has been minimized.

Contributor

jseyfried commented Mar 29, 2017

@petrochenkov I documented the adjust functions and addressed your other comments.

@petrochenkov

This comment has been minimized.

Contributor

petrochenkov commented Mar 29, 2017

@jseyfried
This is only the beginning! I haven't read the "hygienize" and "improve" commits yet 😄

@jseyfried jseyfried force-pushed the jseyfried:decl_macro branch 2 times, most recently from ca341e3 to afe7d89 Mar 30, 2017

@bors

This comment has been minimized.

Contributor

bors commented Mar 30, 2017

☔️ The latest upstream changes (presumably #40224) made this pull request unmergeable. Please resolve the merge conflicts.

@jseyfried jseyfried force-pushed the jseyfried:decl_macro branch from afe7d89 to fbf18a2 Mar 30, 2017

@petrochenkov

This comment has been minimized.

Contributor

petrochenkov commented Mar 30, 2017

FWIW, I finished reading (more or less).
I don't see critical problems, but there is lot of code that I will never understand without weeks of experimenting which I don't have unfortunately, so I can't guarantee anything. I also didn't bother reporting nits, including naming issues.

I'm unhappy with greatly raised complexity and leakage of macros and hygiene into later stages of compilation, I can only hope it will worth it, because the general idea of per-identifier hygiene/resolution/privacy is very attractive.

It looks like private-in-public checks are left alone. They are still unhygienic, right?
This is good for now, "semantic" type privacy seems incompatible with identifier-based privacy implemented here.

The embargo visitor is terribly pessimistic, if I understand it correctly, but it all can be improved in the future.

@petrochenkov I mean something like this:

I see, I though about pattern-patterns, not macro-patterns.

@jseyfried

This comment has been minimized.

Contributor

jseyfried commented Mar 31, 2017

@petrochenkov

I'm unhappy with greatly raised complexity and leakage of macros and hygiene into later stages of compilation

I'm afraid this is unavoidable if we want names in macro definitions to have reliable resolutions (as defined in the draft RFC).

It looks like private-in-public checks are left alone. They are still unhygienic, right?

Yeah, this doesn't address private-in-public hygiene. I'm currently leaning towards:

macro m($i:ident) {
    fn $i(_: <type>) {} // <type> must be accessible from the use site (enforced today)
    fn f(_: <type>){} // <type> must be accessible from the use site *or* the def site
}

After we address #30476, inferred types would also need to be accessible from the use site or the def site.

The embargo visitor is terribly pessimistic

Indeed. However, this pessimism is needed with the current implementation of procedural macros (#40939), since a procedural macro can rename an identifier and leave the hygiene information (in the span) unchanged. Thus, we must be prepared to resolve arbitrary names at the def site of exported declarative macros.

@tikue

This comment has been minimized.

Contributor

tikue commented May 31, 2017

Should this example work?

// src/lib.rs
#![feature(decl_macro)]

pub macro use_fmt {
    () => {
        use std::fmt;
    }
}

// examples/test.rs
#![feature(use_extern_macros)]

extern crate macro_test;

use macro_test::use_fmt;

use_fmt!();

fn main() {}

The error:

$ cargo build --example test
   Compiling macro-test v0.1.0 (file:///.../macro-test)
error[E0432]: unresolved import `std::fmt`
 --> examples/test.rs:7:1
  |
7 | use_fmt!();
  | ^^^^^^^^^^^ Could not find `std` in `{{root}}`
  |
  = note: this error originates in a macro outside of the current crate

error: aborting due to previous error(s)

error: Could not compile `macro-test`.

Invoking the macro in the same crate as it's defined works as expected.

@jseyfried jseyfried deleted the jseyfried:decl_macro branch May 31, 2017

@jseyfried

This comment has been minimized.

Contributor

jseyfried commented May 31, 2017

@tikue Yeah, in general it's a bug if invoking a macro in an external crate behaves differently than invoking it locally. Could you open an issue (just copy the comment) and cc me?

@tikue

This comment has been minimized.

Contributor

tikue commented May 31, 2017

Thanks @jseyfried, will do.

frewsxcv added a commit to frewsxcv/rust that referenced this pull request Jun 2, 2017

Rollup merge of rust-lang#42334 - est31:master, r=jseyfried
Extend the unused macro lint to macros 2.0

Extends the unused macro lint (added in PR rust-lang#41907) to macros 2.0 (added in PR rust-lang#40847).

r? @jseyfried

bors added a commit that referenced this pull request Jun 3, 2017

Auto merge of #42334 - est31:master, r=jseyfried
Extend the unused macro lint to macros 2.0

Extends the unused macro lint (added in PR #41907) to macros 2.0 (added in PR #40847).

r? @jseyfried

bors added a commit that referenced this pull request Jun 5, 2017

Auto merge of #42125 - petrochenkov:privty, r=<try>
Check types for privacy

This PR implements late post factum checking of type privacy, as opposed to early preventive "private-in-public" checking.
This will allow to turn private-in-public checks into a lint and make them more heuristic-based, and more aligned with what people may expect (e.g. reachability-based behavior).

Types are privacy-checked if they are written explicitly, and also if they are inferred as expression or pattern types.
This PR checks "semantic" types and does it unhygienically, this significantly restricts what macros 2.0 (as implemented in #40847) can do (sorry @jseyfried) - they still can use private *names*, but can't use private *types*.
This is the most conservative solution, but hopefully it's temporary and can be relaxed in the future, probably using macro contexts of expression/pattern spans.

Traits are also checked in preparation for [trait aliases](#41517), which will be able to leak private traits, and macros 2.0 which will be able to leak pretty much anything.

This is a [breaking-change], but the code that is not contrived and can be broken by this patch should be guarded by `private_in_public` lint. [Previous crater run](#34537 (comment)) discovered a few abandoned crates that weren't updated since `private_in_public` has been introduced in 2015.

cc #34537 https://internals.rust-lang.org/t/lang-team-minutes-private-in-public-rules/4504
Fixes #30476
Fixes #33479

cc @nikomatsakis
r? @eddyb

bors added a commit that referenced this pull request Jun 19, 2017

Auto merge of #42125 - petrochenkov:privty, r=alexcrichton
Check types for privacy

This PR implements late post factum checking of type privacy, as opposed to early preventive "private-in-public" checking.
This will allow to turn private-in-public checks into a lint and make them more heuristic-based, and more aligned with what people may expect (e.g. reachability-based behavior).

Types are privacy-checked if they are written explicitly, and also if they are inferred as expression or pattern types.
This PR checks "semantic" types and does it unhygienically, this significantly restricts what macros 2.0 (as implemented in #40847) can do (sorry @jseyfried) - they still can use private *names*, but can't use private *types*.
This is the most conservative solution, but hopefully it's temporary and can be relaxed in the future, probably using macro contexts of expression/pattern spans.

Traits are also checked in preparation for [trait aliases](#41517), which will be able to leak private traits, and macros 2.0 which will be able to leak pretty much anything.

This is a [breaking-change], but the code that is not contrived and can be broken by this patch should be guarded by `private_in_public` lint. [Previous crater run](#34537 (comment)) discovered a few abandoned crates that weren't updated since `private_in_public` has been introduced in 2015.

cc #34537 https://internals.rust-lang.org/t/lang-team-minutes-private-in-public-rules/4504
Fixes #30476
Fixes #33479

cc @nikomatsakis
r? @eddyb
@whataloadofwhat

This comment has been minimized.

Contributor

whataloadofwhat commented Jun 24, 2017

I'm not entirely sure if this should be happening but it seems that you can't actually implement a trait from outside of a macro while inside a macro unless the Trait doesn't have anything to implement. While the code in #40847 (comment) seems to compile, it only compiles because fn f(&self) { } has a default implementation. If you change it to fn f(&self); then it no longer works and it complains that f is missing. It's also very misleading because the function that you attempt to implement it with is silently ignored, and the default implementation will run instead.

#![feature(decl_macro)]

trait Tr {
    fn f(&self) {
        println!("This shouldn't happen");
    }
}

pub macro m($t:ty) {
    impl Tr for $t {
        fn f(&self) {
            println!("Run me");
        }
    }
}

struct S;
m!(S);

fn main() {
    S.f(); // "This shouldn't happen"
}
@petrochenkov

This comment has been minimized.

Contributor

petrochenkov commented Jun 24, 2017

@whataloadofwhat
This is certainly a bug.
Ancestors::defs in specialization_graph.rs and its uses seem to not be updated for macro 2.0 hygiene.

bors added a commit that referenced this pull request Jun 26, 2017

Auto merge of #40939 - jseyfried:proc_macro_api, r=nrc
proc_macro: implement `TokenTree`, `TokenKind`, hygienic `quote!`, and other API

All new API is gated behind `#![feature(proc_macro)]` and may be used with `#[proc_macro]`, `#[proc_macro_attribute]`, and `#[proc_macro_derive]` procedural macros.

More specifically, this PR adds the following in `proc_macro`:
```rust
// `TokenStream` constructors:
impl TokenStream { fn empty() -> TokenStream { ... } }
impl From<TokenTree> for TokenStream { ... }
impl From<TokenKind> for TokenStream { ... }
impl<T: Into<TokenStream>> FromIterator<T> for TokenStream { ... }
macro quote($($t:tt)*) { ... } // A hygienic `TokenStream` quoter

// `TokenStream` destructuring:
impl TokenStream { fn is_empty(&self) -> bool { ... } }
impl IntoIterator for TokenStream { type Item = TokenTree; ... }

struct TokenTree { span: Span, kind: TokenKind }
impl From<TokenKind> for TokenTree { ... }
impl Display for TokenTree { ... }

struct Span { ... } // a region of source code along with expansion/hygiene information
impl Default for Span { ... } // a span from the current procedural macro definition
impl Span { fn call_site() -> Span { ... } } // the call site of the current expansion
fn quote_span(span: Span) -> TokenStream;

enum TokenKind {
    Sequence(Delimiter, TokenStream), // A delimited sequence, e.g. `( ... )`
    Term(Term), // a unicode identifier, lifetime ('a), or underscore
    Op(char, Spacing), // a punctuation character (`+`, `,`, `$`, etc.).
    Literal(Literal), // a literal character (`'a'`), string (`"hello"`), or number (`2.3`)
}

enum Delimiter {
    Parenthesis, // `( ... )`
    Brace, // `[ ... ]`
    Bracket, // `{ ... }`
    None, // an implicit delimiter, e.g. `$var`, where $var is  `...`.
}

struct Term { ... } // An interned string
impl Term {
    fn intern(string: &str) -> Symbol { ... }
    fn as_str(&self) -> &str { ... }
}

enum Spacing {
    Alone, // not immediately followed by another `Op`, e.g. `+` in `+ =`.
    Joint, // immediately followed by another `Op`, e.g. `+` in `+=`
}

struct Literal { ... }
impl Display for Literal { ... }
impl Literal {
    fn integer(n: i128) -> Literal { .. } // unsuffixed integer literal
    fn float(n: f64) -> Literal { .. } // unsuffixed floating point literal
    fn u8(n: u8) -> Literal { ... } // similarly: i8, u16, i16, u32, i32, u64, i64, f32, f64
    fn string(string: &str) -> Literal { ... }
    fn character(ch: char) -> Literal { ... }
    fn byte_string(bytes: &[u8]) -> Literal { ... }
}
```
For details on `quote!` hygiene, see [this example](2a43350) and [declarative macros 2.0](#40847).

r? @nrc

bors added a commit that referenced this pull request Jun 27, 2017

Auto merge of #40939 - jseyfried:proc_macro_api, r=nrc
proc_macro: implement `TokenTree`, `TokenKind`, hygienic `quote!`, and other API

All new API is gated behind `#![feature(proc_macro)]` and may be used with `#[proc_macro]`, `#[proc_macro_attribute]`, and `#[proc_macro_derive]` procedural macros.

More specifically, this PR adds the following in `proc_macro`:
```rust
// `TokenStream` constructors:
impl TokenStream { fn empty() -> TokenStream { ... } }
impl From<TokenTree> for TokenStream { ... }
impl From<TokenKind> for TokenStream { ... }
impl<T: Into<TokenStream>> FromIterator<T> for TokenStream { ... }
macro quote($($t:tt)*) { ... } // A hygienic `TokenStream` quoter

// `TokenStream` destructuring:
impl TokenStream { fn is_empty(&self) -> bool { ... } }
impl IntoIterator for TokenStream { type Item = TokenTree; ... }

struct TokenTree { span: Span, kind: TokenKind }
impl From<TokenKind> for TokenTree { ... }
impl Display for TokenTree { ... }

struct Span { ... } // a region of source code along with expansion/hygiene information
impl Default for Span { ... } // a span from the current procedural macro definition
impl Span { fn call_site() -> Span { ... } } // the call site of the current expansion
fn quote_span(span: Span) -> TokenStream;

enum TokenKind {
    Group(Delimiter, TokenStream), // A delimited sequence, e.g. `( ... )`
    Term(Term), // a unicode identifier, lifetime ('a), or underscore
    Op(char, Spacing), // a punctuation character (`+`, `,`, `$`, etc.).
    Literal(Literal), // a literal character (`'a'`), string (`"hello"`), or number (`2.3`)
}

enum Delimiter {
    Parenthesis, // `( ... )`
    Brace, // `[ ... ]`
    Bracket, // `{ ... }`
    None, // an implicit delimiter, e.g. `$var`, where $var is  `...`.
}

struct Term { ... } // An interned string
impl Term {
    fn intern(string: &str) -> Symbol { ... }
    fn as_str(&self) -> &str { ... }
}

enum Spacing {
    Alone, // not immediately followed by another `Op`, e.g. `+` in `+ =`.
    Joint, // immediately followed by another `Op`, e.g. `+` in `+=`
}

struct Literal { ... }
impl Display for Literal { ... }
impl Literal {
    fn integer(n: i128) -> Literal { .. } // unsuffixed integer literal
    fn float(n: f64) -> Literal { .. } // unsuffixed floating point literal
    fn u8(n: u8) -> Literal { ... } // similarly: i8, u16, i16, u32, i32, u64, i64, f32, f64
    fn string(string: &str) -> Literal { ... }
    fn character(ch: char) -> Literal { ... }
    fn byte_string(bytes: &[u8]) -> Literal { ... }
}
```
For details on `quote!` hygiene, see [this example](20a9048) and [declarative macros 2.0](#40847).

r? @nrc

bors added a commit that referenced this pull request Jun 27, 2017

Auto merge of #40939 - jseyfried:proc_macro_api, r=nrc
proc_macro: implement `TokenTree`, `TokenKind`, hygienic `quote!`, and other API

All new API is gated behind `#![feature(proc_macro)]` and may be used with `#[proc_macro]`, `#[proc_macro_attribute]`, and `#[proc_macro_derive]` procedural macros.

More specifically, this PR adds the following in `proc_macro`:
```rust
// `TokenStream` constructors:
impl TokenStream { fn empty() -> TokenStream { ... } }
impl From<TokenTree> for TokenStream { ... }
impl From<TokenKind> for TokenStream { ... }
impl<T: Into<TokenStream>> FromIterator<T> for TokenStream { ... }
macro quote($($t:tt)*) { ... } // A hygienic `TokenStream` quoter

// `TokenStream` destructuring:
impl TokenStream { fn is_empty(&self) -> bool { ... } }
impl IntoIterator for TokenStream { type Item = TokenTree; ... }

struct TokenTree { span: Span, kind: TokenKind }
impl From<TokenKind> for TokenTree { ... }
impl Display for TokenTree { ... }

struct Span { ... } // a region of source code along with expansion/hygiene information
impl Default for Span { ... } // a span from the current procedural macro definition
impl Span { fn call_site() -> Span { ... } } // the call site of the current expansion
fn quote_span(span: Span) -> TokenStream;

enum TokenKind {
    Group(Delimiter, TokenStream), // A delimited sequence, e.g. `( ... )`
    Term(Term), // a unicode identifier, lifetime ('a), or underscore
    Op(char, Spacing), // a punctuation character (`+`, `,`, `$`, etc.).
    Literal(Literal), // a literal character (`'a'`), string (`"hello"`), or number (`2.3`)
}

enum Delimiter {
    Parenthesis, // `( ... )`
    Brace, // `[ ... ]`
    Bracket, // `{ ... }`
    None, // an implicit delimiter, e.g. `$var`, where $var is  `...`.
}

struct Term { ... } // An interned string
impl Term {
    fn intern(string: &str) -> Symbol { ... }
    fn as_str(&self) -> &str { ... }
}

enum Spacing {
    Alone, // not immediately followed by another `Op`, e.g. `+` in `+ =`.
    Joint, // immediately followed by another `Op`, e.g. `+` in `+=`
}

struct Literal { ... }
impl Display for Literal { ... }
impl Literal {
    fn integer(n: i128) -> Literal { .. } // unsuffixed integer literal
    fn float(n: f64) -> Literal { .. } // unsuffixed floating point literal
    fn u8(n: u8) -> Literal { ... } // similarly: i8, u16, i16, u32, i32, u64, i64, f32, f64
    fn string(string: &str) -> Literal { ... }
    fn character(ch: char) -> Literal { ... }
    fn byte_string(bytes: &[u8]) -> Literal { ... }
}
```
For details on `quote!` hygiene, see [this example](20a9048) and [declarative macros 2.0](#40847).

r? @nrc

@petrochenkov petrochenkov referenced this pull request Jul 4, 2017

Closed

Disable macro hygiene #2054

bors added a commit that referenced this pull request Jul 5, 2017

Auto merge of #40939 - jseyfried:proc_macro_api, r=nrc
proc_macro: implement `TokenTree`, `TokenKind`, hygienic `quote!`, and other API

All new API is gated behind `#![feature(proc_macro)]` and may be used with `#[proc_macro]`, `#[proc_macro_attribute]`, and `#[proc_macro_derive]` procedural macros.

More specifically, this PR adds the following in `proc_macro`:
```rust
// `TokenStream` constructors:
impl TokenStream { fn empty() -> TokenStream { ... } }
impl From<TokenTree> for TokenStream { ... }
impl From<TokenKind> for TokenStream { ... }
impl<T: Into<TokenStream>> FromIterator<T> for TokenStream { ... }
macro quote($($t:tt)*) { ... } // A hygienic `TokenStream` quoter

// `TokenStream` destructuring:
impl TokenStream { fn is_empty(&self) -> bool { ... } }
impl IntoIterator for TokenStream { type Item = TokenTree; ... }

struct TokenTree { span: Span, kind: TokenKind }
impl From<TokenKind> for TokenTree { ... }
impl Display for TokenTree { ... }

struct Span { ... } // a region of source code along with expansion/hygiene information
impl Default for Span { ... } // a span from the current procedural macro definition
impl Span { fn call_site() -> Span { ... } } // the call site of the current expansion
fn quote_span(span: Span) -> TokenStream;

enum TokenKind {
    Group(Delimiter, TokenStream), // A delimited sequence, e.g. `( ... )`
    Term(Term), // a unicode identifier, lifetime ('a), or underscore
    Op(char, Spacing), // a punctuation character (`+`, `,`, `$`, etc.).
    Literal(Literal), // a literal character (`'a'`), string (`"hello"`), or number (`2.3`)
}

enum Delimiter {
    Parenthesis, // `( ... )`
    Brace, // `[ ... ]`
    Bracket, // `{ ... }`
    None, // an implicit delimiter, e.g. `$var`, where $var is  `...`.
}

struct Term { ... } // An interned string
impl Term {
    fn intern(string: &str) -> Symbol { ... }
    fn as_str(&self) -> &str { ... }
}

enum Spacing {
    Alone, // not immediately followed by another `Op`, e.g. `+` in `+ =`.
    Joint, // immediately followed by another `Op`, e.g. `+` in `+=`
}

struct Literal { ... }
impl Display for Literal { ... }
impl Literal {
    fn integer(n: i128) -> Literal { .. } // unsuffixed integer literal
    fn float(n: f64) -> Literal { .. } // unsuffixed floating point literal
    fn u8(n: u8) -> Literal { ... } // similarly: i8, u16, i16, u32, i32, u64, i64, f32, f64
    fn string(string: &str) -> Literal { ... }
    fn character(ch: char) -> Literal { ... }
    fn byte_string(bytes: &[u8]) -> Literal { ... }
}
```
For details on `quote!` hygiene, see [this example](20a9048) and [declarative macros 2.0](#40847).

r? @nrc

bors added a commit that referenced this pull request Jul 7, 2017

Auto merge of #42125 - petrochenkov:privty, r=nikomatsakis
Check types for privacy

This PR implements late post factum checking of type privacy, as opposed to early preventive "private-in-public" checking.
This will allow to turn private-in-public checks into a lint and make them more heuristic-based, and more aligned with what people may expect (e.g. reachability-based behavior).

Types are privacy-checked if they are written explicitly, and also if they are inferred as expression or pattern types.
This PR checks "semantic" types and does it unhygienically, this significantly restricts what macros 2.0 (as implemented in #40847) can do (sorry @jseyfried) - they still can use private *names*, but can't use private *types*.
This is the most conservative solution, but hopefully it's temporary and can be relaxed in the future, probably using macro contexts of expression/pattern spans.

Traits are also checked in preparation for [trait aliases](#41517), which will be able to leak private traits, and macros 2.0 which will be able to leak pretty much anything.

This is a [breaking-change], but the code that is not contrived and can be broken by this patch should be guarded by `private_in_public` lint. [Previous crater run](#34537 (comment)) discovered a few abandoned crates that weren't updated since `private_in_public` has been introduced in 2015.

cc #34537 https://internals.rust-lang.org/t/lang-team-minutes-private-in-public-rules/4504
Fixes #30476
Fixes #33479

cc @nikomatsakis
r? @eddyb

bors added a commit that referenced this pull request Oct 6, 2017

Auto merge of #44818 - petrochenkov:astymac2, r=jseyfried
Improve resolution of associated types in declarative macros 2.0

Make various identifier comparisons for associated types (and sometimes other associated items) hygienic.
Now declarative macros 2.0 can use `Self::AssocTy`, `TyParam::AssocTy`, `Trait<AssocTy = u8>` where `AssocTy` is an associated type of a trait `Trait` visible from the macro. Also, `Trait` can now be implemented inside the macro and specialization should work properly (fixes #40847 (comment)).

r? @jseyfried or @eddyb

bors added a commit that referenced this pull request Oct 6, 2017

Auto merge of #44818 - petrochenkov:astymac2, r=jseyfried
Improve resolution of associated types in declarative macros 2.0

Make various identifier comparisons for associated types (and sometimes other associated items) hygienic.
Now declarative macros 2.0 can use `Self::AssocTy`, `TyParam::AssocTy`, `Trait<AssocTy = u8>` where `AssocTy` is an associated type of a trait `Trait` visible from the macro. Also, `Trait` can now be implemented inside the macro and specialization should work properly (fixes #40847 (comment)).

r? @jseyfried or @eddyb

@Geal Geal referenced this pull request Dec 2, 2017

Open

follow and test macros 2.0 #632

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment