Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attribute macros invoked at crate root have issues #41430

Open
abonander opened this issue Apr 20, 2017 · 29 comments
Open

Attribute macros invoked at crate root have issues #41430

abonander opened this issue Apr 20, 2017 · 29 comments
Labels
A-attributes A-macros-2.0 C-bug T-compiler

Comments

@abonander
Copy link
Contributor

@abonander abonander commented Apr 20, 2017

When invoking some attribute procedural macro with inner form, i.e. #![foo_attr] at the crate root, a resolution error occurs:

#![feature(proc_macro)]
#![foo_attr]
//^ ERROR: cannot find attribute macro `foo_attr` in this scope

extern crate foo_macros;
use foo_attr_macro::foo_attr;

This should, ideally, resolve and execute properly.

@abonander
Copy link
Contributor Author

@abonander abonander commented Apr 20, 2017

cc @jseyfried

@jseyfried
Copy link
Contributor

@jseyfried jseyfried commented Apr 20, 2017

@abonander What about this?

#![feature(proc_macro)]

extern crate foo_macros;
use foo_macros::foo_attr;

mod bar {
    #![foo_attr]
}

If compiles, then the issue is that the attribute is at the crate root, not that it is an inner attribute.

@abonander
Copy link
Contributor Author

@abonander abonander commented Apr 20, 2017

@jseyfried Updated issue as this only affects the crate root

abonander added a commit to abonander/rust that referenced this issue Apr 20, 2017
Adds temporary regression test; this ideally should work as-is (rust-lang#41430)

Closes rust-lang#41211
@abonander abonander changed the title Attribute macros invoked with inner-attribute form fail to resolve Attribute macros invoked at crate root fail to resolve Apr 21, 2017
@abonander
Copy link
Contributor Author

@abonander abonander commented Apr 21, 2017

So @jseyfried and I figured out on IRC that this is due to the compiler trying to expand the attribute before the use_extern_macros code gets to see the actual import. I'd really like to make this work as I have some cool use-cases for it. We had a couple ideas for how to get this working but we figured we should expand the discussion on it before taking any action.

My best idea is along the lines of this: if an attribute macro fails to resolve at crate root only (since inner attributes on modules should resolve in the parent's scope), import-process root, get the resolution for the attribute, reinitialize the resolver and copy the resolution for the attribute macro, then expand the root module.

Some concerns from @jseyfried:

I believe that would be doable implementation-wise, but it could lead to coherence issues
For example, if the attribute macro expanded into nothing
Or if it expanded into a different import of itself
The resolution of a name isn't supposed to change as we expand things
It can only go from "indeterminate" to "success" or "indeterminate" to "failed"
If we preserve that with extra checks, what you're proposing could work though, I think

cc @nrc

frewsxcv added a commit to frewsxcv/rust that referenced this issue Apr 22, 2017
Don't panic if an attribute macro fails to resolve at crate root

Adds temporary regression test; this ideally should work as-is (rust-lang#41430)

Closes rust-lang#41211

r? @jseyfried
frewsxcv added a commit to frewsxcv/rust that referenced this issue Apr 22, 2017
Don't panic if an attribute macro fails to resolve at crate root

Adds temporary regression test; this ideally should work as-is (rust-lang#41430)

Closes rust-lang#41211

r? @jseyfried
frewsxcv added a commit to frewsxcv/rust that referenced this issue Apr 22, 2017
Don't panic if an attribute macro fails to resolve at crate root

Adds temporary regression test; this ideally should work as-is (rust-lang#41430)

Closes rust-lang#41211

r? @jseyfried
@abonander
Copy link
Contributor Author

@abonander abonander commented May 15, 2017

@nrc Some input here when you get the chance, I'd like to tackle this soonish.

@nrc
Copy link
Member

@nrc nrc commented May 16, 2017

sorry it took a while to see this - I was away on parental leave and then had to nuke my notifications.

since inner attributes on modules should resolve in the parent's scope

I wonder if this axiom is correct? An alternative seems to be that we could resolve attributes at the scope in which they are written rather than the scope in which they apply. Would that have knock-on effects?

@jseyfried
Copy link
Contributor

@jseyfried jseyfried commented May 16, 2017

resolve attributes at the scope in which they are written rather than the scope in which they apply

Interesting, I like this idea. I'm not aware of any knock-on effects, it should be straightforward to implement, and it would address this use case.

@abonander
Copy link
Contributor Author

@abonander abonander commented May 16, 2017

I was under the impression that inner attributes were syntactic sugar for outer attributes, and it makes sense for outer attributes to resolve in the same scope as the item they're applied to. Crates are just special because there's no outer scope to resolve.

@nrc
Copy link
Member

@nrc nrc commented May 17, 2017

I was under the impression that inner attributes were syntactic sugar for outer attributes

Whether a feature is desugared or not is an implementation detail and we should do what makes the most sense from the user's POV, not the compilers. Desugaring can take place after name resolution if we like, so we don't need to change the implementation either.

@abonander
Copy link
Contributor Author

@abonander abonander commented May 20, 2017

So the plan of action is to just reorder attribute expansion and import processing for a given item? Or just at the crate root for now?

@jseyfried
Copy link
Contributor

@jseyfried jseyfried commented May 20, 2017

@abonander I wouldn't think of this as reordering attribute expansion or import processing, just changing the scope in which e.g. bar resolves in #[foo] mod m { #![bar] #[baz] fn f() {} ... } from that of foo to that of baz.

@jseyfried
Copy link
Contributor

@jseyfried jseyfried commented May 20, 2017

If we make this change, I think we should make the change everywhere (not just at the crate root) for consistency.

@Mark-Simulacrum Mark-Simulacrum added the A-macros-2.0 label Jun 23, 2017
@Mark-Simulacrum Mark-Simulacrum added the C-bug label Jul 27, 2017
@Rantanen
Copy link
Contributor

@Rantanen Rantanen commented Dec 29, 2017

#44660 improves the situation. The following compiles okay on nightly:

#![feature(extern_absolute_paths, proc_macro)]
#![::my_crate::my_attribute]

I believe the absoluteness of the path can be omitted in the future, once the RFC 2126 implementation proceeds further.

However #45458 still makes using the crate-level macros difficult. The issue concerns compiler plugins, but can also be reproduced through attribute macros.

@Rantanen
Copy link
Contributor

@Rantanen Rantanen commented Feb 8, 2018

Moving from #48066 as per @Manishearth wishes. Although I would prefer if the title of this issue was updated - the resolution itself is somewhat "solved" in nightly. Even if the current ::absolute syntax ends up not being the final one, I'd expect the same mechanism to work with whichever syntax the module update ends up implementing.


While normal proc-macro attribute work, crate level proc-macro attributes seem to have several issues. The clearest of these is the order of the attributes and extern crate statements: #41430. #![feature(extern_absolute_paths)] feature can work around this specific issue; However this alone doesn't make proc macros work.

Some of these issues are due to lack of library support (syn), but some are caused by rustc itself.

Identity macro ✔️

First of all: The following proc-macro works just fine with nightly 29c8276:

#[proc_macro_attribute]
fn attribute(_: TokenStream, input: TokenStream) -> TokenStream {
    input
}
#![feature(extern_absolute_paths)]
#![::my_crate::attribute]
fn main() {}

Round trip through FromIterator

As long as the macro doesn't touch the input, everything works just fine. However this isn't really that useful for a macro. Once we try doing anything with the input token stream (short of a .clone()), things start breaking.

#[proc_macro_attribute]
fn attribute(_: TokenStream, input: TokenStream) -> TokenStream {
    TokenStream::from_iter( input.into_iter() )
}
#![feature(extern_absolute_paths)]
#![::my_crate::attribute]
fn main() {}

This results in the following error message

error: expected identifier, found `{`
 --> <macro expansion>:1:1
  |
1 | pub mod  {
  | ^^^ expected identifier

The reason seems to be that the TokenStream the attribute receives looks like:

pub mod {
    ...
}

This is most likely caused by the ast::Crate modeling the items as a Mod with no Ident: https://github.com/rust-lang/rust/blob/master/src/libsyntax/ast.rs#L447-L451

Stripping the pub mod away

Implementing a Crate type for Syn and having its to_tokens skip the pub mod { .. } bits allows the parser to process the output correctly. I'm also hoping this would be the way to go, since the parse_crate_mod method just parses inner attributes and mod items without seemingly caring about the mod keywords or braces.

However now we encounter the following ICE:

thread 'rustc' panicked at 'internal error: entered unreachable code', libsyntax/ext/expand.rs:258:18

error: an inner attribute is not permitted in this context
  |
  = note: inner attributes, like `#![no_std]`, annotate the item enclosing them, and are usually found at the beginning of source files. Outer attributes, like `#[test]`, annotate the item following them.

thread 'rustc' panicked at 'internal error: entered unreachable code', libsyntax/ext/expand.rs:258:18
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::_print
             at libstd/sys_common/backtrace.rs:71
   2: std::panicking::default_hook::{{closure}}
             at libstd/sys_common/backtrace.rs:59
             at libstd/panicking.rs:380
   3: std::panicking::default_hook
             at libstd/panicking.rs:396
   4: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:576
   5: std::panicking::begin_panic
   6: syntax::ext::expand::MacroExpander::expand_crate
   7: rustc_driver::driver::phase_2_configure_and_expand_inner::{{closure}}
   8: rustc_driver::driver::phase_2_configure_and_expand_inner
   9: rustc_driver::driver::compile_input
  10: rustc_driver::run_compiler

error: internal compiler error: unexpected panic

As far as I can tell, the only way for this ICE to happen is for the MacroExpander to successfully expand the attribute. However the code expects the expansion to result in ast::Item(Mod)

I tried compiling a debug version of rustc for this, however then I end up with...

syn::parse(input) panics

At this point my attribute fn does nothing but parses and quotes the input using syn using a custom Crate type that handles pub mod without an ident:

#[proc_macro_attribute]
pub fn attribute(_attr: TokenStream, input: TokenStream) -> TokenStream {
    let file : Crate = parse( input ).unwrap();
    quote!( #file ).into()
}

This results in the following panic:

thread '' panicked at 'proc_macro::__internal::with_sess() called before set_parse_sess()!',

   Compiling test_crate v0.1.0 (file:///C:/Dev/Projects/crate_level_proc_macro/test_crate)
thread '<unnamed>' panicked at 'proc_macro::__internal::with_sess() called before set_parse_sess()!', libproc_macro\lib.rs:851:9
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::windows::backtrace::unwind_backtrace
             at C:\Dev\Projects\rust\src\libstd\sys\windows\backtrace\mod.rs:65
   1: std::sys_common::backtrace::_print
             at C:\Dev\Projects\rust\src\libstd\sys_common\backtrace.rs:71
   2: std::sys_common::backtrace::print
             at C:\Dev\Projects\rust\src\libstd\sys_common\backtrace.rs:58
   3: std::panicking::default_hook::{{closure}}
             at C:\Dev\Projects\rust\src\libstd\panicking.rs:380
   4: std::panicking::default_hook
             at C:\Dev\Projects\rust\src\libstd\panicking.rs:396
   5: std::panicking::rust_panic_with_hook
             at C:\Dev\Projects\rust\src\libstd\panicking.rs:576
   6: std::panicking::begin_panic<str*>
             at C:\Dev\Projects\rust\src\libstd\panicking.rs:537
   7: proc_macro::__internal::with_sess
             at C:\Dev\Projects\rust\src\libproc_macro\lib.rs:851
   8: proc_macro::TokenTree::from_internal
             at C:\Dev\Projects\rust\src\libproc_macro\lib.rs:682
   9: proc_macro::{{impl}}::next
             at C:\Dev\Projects\rust\src\libproc_macro\lib.rs:570
  10: proc_macro2::imp::{{impl}}::next
             at C:\Users\Rantanen\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-macro2-0.2.2\src\unstable.rs:117
  11: proc_macro2::{{impl}}::next
             at C:\Users\Rantanen\.cargo\registry\src\github.com-1ecc6299db9ec823\proc-macro2-0.2.2\src\lib.rs:326
  12: syn::buffer::TokenBuffer::inner_new
             at C:\Users\Rantanen\.cargo\registry\src\github.com-1ecc6299db9ec823\syn-0.12.12\src\buffer.rs:176
  13: syn::buffer::TokenBuffer::new2
             at C:\Users\Rantanen\.cargo\registry\src\github.com-1ecc6299db9ec823\syn-0.12.12\src\buffer.rs:228
  14: syn::synom::{{impl}}::parse2<fn(syn::buffer::Cursor) -> core::result::Result<(crate_level_proc_macro::Crate, syn::buffer::Cursor), syn::error::ParseError>,crate_level_proc_macro::Crate>
             at C:\Users\Rantanen\.cargo\registry\src\github.com-1ecc6299db9ec823\syn-0.12.12\src\synom.rs:221
  15: syn::parse2<crate_level_proc_macro::Crate>
             at C:\Users\Rantanen\.cargo\registry\src\github.com-1ecc6299db9ec823\syn-0.12.12\src\lib.rs:601
  16: syn::parse<crate_level_proc_macro::Crate>
             at C:\Users\Rantanen\.cargo\registry\src\github.com-1ecc6299db9ec823\syn-0.12.12\src\lib.rs:580
  17: crate_level_proc_macro::attribute
             at C:\Dev\Projects\crate_level_proc_macro\src\lib.rs:54
  18: std::panicking::try::do_call
  19: _rust_maybe_catch_panic
  20: <std::thread::local::LocalKey<T>>::with
  21: <syntax_ext::proc_macro_impl::AttrProcMacro as syntax::ext::base::AttrProcMacro>::expand
  22: syntax::ext::expand::MacroExpander::expand
  23: syntax::ext::expand::MacroExpander::expand
  24: syntax::ext::expand::MacroExpander::expand_crate
  25: rustc_driver::driver::count_nodes
  26: rustc_driver::driver::count_nodes
  27: rustc_driver::driver::compile_input
  28: rustc_driver::run_compiler
error: custom attribute panicked
 --> src\main.rs:2:1
  |
2 | #![::crate_level_proc_macro::attribute]
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = help: message: proc_macro::__internal::with_sess() called before set_parse_sess()!

The fact that the same code breaks on my own stage-1 rustc build based on the commit 29c8276ce, while it "works" on nightly 29c8276ce 2018-02-07 is a bit weird. Shouldn't those two compilers have the same code base?

In any case, I can try working around that specific issue by not going through proc_macro2 crate and instead using Syn's parse_str. However...

.to_string() on TokenStream

We can skip proc_macro2 by using the TokenStream::to_string() to turn the stream into rust-code that Syn can parse on its own. This gives us the following attribute function:

#[proc_macro_attribute]
pub fn attribute(_attr: TokenStream, input: TokenStream) -> TokenStream {
    let file : Crate = parse_str( &input.to_string() ).unwrap();
    quote!( #file ).into()
}

Running this results in the following panic

thread '<unnamed>' panicked at 'index out of bounds: the len is 0 but the index is 0',

   Compiling test_crate v0.1.0 (file:///C:/Dev/Projects/crate_level_proc_macro/test_crate)
thread '<unnamed>' panicked at 'index out of bounds: the len is 0 but the index is 0', C:\Dev\Projects\rust\src\liballoc\vec.rs:1551:10
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::windows::backtrace::unwind_backtrace
             at C:\Dev\Projects\rust\src\libstd\sys\windows\backtrace\mod.rs:65
   1: std::sys_common::backtrace::_print
             at C:\Dev\Projects\rust\src\libstd\sys_common\backtrace.rs:71
   2: std::sys_common::backtrace::print
             at C:\Dev\Projects\rust\src\libstd\sys_common\backtrace.rs:58
   3: std::panicking::default_hook::{{closure}}
             at C:\Dev\Projects\rust\src\libstd\panicking.rs:380
   4: std::panicking::default_hook
             at C:\Dev\Projects\rust\src\libstd\panicking.rs:396
   5: std::panicking::rust_panic_with_hook
             at C:\Dev\Projects\rust\src\libstd\panicking.rs:576
   6: std::panicking::begin_panic<alloc::string::String>
             at C:\Dev\Projects\rust\src\libstd\panicking.rs:537
   7: std::panicking::begin_panic_fmt
             at C:\Dev\Projects\rust\src\libstd\panicking.rs:521
   8: std::panicking::rust_begin_panic
             at C:\Dev\Projects\rust\src\libstd\panicking.rs:497
   9: core::panicking::panic_fmt
             at C:\Dev\Projects\rust\src\libcore\panicking.rs:71
  10: core::panicking::panic_bounds_check
             at C:\Dev\Projects\rust\src\libcore\panicking.rs:58
  11: alloc::vec::{{impl}}::index
             at C:\Dev\Projects\rust\src\liballoc\vec.rs:1551
  12: syntax_pos::span_encoding::SpanInterner::get
             at C:\Dev\Projects\rust\src\libsyntax_pos\span_encoding.rs:134
  13: syntax_pos::span_encoding::decode::{{closure}}
             at C:\Dev\Projects\rust\src\libsyntax_pos\span_encoding.rs:109
  14: syntax_pos::span_encoding::with_span_interner::{{closure}}
             at C:\Dev\Projects\rust\src\libsyntax_pos\span_encoding.rs:144
  15: std::thread::local::LocalKey<core::cell::RefCell<syntax_pos::span_encoding::SpanInterner>>::try_with
             at C:\Dev\Projects\rust\src\libstd\thread\local.rs:377
  16: std::thread::local::LocalKey<core::cell::RefCell<syntax_pos::span_encoding::SpanInterner>>::with<core::cell::RefCell<syntax_pos::span_encoding::SpanInterner>,closure,syntax_pos::SpanData>
             at C:\Dev\Projects\rust\src\libstd\thread\local.rs:290
  17: syntax_pos::span_encoding::with_span_interner
             at C:\Dev\Projects\rust\src\libsyntax_pos\span_encoding.rs:144
  18: syntax_pos::span_encoding::decode
             at C:\Dev\Projects\rust\src\libsyntax_pos\span_encoding.rs:109
  19: syntax_pos::span_encoding::Span::data
             at C:\Dev\Projects\rust\src\libsyntax_pos\span_encoding.rs:47
  20: syntax_pos::span_encoding::Span::lo
             at C:\Dev\Projects\rust\src\libsyntax_pos\lib.rs:196
  21: syntax::print::pprust::State::print_item
             at C:\Dev\Projects\rust\src\libsyntax\print\pprust.rs:1173
  22: syntax::print::pprust::item_to_string::{{closure}}
             at C:\Dev\Projects\rust\src\libsyntax\print\pprust.rs:332
  23: syntax::print::pprust::to_string
             at C:\Dev\Projects\rust\src\libsyntax\print\pprust.rs:165
  24: syntax::print::pprust::item_to_string
             at C:\Dev\Projects\rust\src\libsyntax\print\pprust.rs:331
  25: syntax::print::pprust::token_to_string
             at C:\Dev\Projects\rust\src\libsyntax\print\pprust.rs:269
  26: syntax::print::pprust::PrintState::print_tt<syntax::print::pprust::State>
             at C:\Dev\Projects\rust\src\libsyntax\print\pprust.rs:794
  27: syntax::print::pprust::PrintState::print_tts<syntax::print::pprust::State>
             at C:\Dev\Projects\rust\src\libsyntax\print\pprust.rs:818
  28: syntax::print::pprust::tokens_to_string::{{closure}}
             at C:\Dev\Projects\rust\src\libsyntax\print\pprust.rs:320
  29: syntax::print::pprust::to_string
             at C:\Dev\Projects\rust\src\libsyntax\print\pprust.rs:165
  30: syntax::print::pprust::tokens_to_string
             at C:\Dev\Projects\rust\src\libsyntax\print\pprust.rs:320
  31: syntax::tokenstream::{{impl}}::fmt
             at C:\Dev\Projects\rust\src\libsyntax\tokenstream.rs:555
  32: core::fmt::{{impl}}::fmt<proc_macro::TokenStream>
             at C:\Dev\Projects\rust\src\libcore\fmt\mod.rs:1566
  33: core::fmt::Formatter::run
             at C:\Dev\Projects\rust\src\libcore\fmt\mod.rs:1084
  34: core::fmt::write
             at C:\Dev\Projects\rust\src\libcore\fmt\mod.rs:1030
  35: core::fmt::Write::write_fmt<alloc::string::String>
             at C:\Dev\Projects\rust\src\libcore\fmt\mod.rs:226
  36: alloc::string::{{impl}}::to_string<proc_macro::TokenStream>
             at C:\Dev\Projects\rust\src\liballoc\string.rs:2054
  37: crate_level_proc_macro::attribute
             at C:\Dev\Projects\crate_level_proc_macro\src\lib.rs:54
  38: std::panicking::try::do_call
  39: _rust_maybe_catch_panic
  40: <std::thread::local::LocalKey<T>>::with
  41: <syntax_ext::proc_macro_impl::AttrProcMacro as syntax::ext::base::AttrProcMacro>::expand
  42: syntax::ext::expand::MacroExpander::expand
  43: syntax::ext::expand::MacroExpander::expand
  44: syntax::ext::expand::MacroExpander::expand_crate
  45: rustc_driver::driver::count_nodes
  46: rustc_driver::driver::count_nodes
  47: rustc_driver::driver::compile_input
  48: rustc_driver::run_compiler
error: custom attribute panicked
 --> src\main.rs:2:1
  |
2 | #![::crate_level_proc_macro::attribute]
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: Could not compile `test_crate`.

I suspect this has something to do with the token stream including those pub mod bits, which do not have proper span information. The panic seems to result from the pretty printer trying to fetch SpanData for tokens that don't have any.


That's as far as I've gotten with this. I still believe crate-level macro attributes are a strong feature that is worth pursuing.

My own use case for these is to emit some FFI entry point functions, which enable the crate to be used as a COM library. While these could be done from a normal macro, an attribute would make it feel more declarative.

These attributes also came up in custom test frameworks. A while back I was playing around with a Catch-inspired ratcc test framework, where I would have loved to to use a crate-level attribute to alter the crate contents.

And continuing on the test framework theme, they also came up in the eRFC discussion for custom test frameworks: rust-lang/rfcs#2318

Given the amount of different panics I encountered in trying to work around the issues hints that they are currently far from supported. However the fact that an identity attribute works makes me hopeful that if the parsing could be fixed so that a TokenStream could be parsed into an ast::Crate (or similar), then the rest of the macro pipeline would "just work".

Edit: Fixed eRFC link

@Manishearth Manishearth changed the title Attribute macros invoked at crate root fail to resolve Attribute macros invoked at crate root have issues Feb 8, 2018
@abonander
Copy link
Contributor Author

@abonander abonander commented Apr 4, 2018

A crate's token-stream should look exactly like the source; we shouldn't be emitting pub mod {} around its contents.

@abonander
Copy link
Contributor Author

@abonander abonander commented Apr 4, 2018

I think the solution here would be to introduce Crate variants for Expansion[Kind]/Annotatable/Nonterminal and handle them accordingly. However, syn would have to know to parse multiple items or else most users won't see the whole input.

@abonander
Copy link
Contributor Author

@abonander abonander commented Apr 4, 2018

cc @petrochenkov

@abonander
Copy link
Contributor Author

@abonander abonander commented Apr 4, 2018

Or would it be preferable to pass a syntatically correct representation of the crate as some pub mod krate {}?

@Rantanen
Copy link
Contributor

@Rantanen Rantanen commented Apr 4, 2018

However, syn would have to know to parse multiple items or else most users won't see the whole input.

I believe syn already handles "crate" properly as https://docs.rs/syn/0.12/syn/struct.File.html

Or would it be preferable to pass a syntatically correct representation of the crate as some pub mod krate {}?

From a user perspective, I found the pub mod {} a bit confusing at first, given those tokens do not exist in the parsed input. However if this issue didn't exist and I could have just used syn, I would have never even encountered the pub mod { .. } syntax as a user.

So while a token stream without pub mod {} would be more pure, I don't know if it would be worth a much more complex implementation at least from user perspective.

@abonander
Copy link
Contributor Author

@abonander abonander commented Apr 4, 2018

I think for sanity's sake we should pass the crate's tokens just as they appear in the source, which would mean handling a crate not like a module but like its own thing.

@abonander
Copy link
Contributor Author

@abonander abonander commented Apr 27, 2018

In #50101 @alexcrichton stated that he doesn't expect support for proc-macros on the crate root to be stabilized in the foreseeable future, that we should address the possible use-cases individually instead of allowing an attribute to completely rewrite the crate.

@Rantanen
Copy link
Contributor

@Rantanen Rantanen commented Apr 27, 2018

The use case that prompted me to examine this in the first place was prototyping parts of the test framework pre-RFC. That RFC suggested a whole-crate proc-macro as one option for custom test framework. It even goes as far as stating the following for one of the options:

This assumes that #![foo] ("inner attribute") macros work on modules and on crates.

Currently the general understanding among people seems to be that inner attribute proc macros not working on modules and crates is a bug instead of an unsupported feature. If I remember right they do work on modules after all; Allowing people to wrap the whole crate in a mod statement to achieve the same result.

Though I do appreciate @alexcrichton's concerns over this issue. Just to keep the discussion here I'll copy his response from #50101 here:

Er sorry yeah, let me clarify. Right now for "Macros 1.2" were very unlikely to stabilize attribute invocations on modules. It's a weird question of what does #[foo] mod foo; receive? Does it receive mod foo ;? The contents of the module foo?

Something like entire crate expansion also has huge repercussions on hygiene as well as whether it's feasible/quick. Entire crate expansion runs a risk of being extremely slow (we have to serialize back and forth with the procedural macro) and highly non-incremental (it's a complete black box to the compiler). I'd imagine that we're very far away from stabilizing this functionality, if at all.

In that sense I'd personally be wary of landing this functionality in the compiler, even on nightly. I think it'd be best to take other avenues of attack for use cases that would otherwise require entire-crate expansion.

@Manishearth
Copy link
Member

@Manishearth Manishearth commented Apr 27, 2018

@Rantanen
Copy link
Contributor

@Rantanen Rantanen commented Apr 27, 2018

@Manishearth Doesn't that result in the same hygiene/incremental issues? Referring here to @alexcrichton's comment:

Something like entire crate expansion also has huge repercussions on hygiene as well as whether it's feasible/quick. Entire crate expansion runs a risk of being extremely slow (we have to serialize back and forth with the procedural macro) and highly non-incremental (it's a complete black box to the compiler).

I know one of the options for the test framework RFC is to just generate a main function. The "Only add stuff"-kind of proc macro does somewhat circumvent the hygiene/incrmental compilation issues, but that's just one option mentioned in the RFC - all the other options face the same issues I'd imagine.

Essentially what I'm after here is: If those issues are solved/resolved/accepeted in relation to the the test RFC, doesn't that apply here as well? Although what I suspect is that given the issue raised here, the test RFC will end up going with the "just generate main()" option.

@Manishearth
Copy link
Member

@Manishearth Manishearth commented Apr 27, 2018

Slow and non-incremental matters less for testing.

"just add a main function" is none of the options, all of the options already have to run some kind of whole crate proc macro to rewrite things to be pub. This is what libtest already does, it folds the entire crate. The question debated in the RFC is whether or not this should be exposed to the user as a whole crate proc macro API.

The hygiene issue is still a thing, but if folks just use the "generate main" API we put on crates.io that's still not a problem. And most of the frameworks which wish to do something more complex than that probably won't need to worry as much about hygiene.

@Manishearth
Copy link
Member

@Manishearth Manishearth commented Apr 27, 2018

Well, that's not entirely true -- the "just generate main" API would have less of a serialization overhead. But that's about it.

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Apr 27, 2018

Another reason that I don't think that this is a great idea is #43081 right now. Any change to the AST would lose span information for the entire crate. I think that's basically a showstopper until we fix that.

@vakaras
Copy link
Contributor

@vakaras vakaras commented Feb 29, 2020

This is what libtest already does, it folds the entire crate.

Could someone point me to the code that does this? (I could not find it myself.) We want to update the Prusti verifier to work with the latest version of the Rust compiler. However, one of the blocking issues we have is that I cannot find any more a way to obtain a mutable reference into an AST. We used to rewrite AST to type-check specifications. Treating each specification as a separate procedural macro invocation is very problematic because we need to maintain a global state.

@Robbepop
Copy link
Contributor

@Robbepop Robbepop commented Nov 18, 2020

Since #43081 (comment) mentions that #43081 is likely to be fixed soon for all stable Rust syntax we can reiterate on this issue. I'd love to hear what else is blocking this feature and if there are any other show stoppers, bugs or design considerations before we can get there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-attributes A-macros-2.0 C-bug T-compiler
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants