Borrow #run idea from JAI

[ckaran]

I've been trying to figure out macros in Rust, including some of the tracking issues for Macros 2.0, and have come to the conclusion that JAI's #run mechanism may be the better way to go (see https://github.com/BSVino/JaiPrimer/blob/master/JaiPrimer.md, and search for #run).

The basic idea is that the compiler has an interpreter for the language built into itself. Whenever you reach some code marked with #run, the compiler executes the code immediately as a part of the compilation process. My thought is that this may be a good way to deal with macros. The process would be something like the following:

• The compiler defines a fake crate that is implicitly in scope whenever #run is encountered. This crate provides the communications link between the #run interpreted code and the compiler.
• When the compiler encounters #run, it starts up a MIRI interpreter instance which takes over. The code has access to the current state of the compiler (including the abstract syntax tree, etc.) via the fake crate's features. The code can mutate the compiler's state (including the AST) via the crate.
• When the execution finishes, compilation goes on using the compiler's newly mutated state. If nothing was mutated via the crate, then the entire #run is a no-op.
• MIRI interpretation is recursive; if another #run is encountered before the first #run is finished, then a new MIRI interpreter instance is spun up, and the steps above follow suit. This might be used in cases where you have a vec!() within the interpreted code.

Advantages to this approach

• We can get rid of the macro mini-language entirely; everything is in plain old rust.
• Full power of rust. In theory, you could #run your entire program (although probably more slowly than if it were compiled).

Issues I can think off off-hand:

• Depending on how the compiler was written, this may or may not be possible. I'm still learning rust, and don't have enough experience to know if this is a good or bad idea.
• Compilation might hang; it's pretty easy to write bad code that just hangs, or effectively does a DOS attack against the computer its being compiled on, etc.
• It's yet another macro-like system to keep track of. Thanks to rust's stability guarantees, nothing that is in stable can be removed or changed until Rust 2.0 (and probably not until after that).

The only real issue that I see is the middle one; it is a security issue. However, it would have limited effect as it we're talking about compile-time execution, which means on a developer's machine rather than across numerous machines on the internet.

Thoughts, critique, comments?

[sfackler]

You can already do this, just not inline in the crate being built: https://doc.rust-lang.org/book/ch19-06-macros.html#how-to-write-a-custom-derive-macro

[ckaran]

Can you clarify one thing for me? Are you able to walk backwards through the TokenStream to get outside of the block that the macro is being applied to? If you can, then you're right, and my proposal is already fully covered.

[sfackler]

You're given access to whatever the macro is applied to:

#[my_custom_macro] // this can read and modify the module declaration and everything in it
mod foo {
    ...
}

my_custom_macro! {
    // this can only modify the stuff in the block here
    ...
}

[ckaran]

That's what I thought. OK, how do I handle the following cases from within a macro?

• Examining the crate structure so I can have hard-coded customized strings that depend on organization of the crate?
• Reading in some data in another file and loading its contents into the file (handy when you've generated lookup tables somewhere else)
• Reading in an external file that is somewhere in the file system, realizing the lookup tables weren't generated, and throwing a compilation error because the file can't be found?

That said, it looks like I need to re-read everything about macros. The last time I looked at them was more than a year ago, when I started working on developing macros for marshaling crate.

[Centril]

If you want to continue this conversation I recommend doing so on http://internals.rust-lang.org/. This is not the place for speculative new designs.

[ckaran]

Understood, I'll move the conversation over there and reference this issue.

[ckaran]

Now moved to https://internals.rust-lang.org/t/proposal-borrow-run-idea-from-jai/9226