-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Rust edition upgrade, DX fix for generated token macros (#2)
# Breaking changes ## Generated token macros are no longer `#[macro_export]` The `Token` derive macro generates a `macro_rules!` macro for each of the enum variants. E.g., for Lox's `Token` enum in `examples/lox`: ```rs // examples/lox/src/tokens.rs #[derive(DebugLispToken, PartialEq, Token, Lexer)] pub enum Token { #[subset_of(Ident)] #[pattern = "and|class|else|false|for|fun|if|nil|or|print|return|super|this|true|var|while"] Keyword(Substr, Span), #[pattern = "[a-zA-Z_][a-zA-Z0-9_]*"] Ident(Substr, Span), #[pattern = r"[(){}]"] Brace(Substr, Span), #[pattern = "[,.;]"] Punct(Substr, Span), #[pattern = "[=!<>]=?"] #[pattern = "[-+*/]"] Operator(Substr, Span), #[pattern = "[0-9]+"] NumLit(Substr, Span), #[pattern = r#""[^"]*""#] StrLit(Substr, Span), } ``` We get `macro_rules!` macros named `keyword`, `ident`, `brace`, `punct`, `operator`, `num_lit` and `str_lit`. These are mostly useful for the `Parse` implementations, e.g.: ```rs // examples/lox/src/decl.rs impl Parse for ClassDecl { type Stream = ParseStream; fn parse(input: &mut Self::Stream) -> Result<Self> { // ... let superclass = if input.check(operator![<]) { input.consume(operator![<])?; Some(input.consume_kind(TokenKind::Ident)?) } else { None }; // ... } } ``` Previously, those generated macros were declared like this: ```rs #( #[macro_export] macro_rules! #snake_case_variant_ident { // ... } )* ``` That has always [caused some headaches](rust-lang/rust#52234 (comment)) which I was never really sure how to deal with. In the examples here, and in my consuming projects, I had resorted to using `*` imports everywhere to work around the problem (in hindsight though, I think that likely just hides the lint by obscuring what we're doing instead of actually addressing the issue). This PR attempts an alternative solution. `#[macro_export]` is intended for libraries to expose macros to external consumers, but I don't foresee the macros generated by the `Token` derive being actually useful in that context. So instead, the generated macros are now declared like this: ```rs #( macro_rules! #snake_case_variant_ident { // ... } pub(crate) use #snake_case_variant_ident; )* ``` This is a breaking change for two reasons: 1. If you were depending on those `#[macro_export]` attributes to make the generated macros available to external consumers of your library, that is no longer going to work. Again, I don't imagine this was actually a real-world use case for anyone, but I've been wrong before! Let me know if this is a problem for you and I'll see what we can do about it. 2. If you had been importing those macros from the root of your crate, but your `Token` enum is _not_ declared in the crate root, you'll need to update your import paths to instead import them from the module where the enum is declared. E.g.: ```rs mod token { use gramatika::{DebugLisp, Span, Substr, Token as _}; #[derive(DebugLispToken, PartialEq, Token, Lexer)] pub enum Token { #[pattern = ".+"] Word(Substr, Span), } } mod foo { // use crate::word; // 👎 use crate::token::word; // 👍 } ``` On the bright side, tools like `rust-analyzer` should now find and automatically suggest the correct import paths for those macros, so the fastest way to migrate will probably be to just delete your existing `use` statement and invoke your editor's suggestion feature to re-import any unresolved symbols from their correct paths. # Other changes * Updated the Rust edition to 2021 and fixed any resulting errors * Fixed any new `clippy` lints as a result of upgrading my environment to v1.77.2 * Performed some low-hanging-fruit dependency upgrades. `regex-automata` and `syn` are still out of date for now — I attempted to update the former, but couldn't figure out how to migrate after ~10 minutes of poking around, and unfortunately I have other priorities that need to take precedence. Didn't even attempt `syn` because it's a major version bump, and that crate is basically the backbone of this whole project, so it'll have to wait for now.
- Loading branch information
1 parent
c51ef06
commit 6f3737e
Showing
20 changed files
with
104 additions
and
80 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,4 @@ | ||
use crate::{Lexer, Token}; | ||
use crate::{tokens::Token, Lexer}; | ||
use gramatika::{ArcStr, Lexer as _}; | ||
|
||
#[test] | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,6 +1,6 @@ | ||
use gramatika::{ParseStream, ParseStreamer}; | ||
|
||
use crate::*; | ||
use crate::stmt::Program; | ||
|
||
#[test] | ||
fn print_stmt() { | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.