Consider adding "generate Rust source code" mode

[matklad]

From reddit:

Question: would you consider adding a traditional code generation API in addition to proc macro?

The problem with proc macros is that if you build a lexer using Logos, then all of your consumers will have to compile Logos itself and associated proc-macro machinery. In contrast, if you literally generate the lexer’s code and commit it to the repo, lexer consumers wouldn’t need any additional dependencies.

Logging the issue so that I can be notified if/when it is fixed, and switch https://github.com/matklad/tom and other stuff to logos :)

[matklad]

A sketch of the API:

Ideally, we use a custom attribute to tell logos "do a real code-gen for this token enum", but custom attributes are unstable. So, I suggest using something like this instead: a token is defined in a foo.in.rs file, using the usual syntax:

#[derive(Logos, Debug, PartialEq)]
enum Token {
    #[end]
    End,
    #[error]
    Error,
    #[token = "fast"]
    Fast,
    #[token = "."]
    Period,
    #[regex = "[a-zA-Z]+"]
    Text,
}

logos-codegen export a function fn codegen(in: &Path, out: &Path) -> io::Result<()>, which reads the in file using syn, and generates to out both the generated lexer and the Token enum, with logos-specific attributes stripped. That is,

#[derive(Debug, PartialEq)]
enum Token {
    End,
    Error,
    Fast,
    Period,
    Text,
}

impl logos::Logos for Token {
    ...
}

Note that codegen should avoid writing the file if the content on disk is equal to the generated one.

[rljacobson]

@matklad Have you made any progress on this?

[matklad]

Nope, I also don’t really plan to.

[jeertmans]

Seems like closed by #248