Skip to content
Luther is an embedded lexer generator for stable Rust.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.cargo
ci
luther-derive
src
tests
tools/testsuite
.gitignore
.travis.yml
CODE_OF_CONDUCT.md
CONTRIBUTING.md
Cargo.toml
LICENSE-APACHE-2.0
LICENSE-MIT
README.md

README.md

Luther

Luther is an embedded lexer generator for stable Rust.

Build Status Coverage Status Latest Version Rust documentation

Luther generates the lexer through its macros 1.1 derive implementation in the luther-derive crate. You annotate your token enum with regular expressions (through the #[luther(...)] attribute) and then #[derive(Lexer)] on it. Unlike many other approaches in Rust to lexing (or tokenizing), Luther does not operate on &str but rather on char iterators. The luther::spanned module, though, contains extension traits to produce such char iterators from a &str or from a std::io::Read implementation.

Usage

Add this to your Cargo.toml:

[dependencies]
luther="0.2"
luther-derive="0.2"

and this to your crate root:

extern crate luther;
#[macro_use]
extern crate luther_derive;

Example

extern crate luther;
#[macro_use]
extern crate luther_derive;

use luther::spanned::StrExt;

#[derive(Lexer)]
enum Token {
    #[luther(regex="ab")]
    Ab,

    #[luther(regex="acc*")]
    Acc,
}

fn main() {
    use luther::Lexer;

    let input = "abacaccabacccc".spanned_chars();   // from luther::spanned::StrExt

    let tokens = Token::lexer(input)
        .map_span(|s| s.into_inner());

    // use tokens
}

The tokens iterator from the above example will yield the following tokens (together with their start and end locations):

  • Token::Ab
  • Token::Acc
  • Token::Acc
  • Token::Ab
  • Token::Acc

The procedural macro implementation that provides the #[derive(Lexer)] and recognized the #[luther(...)] attributes is in the luther-derive crate.

The intention is for the tokens iterator from the above example to be a suitable candidate for an external lexer for the parser generator Lalrpop.

License

Luther is licensed under either of

at your option.

Contribution

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.

You may wish to review our Contributing Guidelines before making a contribution.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Luther by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

You can’t perform that action at this time.