Skip to content

BenHanson/lexertl14

master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 

lexertl14: The Modular Lexical Analyser Generator

lexertl is a header-only library for writing lexical analysers. With lexertl you can:

  • Build lexical analysers at runtime
  • Scan Unicode and ASCII input
  • Scan from files or memory
  • Generate C++ code or even write your own code generator

Construct a Lexer and Tokenise input

#include <lexertl/generator.hpp>
#include <lexertl/lookup.hpp>
#include <iostream>

int main()
{
    lexertl::rules rules;
    lexertl::state_machine sm;

    rules.push("[0-9]+", 1);
    rules.push("[a-z]+", 2);
    lexertl::generator::build(rules, sm);

    std::string input("abc012Ad3e4");
    lexertl::smatch results(input.begin(), input.end());

    // Read ahead
    lexertl::lookup(sm, results);

    while (results.id != 0)
    {
        std::cout << "Id: " << results.id << ", Token: '" <<
            results.str () << "'\n";
        lexertl::lookup(sm, results);
    }

    return 0;
}

The same thing using lexertl::iterator:

#include <lexertl/generator.hpp>
#include <lexertl/iterator.hpp>
#include <iostream>

int main()
{
    lexertl::rules rules;
    lexertl::state_machine sm;

    rules.push("[0-9]+", 1);
    rules.push("[a-z]+", 2);
    lexertl::generator::build(rules, sm);

    std::string input("abc012Ad3e4");
    lexertl::siterator iter(input.begin(), input.end(), sm);
    lexertl::siterator end;

    for (; iter != end; ++iter)
    {
        std::cout << "Id: " << iter->id << ", Token: '" <<
            iter->str() << "'\n";
    }

    return 0;
}

More examples and documentation

See http://www.benhanson.net/lexertl.html for full documentation and more usage examples.

About

C++14 version of lexertl

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages