Join GitHub today
GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
|Failed to load latest commit information.|
Simple table-driven lexer in Erlang. Works by building an NFA given the rules, then converting the NFA into a DFA. If multiple rules can match, a priority can be specified. Ambiguous matches yield compile-time errors. Note: building a lex table requires about 50ms, so it's possible to just do it dynamically if the lexing task is substantial. Advantage: generate and run your lexer on the fly. See lex.erl for some examples. lex:with(Rules, String) is the easiest entrypoint. Drawbacks: lexing errors currently not very legible. TODO: - better error messages, at compile time and runtime - better support for incremental lexing - better support lib for writing lexers - allow UTF-8 literals and text in regexps * regexp_parse has to be extended * possibly also matching characters, if needed (in particular, matching one character = a valid sequence of UTF-8 bytes) - allow full UTF-8, character classes, etc. - code generation * emit a table as a constant data structure, and the lex driver with it, and a suitable API * maybe: emit leex-style full code for the lexer using the table - worth checking which is faster and smaller * easier to distribute than being required to use erlang-lex