Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
branch: master
Fetching contributors…

Cannot retrieve contributors at this time

30 lines (24 sloc) 1.22 kB
Simple table-driven lexer in Erlang.
Works by building an NFA given the rules, then converting the NFA
into a DFA. If multiple rules can match, a priority can be specified.
Ambiguous matches yield compile-time errors.
Note: building a lex table requires about 50ms, so it's possible to
just do it dynamically if the lexing task is substantial.
Advantage: generate and run your lexer on the fly. See lex.erl for
some examples. lex:with(Rules, String) is the easiest entrypoint.
Drawbacks: lexing errors currently not very legible.
TODO:
- better error messages, at compile time and runtime
- better support for incremental lexing
- better support lib for writing lexers
- allow UTF-8 literals and text in regexps
* regexp_parse has to be extended
* possibly also matching characters, if needed (in particular, matching
one character = a valid sequence of UTF-8 bytes)
- allow full UTF-8, character classes, etc.
- code generation
* emit a table as a constant data structure, and the lex driver with it,
and a suitable API
* maybe: emit leex-style full code for the lexer using the table
- worth checking which is faster and smaller
* easier to distribute than being required to use erlang-lex
Jump to Line
Something went wrong with that request. Please try again.