Stateful Lexing Support #706
Comments
nwalfield
added a commit
to nwalfield/lalrpop
that referenced
this issue
Jun 26, 2024
In some languages, it is necessary for the parser to control the lexer's mode. This is possible, but non-obvious. This commit adds a simple example that shows how to parse the "list of length-value" language in which values are encoded as a length, followed by a colon, and then the literal number of bytes, e.g., `2:hi`. To parse this, the parser needs to tell the lexer to return the next `n` literal bytes when it sees the length prefix. See lalrpop#802, lalrpop#706.
github-merge-queue bot
pushed a commit
that referenced
this issue
Jul 8, 2024
…916) * Add an example showing how the parser can control the lexer's mode. In some languages, it is necessary for the parser to control the lexer's mode. This is possible, but non-obvious. This commit adds a simple example that shows how to parse the "list of length-value" language in which values are encoded as a length, followed by a colon, and then the literal number of bytes, e.g., `2:hi`. To parse this, the parser needs to tell the lexer to return the next `n` literal bytes when it sees the length prefix. See #802, #706. * run clippy --fix * remove explicit into_iter --------- Co-authored-by: Patrick LaFontaine <32135464+Pat-Lafon@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The custom lexer page of the Lalrpop tutorial mentions:
But does not give any say if Lalrpop supports stateful lexers.
I'm coming from Haskell's Alex lexer which has a feature called "start codes" for this. You can annotate lexer rules with start codes so that they only apply when the lexer's state matches that start code. You can then push and pop start codes from a stack to set the state of the lexer.
This is a really helpful in certain situations such as lexing string templates. Does Lalrpop supports anything like this?
The text was updated successfully, but these errors were encountered: