Unfeeding Tokens

[aliyildiz1980]

I am using lark-parser for a project I am working on. While parsing the input text, if an unexpected token is received, the token needs to be fixed with heuristic methods (that is, the token is converted to another token and/or additional tokens are feeded). To accomplish this, I coded custom lexer. It works really good. To carry my work to one upper level, I need to unfeed tokens. But I couldn't find any functions in api to accomplish this. Is unfeeding tokens possible?

Example:
Input text receied: ab1
a is received and feeded. OK
b is received and feeded. OK.
1 is received but not OK. Unfeed b (tree must be reverted too), feed 1 and feed b again.

One way is to extend the language to parse ambiguous inputs. But I don't want to use this method because the language is already very complex.

[MegaIng]

No, this is not really possible, primarily because it is not always unambiguous what exactly that means, and sometimes there is even a loss of information. You can instead copy the state after every successfully fed token and then manually roll back. (using the new Interactive Parser interface)

[aliyildiz1980]

Thanks for fast reply. Is there a documentation? Which data do I need to copy and roll back? Only lexer_state and parser_state?

[MegaIng]

Small example: (Lark.parse_interactive InteractiveParser/ImmutableInteractiveParser

# as_immutable is currently undocumented.
base_puppet = parser.parse_interactive(start='start').as_immutable()

history = [base_puppet]

for t in tokens:
    try:
        new = history[-1].feed_token(t)
    except UnexpectedToken:
         pass
    else:
         history.append(new)

[erezsh]

@MegaIng I think we should add a method to the interactive parser for consuming the next token.

Something like InteractiveParser.feed_from_lexer(), which will advance by getting the next token from the lexer. We can add a keyword all=True for getting all the tokens, which can replace exhaust_lexer().

[aliyildiz1980]

Thanks for your suggestions. I have one more question. How can I save state, if I use Earley parser (to take advantage of ambiguous parsing)? Current version raises exception saying parse_interactive is not supported with Earley.

[MegaIng]

@aliyildiz1980 You can't at the moment.