Fallback

[gwenn]

lalrpop

[tarqd]

I'm using this method for my fork for MySQL grammar support. Is this what you're going for?

Id: Name = {
    "Id" => <>.to_owned(),
    AsStr< Fallback > => <>.to_owned()
};
# list of all tokens that can fallback to an Id 
Fallback = {
  # shortened list for brevity
  "extended",
  "partitions"
};
AsStr<T> : &'input str  = {
  # get original text from buffer (passed in from grammar parameters)
  <start:@L> T <end:@R> => &text[start..end]
};

[gwenn]

For me, only the parser can choose to fallback properly.
You can take look at lemon (search "iFallback").

Or you have to tweak your grammar like you suggest or me or here.
But, at least for the SQLite grammar, this generates conflicts.

And I do not feel able to add fallback to lalrpop. I don't even know if it is feasible.
So I am trying to make lemon generate Rust code here...
I already made lemon generate Java code.
Lemon is so fast (compare to lalrpop) and so terse !

[tarqd]

Do you mind giving me an example of a conflict for SQLite? There's a lot of underdocumented features in lalrpop so maybe I can hack together a workaround. I have to admit the codebase for lalrpop is hard to grok for me (lot of concepts that are frankly outside my comfort zone and I'm very new to rust).

I'd love to use your Lemon implementation instead of lalrpop if you're still working on it (there are modified sqlite lemon grammars for MySQL in the wild already so that'd save me a lot of time) and speed is definitely a concern for my use-case. If you think that's the best route compared to lalrpop I'd be happy to contribute what I can to lemon-rs

[gwenn]

You just have to uncomment the lines here and you will have conflicts.

And I suggest you keep using lalrpop because I don't want to waste your time.
First, I want to do a streaming lexer (with lalrpop lexer and SQLite tokenizer you have to load the whole content in memory even if the SQL dump is huge). But perf are not good when comparing a simple CSV tokenizer against rust-csv.
Then, I will have to fix the memory leak in lemon-rs (union with drop fields).
And finally, I need to rewrite the SQLite grammar with Rust actions.
Maybe next year there will be something usable ...