Odd behaviors with lexemes at end of greedy grammars

[hudson-ai]

g = greedy_grammar(body=lexeme("\d+"))

(
    g.match("123") is not None,
    g.match("123 ") is not None,
    g.match("123  ") is not None,
    (g+"x").match("123x") is not None
)
>> (False, False, False, True)

Only matches when not at end of grammar (I suppose an EOS would work?)

g = greedy_grammar(body=lexeme("\d+"), skip_regex='\s')

(
    g.match("123") is not None,
    g.match("123 ") is not None,
    g.match("123  ") is not None,
    (g+"x").match("123x") is not None
)
>> (False, True, False, True)

Requires single trailing whitespace to match

g = greedy_grammar(body=lexeme("\d+"), skip_regex='\s*')

(
    g.match("123") is not None,
    g.match("123 ") is not None,
    g.match("123  ") is not None,
    (g+"x").match("123x") is not None
)
>> (False, False, True, True)

Requires two trailing whitespace to match

[mmoskal]

@hudson-ai there is still an issue with the captures not being generated - I guess we can have either the parser generate partial captures (here "1", "12", "123") or you feeding EOS before you get the captures

[hudson-ai]

Thanks! I think that the current capture approach is fine... May need to expose something for partial captures in the future (not sure if there is actually a valid use-case for this..?), but for now I can try and manually inject an EOS