Process Known Single-Character Lexemes Immediately #16
Conversation
|
Hmm looks like this is going to be more complicated than I'd hoped. |
Also avoid building a new set every time
|
It's worth noting that this PR doesn't solve this issue for binary inputs; the codecs wrapper is unfortunately greedy in its own right. |
|
Looks like the tests aren't increasing coverage as much as expected. Odd. Another note on these tests: they've uncovered something else really oddly ugly; suppressed UnexpectedSymbol errors that only come up if I iterate over
|
|
Hi, I'm off until next week, so I won't be able to pay close attention to this until then. Coverage is bogus (don't know why), so don't mind it too much. Is this new code effectively faster? You could try running the benchmarks to get an idea. |
|
@rtobar I wouldn't expect much difference under normal conditions and the benchmarks seem to support this. |
The new set of tests introduced in the previous commit highlighted a corner case that wasn't properly covered, and that was mentioned in #16: the yield expression within the try/expect block can also actually raise an exception if the generator is being destroyed before it is consumed fully (e.g., if manually iterated over via next() and then discarded). In these cases an UnexpectedSymbol exception would be raised during the destruction of the generator object, which would be ignored by the interpreter. The solution is simple: include *only* the number parsing logic within the try/catch, leaving the yield expression outside to get a normal logic flow. Signed-off-by: Rodrigo Tobar <rtobar@icrar.org>
|
Closing, although all changes made it to the repo via manual cherry picking plus some adjusting. |
Addresses #15 by checking for the existence of terminal lexemes.