lalrpop exploration

[CrockAgile]

I just read thru the lalrpop documentation and it seems worth investigating the readability & performance comparison to the current nom usage. This exploration task could just be done on its own branch and never merged and it would serve its purpose. But if it stumbles into some improvements, then it could be up for consideration.

[shnewto]

@CrockAgile I'll plan on taking some time to read the lalrpop docs. I'm definitely interested potential for value it would add to this project. Keep me posted on any details.

[shnewto]

Just remembered this issue and something I'd bumped into trying to work with significant whitespace reading the Writing a custom lexer chapter in the lalrpop book. Looks like there's a way work around it (the point of that chapter) but it wasn't the way I'd hoped. I think it's a deal breaker for me, for now.

By default, LALRPOP generates a tokenizer that skips all whitespace -- including newlines. What we want is to capture whitespace characters and ignore the rest as comments, and LALRPOP does the opposite of that.

At the moment, LALRPOP doesn't allow you to configure the default tokenizer. In the future it will become quite flexible, but for now we have to write our own.