Linear list of tokens

[LoganDark]

Currently, LexLua returns a large list full of tokens, each token potentially containing more lists of tokens, for example LeadingWhite full of more tokens that have to be plucked out manually and all sorts of weird things like that. Additionally, the order of LeadingWhite cannot be relied on by index (not even reversed), as I figured out earlier (see issue #18). Also, comments are preserved incorrectly in LeadingWhite (#20).

I'm looking to get a list containing tokens in the order they are encountered. That means by looping through the list and adding up the Data of each token, I'll be able to reconstruct the original input string passed to LexLua exactly.

Is there a ready-made function to create that list for me? If not, how would I create such a function?

[LoganDark]

I ended up rewriting the lexer from scratch myself. I could share the code if anyone's interested.

[tst2005]

@LoganDark : I'm always interested by Lexer/Parser :D

[LoganDark]

https://pastebin.com/edyuQ5xY

[tst2005]

Can you publish it in a repository ? with LICENSE etc ?

[LoganDark]

https://github.com/LoganDark/lua-lexer

[tst2005]

thanks!