Error: too many pushed back tokens (internal error)

[cdsousa]

Hi,

With Julia 0.4.1, I get ERROR: too many pushed back tokens (internal error)
Whenever I try to parse anything, e.g.,

import JuliaParser.Parser
ast = Parser.parse("a=1")

[EricForgy]

Me too (0.4.2)

[eschnett]

Would this be easy to correct, or does it require some more fundamental change? Can you give a pointer for where to start looking?

[Ismael-VC]

@eschnett and @EricForgy I have solved that one and 3 other bugs in #33 but I accidentally messed up my PF closed and I don't know what to do to reopen it, @jakebolewski didn't answer back.

This 4 fixes are needed in order to be able to load the module again:

• Fix StreamToken constructor call, see Fix TokenStream constructor, fixes #29 #31 & Error: too many pushed back tokens (internal error) #29
• Fix utf8sizeof import.
• Fix testfile abs path.
• Fix line_number_node, see New error on 0.4 MichaelHatherly/Lexicon.jl#131 & MichaelHatherly/Docile.jl@a2509fb

You can see the diff here: https://github.com/jakebolewski/JuliaParser.jl/pull/33/files

I have been able to continue with my Julieta experiment (Julia written in any natural language, currently testing with Spanish and Esperanto) after those changes.

Should I open yet another new PR with the same changes, do anyone else can reopen or merge #33?

There are still lots of errors, but that is because tests compare ASTs with norm_ast which yields different results, even with that, the parsed AST is still able to be evaluated successfully, but that is probably for another issue/PR.

[jakebolewski]

I have been distracted by life, but hopefully over the next couple of days I can help resolve these issues.

[EricForgy]

I have been distracted by life

I know the feeling :) Hope all is well @jakebolewski 🙏

[Keno]

Fixed by #33.