Broken nesting parsing in tokadd_string #20

Closed
adambeynon opened this Issue Oct 9, 2011 · 5 comments

Projects

None yet

3 participants

Various parse issues are present in tokadd_string. Simple example, this parses:

%Q(
    {
      #{yield `baz`};
    }
)

but this doesnt:

%Q{
    {
      #{yield `baz`};
    }
}

The resulting error is that it is getting to the EOF without closing the string:

SyntaxError for _opal/test2.rb: unterminated string meets end of file. near line 1: ""
done

Observing the tokens gathered in #advance it looks like the "`" following "baz" is being parsed as string content.

@zenspider zenspider was assigned Oct 11, 2011
Owner

Someone smarter and/or more motivated than me is going to have to fix this. The string parsing code in ruby and ruby_parser is ghetto.

Nevermind the previous comment. Was too tired.

@zenspider, you might recall that I was going to rewrite the lexer in Ragel. I've almost finished that, and it parses this input without problems:

whitequark@raidho:~/ruby_parser (ragel){*}$ ./t.sh heredoc4.rb 
TESTCASE:
%Q{
    {
      #{yield `baz`};
    }
}

EMIT tSTRING_BEG "%Q"
EMIT tSTRING_CONTENT "\n"
EMIT tSTRING_CONTENT "    {\n"
EMIT tSTRING_CONTENT "      "
EMIT tSTRING_DBEG "\#{"
EMIT kYIELD "yield"
EMIT tXSTRING_BEG "`"
EMIT tSTRING_CONTENT "baz"
EMIT tSTRING_END "`"
EMIT tRCURLY "}"
EMIT tSTRING_CONTENT ";\n"
EMIT tSTRING_CONTENT "    }\n"
EMIT tSTRING_END "}"
EMIT tNL nil

The new lexer currently passes almost all tests:

291 tests, 1682 assertions, 13 failures, 5 errors, 0 skips

It does not yet support stack states (cond, cmdarg, tern in the old one), as I'm looking for a way to do this better, and all expr_ states except {end, beg, fname} are slightly off. Nevertheless, the current code serves as a proof of concept that Ragel lexers can work on Ruby.

What do you think about my work?

Owner

Once I fix up a few existing grammar bugs I'll be poking at your ragel-based lexer.

Owner

Holy fuck. I fixed it!

@zenspider zenspider closed this Jul 8, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment