Parser balks at long tokens (regression)

[opeongo]

The javacc parser (or more specifically the JavaCharStream class) has changed since bsh-2.0b5 and it appears that it can no longer handle arbitrarily sized input tokens. Previously the JavaCharStream implementation had working code that would expand the internal buffer as required, thus supporting arbitrarily sized tokens found in the input stream. It appears that in the switch from JavaCC to ParserGeneratorCC this capability no longer works. There is code in the class to increase the size of the internal buffer, however here is an example of a long string assignment that the current parser cannot handle:
parser1.bsh.txt

[opeongo]

I found a fix for this problem. If consists of commenting out two lines in the AbstractCharStream class. See my comments on the ParserGeneratorCC site about this.

These changes can be made to the template in the upstream ParserGeneratorCC project, or they can be made to the generated AbstractCharStream.java file that shows up in the targets/generated-sources/javacc/bsh folder . I just do not know how to make these changes so that they persist. Right now any changes will get clobbered if javacc is rerun.

[nickl-]

I wonder if that wasn't a patch in the original javacc that beanshell was using. This is a difficult one, lets hope they fix it upstream.

[revusky]

Nick, it's really pretty safe to say that none of these things will ever be "fixed upstream". Well, not unless you finally pick up the ball and move over to the actively developed/maintained branch of development, which is CongoCC.

And you should really do that sooner rather than later, because it's the same amount of work to do it now as later, and you get the benefits of the transition sooner. Are your collaborators aware of this branch?