JavaCC 21

[revusky]

Since you guys are heavy users of that legacy dinosaur JavaCC project, I thought you might find an update on the work going on in the space to be useful. Or interesting, at least. I'm referring to JavaCC 21 which is my second run at reviving JavaCC. The first attempt was back in 2008 and I decided, after dropping it for a decade to pick it up again. Anybody interested can read the whole history of that here.

There are a lot of things to tell you about but this may be the biggest one:

You know, there's this very longstanding bug in JavaCC that nested lookahead does not work properly. That is basically a showstopper in terms of any very sophisticated use of the tool. You can't use LOOKAHEAD recursively. It's right there in their FAQ.

Anyway, on my last iteration of refactoring and cleaning up the code, I realized that nailing that bug was now low-hanging fruit. So I fixed it.

I hope that piques people's interest. There are a bunch of other quite nifty improvements that came on line in the last few months. For example, this new up-to-here marker construct or this new feature I call Lookbehind.

Well, that's enough said for now. Pardon the shameless plug, but you have to understand that this project is at a severe disadvantage visibility-wise so there is really little choice. Besides, somebody should break it to you guys that you are using something that is totally obsolete!

[AnEmortalKid]

@revusky , thank you for the revival of javacc21 and letting us know it is ready to give a spin. Nice interesting bit of history there! Also, thanks for all the effort and time spent carrying the torch.

[AnEmortalKid]

@revusky any chance the jar can be grabbed from maven?

It looks like we depend on a plugin: https://github.com/JSQLParser/JSqlParser/blob/master/pom.xml#L126 but at first glance and super quick inspection, we might be able to JUST change the dependency. Otherwise, we might have to do a script/something custom to call to the jar.

Follow up, did you end up ripping out JAVACODE? https://parsers.org/t/about-to-rip-out-javacode-productions-does-anybody-object-to-this/34

We use them in a few instances: https://github.com/JSQLParser/JSqlParser/blob/master/src/main/jjtree/net/sf/jsqlparser/parser/JSqlParserCC.jjt#L2498

[revusky]

@revusky any chance the jar can be grabbed from maven?

Well, at this precise moment, no, because it's not there. But that may well change soon That said, I honestly don't know why downloading something from Maven Central is of any importance to anybody. You're not the first person to bring this up, but it drives me a bit crazy, because I just cannot fathom why it matters to anybody which server you download something from. But... then again, I have been basically out of things for a decade or so, 2009-2019, so some things changed in the field that I don't really understand. (Yet?) And beyond that, to the extent that I even understand the concept of Maven Central, I really don't like it that much. (But that is another conversation.)

Follow up, did you end up ripping out JAVACODE? https://parsers.org/t/about-to-rip-out-javacode-productions-does-anybody-object-to-this/34

We use them in a few instances: https://github.com/JSQLParser/JSqlParser/blob/master/src/main/jjtree/net/sf/jsqlparser/parser/JSqlParserCC.jjt#L2498

Well, the whole JAVACODE thing is really a non-issue but I see it as this sort of weird source of confusion (which is precisely why I removed it!) but then I started writing an explanation of the issue and finally just figured that I'd write a blog post about it, so the explanation is there and easily findable...

https://javacc.com/2020/10/18/javacode-productions-redux/

[revusky]

Oh, another thing that should be of great interest to people int this community is that I've finally refactored the code so that this problem is tractable: #677

Just in the last few days, actually! Actually, the whole problem of context-dependent tokenizing is basically intractable in legacy JavaCC -- well, at least assuming that you make any significant use of LOOKAHEAD. If your grammar is LL(1) throughout, then the existing lexical state machinery works well enough, I grant. But in terms of interactions with LOOKAHEAD, basically the whole thing is FUBAR broken.

The funny thing is that Tobias Warnecke showed up and was asking me about this whole issue some months ago. I offered him a solution that I now realize doesn't really work javacc21/javacc21#25 (comment)

I mean, the above "solution" doesn't work (at least not reliably!) because this kind of stuff is pretty broken in legacy JavaCC and has been until very recently in my version too. You can expect a blog post about this fairly soon. Probably more lengthy and detailed than the one I wrote about JAVACODE productions getting ripped out!

The problem is now tractable in JavaCC21, but I'm still working on a more elegant general solution to the issue of context-dependent tokenization. Surely you understand that these sorts of issues will never be addressed in the legacy JavaCC project. Sun Microsystems open-sourced JavaCC in 2003 and, as far as I can tell, they put out all these versions, gearing up for an 8.x release cycle, but all of that alleged development work does not encapsulate a single new idea... I guess that could come off as me running down my "competition", but I feel like I'm just stating the plain facts..

[AnEmortalKid]

That said, I honestly don't know why downloading something from Maven Central is of any importance to anybody. You're not the first person to bring this up, but it drives me a bit crazy, because I just cannot fathom why it matters to anybody which server you download something from.

Probably just the easiest thing to integrate into my build cause of the pom :) I guess it doesn't HAVE to be hosted on central, any repository that can support downloading through maven should be fine.

The funny thing is that Tobias Warnecke showed up

Yep, Tobias knows the most about how all of this crazy stuff works.

[revusky]

Okay, I wrote a new blog post about this whole issue of context-sensitive tokenization. The long and short of that is that this is something that in legacy JavaCC, simply does not work! I explain this in quite nitty-gritty detail here: https://javacc.com/2020/10/20/context-sensitive-tokenization/

Now, the point of that post for you guys is that, even though it may not seem so at first glance, the machinery put in place to handle that problem also applies to this problem: javacc21/javacc21#25 (comment)

I mean, if you're going to handle square brackets differently depending on context, this is basically intractable in legacy JavaCC. I honestly don't know how ANTLR is on this sort of thing, but legacy JavaCC just drops the ball completely. And that is basically fixed in JavaCC21 and that is what the above-linked blog post explains, though one might have to read it carefully to get the point.

Also, regarding the JAVACODE productions, did my previous blog post answer the question for you? https://javacc.com/2020/10/18/javacode-productions-redux/

[AnEmortalKid]

@revusky , thank you for that blog post (and for writing the code i'll need to change there :) ). That definitely answers my question!

[revusky]

@revusky , thank you for that blog post (and for writing the code i'll need to change there :) ). That definitely answers my question!

Good. I did write that to answer your question, but also because I figured that I would need to answer the same question later on occasion, so I figured I might as well write a reasonably detailed explanation of the situation.

I hope you can see why I removed that JAVACODE nonsense. Not only does it not really do anything but, as far as I can see, it can only be a source of confusion. So, while I'm adding much-needed new features to the tool, I'm also streamlining away that kind of useless old stuff that just doesn't do anything useful...

Let's see... in other matters, I was looking at your grammar a bit. I guess I'll just point out some simple things, since one should pick the low-hanging fruit first.

This whole disposition of having a BaseToken class to be the superclass of Token, so you can add in a couple of fields... there is no need for that. I'm referring to this

If all you want to do is insert a couple of fields in your Token class, you can just use a code injection, a trivial one at that:

 INJECT Token : {public int absoluteBegin, absoluteEnd;}

So the BaseToken.java is no longer needed and you also don't need that TOKEN_EXTENDS=BaseToken line in the options. In fact, that option doesn't even exist in JavaCC21. By the way, if you really need a generated class to extend some class or other (or implement an interface) or stick an annotation up top, you could also use an injection, so if you really did need to have a BaseToken class that Token extends (you don't need that!) you could always just use an injection like this:

 INJECT TOKEN : extends BaseToken

But you don't need that. Better just to inject the two fields you want in there. Though IMHO you might consider encapsulating the fields, i.e.

  INJECT Token : {
          private int absoluteBegin, absoluteEnd;
 
           public int getAbsoluteBegin() {return absoluteBegin;}
           public void setAbsoluteBegin(int absoluteBegin) {this.absoluteBegin = absoluteBegin;}

           public int getAbsoluteEnd() {return absoluteEnd;}
           public void setAbsoluteEnd(int absoluteEnd) {this.absoluteEnd = absoluteEnd;}
  }

That gives you more flexibility to refactor things moving forward, but that's up to you...

As for the whole question of all this plethora of settings in legacy JavaCC, this page should be a good resource.

This page and this one will likely be useful.

Well, you'll run into some other things, like the way a lot of that old crufy JJtree stuff has been reworked. All that stuff of generating some XXXVisitor.java file and all that, that's gone, but there is a far more elegant solution built in. But... when you get to that, feel free to drop me a note and ask me any questions. I'm really quite happy to help. Besides, it gives me feedback about what things are confusing to people who are trying to upgrade.