Provide the ability to disable scanner buffer growth

[madrob]

The Lucene project hacks generated scanners to do this - see https://issues.apache.org/jira/browse/LUCENE-5897 and https://issues.apache.org/jira/browse/LUCENE-5400 for background.

[lsf37]

Logging some design thoughts here:

If we remove buffer growth, we are obviously putting a limit on the length of returned tokens (that is the whole purpose). That means, there is a new error condition when we reach a token of that length. I think it would not make sense for the scanning engine to try to recover from that condition by itself. The application might treat this as an unrecoverable error and just shut down the scanner or try to handle it in some way by returning the text matched so far, or anything in between.

If we just removed the buffer growth code, generated scanners will currently produce an IOException, because it will request 0 characters from the Reader and will therefore get 0 characters back, which is already a checked error case.

In terms of recovery options for the user, depending on what kind of state the scanner keeps, one could for instance yyreset() the scanning engine with the current Reader again and keep scanning. Line/column etc counting will be reset and lost, though. One could also add a scanner method (in standard lexer user class code) that only resets the buffer-related fields (zzEndRead, zzMarkedPos, etc), but not other state in the scanner. Depending on what the scanner does, this could be safe and keep counts etc up to date. It would lose content of the offending error token.

What would not be easily possible is figuring out what kind of token it was that created the error -- because scanning is not finished, multiple expressions could still be matching simultaneously, some expressions might have matched but are not the longest match, or no expression might have matched yet, but multiple are still possible to match. For now, I'm going to leave all these unexplored, because the use case is mostly guarding against malicious input where it would be Ok to abort.

[lsf37]

In terms of spec language, I'm currently thinking of introducing a directive

%token_size_limit (<identifier>|<number>)

where <number> would be a compile-time limit on the buffer size, and <identifier> could be either a constant or a mutable field that can be declared in and adjusted from user code at runtime. Should probably be a long ident instead of just ident.