[Feature Request]: Ability to use `$` in regexp

[hackwaly]

Currently $ is a special rule, not a regexp. Thus we can't represent the follow:

    '"' character* ('\n' | $) {
        error lexbuf "unclosed string literal"
    }

[skvadrik]

What would unquoted $ mean in a regexp? Do you want it to mean the end of input?

[hackwaly]

Yes.

[skvadrik]

Ok, I need to think how it can be implemented. It will require changes throughout the RE -> NFA -> TDFA construction pipeline. The main difficulty seems to be that final states for the same rule will have to be split into those requiring end-of-input check and those that don't need it. I'll give it more thought, but no quick fix for this one. :)

[hackwaly]

In ocamllex and sedlex, eof is special pseudo character which value is -1. YYPEEK returns -1 if it reaches end of input. But it is a challenge in re2c which output char ranges for pattern. So -1 cannot fit well as a char type.

sedlex solves this by using int type for yych and generate "binary search optimized if chains" instead of pattern matching for character set match testing.

[eilvelia]

In case the input is expected to be valid UTF-8, there's a possibility of using an invalid UTF-8 code unit to indicate EOF, e.g. \xFF. I think something like that is already possible to implement using re2c, although it would be less efficient because of bounds checking on every get and a bit less nice to use. For example:

let peek str i =
  if i >= String.length str then '\xFF' else String.unsafe_get str i

(*...*)

    '"' character* [\n\xFF] {
        error lexbuf "unclosed string literal"
    }

upd: It looks like with --utf8, this is parsed as U+00FF and not literal 0xFF, sadly.

[skvadrik]

I'm taking a stab at this; here's a few notes:

• Implementation. Although we cannot use a special end-of-input symbol in the generated code, it's still possible to use it for the internal implementation (add extra TNFA / TDFA transitions on this symbol). This is only an implementation detail, but it simplifies certain things a lot; e.g. TDFA minimization will just work without any changes, etc.

• Backwards compatibility. Allowing $ appear in any rule means that the end-of-input rule $ { ... } is no longer a special rule (as, for example, the default * rule); it's just a regular rule. And there is a case when special rules are treated differently from normal rules: if there is a rules block X that gets reused by a block Y, normal rules from Y are appended to the inherited normal rules from X as if the two blocks are concatenated (or as if the X block was inserted in place of the !use directive). But special rules from the Y block override inherited rules. Consider the example below:

   /*!rules:re2c:X
     re2c:eof = 0;
  
     [a] { /* X-a */ }
     [b] { /* X-b */ }
     *   { /* X-default */ }
     $   { /* X-eof */ }
   */
  
   /*!re2c:Y
     [a] { /* Y-a */ }
     !use:X;
     [b] { /* Y-b */ }
     * { /* Y-default */ }
     $ { /* Y-eof */ }
   */
  

  Currently Y-a, X-b, Y-default and Y-eof rules takes priority, but when $ becomes a normal rule, X-eof will take priority.

  This is a breaking change, but it's a rare case and I'm thinking of emitting an error in cases when there is a shadowed $ rule (rather than just a warning, as we do now for shadowed rules). This ways the users affected by the change will notice it.

  We can, of course, keep backward compatible behavior for just $ { ... }, but it will be weird if it differs from "" $ { ... } or e.g. $ | "a" { ... } in this regard, probably a worse inconsistency than the breakage discussed above.

[skvadrik]

This was done and released in 4.4, see https://re2c.org/releases/release_notes.html#release-4-4.