Add support for validating Regexp in Ripper

[grddev]

This represents a stab at implementing the remaining Ripper compatibility as mentioned in #4898.

This uses the fact that Regexp tokenization is handled by a single StringTerm, and thus all tSTRING_CONTENT fragments are easily collectable until the tREGEXP_END comes with the options that we need for validation.

The validation itself is a copied/simplified version of what is performed by the main parser, as large parts the validation depended on the AST structure, which we do not have here.

Technically, this doesn't perform the validation at the same point in time as the main parser, as it performs the validation when encountering the tREGEXP_END token rather than when processing the regexp rule.

I speculate that the difference doesn't really matter given that the only thing we could do with the tREGEXP_END token is to apply the regexp rule.

In order to reduce copy-paste between the main parser and Ripper, I opted to shift some Regexp-related code into the Lexer. In theory, the code doesn't belong in the lexer, but putting it in the Lexer has some benefits. First, it is a component that is shared in a reasonable way between the two parsers. Second, it is essentially required by the proposed implementation, as the new validation takes place effectively inside the Lexer.

Unsurprisingly, it turns out that the coverage for Ripper parsing of Regexp isn't very extensive, and I haven't had time to put the code through any additional tests.

[grddev]

Forgot to mention, but this does nothing to address local variables as mentioned in #4898, but some quick testing seemed to indicate that MRI didn't handle this either (as the below output has a vcall rather than a var_ref towards the end.

% RBENV_VERSION=2.4.2 ruby -rripper -e 'p Ripper.sexp("/\$(?<dollars>\d+)\.(?<cents>\d+)/ =~ \"$3.67\"\ndollars")'
[:program, [[:binary, [:regexp_literal, [[:@tstring_content, "$(?<dollars>d+).(?<cents>d+)", [1, 1]]], [:@regexp_end, "/", [1, 29]]], :=~, [:string_literal, [:string_content, [:@tstring_content, "$3.67", [1, 35]]]]], [:vcall, [:@ident, "dollars", [2, 0]]]]]

[enebo]

Really I just want a comment on that null fragment if since it is not obvious what scenario that happens in.

[enebo]

Is this only possible for /#{ffofofofo}/ and /@{gfogfogogog}/? If so can you add a comment here as to what scenario this is for? Without printing out the lex stream I would almost think this was not possible. A comment will help later on.

[grddev]

Rewrote the implementation with an additional regexpDynamic variable instead to make the logic simpler to follow instead on commenting on the cryptic logic.

[enebo]

Note for myself. We should make sure CR is calculated as we build up string so we do not need to rescan all bytelists after creation.

[enebo]

Since 9.2 will be considered a major version you can remove this from ParserSupport if you want (or not) and just make callers call lexer directly. Optional.

[grddev]

Done

[enebo]

@grddev sorry our last merge broke this. I did not notice you had addressed the comments so I spaced out landing this. Can you update it?

[grddev]

Rebased the changes on top of master

[enebo]

@grddev thanks. sorry I lost this one in the weeds :)