UTF8-escaped characters mess up locations when they are part of a character class

[Earlopain]

Consider the following example:

require "regexp_parser"

ascii_escaped = <<~'REGEXP'
  [\.][b][\.]
REGEXP
utf8_escaped = <<~'REGEXP'
  [\（][b][\（]
REGEXP

def loc(node, index)
  [node.expressions[index][0].ts, node.expressions[index][0].te]
end

ascii_parsed = Regexp::Parser.parse(ascii_escaped)
utf8_parsed = Regexp::Parser.parse(utf8_escaped)

pp ["ascii", loc(ascii_parsed, 0), loc(ascii_parsed, 1), loc(ascii_parsed, 2)]
pp ["utf8", loc(utf8_parsed, 0), loc(utf8_parsed, 1), loc(utf8_parsed, 2)]

It has the following output, printing start/end locations for the content of each character class:

["ascii", [1, 3], [5, 6], [8, 10]]
["utf8", [1, 3], [7, 8], [10, 12]]

Notice how the locations are different. Both regexp have the same source length (12 in this case), yet they report different locations for the content of the character classes. More specifically, if you append more utf8-escaped character classes you get locations that are outside the length of the regexp.

If I switch out the character classes for a capture group, the locations are correct again.

[Earlopain]

I missed this beforehand (not sure how), but the much more simple repro is to simply look at the expressions:

pp Regexp::Parser.parse(<<~'REGEXP').expressions[0].expressions
  [\（]
REGEXP

[#<Regexp::Expression::EscapeSequence::Literal:0x00007fc855a3b360 @text="\\\xEF">,
 #<Regexp::Expression::Literal:0x00007fc855a3af00 @text="\xBC">,
 #<Regexp::Expression::Literal:0x00007fc855a45d38 @text="\x88">]

Makes it obvious where the extra offset comes from

[jaynetics]

Thank you very much for the report and fix! ❤️ I've just released it as v2.11.3.

[jaynetics]

ℹ️ If anyone finds this issue later and is confused by the above examples, what looks like a space and a parentheses ( () is actually a single char, U+FF08 Fullwidth Left Parenthesis (（).

[Earlopain]

Thanks for the release!