UTF-16LE Regexp not working as expected

[zako42]

Not able to create a UTF-16LE encoded Regexp object. I tried the following in Jruby 1.7.19 in IRB:

abc8 = "abc"
abc8.encoding  # <Encoding:UTF-8>
regex8 = Regexp.new(abc8)  # succeeds

abc16 = "abc".encode("UTF-16LE")
abc16.encoding  # <Encoding:UTF-16LE>
regex16 = Regexp.new(abc16)
# in MRI 1.9.3-p545 this succeeds
# in jruby 1.7.19:  Encoding::CompatibilityError: incompatible character encodings: US-ASCII and UTF-16LE

[headius]

Works ok on master, but that's not too surprising (reimplemented encodings since 1.7).

I would suspect the problem is in our Regexp encoding negotiation (prepareRegexp and such); it's not allowing this combination when it should.

[azolotko]

Here's the stack trace:

Encoding::CompatibilityError: incompatible character encodings: US-ASCII and UTF-16LE
    from org/jruby/RubyString.java:1228:in `%'
    from org/jruby/RubyKernel.java:986:in `sprintf'
    from org/jruby/RubyKernel.java:615:in `printf'
    from org/jruby/RubyKernel.java:1511:in `loop'
    from org/jruby/RubyKernel.java:1274:in `catch'
    from org/jruby/RubyKernel.java:1274:in `catch'
    from /Users/azolotko/.rbenv/versions/jruby-1.7.20/bin/jirb:13:in `(root)'

As you can see, the error is raised when IRB tries to printf the result of the evaluation, i.e. => /abc/.

The actual difference between JRuby and MRI seems to be in the result of the Regexp#inspect:
regex16.inspect in JRuby 1.7.20 returns "\u612F\u6200\u6300\u2F00", when MRI does "/a\x00b\x00c\x00/"

[azolotko]

JRuby 1.7.20:

>> regex16.inspect.encoding
=> #<Encoding:UTF-16LE>

MRI 2.2.2p95:

irb(main):004:0> regex16.inspect.encoding
=> #<Encoding:US-ASCII>

MRI forces US-ASCII if the Regexp encoding is not compatible with ASCII: https://github.com/ruby/ruby/blob/ruby_2_2/re.c#L434
https://github.com/ruby/ruby/blob/ruby_2_2/re.c#L411

But JRuby uses the original encoding:
https://github.com/jruby/jruby/blob/jruby-1_7/core/src/main/java/org/jruby/RubyRegexp.java#L1843
https://github.com/jruby/jruby/blob/jruby-1_7/core/src/main/java/org/jruby/RubyRegexp.java#L1314

[enebo]

@azolotko you did such a good job find this is there any chance you can submit a pull request too?

[azolotko]

@enebo appendRegexpString19 looks scary. Let's see if I can do this.

[azolotko]

Sorry guys, I didn't manage to tackle the issue and can't spend more time on this. Maybe someone else will have a better luck.