JRuby isn't escaping strings correctly

[nevans]

I've encounted a weird issue that I don't know how to debug. This is using JRuby 10.0.2.0. I used the version installed by ruby-build using rbenv install.

There's a rake task in ruby/net-imap that generates some code (mostly regular expressions and hashes) from RFC tables. First it parses the RFC and writes the data into a JSON file. Then it reads that JSON file and generates the code. It was broken into two parts so rake can conditionally run the generator only if the JSON or the generator itself has changed.

Anyway, it's not generating the JSON correctly, and I can't figure out why. To replicate:

From a clone of https://github.com/ruby/net-imap, run bundle exec rake saslprep_rb.

To dig deeper, run irb, and from that irb console, do the following:

load "./rakelib/string_prep_tables_generator.rb"
generator = StringPrepTablesGenerator.new
rfc_path  = generator.rfc_filename
rfc_text  = File.read rfc_path
parsed    = generator.send(:parse_rfc_text, rfc_text)

This parsed variable now holds the data that will be generated and written to disk and then reread in a later step. But a couple of strings toward the bottom show the issue:

irb(main):052> d_1 = parsed["titles"]["D.1"]
=> "Characters with bidirectional property \"R\" or \"AL\""
irb(main):053> d_1 == d_1.dump.undump
=> true
irb(main):054> d_1.bytes == d_1.dump.undump.bytes
=> true
irb(main):055> d_1.encoding == d_1.dump.undump.encoding
=> true
irb(main):060> puts JSON.generate d_1.dump.undump
"Characters with bidirectional property \"R\" or \"AL\""
=> nil
irb(main):061> puts JSON.generate d_1
"Characters with bidirectional property "R" or "AL\""
=> nil

Same with:

irb(main):065> d_2 = parsed["titles"]["D.2"]
=> "Characters with bidirectional property \"L\""
irb(main):067> puts JSON.generate d_2.dump.undump
"Characters with bidirectional property \"L\""
=> nil
irb(main):068> puts JSON.generate d_2
"Characters with bidirectional property "L\""
=> nil

What's going on with the quoting there?! What's different about the string and its dump.undump counterpart?

[byroot]

I'm sorry but I have a bit of a hard time understanding the issue (granted it's early here).

Anyway you could reduce this further? E.g. dump the problematic string in a byte array so you can rebuild it in a simple standalone script or something?

Because at that point it's not clear if there is a problem with how JSON created the string or if it's an issue with JRuby's dump/undump methods.

[nevans]

I'm sorry but I have a bit of a hard time understanding the issue

Me too! It's weird!

Anyway you could reduce this further? E.g. dump the problematic string in a byte array so you can rebuild it in a simple standalone script or something?

That's the part where I got stumped: the bad string and good string have the exact same byte array and the same encoding. I didn't print out the bytes because the string is exactly what it looks like: all of the characters are ASCII.

There must be some underlying JRuby string representation differences, but I don't know how to debug that.

I could probably eventually reduce the code to a bare-bones reproduction script, but... it was late on a Friday, and it can be reproduced with only a few lines (in the net-imap repo).

Because at that point it's not clear if there is a problem with how JSON created the string or if it's an issue with JRuby's dump/undump methods.

I used "dump.undump" to create an identical (surface level) string without the problematic (underlying hidden) state. Also, because #dump does something very similar to JSON string encoding, I wanted to see if it also exhibits the bug (it doesn't). So, I don't know if the bug is in JRuby's json encoder or if its handling of global regexp match vars ($1, $2, etc) created a string with corrupted internals... or something else entirely. But, other methods (like String#dump) seem unbothered by the string, so I'm guessing JRuby has different valid internal string representations and the bug is in the JSON encoder for one of them.

[byroot]

Yeah, that's likely require too much JRuby specific knowledge for me to figure out, so I'll defer to @headius here, but I think you could help him help you by reducing the repro further when you get the time.

[nevans]

I was gonna say, I'll get to that later. But I was able to try out my simple hypothesis while lying in bed using my phone (termux => ssh to laptop with JRuby). 🙂

irb(main):001* 'String with text: "I\'m extracting this text."  This too.' =~
irb(main):002>   /text: (".*")  (\w.*)/
=> 12
irb(main):003> quoted = $1
=> "\"I'm extracting this text.\""
irb(main):004> other = $2
=> "This too."
irb(main):005> require "json"
=> true

irb(main):009> puts JSON.pretty_generate quoted: quoted.dump.undump, other: other.dump.undump
{
  "quoted": "\"I'm extracting this text.\"",
  "other": "This too."
}
=> nil
irb(main):010> puts JSON.pretty_generate quoted:, other:
{
  "quoted": ""I'm extracting this text.\"",
  "other": "This too."
}
=> nil

[nevans]

So it looks to me like it is somehow related to the regexp match extraction. FWIW, global vars vs MatchData makes no difference:

irb(main):011> match = 'String with text: "I\'m extracting this text."  This too.'.match(/text: (".*")  (\w.*)/)
=>
#<MatchData
...
irb(main):012> puts JSON.pretty_generate quoted: match[1], other: match[2]
{
  "quoted": ""I'm extracting this text.\"",
  "other": "This too."
}
=> nil

[tompng]

Maybe it is related to shared substring

irb(main):001> puts 'abcd"efg'[2, 4].to_json
"cd"e"
=> nil