URI.unescape "extension" fails with Unicode input

[kivikakk]

Summary

This is a fix for a bug @tenderlove and I have been stepping through together.

URI.unescape in Rails 4.2 throws an Encoding::CompatibilityError if the (UTF-8 tagged) argument contains actual Unicode characters. This doesn't happen on 3.x; compare the monkey-patches: Turns out this was a patch made in our own application. Looks like we should be able to just pull this across.

3.x patched

      def unescape(str, escaped = /%[a-fA-F\d]{2}/)
        # TODO: Are we actually sure that ASCII == UTF-8?
        # YK: My initial experiments say yes, but let’s be sure please
        enc = str.encoding
        enc = Encoding::UTF_8 if enc == Encoding::US_ASCII
        str.dup.force_encoding(Encoding::ASCII_8BIT).gsub(escaped) { [$&[1, 2].hex].pack(‘C’) }.force_encoding(enc)
      end

4.2

    def unescape(str, escaped = /%[a-fA-F\d]{2}/)
      # TODO: Are we actually sure that ASCII == UTF-8?
      # YK: My initial experiments say yes, but let’s be sure please
      enc = str.encoding
      enc = Encoding::UTF_8 if enc == Encoding::US_ASCII
      str.gsub(escaped) { [$&[1, 2].hex].pack(‘C’) }.force_encoding(enc)
    end

The issue is that [$&[1, 2].hex].pack('C') returns an ASCII-8BIT tagged string, which we then fail to gsub into str. This wasn't a problem in the 3.x patched variant where the string was tagged as ASCII-8BIT anyway.

This PR opens by correcting the test; parser.escape(str) returns an US-ASCII (!) tagged string, so parser.unescape succeeds for similar reasons as why the 3.x patched variant succeeded. This corrects the test to resemble the actual use-case: passing UTF-8 tagged strings into URI.unescape.

[matthewd]

str.dup.force_encoding(Encoding::ASCII_8BIT)

I can't find that in 3-2-stable 😕

[kivikakk]

I can't find that in 3-2-stable 😕

Ugh, don't tell me this is in one of our own patches to activesupport … investigating.

[kivikakk]

Not only is it from a patch in activesupport made in our application, it was me who made the change nearly two years ago. My goodness.

So this has been in Rails forever — I think it's a valid bug nonetheless.

[matthewd]

Yeah, I agree it sounds like a sensible change.. that's why I tried to blame-chase why it'd gone away 😅

[kivikakk]

Haha, right? I'll let the latest commit fail CI so we can red–green this properly, then push the fix commit.

[kivikakk]

This is ready for review. There's two failing builds (one required), but it looks like it's a timeout (and I can't restart it).

[kivikakk]

Someone reran the failing builds! Thanks! ❤️

[eileencodes]

Thanks @kivikakk ❤️

[tenderlove]

@kivikakk it seems like this code is almost entirely lifted from upstream. Does this seem like something we should upstream so we can rm the monkeypatch in Rails?

[eileencodes]

@tenderlove this is Rails 😉

[eileencodes]

Oh gosh you meant upstream to Ruby. 😳

[kivikakk]

I dug into it a bit — URI.unescape was deprecated in 2009 (lol), see ruby/ruby@238b979. CGI.uenscape is recommended instead, which doesn't have this bug (it calls string.tr(…).b before gsub. Should we still consider submitting a patch?

[kivikakk]

Patch opened upstream: https://bugs.ruby-lang.org/issues/14586

After discussion with @tenderlove, it appears the following is the case:

• This problem exists in upstream Ruby. No-one is currently relying on the behaviour introduced by this PR, since it's always been broken.
• Since we're monkey-patching Ruby itself, we should try to make the fix there instead of monkey-patching.

So the best course of action looks like:

• Back this PR out before anyone comes to rely on the newly-fixed behaviour.
• Try to get the patch across the line in Ruby.
• Once (if) it's in Ruby-next, reintroduce this monkey-patch; this way the following combinations will work: old Rails, new Ruby; new Rails, old Ruby; new Rails, new Ruby. (Only old Ruby, old Rails won't work.)
• Keep the monkey-patch in our application. We can remove it if we get a new Rails with the monkey-patch.

Alternatively, we can just keep this PR in and hope Ruby might take the patch, but keeping the diff of core libraries between Ruby and Rails seems like something to aim for.

How does this sound?

[matthewd]

CGI.unescape is recommended instead

Do we call this monkey-patched method internally? If so, should we maybe be using CGI.unescape instead?

[kivikakk]

Do we call this monkey-patched method internally? If so, should we maybe be using CGI.unescape instead?

Nope — this is only a convenience patch that our users might be relying on.

[kivikakk]

This has been patched in Ruby! 🎉

So, I think we can keep this PR in.

[tenderlove]

@kivikakk I think even with the patch in Ruby, this line will never un-monkeypatch URI because the default encoding is UTF-8, but parser.escape will return a US-ASCII string (causing parser.unescape to return a wrongly encoded US-ASCII string). Would you mind fixing that?

[kivikakk]

Good catch, I'll open a PR shortly. 👍

[kivikakk]

→ #32210