Dealing with non-ascii URIs (IRIs) in redirects

[jgr]

I'm running into problems when getting a URI that happens to redirect to a URI with non-ascii characters in it (an IRI from what I understand). A simple, repeatable case (in this case using http.rb 0.7.3 and Ruby 2.2.0) is:

require 'http'

REDIRECTS_TO_UMLAUT_URL = 'http://www.amazon.co.uk/dp/B00C6Q3SEQ'
http_client = HTTP.follow(true)
http_client.get(REDIRECTS_TO_UMLAUT_URL)

Which results in this error:

/Users/jamesrucker/.rbenv/versions/2.2.0/lib/ruby/2.2.0/uri/generic.rb:1100:in `rescue in merge': URI must be ascii only "http://www.amazon.co.uk/Bianka-Minte-K\xC3\xB6nig-Emilia-Sch\xC3\xBCle/e/B001K1JKKY/digital/ref=ntt_mp3_rdr/276-0649186-0513969?_encoding=UTF8&sn=d" (URI::InvalidURIError)
    from /Users/jamesrucker/.rbenv/versions/2.2.0/lib/ruby/2.2.0/uri/generic.rb:1097:in `merge'
    from /Users/jamesrucker/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/http-0.7.3/lib/http/request.rb:87:in `redirect'
    from /Users/jamesrucker/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/http-0.7.3/lib/http/redirector.rb:47:in `follow'
    from /Users/jamesrucker/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/http-0.7.3/lib/http/redirector.rb:22:in `perform'
    from /Users/jamesrucker/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/http-0.7.3/lib/http/client.rb:35:in `request'
    from /Users/jamesrucker/.rbenv/versions/2.2.0/lib/ruby/gems/2.2.0/gems/http-0.7.3/lib/http/chainable.rb:16:in `get'
    from ./fetch_umlaut_url.rb:16:in `<main>'

From doing a bit of googling, it sounds like two possible solutions are to call URI.encode before URI.parse (which isn't foolproof...it will encode hash fragments for instance) or to use the addressable gem, neither of which seemed like great options for a PR. Is there a more reasonable solution?

[tarcieri]

Kinda sad this just doesn't work with stdlib URI.

I'm not opposed to using a gem if it has better handling of non-ASCII URIs.

[ixti]

@tarcieri we can actually support Addressable when it's available. I mean do not depend on it directly, but provide different URI parser backends...

[ixti]

Some kind of Multi-URI :D

[ixti]

Oh, just realized (after reading mentioned SO thread), there's actually a solution:

def normalize_uri(uri)
  return uri if uri.is_a? URI

  uri = uri.to_s
  uri, *tail = uri.rpartition "#" if uri["#"]

  URI(URI.encode(uri) << Array(tail).join)
end