Jsoup can't connect to URL with Unicode path

[johannroux]

Hello,

Testing with Jsoup 1.15.4 (latest on this date), it seems Jsoup is unable to retrieve pages published on URLs with a Unicode path.

Description of the problem

val url = "https://example.com/unicode/שלום"

val doc: Document = Jsoup.connect(url)
    .followRedirects(true)
    .get()

returns:

org.jsoup.HttpStatusException: HTTP error fetching URL. Status=400, URL=[https://example.com/unicode/שלום]

Expected behaviour

Jsoup should handle the encoding on its side and connect without problem.

Already-tested ideas

I tried the following:

Encode the whole URL with URI.toASCIIString()

val url = "https://example.com/unicode/שלום"
val encodedUrl = URI(url).toASCIIString()

val doc: Document = Jsoup.connect(encodedUrl)
    .followRedirects(true)
    .get()

returns:

org.jsoup.HttpStatusException: HTTP error fetching URL. Status=404, URL=[https://example.com/unicode/%25D7%25A9%25D7%259C%25D7%2595%25D7%259D]

Encode the whole URL with URLEncoder

val url = "https://example.com/unicode/שלום"
val encodedUrl = URLEncoder.encode(url, StandardCharsets.UTF_8.toString())

val doc: Document = Jsoup.connect(encodedUrl)
    .followRedirects(true)
    .get()

returns:

java.lang.IllegalArgumentException: The supplied URL, 'https%3A%2F%2Fexample.com%2Funicode%2F%D7%A9%D7%9C%D7%95%D7%9D', is malformed. Make sure it is an absolute URL, and starts with 'http://' or 'https://'. See https://jsoup.org/cookbook/extracting-data/working-with-urls

Encode just the path with URLEncoder

val url = "https://example.com/unicode/" + URLEncoder.encode("שלום", StandardCharsets.UTF_8.toString()) // yields https://example.com/unicode/%D7%A9%D7%9C%D7%95%D7%9D

val doc: Document = Jsoup.connect(url)
    .followRedirects(true)
    .get()

returns:

org.jsoup.HttpStatusException: HTTP error fetching URL. Status=404, URL=[https://example.com/unicode/%25D7%25A9%25D7%259C%25D7%2595%25D7%259D]

It seems that Jsoup is doing a double encoding on its end, but I might be wrong.

Thank you in advance for your help!

[jhy]

Thanks, fixed! The first form now works correctly by normalizing the input path.

The double escaping issue was fixed in #1902.

Specifically, the first form emits:

HTTP error fetching URL. Status=404, URL=[https://example.com/unicode/%D7%A9%D7%9C%D7%95%D7%9D]

Which is the correctly encoded URL, and the status that example.com returns.

[jhy]

Also with 0121311, if the query string contains non-ascii, we normalize that to ascii. Any existing escapes are preserved (which is why this impl is more complicated than just decoding the URL components and then constructing a URI and letting that encode -- existing escapes would get incorrectly smooshed)

[johannroux]

Thank you very much @jhy! When is the next release planned? 🙂