Use java.net.URLDecoder to fix UTF-8 related issues

[sinkuu]

Hi, I found MultiProtocolURL.unescape has a bug that eliminates some characters from non-ASCII input. i.e. new MultiProtocolURL("URL containing non-ASCII") sometimes breaks URL.

Example: new MultiProtocolURL("http://www.example.com/q=東方儚月抄").toString() => "http://www.example.com/q=東儚月"

This PR changes MultiProtocolURL to use the URL decoding functionality from the standard library.

[luccioman]

Hi @sinkuu, thanks for your report and also for detailing a bit more your example.

Given the core nature of this code, we should be careful with any eventual side-effects. Would you have also some higher level failing scenarios examples to provide?

[sinkuu]

URLDecoder.decode seems to be equivalent to unescape (except this bug), but escape does more than URLEncoder.encode (detecting &, and leaving some more punctuations as-is).

I've reverted escape change.

Would you have also some higher level failing scenarios examples to provide?

I found this bug via OpenSearch integration - when I search with a keyword containing non-ASCII, broken query is sent to OpenSearch provider.

[luccioman]

Ok thanks, I also effectively reproduce this bug with the OpenSearch connector.
I also found that remote Solr requests to other peers do not suffer from this as the Solrj client internally uses URLDecoder and URLEncoder.

[luccioman]

Hi @sinkuu , I found no regression after running hopefully enough high-level tests. By the way I added some unit tests and modifications to your proposal : if you have some time to perform some tests yourself, your feedback will be welcome.
Best regards