Erroneous "0" returned where empty string expected in call through the RestTemplate [SPR-9832]

[spring-projects-issues]

Mattias Severson opened SPR-9832 and commented

The RestTemplate erroneously infers "0" for an empty string values.

1. Execute the attached test RestTemplateTest using the attached pom.xml file.
2. The test fails, the response from the RestTemplate is {"lhs":"1 U.S. dollar","rhs":"1 U.S. dollar","error":"0","icc":true}

For comparison, change the spring version to any 3.0.x version, or perform a request to http://www.google.com/ig/calculator?q=1USD=?EUR from a browser. The response is {lhs: "1 U.S. dollar",rhs: "0.774833411 Euros",error: "",icc: true}

---

Affects: 3.1 RC1, 3.1.2

Attachments:

• pom.xml (2.74 kB)
• RestTemplateTest.java (1.02 kB)

Issue Links:

• #14603 UriComponentsBuilder incorrectly disallows "=" as query parameter value ("is duplicated by")
• #14424 RestTemplate fails to correctly parse some HTTP URI parameters ("is duplicated by")
• #14109 UriComponentsBuilder.query(String) does not always match properly the query ("is duplicated by")

Referenced from: commits 0721146, 88906d2

[spring-projects-issues]

Rossen Stoyanchev commented

Repro project: https://github.com/SpringSource/spring-framework-issues/tree/master/SPR-9832

[spring-projects-issues]

Rossen Stoyanchev commented

From an brief look the response is of content type "text/html" and is processed with StringHttpMessageConverter, which simply turns the response body into a String. At the moment I fail to see what could be causing this issue.

Also I'm not sure why the subject is about json-lib or how this is connected to JSON deserialization?

[spring-projects-issues]

Rossen Stoyanchev commented

Mattias Severson, could you clarify the subject? I.e. how json-lib and JSON deserialization are related? The server response is of Content-Type "text/html" (`curl -v http://www.google.com/ig/calculator?q=1USD=?EUR') and is processed with StringHttpMessageConverter.

If you can confirm that I will update the title. Also have you found any other ways in which the issue can be reproduced or exhibits itself?

[spring-projects-issues]

Mattias Severson commented

Sorry about the late reply. I am currently on vacation and I don't have my laptop with me, so unfortunately I cannot do any investigation at this time. You are probably right that the title is misleading, feel free to update it.

[spring-projects-issues]

Mattias Severson commented

Rossen Stoyanchev, as you stated, the json-lib can be ruled out.

Additionally, it seems like the error was introduced in the 3.1.0.RELEASE (I have verified that the test passes using the 3.0.0.RELEASE and 3.0.7.RELEASE, but fails using the 3.1.0.RELEASE, 3.1.1.RELEASE and 3.1.2.RELEASE).

[spring-projects-issues]

Rossen Stoyanchev commented

I'm a bit stumped by this issue. What's odd is the server returns "text/html" content and the StringHttpMessageConverter simply turns the InputStream into a String without ever parsing it or trying to interpret it.

I created a project to try to replicate the behavior by creating a controller method that returns the same content as the Google API but I wasn't able to reproduce it. See RestTemplateTests and TestController.

[spring-projects-issues]

Rossen Stoyanchev commented

Root cause: thanks to Phil Webb for pointing out that the issue is with the way the query string is parsed. Given "q=1USD=?EUR", the value of "q" is "1USD=?EUR", which contains a reserved character "=". Ideally, for such cases, where reserved characters are expected, it is recommend to use a URI variable:

"http://www.google.com/ig/calculator?q={value}"

This ensures the URI can be parsed correctly and the value encoded so the resulting URI would be:

http://www.google.com/ig/calculator?q=1USD%3D?EUR

Spring 3.0 vs 3.1: in Spring 3.0, the UriTemplate encoded the query string as a whole without attempting to parse it, which meant that query parameters could not be fully encoded. Spring 3.1 introduced UriComponentsBuilder (also used inside UriTemplate), which does parse the query string and fully encodes query parameter names and values. However, when the query string contains reserved characters there is ambiguity in the parsing.

Fix: I've updated the regular expression used to parse query params to give more intuitive behavior in some common cases and to fix the regression. Below is the comment from commit 072114.

Improve regex for parsing query params

Previously UriComponentsBuilder used a regular expression for parsing
query name-value pairs where both name and value were expected to not
contain neither '&', not '='. The idea is that the presence of reserved
characters makes it impossible to guess correctly how to parse the
query string (e.g. a=b&c).

This change relaxes the constraint on query param values, allowing them
to contain '='. In effect '&' is the ultimate separator of name-value
pairs, and any '=' in values is ignored. For example "q=1USD=?EUR" is
interpreted as "q equals '1USD=?EUR'".