Allow to make FormHttpMessageConverter add the charset in the Content-Type header [SPR-16613]

[spring-projects-issues]

Mauro Molinari opened SPR-16613 and commented

When writing out a form that does not need multipart encoding, FormHttpMessageConverter does this in org.springframework.http.converter.FormHttpMessageConverter.writeForm(MultiValueMap<String, String>, MediaType, HttpOutputMessage): if the request headers do not specify a Content-Type (which is the case most of the times, when you use RestTemplate and the RequestEntity is built automatically from the supplied HTTP method and body object), application/x-www-form-urlencoded is added as the content type and the charset used for encoding parameter values is by default (if not specified otherwise) UTF-8.
So, the default behaviour of FormHttpMessageConverter is to encode parameters with UTF-8 and assume the server does the same (to decode them) when just application/x-www-form-urlencoded is specified as a content type.
However, there are cases in which the server may use another "implicit" encoding for this kind of data: Tomcat, for instance, uses ISO-8859-1 in its default configuration (or at least it did so until version 6 or 7, I've not checked what happens in more recent versions). Today I also encountered another web server, running PHP scripts, that assumes ISO-8859-1 if no encoding is specified in the Content-Type.

The solution to this problem is, either:

• set a different charset on the FormHttpMessageConverter, like ISO-8859-1 in the above example: this is however cumbersome (it requires to cycle through all the default registered HttpMessageConverters on the RestTemplate to find the correct one and change the encoding) and it also limits the characters that can be correctly encoded and transferred to the server (as an example: ISO-8859-1 is missing the € sign)
• otherwise, you have to find a way to make FormHttpMessageConverter explicitly set the charset in the Content-Type, like this: Content-Type: application/x-www-form-urlencoded;charset=UTF-8

IMHO, I can't think of a good reason for which FormHttpMessageConverter shouldn't do the latter by default (compatibility reasons with very old servers??). In my real-world scenario I see that I can make the server correctly use UTF-8 to decode my form data when I explicitly set the charset in the content type, overriding the default ISO-8859-1 that is used otherwise.

Extending FormHttpMessageConverter to do this is really a pain, because this class was not designed with extensibility in mind: there's not getter for the private charset field, many methods (like writeForm(...) and isMultipart(...)) are private, no getter for partConverters, etc.

---

Affects: 5.0.4

Referenced from: commits 2e4963f, 5861e96, 1897d8e

0 votes, 5 watchers

[spring-projects-issues]

Andy Wilkinson commented

The fix for this issue has necessitated a change in Spring REST Docs. It previously checked for form-encoded requests using the following:

MediaType.APPLICATION_FORM_URLENCODED.equals(result.getRequestHeaders().getContentType())

As of this change, the check returns false. I've updated it to the following:

MediaType.APPLICATION_FORM_URLENCODED.isCompatibleWith(result.getRequestHeaders().getContentType())

That's probably what it should have been all along, but I thought it worth noting here in case anyone else has the same problem.

[spring-projects-issues]

Mauro Molinari commented

Thank you Rossen for the fix!

[spring-projects-issues]

Rossen Stoyanchev commented

Since this is merely a convenience, I've removed 4.3.1.5 from the fix version, leaving it in 5.0.5 only. The likelihood of breaking existing tests, and potentially large number of them is high.