add utf-8 charset parameter to the content-type header for 'cf login'

[boyang9527]

cf login send request to login server without charset parameter in content-type header, the login server can not determine the charset for correct decoding. We need Content-Type: application/x-www-form-urlencoded; charset=utf-8 in the following request

REQUEST: [2016-08-29T11:07:38-07:00]
POST /UAALoginServerWAR/oauth/token HTTP/1.1
Host: login.ng.bluemix.net
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/x-www-form-urlencoded
User-Agent: go-cli 6.18.0+b22884b / darwin

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/129884299

The labels on this github issue will be updated when the story is started.

[dkoper]

Hi Bo Yang,

According to the HTML5 RFC the charset parameter is not supported there. The request body only uses us-ascii characters after all.
The login server should be configured to assume utf-8.

Regards,
Dies Koper
CF CLI PM

[msmolny]

Hi Dies,
the link you reference only describes how the enctype attribute of an HTML5 form element should be handled. You are right, HTML5 does not allow to prescribe any encoding in the HTML5 form. This does simply mean that the client has to specify the correct, client-chosen charset.

This issue is about the "Content-Type" HTTP header. And here, the charset parameter is supported. Please refer to: https://www.ietf.org/rfc/rfc2068.txt

Here, it is clearly stated:

When no explicit charset parameter is provided by the sender, media subtypes of the "text" type are defined to have a default charset value of "ISO-8859-1" when received via HTTP. Data in character sets other than "ISO-8859-1" or its subsets MUST be labeled with an appropriate charset value.

So for any HTTP 1.1 request received without an enctype in the content type, the server must assume ISO-8859-1.

[dkoper]

Hi @msmolny

The section you quoted refers to media subtypes of the "text" type, e.g. "text/xml". This media subtype supports a "charset" parameter, and this clause is saying that *for this media subtype, "charset" must be included if the data is not ISO-8859-1 compatbile. (This rule has been slightly redefined in rfc6657, but not relevant to this discussion).

Comparing the registered media types application/x-www-form-urlencoded and a "text" type, e.g. text/html, there seems to be no ambiguity on which allows a "charset" parameter and which does not.

What is your issue exactly? What character set is your login server falling back to now (if any) and why?

[dkoper]

While investigating this issue, I found a number of (non-authoritive) pages, most suggesting to assume utf-8:

• http://lucumr.pocoo.org/2013/7/19/application-mimetypes-and-encodings/
• https://tools.ietf.org/html/draft-hoehrmann-urlencoded-01#page-4 <- expired memo
• http://htmlpurifier.org/docs/enduser-utf8.html#whyutf8-forms-urlencoded
• http://stackoverflow.com/questions/16819502/application-x-www-form-urlencoded-and-charset-utf-8

[msmolny]

Hi @dkoper ,

What is your issue exactly? What character set is your login server falling back to now (if any) and why?

The Login Server is Java based, so all character encoding and decoding happens through the Web Container. Our code just accesses the parameters and get them decoded. In our case, this is a WebSphere Liberty server. And it looks like the WebSphere Liberty Web Container decided to use ISO-8859-1 if no charset is set to the content type http header. When manually adding that charset, the french character è is decoded correctly, without that charset, this is decoded incorrectly.

I can try to do a test with Apache Tomcat which is used by UAA just to make sure that this is not a WebSphere Liberty specific behavior, but also applies to other Web Containers. And if Apache Tomcat behaves like WebSphere Liberty, this would explain maybe even this issue which magically seem to be disappeared? https://www.pivotaltracker.com/n/projects/997278/stories/114531721

[msmolny]

I just made a quick test with a simple JSP:

<%@ page contentType="text/html"%>

<%@ taglib uri="http://java.sun.com/jsp/jstl/core" prefix="c" %>
<html>
  <body>
    Parameter echo content: <c:out value="${param.echo}"/>
  </body>
</html>

Calling WebSphere Liberty with
curl -H "Content-Type: application/x-www-form-urlencoded" -d "echo=Hall%C3%A9" "http://127.0.0.1/test/index.jsp"
and
curl -H "Content-Type: application/x-www-form-urlencoded; charset=ISO-8859-1" -d "echo=Hall%C3%A9" "http://127.0.0.1/test/index.jsp"
gives me back the identical response.

Sending
curl -H "Content-Type: application/x-www-form-urlencoded; charset=UTF-8" -d "echo=Hall%C3%A9" "http://127.0.0.1/test/index.jsp"
returns a different response.

=> Default WebSphere Liberty Web Container behavior for application/x-www-form-urlencoded without a charset specified: ISO-8859-1

I just rerun the same tests with Apache Tomcat 8.5, and it behaves identical. So I think it is at least very common for Java based Web Containers to use ISO-8859-1 for application/x-www-form-urlencoded if no charset is specified.

[dkoper]

True, the servers seem lenient to non-complying clients. But they also provide options to deal with compliant clients:
http://wiki.apache.org/tomcat/FAQ/CharacterEncoding#Q3
http://balusc.omnifaces.org/2009/05/unicode-how-to-get-characters-right.html#JSPServletRequest
http://fazlansabar.blogspot.com.au/2012/06/how-to-enable-utf-8-support-on-tomcat.html
http://stackoverflow.com/a/2419142/1227328

[msmolny]

@dkoper

http://fazlansabar.blogspot.com.au/2012/06/how-to-enable-utf-8-support-on-tomcat.html

Update Tomcat's server.xml
...
IMPORTANT NOTE: POST requests will have NO effect by this change.
...

The suggested way with a filter is not switching a default, but overwriting the encoding always. This is a last resort solution.

http://wiki.apache.org/tomcat/FAQ/CharacterEncoding#Q3

POST requests should specify the encoding of the parameters and values they send. Since many clients fail to set an explicit encoding, the default is used (ISO-8859-1). In many cases this is not the preferred interpretation so one can employ a javax.servlet.Filter to set request encodings.

Thanks for this FAQ. This FAQ confirms this issue. Clients SHOULD specify the encoding, and default is ISO-8859-1. That's basically what I say. Having a filter is there to fix the behavior in case you cannot change the client.

And this is exactly what I want to do with this issue. Please specify an encoding in the request header in CF CLI, the client.

[fhanik]

The challenge here is that the MIME type application/x-www-form-urlencoded does not support a charset parameter according to spec. So most user agent's don't send. A browser does not.

We have confirmed that the CLI is working properly within cf-release.

cf auth 琳贺 koala
API endpoint: https://api.identity.cf-app.com 
Authenticating...
OK
Use 'cf target' to view or set your target org and space

In the server side, the server has to perform that translation. If the cf cli added the charset parameter, it would essentially be non standard. And there is no guarantee that the server understands it, a server may not look for the charset parameter for this content type.

In the UAA, we simply add a filter that does the character encoding conversion
https://github.com/cloudfoundry/uaa/blob/develop/server/src/main/java/org/cloudfoundry/identity/uaa/authentication/UTF8ConversionFilter.java#L37-L37

Reference: Encoding algorithm - https://www.w3.org/TR/html5/forms.html#application/x-www-form-urlencoded-encoding-algorithm

[dkoper]

@fhanik Thanks for looking up what UAA does!

Closing, we don't want to introduce non-standard behaviour in the cf CLI, especially when there is a recommended and working solution to address this on the server side.