Flask 0.10.x and Werkzeug 0.9.x encoding issue

[maxcountryman]

This appears to be an issue with how Werkzeug is encoding headers, new to the version series 0.9.x:

>>> from flask import Flask
>>> app = Flask(__name__)
>>> app.test_client().get('/', headers=[('User-Agent', u'中文')])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/werkzeug/test.py", line 760, in get
    return self.open(*args, **kw)
  File "/Library/Python/2.7/site-packages/flask/testing.py", line 108, in open
    follow_redirects=follow_redirects)
  File "/Library/Python/2.7/site-packages/werkzeug/test.py", line 724, in open
    environ = args[0].get_environ()
  File "/Library/Python/2.7/site-packages/werkzeug/test.py", line 573, in get_environ
    for key, value in self.headers.to_wsgi_list():
  File "/Library/Python/2.7/site-packages/werkzeug/datastructures.py", line 1185, in to_wsgi_list
    return [(k, v.encode('latin1')) for k, v in self]
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 0-1: ordinal not in range(256)

This affects maxcountryman/flask-login#78.

[DasIch]

Is a non-latin1 User-Agent even allowed?

[maxcountryman]

@DasIch no idea, but this worked with Flask 0.9 and Werkzeug 0.8.x, fwiw.

[mitsuhiko]

HTTP does not allow non latin1 headers out of the box. We changed the default encoding to latin1 because that's what WSGI on Python 3 does as well. For certain headers custom behavior is applied. For instance setting cookies to utf-8 values is allowed by tunneling it through latin1.

The correct solution for non latin1 headers is this RFC: http://tools.ietf.org/html/rfc5987 - we might implement that at a later point.

[maxcountryman]

It would be nice if this bug were kept open given that this breaks functionality from werkzeug 0.8.x; at least until RFC 5987 is implemented.