Headers are returned as unicode string from Response.from_file with Python 2.7.3 #99

Closed
cdman opened this Issue Mar 25, 2013 · 4 comments

Projects

None yet

3 participants

@cdman
cdman commented Mar 25, 2013

The situation in $subject makes it impossible to return the resulting object to the WSGI stack on GAE, since it expects all headers to be str (not unicode). I think the fix is to add something like the following around line 190 in webob/response.py:

        headerlist.append((
            native_(header_name, 'utf-8'),
            native_(value, 'utf-8')
        ))
@mcdonc
Member
mcdonc commented Apr 3, 2013

Definitely a bug on Python 2, thank you for the report. I'm not sure it shouldn't be native_(foo, 'latin-1') however. I dont know how this gets used, or what uses it, or what the expectations are for decoding headers with nonascii chars. Typically the HTTP spec waffles about it but makes some vague assertions that response header values should be encoded as latin-1, but I always have to go swap the entire spec into my head to remember.

@cdman
cdman commented Apr 6, 2013

Couple of more updates. My usecase is to be able to return a stored response from a WSGI app. A very simple test code I came up with is the following:

import io
import webapp2
from webob.compat import bytes_

STORED_RESPONSE = io.BytesIO(bytes_(str( webapp2.Response('test') )))

class MainApp(object):
    def __call__(self, environ, start_response):
        resp = webapp2.Response.from_file(STORED_RESPONSE)
        start_response(resp.status, resp.headerlist)
        return resp.app_iter

application = MainApp()

This errors out like this (traceback edited for my privacy :-)):

Traceback (most recent call last):
  File ".../google/appengine/runtime/wsgi.py", line 223, in Handle
    result = handler(dict(self._environ), self._StartResponse)
  File "/home/cdman/udacity/mini_gae/main.py", line 10, in __call__
    start_response(resp.status, resp.headerlist)
  File ".../google/appengine/runtime/wsgi.py", line 152, in _StartResponse
    (_GetTypeName(name), name))
InvalidResponseError: header names must be str, got 'unicode' (u'Content-Type')

As far as expected encoding it looks like you remember correctly and Latin-1 is expected: http://stackoverflow.com/a/4410331/1265

Also, as far usage of Response.from_file, I couldn't find any production code (aside from my own). All that came up were test code and I think that backward compatibility wouldn't be a big issue (especially given the fact 'foo' == u'foo' under Python). My searches:

@mcdonc
Member
mcdonc commented Aug 10, 2013

Marking this as "sprintable" for upcoming sprints.

@mcdonc
Member
mcdonc commented Aug 10, 2013

FTR, I believe the original fix mentioned (delta swapping out 'utf-8' for 'latin-1') is correct.

@deuxpi deuxpi added a commit to deuxpi/webob that referenced this issue May 28, 2014
@deuxpi deuxpi Native `str` strings from Response.from_file
Make sure that header names extracted with Response.from_file are `str`
instances otherwise most WSGI implementations will reject the response.

Should fix issue Pylons#99
6770d4a
@deuxpi deuxpi added a commit to deuxpi/webob that referenced this issue May 28, 2014
@deuxpi deuxpi Native `str` strings from Response.from_file
Make sure that header names extracted with Response.from_file are `str`
instances otherwise most WSGI implementations will reject the response.

Should fix issue Pylons#99
ee70769
@deuxpi deuxpi added a commit to deuxpi/webob that referenced this issue May 28, 2014
@deuxpi deuxpi Native `str` strings from Response.from_file
Make sure that header names extracted with Response.from_file are `str`
instances otherwise most WSGI implementations will reject the response.

Should fix issue Pylons#99
904788b
@bertjwregeer bertjwregeer added this to the Version 1.5 milestone Mar 23, 2015
@bertjwregeer bertjwregeer added a commit that closed this issue Apr 4, 2015
@bertjwregeer bertjwregeer Merge remote-tracking branch 'origin/pr/150'
This makes sure that the header extracted with Response.from_file are
latin1 encoded str instances.

Fixes #99
1536942
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment