Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Backwards Compatibility issue: remove chunked encoding support from WebOb (unless explicitly flagged by WSGI server) #279
The problem with loosely defined specs
Unfortunately the WSGI spec does not have a good way to allow Chunked Encoding if you read the spec loosely. This is an issue because to support it you need to read all of the "should"'s as "must".
That would have been the correct thing to do anyway, but alas we are stuck with what we've got for now.
For example, to showcase the issue:
Will cause the server to hang forever as long as the client keeps the connection open.
This means when you call
Good servers like waitress don't do this, and we can always call
Choice number One (arguably the most correct choice)
Currently WebOb tries to be too smart for it's own good and that gets it into trouble...
I'd like to turn back the clock, remove any and all support for Chunked Encoding unless there is a flag in the environment that states it is supported. #278 is one such proposal.
Choice number Two
We completely ignore
I don't know of a good way of verifying if we are running under
Chunked encoding is apparently being used more and more by mobile clients, versus your run off the mill browser, but I imagine it has other uses too. While I am loathe to remove support for something, I do believe it should be better implemented. I'd like to allow a request body on all request methods (spec says it's allowed, just doesn't have any defined semantics) but I don't want to break WebOb on servers that don't implement PEP-3333 fully.
I'd love feedback on this, and ideas on how to move this forward.
What I would like to do, to give control back to the user ultimately, is ship a piece of middleware that can easily be loaded into the users WSGI pipeline that tells WebOb it is safe to consume
This way even if there is no way to check that a server supports
ISTM that https://gist.github.com/mitsuhiko/5721547 is already implemented in werkzeug and the only issue is that no WSGI servers (I checked werkzeug, waitress, wsgiref, gevent, gunicorn, uwsgi) actually define
So if I understand this correctly, if you update webob to depend on
Yes. Unfortunately due to PEP3333 not properly defining requirements the Content-Length is the only way to know that a request has a body. The only time a client doesn't send a Content-Length is if the request is chunked, in the case of a HTTP -> WSGI the HTTP server is expected to buffer the full chunked request and then pass the WSGI application the full buffered request and set the Content-Length into the environ.
Graham Dumpleton on Twitter suggested creating a wsgi.org proposal instead of just an ad-hoc
My other proposal is to add a middleware to WebOb that sets
To me, it seems like adding support for
Yeah, Graham said something similar about web-sig. I was going to just submit it, let others fight about it. Shipping code wins.
As a reminder to myself, need to make sure I support what Graham has in mod_wsgi as well, there is a flag that is set if the user turned on Chunked support.
This request will send empty body (Content-Length: 0)
And according to this note in pep 3333
this line should return immediately:
It it hangs, then it is just a little but in wsgiref
If client is using Chunked Encoding, there should be header "Transfer-Encoding: chunked" in request, and .read() will block until client send a chunk. What exactly wrong with it?
Also, the WSGI spec does not allow for Hop-by-Hop headers, which is what
Sorry, this feature has been removed, and I am not currently interested in re-visiting it, especially with all of the information that is available above!