Support chunked transfer encoding

[jan-auer]

This PR introduces support for requests with chunked transfer encoding to WSGIRequestHandler by wrapping the input in a DechunkedInput. Chunks are unrolled in the following way

• Read a line to get the chunk's size in bytes
• Read the chunk data, but only to the length of the buffer
• Reduce the chunk size by the number of bytes read
• At the end of the chunk, skip the chunk's final newline

Once a chunk of size zero is encountered, a final newline is read, and then the input is marked as done. All further requests to readinto will exit early.

There is only a test for the success case. It needs to issue a manual HTTP to achieve a proper chunked multipart payload.

Note: For some reason, werkzeug crashes when parsing the response body if a Content-Length header is present. If you would like this fixed in this PR too, let me know. RFC 2616, Section 4.4 states that the Content-Length header must be ignored with any Transfer-Encoding other than identity.

Fixes #1149

[davidism]

Thanks for working on this!

[davidism]

There were some issues with our tests on Travis, you'll need to merge in master to fix the unrelated failing tests.

[mitsuhiko]

Generally looks good. Needs some fixes for the test failures and I will look into what uses the content length currently.

[davidism]

Hold off merging until I roll back the changes mentioned in #1149 (#1126), since they're related.

[jan-auer]

Pushed some updates:

• The DechunkedInput read only one chunk at a time and then immediately returned the buffer. Now it tries to fill the buffer and put as many chunks (or parts of it) as possible
• Also added some comments and a docstring to make the control flow more clear. It's still a bit weird on first sight, but someone familiar with the spec should understand it easily
• Fixed the bug where the Content-Length header crashed the form parser. The problem is that the content length header includes chunk headers and newlines, but the payload returned by DechunkedInput does not. The form parser, however, wraps the input in a LimitedStream if a Content-Length header is present, which asserts they are the same. By removing the content length, the form parser does not apply LimitedStream anymore and the problem goes away. Also, this is more spec compliant.
• Replaced six by an explicit import based on the python version. Could have put it in _compat like suggested, but since httplib/http.client is only used in the test, it's easer this way.
• Finally, the test had some unused code which is removed now.

[davidism]

Reverted #1126 in #1200, please merge master. Still working on unrelated test failures.

[jan-auer]

Alright, here you go. FWIW, I also ran some manual tests with uwsgi 2.0.15 and master in both python 2.7 and 3.6 and it seemed to work just fine.

Just one thing I'm still wondering about is whether we need to take care of infinite streams in DechunkedInput, or is something else already dealing with that?

[davidism]

This shouldn't affect other WSGI server support of chunked encoding, but thanks for testing it.

#1126 was supposed to handle infinite streams by setting a hard limit on the content read in, but that was based on some incorrect assumptions from me. I can't recall any special handling in Gunicorn from when I was looking a while ago, but might be good to check.

[davidism]

I merged master again and fixed some 2.6 and style issues. Travis should pass now.

[jan-auer]

Whoop, it's all green! 🎉

[davidism]

Would you mind looking at how this behaves with infinite streams (not chunked, but no content length) versus how Gunicorn behaves?

[jan-auer]

Sure 👍 Please let me know if you or @mitsuhiko have a perspective on this, though.

[mitsuhiko]

I will merge this in the current state now. I think we can look at the edge cases once we see how others are doing it. This is basically just the local dev setup anyways.