Support for "Expects" http header

[edevil]

It would be nice to have support for the Expect -> 100 Continue flow. This is specially important when we are doing uploads, since we will not need to transfer the whole data before we encounter a 401, for example.

A typical use-case would be a streaming upload, with the data coming from a client and proxied on-the-fly to our destination server. Requests would read the data from the input source socket only if the destination server had already sent the 100-Continue header. When we are dealing with S3, 401 errors are common, and if the data is not read we can retry the request.

Requests can check the request headers, and if it finds an "Expect" header it will wait for the 100 Continue response from the server. If it does not come, this error should flow to the caller in a way that it can distinguish a network problem from a failed expectation error.

[kennethreitz]

This would be wonderful.

I believe @durin42 has some thoughts about this as well.

[durin42]

nod We can do this, but it'll be easiest with a rewritten http client library. I've got one that's mostly API compatible with httplib (http://code.google.com/p/httpplus/) that I'm writing as part of my 20% work on Mercurial that we could try.

Note that the flow proposed in the initial bug isn't strictly RFC compliant - you can't wait indefinitely for the response to the Expect: header. You have to wait an unspecified delay and then continue optimistically, as servers SHOULD (not MUST) understand Expect: headers. Reverse proxies are allowed to strip Expect headers entirely, and many do so.

@kennethreitz I don't have any particular time to dedicate to this, but I'm happy to chat more about what we can do to make this better. Are you going to be at OSCON?

[sigmavirus24]

Since we now have streaming uploads (if I remember correctly) is anyone interested in taking care of this?

[durin42]

@kennethreitz do you have support for streaming uploads? I've not followed development of requests closely.

[kennethreitz]

@durdin42 yessir! http://docs.python-requests.org/en/latest/user/advanced/#streaming-uploads

[Lukasa]

I haven't really thought about this for any length of time, but is this a Requests-level issue or a urllib3-level issue? Put another way, does it belong in the HTTP Adapter or in urllib3?

[kennethreitz]

I think we can handle it on our end.

[durin42]

Having looked at this a little this morning, it'll be very challenging to implement fully proper expect header handling, because some servers assume that if you send an expect header you can handle an early response, and can close the socket early in a way that breaks httplib.

That said, you should be able to fix this by waiting ~500ms after you send headers and peeking for a response, and if you have a non-100-continue response pretend to the httplib guts that the request body was completely sent.

[durin42]

In particular, if you tried to read from the low_conn.sock() here with a timeout, I think that might be the right jumping-off point:

https://github.com/kennethreitz/requests/blob/master/requests/adapters.py#L185

[sigmavirus24]

Ah time.sleep the cause of and solution to, all of life's problems.

We could do:

if 'expects' in request.headers:
        time.sleep(0.5)

So we don't degrade performance for other cases, correct?

[edevil]

Any progress on this?

[Lukasa]

@edevil Nope. Feel free to pick it up if you want something to hack on. =)

[aseemk]

I don't have the Python/requests/httplib proficiency to do this, but as a user and big fan of HTTPie, I'd love to see this implemented. =) Thanks for the consideration, and good luck with the implementation!

[sigmavirus24]

So I'm starting to wonder if this is entirely an adapter-level change. I hadn't thought much about this, but I think it would be easiest and make the most sense to be on the adapter level. My reasoning why:

• The adapter handles the connection details
• The adapter returns a response object not a partial response object
• The session expects a response and so the fact that we'd have to handle a non-100-continue response should be handled before we return a response to the session.

[sridevikoushik31]

Any update on this?
This would be very useful for our use case (Uploading large objects to openstack swift.)
We're awaiting this feature.

[Lukasa]

There's been no progress on this, and it's not high on the list of priorities for any of the core development team. This is only likely to happen any time soon if someone else develops it. =)

[thegeekchick]

Are there any updates on this bit? We're facing an issue and 100-continue can help make our lives much better.

[Lukasa]

See my above comment.

[sigmavirus24]

For those interested, this is probably not possible with our current stack. We talked with @durin42 tonight and it seems that httplib just swallows the 100-Continue and does not really wait for it either. This may be far more difficult than originally thought.

[Lukasa]

To continue keeping this issue up to date, the HTTPbis WG is considering deprecating the entire 1XX series of status codes as an erratum to RFC 723[0-5]. It's not entirely clear that this will happen, but what is clear is that the prevailing opinion in the WG is that 100 Continue is bad.

The most recent relevant WG thread is here. Note that the discussion is heavily informed by HTTP/2, which has no need of the 100 Continue mechanism. It's not clear exactly where this will go, but it puts the 1XX codes at risk.

[Lukasa]

I think this issue can now be characterised as "impossible without massive rewrite of httplib". For that reason, I want to close this down: having issues we cannot take action on is problematic.

[edevil]

If anyone needs a feature like this I ended up going with a patched httplib:

http://bugs.python.org/issue1346874

[cancan101]

@edevil Id your patched httplib available anywhere for as a package?

BTW, Amazon S3 recommends using 100-continues: http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTRedirect.html

[edevil]

@cancan101 Sorry, no. But you have the patch in the ticket: http://bugs.python.org/file26357/issue1346874-273.patch

[stuart-mclaren]

This is how this is supported in the AWS s3 python cli:

https://github.com/boto/botocore/blob/develop/botocore/awsrequest.py#L57

This would be useful for OpenStack Swift for the same reasons it's useful for s3: you get a 401 back before you've uploaded any data. This prevents wasted (potentially large) uploads and allows you to retry if your input is a stream.

[Lukasa]

We're desperately trying to avoid further patching httplib. I highly recommend trying to make this enhancement upstream if OpenStack values it: that would make it much easier for us to backport that support to earlier versions.

[stuart-mclaren]

upstream: ie httplib?

[Lukasa]

Correct. =)

[stuart-mclaren]

Right, it would definitely be nice and clean to have it in there.

The patch to httplib referenced above (http://bugs.python.org/issue1346874) has been up since 2005!

Have the requests folks had any luck in fixing functionality gaps by pushing stuff upstream?

[Lukasa]

Some, but it's not been as fast as we'd like in all cases. However, our only other option is to jettison httplib entirely, which is a bit excessive and puts a substantial maintenance burden on us. Where possible, working with upstream is the ideal thing to do.

[redixin]

There is a code which implements chunked sending [0]

Why don't we extend it to support 100-continue?

[0] https://github.com/kennethreitz/requests/blob/v2.11.1/requests/adapters.py#L434-L448

[sigmavirus24]

@redixin httplib never exposes a 100-continue to us. It's not a fully-fledged response per the RFC so we never see it. We can't extend code to support something if we never see it.

[redixin]

@sigmavirus24 what do you mean "never exposes"? We are sending data directly to socket, and then reading response [0]. We can read and write anything we want. This may be very simple patch, just few more lines and tests.

[0] https://github.com/kennethreitz/requests/blob/v2.11.1/requests/adapters.py#L434-L465

[sigmavirus24]

what do you mean "never exposes"?

httplib literally sees the 100 Continue come from the server and trashes it without letting its users see it.

We are sending data directly to socket

I can assure you that lowconn is not a socket. Requests does not ever touch a socket object. lowconn is an HTTPConnection from urllib3 which wraps an httplib.HTTPConnection which encapsulates a socket object eventually.

We can read and write anything we want. This may be very simple patch, just few more lines and tests.

I assure you that we can not.

Further, that particular section of code you are referring to is only conditionally used when the body is an iterator and we cannot determine the total length of the body.

[Lukasa]

We can read and write anything we want. This may be very simple patch, just few more lines and tests.

@redixin If you believe that to be true, show us the code. Write that patch, write those tests, demonstrate that it works. But currently anyone else who would write that code believes that it is a much more substantial patch than that, and is unwilling to contribute such a patch.

[jallirs]

Are changes to the python 2.7.x being entertained? I'm not doubting it needs to be done as a change in httplib, but it may be difficult as I understand. Is there a bug filed in python upstream for this?

[jallirs]

nvm, found it, it's open! as referenced above, since 2005.

https://bugs.python.org/issue1346874

[sigmavirus24]

@jallirs to be clear, Python 2.7 will likely never see this and the earliest version of Python to see this would be in the 3.x series (3.7, 3.8, etc.) It's not something that would likely ever be backported.