boto3 and eventlet don't work together

[vanadium23]

I've some problem to make eventlet and boto3 work together. Purpose is to upload file to Amazon S3.
Here is a code example for debug purpose:

from eventlet import monkey_patch

monkey_patch()

from boto3.session import Session
from uuid import uuid4


AWS_S3_ACCESS_KEY_ID = '....'
AWS_S3_SECRET_ACCESS_KEY = '....'
AWS_DEFAULT_REGION = '....'
AWS_S3_BUCKET_NAME = '....'

session = Session(
    aws_access_key_id=AWS_S3_ACCESS_KEY_ID,
    aws_secret_access_key=AWS_S3_SECRET_ACCESS_KEY,
    region_name=AWS_DEFAULT_REGION)
s3 = session.resource('s3')
client = s3.Bucket(AWS_S3_BUCKET_NAME)

filedata = b'test\r\n'
filename = '{}.txt'.format(str(uuid4()))
client.Object(filename).put(Body=filedata)

On python 2.7.11 and eventlet 0.19.0 it works fine, but after switching to python 3.5.1 it become stuck on epoll_wait operation.

Here is strace for python 2.7.11:

epoll_ctl(4, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLPRI|EPOLLERR|EPOLLHUP, {u32=3, u64=13940307706339393539}}) = 0
epoll_wait(4, [{EPOLLIN, {u32=3, u64=13940307706339393539}}], 1023, 999) = 1
epoll_ctl(4, EPOLL_CTL_DEL, 3, 0x7fffd6401260) = 0
write(1, "Waited for 100 Continue response"..., 34Waited for 100 Continue response.
) = 34
read(3, "\27\3\3\0@", 5)                = 5
read(3, "C\371A^1y]q\3248m\17\220O\325\220_\224\342\275\211-x\tJ\370\216\331q\27\254\266"..., 64) = 64
write(3, "\27\3\3 0IAV\330\362\\B\270D}\\b\230\230\341\320\237\207\371\357\311^$\16\364}\34"..., 8245) = 8245

And for python 3.5.1:

epoll_ctl(5, EPOLL_CTL_ADD, 4, {EPOLLIN|EPOLLPRI|EPOLLERR|EPOLLHUP, {u32=4, u64=17460689526090366980}}) = 0
epoll_wait(5, [{EPOLLIN, {u32=4, u64=17460689526090366980}}], 1023, 1000) = 1
epoll_ctl(5, EPOLL_CTL_DEL, 4, 0x7ffe9fe8bdf0) = 0
write(1, "Waited for 100 Continue response"..., 34Waited for 100 Continue response.
) = 34
epoll_ctl(5, EPOLL_CTL_ADD, 4, {EPOLLIN|EPOLLPRI|EPOLLERR|EPOLLHUP, {u32=4, u64=17460689526090366980}}) = 0
epoll_wait(5, [{EPOLLIN, {u32=4, u64=17460689526090366980}}], 1023, 932) = 1
epoll_ctl(5, EPOLL_CTL_DEL, 4, 0x7ffe9fe8b410) = 0
read(4, "\27\3\3\0@", 5)                = 5
read(4, "\200\231T!\vM\327\213\363Vn\245\312#\350O\305\0\315\276*f\246f\204\301\236\v\24j\262\366"..., 64) = 64
epoll_ctl(5, EPOLL_CTL_ADD, 4, {EPOLLIN|EPOLLPRI|EPOLLERR|EPOLLHUP, {u32=4, u64=17460689526090366980}}) = 0
epoll_wait(5, [], 1023, 932)            = 0  <-- can be several lines like this 
epoll_wait(5, 

Currently I ran out of ideas what has break, but as far as I can see there is problem, that socket wait on empty events.

[temoto]

Thanks for reporting this.
Can you test on different Linux (kernel) version? On different Python3 (say 3.4)?

[vanadium23]

Tested against different python version: 3.4.4, 3.3.6 on kernel 4.5.1-1-ARCH - bug reproduced. Will test against different kernel versions later.

[vanadium23]

Tested with python 3.4.4 on ubuntu trusty (14.04.4 LTS) with kernel 3.13.0-86-generic. Same result :(

[vanadium23]

Update: error is something about 100-Expect header for S3 service near custom implementation in botocore: https://github.com/boto/botocore/blob/develop/botocore/awsrequest.py#L162

[acaos]

Please check the fix I have placed in #315 . Your issue appears to be similar to the one I encountered (SSL sessions through the requests module may hang). I verified that botocore does use the requests module.

You could try this immediately after your eventlet.monkey_patch() to see if it solves the issue:

if True:
    import eventlet.green.ssl

    def _green_ssl_recv_into (self, buffer, nbytes=None, flags=0):
        if self._sslobj:
            if flags != 0:
                raise ValueError(
                    "non-zero flags not allowed in calls to recv_into() on %s" %
                    self.__class__)
            if nbytes is None:
                if buffer:
                    nbytes = len(buffer)
                else:
                    nbytes = 1024
            read = self.read(nbytes, buffer)
            return read
        else:
            while True:
                try:
                    return eventlet.green.ssl.socket.recv_into(self, buffer, nbytes, flags)
                except eventlet.green.ssl.orig_socket.error as e:
                    if self.act_non_blocking:
                        raise
                    erno = eventlet.green.ssl.get_errno(e)
                    if erno in eventlet.green.ssl.greenio.SOCKET_BLOCKING:
                        try:
                            eventlet.green.ssl.trampoline(
                                self, read=True,
                                timeout=self.gettimeout(), timeout_exc=eventlet.green.ssl.timeout_exc('timed out'))
                        except eventlet.green.ssl.IOClosed:
                            return b''
                    elif erno in eventlet.green.ssl.greenio.SOCKET_CLOSED:
                        return b''
                    raise

    eventlet.green.ssl.GreenSSLSocket.recv_into = _green_ssl_recv_into

[temoto]

@vanadium23 is it possible to test without SSL? Some kind of reliable naked-to-ssl proxy between eventlet and target API. I could add ssl support to my meetup proxy gist if it would help.

[vanadium23]

@temoto I've tested without SSL and it worked. So the problem is in SSL hanging.

[temoto]

Fix from @acaos works. This issue is merged into #315

[m3talstorm]

@acaos's fix no longer works it seems

>>> import eventlet
>>> eventlet.monkey_patch()
>>> import eventlet.green.ssl
>>> def _green_ssl_recv_into (self, buffer, nbytes=None, flags=0):
...     if self._sslobj:
...         if flags != 0:
...             raise ValueError(
...                 "non-zero flags not allowed in calls to recv_into() on %s" %
...                 self.__class__)
...         if nbytes is None:
...             if buffer:
...                 nbytes = len(buffer)
...             else:
...                 nbytes = 1024
...         read = self.read(nbytes, buffer)
...         return read
...     else:
...         while True:
...             try:
...                 return eventlet.green.ssl.socket.recv_into(self, buffer, nbytes, flags)
...             except eventlet.green.ssl.orig_socket.error as e:
...                 if self.act_non_blocking:
...                     raise
...                 erno = eventlet.green.ssl.get_errno(e)
...                 if erno in eventlet.green.ssl.greenio.SOCKET_BLOCKING:
...                     try:
...                         eventlet.green.ssl.trampoline(
...                             self, read=True,
...                             timeout=self.gettimeout(), timeout_exc=eventlet.green.ssl.timeout_exc('timed out'))
...                     except eventlet.green.ssl.IOClosed:
...                         return b''
...                 elif erno in eventlet.green.ssl.greenio.SOCKET_CLOSED:
...                     return b''
...                 raise
...
>>> eventlet.green.ssl.GreenSSLSocket.recv_into = _green_ssl_recv_into
>>>
>>>
>>> import boto3
>>> s3 = boto3.resource('s3')
>>> s3.meta.client.upload_file('/tmp/somefile.png', 'somebucket', 'somefile.png')
<<< This will hang >>>

[twslankard]

I know this issue has been closed for a while, but this problem is still affecting me (as of eventlet 0.25.1). Does anyone know if there is a workaround?

[DanielMS93]

has there been any progress fixing this issue? it is a blocker for a current project i am working on