New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

boto3 and eventlet don't work together #313

Closed
vanadium23 opened this Issue May 15, 2016 · 8 comments

Comments

Projects
None yet
3 participants
@vanadium23

vanadium23 commented May 15, 2016

I've some problem to make eventlet and boto3 work together. Purpose is to upload file to Amazon S3.
Here is a code example for debug purpose:

from eventlet import monkey_patch

monkey_patch()

from boto3.session import Session
from uuid import uuid4


AWS_S3_ACCESS_KEY_ID = '....'
AWS_S3_SECRET_ACCESS_KEY = '....'
AWS_DEFAULT_REGION = '....'
AWS_S3_BUCKET_NAME = '....'

session = Session(
    aws_access_key_id=AWS_S3_ACCESS_KEY_ID,
    aws_secret_access_key=AWS_S3_SECRET_ACCESS_KEY,
    region_name=AWS_DEFAULT_REGION)
s3 = session.resource('s3')
client = s3.Bucket(AWS_S3_BUCKET_NAME)

filedata = b'test\r\n'
filename = '{}.txt'.format(str(uuid4()))
client.Object(filename).put(Body=filedata)

On python 2.7.11 and eventlet 0.19.0 it works fine, but after switching to python 3.5.1 it become stuck on epoll_wait operation.

Here is strace for python 2.7.11:

epoll_ctl(4, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLPRI|EPOLLERR|EPOLLHUP, {u32=3, u64=13940307706339393539}}) = 0
epoll_wait(4, [{EPOLLIN, {u32=3, u64=13940307706339393539}}], 1023, 999) = 1
epoll_ctl(4, EPOLL_CTL_DEL, 3, 0x7fffd6401260) = 0
write(1, "Waited for 100 Continue response"..., 34Waited for 100 Continue response.
) = 34
read(3, "\27\3\3\0@", 5)                = 5
read(3, "C\371A^1y]q\3248m\17\220O\325\220_\224\342\275\211-x\tJ\370\216\331q\27\254\266"..., 64) = 64
write(3, "\27\3\3 0IAV\330\362\\B\270D}\\b\230\230\341\320\237\207\371\357\311^$\16\364}\34"..., 8245) = 8245

And for python 3.5.1:

epoll_ctl(5, EPOLL_CTL_ADD, 4, {EPOLLIN|EPOLLPRI|EPOLLERR|EPOLLHUP, {u32=4, u64=17460689526090366980}}) = 0
epoll_wait(5, [{EPOLLIN, {u32=4, u64=17460689526090366980}}], 1023, 1000) = 1
epoll_ctl(5, EPOLL_CTL_DEL, 4, 0x7ffe9fe8bdf0) = 0
write(1, "Waited for 100 Continue response"..., 34Waited for 100 Continue response.
) = 34
epoll_ctl(5, EPOLL_CTL_ADD, 4, {EPOLLIN|EPOLLPRI|EPOLLERR|EPOLLHUP, {u32=4, u64=17460689526090366980}}) = 0
epoll_wait(5, [{EPOLLIN, {u32=4, u64=17460689526090366980}}], 1023, 932) = 1
epoll_ctl(5, EPOLL_CTL_DEL, 4, 0x7ffe9fe8b410) = 0
read(4, "\27\3\3\0@", 5)                = 5
read(4, "\200\231T!\vM\327\213\363Vn\245\312#\350O\305\0\315\276*f\246f\204\301\236\v\24j\262\366"..., 64) = 64
epoll_ctl(5, EPOLL_CTL_ADD, 4, {EPOLLIN|EPOLLPRI|EPOLLERR|EPOLLHUP, {u32=4, u64=17460689526090366980}}) = 0
epoll_wait(5, [], 1023, 932)            = 0  <-- can be several lines like this 
epoll_wait(5, 

Currently I ran out of ideas what has break, but as far as I can see there is problem, that socket wait on empty events.

@temoto

This comment has been minimized.

Show comment
Hide comment
@temoto

temoto May 15, 2016

Member

Thanks for reporting this.
Can you test on different Linux (kernel) version? On different Python3 (say 3.4)?

Member

temoto commented May 15, 2016

Thanks for reporting this.
Can you test on different Linux (kernel) version? On different Python3 (say 3.4)?

@vanadium23

This comment has been minimized.

Show comment
Hide comment
@vanadium23

vanadium23 May 16, 2016

Tested against different python version: 3.4.4, 3.3.6 on kernel 4.5.1-1-ARCH - bug reproduced. Will test against different kernel versions later.

vanadium23 commented May 16, 2016

Tested against different python version: 3.4.4, 3.3.6 on kernel 4.5.1-1-ARCH - bug reproduced. Will test against different kernel versions later.

@vanadium23

This comment has been minimized.

Show comment
Hide comment
@vanadium23

vanadium23 May 16, 2016

Tested with python 3.4.4 on ubuntu trusty (14.04.4 LTS) with kernel 3.13.0-86-generic. Same result :(

vanadium23 commented May 16, 2016

Tested with python 3.4.4 on ubuntu trusty (14.04.4 LTS) with kernel 3.13.0-86-generic. Same result :(

@vanadium23

This comment has been minimized.

Show comment
Hide comment
@vanadium23

vanadium23 May 16, 2016

Update: error is something about 100-Expect header for S3 service near custom implementation in botocore: https://github.com/boto/botocore/blob/develop/botocore/awsrequest.py#L162

vanadium23 commented May 16, 2016

Update: error is something about 100-Expect header for S3 service near custom implementation in botocore: https://github.com/boto/botocore/blob/develop/botocore/awsrequest.py#L162

@acaos

This comment has been minimized.

Show comment
Hide comment
@acaos

acaos May 17, 2016

Please check the fix I have placed in #315 . Your issue appears to be similar to the one I encountered (SSL sessions through the requests module may hang). I verified that botocore does use the requests module.

You could try this immediately after your eventlet.monkey_patch() to see if it solves the issue:

if True:
    import eventlet.green.ssl

    def _green_ssl_recv_into (self, buffer, nbytes=None, flags=0):
        if self._sslobj:
            if flags != 0:
                raise ValueError(
                    "non-zero flags not allowed in calls to recv_into() on %s" %
                    self.__class__)
            if nbytes is None:
                if buffer:
                    nbytes = len(buffer)
                else:
                    nbytes = 1024
            read = self.read(nbytes, buffer)
            return read
        else:
            while True:
                try:
                    return eventlet.green.ssl.socket.recv_into(self, buffer, nbytes, flags)
                except eventlet.green.ssl.orig_socket.error as e:
                    if self.act_non_blocking:
                        raise
                    erno = eventlet.green.ssl.get_errno(e)
                    if erno in eventlet.green.ssl.greenio.SOCKET_BLOCKING:
                        try:
                            eventlet.green.ssl.trampoline(
                                self, read=True,
                                timeout=self.gettimeout(), timeout_exc=eventlet.green.ssl.timeout_exc('timed out'))
                        except eventlet.green.ssl.IOClosed:
                            return b''
                    elif erno in eventlet.green.ssl.greenio.SOCKET_CLOSED:
                        return b''
                    raise

    eventlet.green.ssl.GreenSSLSocket.recv_into = _green_ssl_recv_into

acaos commented May 17, 2016

Please check the fix I have placed in #315 . Your issue appears to be similar to the one I encountered (SSL sessions through the requests module may hang). I verified that botocore does use the requests module.

You could try this immediately after your eventlet.monkey_patch() to see if it solves the issue:

if True:
    import eventlet.green.ssl

    def _green_ssl_recv_into (self, buffer, nbytes=None, flags=0):
        if self._sslobj:
            if flags != 0:
                raise ValueError(
                    "non-zero flags not allowed in calls to recv_into() on %s" %
                    self.__class__)
            if nbytes is None:
                if buffer:
                    nbytes = len(buffer)
                else:
                    nbytes = 1024
            read = self.read(nbytes, buffer)
            return read
        else:
            while True:
                try:
                    return eventlet.green.ssl.socket.recv_into(self, buffer, nbytes, flags)
                except eventlet.green.ssl.orig_socket.error as e:
                    if self.act_non_blocking:
                        raise
                    erno = eventlet.green.ssl.get_errno(e)
                    if erno in eventlet.green.ssl.greenio.SOCKET_BLOCKING:
                        try:
                            eventlet.green.ssl.trampoline(
                                self, read=True,
                                timeout=self.gettimeout(), timeout_exc=eventlet.green.ssl.timeout_exc('timed out'))
                        except eventlet.green.ssl.IOClosed:
                            return b''
                    elif erno in eventlet.green.ssl.greenio.SOCKET_CLOSED:
                        return b''
                    raise

    eventlet.green.ssl.GreenSSLSocket.recv_into = _green_ssl_recv_into
@temoto

This comment has been minimized.

Show comment
Hide comment
@temoto

temoto May 19, 2016

Member

@vanadium23 is it possible to test without SSL? Some kind of reliable naked-to-ssl proxy between eventlet and target API. I could add ssl support to my meetup proxy gist if it would help.

Member

temoto commented May 19, 2016

@vanadium23 is it possible to test without SSL? Some kind of reliable naked-to-ssl proxy between eventlet and target API. I could add ssl support to my meetup proxy gist if it would help.

@vanadium23

This comment has been minimized.

Show comment
Hide comment
@vanadium23

vanadium23 May 23, 2016

@temoto I've tested without SSL and it worked. So the problem is in SSL hanging.

vanadium23 commented May 23, 2016

@temoto I've tested without SSL and it worked. So the problem is in SSL hanging.

@temoto

This comment has been minimized.

Show comment
Hide comment
@temoto

temoto May 23, 2016

Member

Fix from @acaos works. This issue is merged into #315

Member

temoto commented May 23, 2016

Fix from @acaos works. This issue is merged into #315

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment