Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python 3.7 - wrap_socket() got an unexpected keyword argument '_context' #526

Open
Nexodaru opened this issue Sep 29, 2018 · 20 comments

Comments

@Nexodaru
Copy link

commented Sep 29, 2018

I develop an app with flask-socket.io and because of the new merge of #506 i thought i can upgrade the python version to 3.7. I used eventlet.monkey_patch. Not sure if it is an issue of eventlet, but on connection i get following Trace:

message handler error
Traceback (most recent call last):
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/engineio/server.py", line 411, in _trigger_event
    return self.handlers[event](*args)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/socketio/server.py", line 518, in _handle_eio_message
    self._handle_connect(sid, pkt.namespace)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/socketio/server.py", line 421, in _handle_connect
    self.environ[sid]) is False:
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/socketio/server.py", line 495, in _trigger_event
    event, *args)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/flask_socketio/namespace.py", line 26, in trigger_event
    *args)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/flask_socketio/__init__.py", line 632, in _handle_event
    ret = handler()
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/server/auth/__init__.py", line 10, in wrapped
    if not current_user.is_authenticated:
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/werkzeug/local.py", line 347, in __getattr__
    return getattr(self._get_current_object(), name)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/werkzeug/local.py", line 306, in _get_current_object
    return self.__local()
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/flask_login/utils.py", line 26, in <lambda>
    current_user = LocalProxy(lambda: _get_user())
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/flask_login/utils.py", line 335, in _get_user
    current_app.login_manager._load_user()
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/flask_login/login_manager.py", line 355, in _load_user
    return self._load_from_request(request)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/flask_login/login_manager.py", line 412, in _load_from_request
    user = self.request_callback(request)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/geocatch_persistence/login_manager.py", line 35, in load_user_from_request
    return user or validate_token(token)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/geocatch_persistence/login_manager.py", line 69, in validate_token
    current_app.config.get("GOOGLE_CLIENT_ID"))  # type: dict
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/google/oauth2/id_token.py", line 141, in verify_oauth2_token
    certs_url=_GOOGLE_OAUTH2_CERTS_URL)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/google/oauth2/id_token.py", line 120, in verify_token
    certs = _fetch_certs(request, certs_url)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/google/oauth2/id_token.py", line 94, in _fetch_certs
    response = request(certs_url, method='GET')
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/google/auth/transport/requests.py", line 120, in __call__
    **kwargs)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 849, in _validate_conn
    conn.connect()
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/urllib3/connection.py", line 356, in connect
    ssl_context=context)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 359, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/eventlet/green/ssl.py", line 426, in wrap_socket
    return GreenSSLSocket(sock, *a, _context=self, **kw)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/eventlet/green/ssl.py", line 79, in __init__
    *args, **kw
TypeError: wrap_socket() got an unexpected keyword argument '_context'

@nat-goodspeed

This comment has been minimized.

Copy link
Contributor

commented Oct 2, 2018

@olii

This comment has been minimized.

Copy link

commented Oct 11, 2018

Minimal example

with current eventlet master branch (cf47cb5) and Python interpreter version:

Python 3.7.0 (default, Jul 15 2018, 10:44:58) 
[GCC 8.1.1 20180531] on linux
Example # 1
import eventlet
eventlet.monkey_patch()

import socket
import ssl

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
wrappedSocket = ssl.wrap_socket(sock)
Example # 2 (with requests library from master branch):
import eventlet
eventlet.monkey_patch()

import requests
print(requests.get('https://www.google.com/').status_code)

Both examples fail with the exception: TypeError: wrap_socket() got an unexpected keyword argument '_context'

There were some breaking changes in Python 3.6 -> 3.7 in ssl module.
3.6 -> https://github.com/python/cpython/blob/3.6/Lib/ssl.py
3.7 -> https://github.com/python/cpython/blob/3.7/Lib/ssl.py

@Dormouse759

This comment has been minimized.

Copy link
Contributor

commented Oct 11, 2018

I am aware of this. Unfortunately, I currently don't have the capacity to pay attention to this issue.
It could take some week or two until I become.

Let me at least explain some background on the issue:
SSL changed the API, the socket doesn't have public constructor and cannot be used by subclasses anymore.
This was worked around by #506 and the fix is not perfect yet.
This seems like argument handling failure at https://github.com/eventlet/eventlet/pull/506/files#diff-72b0a51127e4186c413957d1d5a59798R70

@luanjunyi

This comment has been minimized.

Copy link
Contributor

commented Dec 9, 2018

I've a fix: #531. @Dormouse759 PTAL

@nat-goodspeed

This comment has been minimized.

Copy link
Contributor

commented Mar 4, 2019

I'm closing this on the assumption that luanjunyi's fix works. Please reopen if it doesn't address the problem.

@MertcanGokgoz

This comment has been minimized.

Copy link

commented Apr 12, 2019

Fix Doesnt work on Py 3.7.3

[2019-04-12 21:25:29,814: INFO/MainProcess] mingle: searching for neighbors
[2019-04-12 21:25:30,831: INFO/MainProcess] mingle: all alone
[2019-04-12 21:25:30,845: INFO/MainProcess] pidbox: Connected to amqp://guest:**@127.0.0.1:5672//.
[2019-04-12 21:25:30,848: WARNING/MainProcess] c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\celery\fixups\django.py:202: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2019-04-12 21:25:30,848: INFO/MainProcess] celery@DESKTOP-D52B9F2 ready.
[2019-04-12 21:26:01,042: INFO/MainProcess] Received task: main.task.mul[3112d374-cdc6-4202-955f-e33f416914f2]
[2019-04-12 21:26:01,085: INFO/MainProcess] Task main.task.mul[3112d374-cdc6-4202-955f-e33f416914f2] succeeded in 0.031000000002677552s: True
[2019-04-12 21:26:01,086: INFO/MainProcess] Received task: main.task.ruun[98e6ddba-0b63-41cc-bb00-6d8868b715f3]
[2019-04-12 21:26:01,086: INFO/MainProcess] Received task: main.task.ruun[f45e4e2f-b58d-4d16-8d06-a1eba4d66d9a]
[2019-04-12 21:26:01,094: WARNING/MainProcess] https://www.hepsiburada.com/xiaomi-10000-mah-versiyon-3-tasinabilir-sarj-cihazi-gumus-ince-ve-hafif-kasa-p-HBV00000B8GOR
[2019-04-12 21:26:01,133: WARNING/MainProcess] https://www.hepsiburada.com/apple-iphone-6-32-gb-apple-turkiye-garantili-p-HBV000003XBYY
[2019-04-12 21:26:01,499: ERROR/MainProcess] Task main.task.ruun[f45e4e2f-b58d-4d16-8d06-a1eba4d66d9a] raised unexpected: TypeError("wrap_socket() got an unexpected keyword argument '_context'")
Traceback (most recent call last):
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\celery\app\trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\celery\app\trace.py", line 648, in __protected_call__
    return self.run(*args, **kwargs)
  File "E:\Github\pirays\main\task.py", line 10, in ruun
    management.call_command('crawler', '{}'.format(x))
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\django\core\management\__init__.py", line 148, in call_command
    return command.execute(*args, **defaults)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\django\core\management\base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "E:\Github\pirays\main\management\commands\crawler.py", line 17, in handle
    ProductHistory(**parser.main(x)).save()
  File "E:\Github\pirays\main\management\commands\parsers\hepsiburadacom.py", line 13, in main
    response = requests.request("GET", url, headers=headers)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\adapters.py", line 449, in send
    timeout=timeout
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 839, in _validate_conn
    conn.connect()
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connection.py", line 344, in connect
    ssl_context=context)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\util\ssl_.py", line 344, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\eventlet\green\ssl.py", line 438, in wrap_socket
    return GreenSSLSocket(sock, *a, _context=self, **kw)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\eventlet\green\ssl.py", line 76, in __new__
    *args, **kw
TypeError: wrap_socket() got an unexpected keyword argument '_context'
[2019-04-12 21:26:01,610: ERROR/MainProcess] Task main.task.ruun[98e6ddba-0b63-41cc-bb00-6d8868b715f3] raised unexpected: TypeError("wrap_socket() got an unexpected keyword argument '_context'")
Traceback (most recent call last):
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\celery\app\trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\celery\app\trace.py", line 648, in __protected_call__
    return self.run(*args, **kwargs)
  File "E:\Github\pirays\main\task.py", line 10, in ruun
    management.call_command('crawler', '{}'.format(x))
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\django\core\management\__init__.py", line 148, in call_command
    return command.execute(*args, **defaults)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\django\core\management\base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "E:\Github\pirays\main\management\commands\crawler.py", line 17, in handle
    ProductHistory(**parser.main(x)).save()
  File "E:\Github\pirays\main\management\commands\parsers\hepsiburadacom.py", line 13, in main
    response = requests.request("GET", url, headers=headers)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\adapters.py", line 449, in send
    timeout=timeout
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 839, in _validate_conn
    conn.connect()
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connection.py", line 344, in connect
    ssl_context=context)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\util\ssl_.py", line 344, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\eventlet\green\ssl.py", line 438, in wrap_socket
    return GreenSSLSocket(sock, *a, _context=self, **kw)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\eventlet\green\ssl.py", line 76, in __new__
    *args, **kw
TypeError: wrap_socket() got an unexpected keyword argument '_context'

worker: Hitting Ctrl+C again will terminate all running tasks!

worker: Warm shutdown (MainProcess)

(pirays_env) E:\Github\pirays>
@kashirin-alex

This comment has been minimized.

Copy link

commented Apr 12, 2019

Looks like as wrap_socket does not accept anything but https://github.com/python/cpython/blob/42b1d6127bd8595522a78a75166ebb9fba74a6a2/Lib/ssl.py#L1199

and in-case there is a ready ctx assign it to the class(GreenSSLSocket) directly
https://github.com/python/cpython/blob/42b1d6127bd8595522a78a75166ebb9fba74a6a2/Lib/ssl.py#L821
at

if _is_under_py_3_7:

if '_context' in kw:
     self._context = kw['_context']

and use it to init(wrap) - as (https://github.com/python/cpython/blob/42b1d6127bd8595522a78a75166ebb9fba74a6a2/Lib/ssl.py#L1219)
for new -

ret = _original_wrap_socket(

to be, without *args or **kwargs:

  ret = context.wrap_socket(
        sock=sock, server_side=server_side,
        do_handshake_on_connect=do_handshake_on_connect,
        suppress_ragged_eofs=suppress_ragged_eofs
    )

predefined ready context must be allowed form of usage

I'm just yet using Py3.7+ to make a ready version for commit. (sorry)

A working Example of predefined Context should be remain well as well with 3.7+:

import eventlet
import ssl
import socket


cipher_list = "ECDH+ECDSA+AESGCM:" \
 \
              "ECDH+ECDSA+AES:" \
              "ECDH+AESGCM:" \
              "ECDH+AES:" \
 \
              "DH+ECDSA+AESGCM:" \
              "DH+ECDSA+AES:" \
              "DH+AESGCM:" \
              "DH+AES:" \
 \
              "RSA+AESGCM:" \
              "RSA+AESCBC:" \
 \
              "!aNULL:!MD5:!DSS:"

host = 'thither.direct'
port = 443

ips = socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM)
af, socktype, proto, canonname, sa = ips[0]
r = socket.socket(af, socktype, proto)

r.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 1024)
r.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 16384)
r.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
r.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
r.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)

r.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 10)
r.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 5)
r.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, 1)
r.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

r.connect(sa)

ctx = ssl.create_default_context(purpose=ssl.Purpose.SERVER_AUTH)
ctx.options |= (ssl.OP_NO_SSLv2 | ssl.OP_NO_SSLv3 | 
                ssl.OP_NO_TLSv1 | ssl.OP_NO_TLSv1_1 | ssl.OP_NO_COMPRESSION)
ctx.set_ciphers(cipher_list)

ctx.set_npn_protocols(['http/1.1', 'h2'])
ctx.set_alpn_protocols(['http/1.1', 'h2'])


def cb(sock, domainname, ctx): print (sock, domainname, ctx)


ctx.set_servername_callback(cb)

r = eventlet.wrap_ssl(r, server_hostname=host, _context=ctx)

print ('version', r.version())

print ('selected_npn_protocol', r.selected_alpn_protocol())
print ('selected_npn_protocol', r.selected_npn_protocol())
@kashirin-alex

This comment has been minimized.

Copy link

commented Apr 14, 2019

or More correct, it is to use predefined-context as follow:

from eventlet.green import ssl

ctx = ssl.create_default_context(...... )
r = ctx.wrap_socket(r, server_hostname=host)
@nat-goodspeed

This comment has been minimized.

Copy link
Contributor

commented Apr 30, 2019

Please be patient with my attempt to catch up...

  • urllib3 instantiates a context object, presumably using util.ssl_.create_urllib3_context(). Of course, with eventlet, this is a GreenSSLContext.
  • Eventually VerifiedHTTPSConnection.connect() calls ssl_wrap_socket() with that GreenSSLContext instance.
  • ssl_wrap_socket() calls context.wrap_socket(), which reaches GreenSSLContext.wrap_socket().
  • GreenSSLContext.wrap_socket() simply instantiates GreenSSLSocket(), passing _context=self to set the intended context.
  • GreenSSLSocket.__new__(), not having a _context argument itself, tries to pass through _context to ssl.wrap_socket() ...
  • but as ssl.wrap_socket() no longer has a _context argument, we get TypeError.

Thing is, Python 3.7.3's ssl.wrap_socket() is going to instantiate a whole new SSLContext anyway, and call its wrap_socket() method.

Specifically:

  • ssl_wrap_socket() calls GreenSSLContext.wrap_socket(sock, server_hostname=).
  • GreenSSLSocket.__new__() defines a lot of parameters, but most of them are defaulted. In fact, since server_hostname isn't one of them, the only explicit __new__() parameter with a non-default value is sock.
  • So when __new__() calls _original_wrap_socket() (aka ssl.wrap_socket()), it passes __new__()'s default values for most everything: keyfile=None, certfile=None, server_side=False, cert_reqs=CERT_NONE, ssl_version=PROTOCOL_SSLv23, ca_certs=None. It explicitly passes do_handshake_on_connect=False of course, along with server_hostname and _context courtesy of **kw.
  • That means that ssl.wrap_socket() would bypass its validation (server_side and keyfile are both None).
  • It would set the new SSLContext instance's verify_mode to CERT_NONE. (But we don't want a new SSLContext anyway, we want our existing GreenSSLContext instance.)
  • It would bypass the next several method calls because ca_certs, certfile and ciphers are all None.
  • In other words, all we really want from ssl.wrap_socket() is to call the (original) SSLContext.wrap_socket().

Would it be reasonable to change GreenSSLSocket.__new__() so that instead of calling _original_wrap_socket(), it calls super(GreenSSLSocket, self).wrap_socket() instead, adjusting parameters accordingly? (Does super() work properly within __new__()?)

Would that work even in earlier Python library versions?

@kashirin-alex

This comment has been minimized.

Copy link

commented Apr 30, 2019

A suggestion, before changing/adjusting to new py version,

Instead wrapping and parenting, which is followed by memleaks, GreenSocket is seen intensively with objgraph.get_leaking_objects()

a One Class can handle it all, while some convenient-methods to patching can be useful
https://github.com/kashirin-alex/eventlet/blob/master/eventlet/greenio/ultra.py

the use case with the UltraGreenSocket,

from eventlet.greenio.ultra import UltraGreenSocket
# some of original pkg
from eventlet.greenio.ultra import socket  
from eventlet.greenio.ultra import ssl
from eventlet.greenio.ultra import SSL

r = UltraGreenSocket(socket.AF_INET, socket.SOCK_STREAM)
#    or
r =  UltraGreenSocket(fd=a_socket)
#    or, init with SSL socket
r =  UltraGreenSocket(fd=ctx.wrap_socket(r, **kw))
#    or
with UltraGreenSocket(af, socket.SOCK_STREAM) as r:
   r.conn ... ()
   r.ssl_wrap(ctx)
   r.ssl_unwrap()

What I mean patch/convenient methods, it is UltraGreenSocket to be the socket.socket
and convenient are the ssl.wrap_socket and SSL.wrap_socket to do a wrap around ultra_fd.wrap(ctx) returning ultra_fd
eg:

def pyopenssl_wrap(self, ultra_fd, **kw):
    self.socket_wrap(ultra_fd)
    ultra_fd.ssl_wrap(self, **kw)
    return ultra_fd
SSL.Context.socket_wrap = pyopenssl_wrap

# and another for ssl.wrap_socket with default or predefined ctx

considering the class all sums down at https://github.com/kashirin-alex/eventlet/blob/master/eventlet/greenio/ultra.py#L365

A simple test-check-example:

# BASIC TESTS:


# plain client-connection

###
import socket
from eventlet.greenio.ultra import UltraGreenSocket
host = b'www.thither.direct'
port = 80

ips = socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM)
af, socktype, proto, canonname, sa = ips[0]
r = UltraGreenSocket(af, socktype, proto)

r.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 1024)
r.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 16384)
r.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
r.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
r.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)

r.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 10)
r.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 5)
r.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, 1)
r.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

r.connect(sa)

r.sendall(b'GET / HTTP/1.1\r\nHost: '+host+b'\r\n\r\n')
s = r.recv(64000)
len(s)
headers, body = s.split(b'\r\n\r\n', 1)
print (headers)
print (body[:20], body[-20:])

r.close()
###

# ssl client-connection

###
from eventlet.greenio.ultra import UltraGreenSocket, socket, ssl

cipher_list = b"ECDH+ECDSA+AESGCM:ECDH+ECDSA+AES:ECDH+AESGCM:ECDH+AES:" \
              b"DH+ECDSA+AESGCM:DH+ECDSA+AES:DH+AESGCM:DH+AES:" \
              b"RSA+AESGCM:RSA+AESCBC:" \
              b"!aNULL:!MD5:!DSS:"
host = b'www.thither.direct'
port = 443

ips = socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM)
af, socktype, proto, canonname, sa = ips[0]
r = UltraGreenSocket(af, socktype, proto)

r.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 1024)
r.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 16384)
r.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
r.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
r.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)

r.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 10)
r.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 5)
r.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, 1)
r.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

r.connect(sa)


ctx = ssl.create_default_context(purpose=ssl.Purpose.SERVER_AUTH)
ctx.options |= (ssl.OP_NO_SSLv2 | ssl.OP_NO_SSLv3 | 
                ssl.OP_NO_TLSv1 | ssl.OP_NO_TLSv1_1 | ssl.OP_NO_COMPRESSION)
ctx.set_ciphers(cipher_list)

ctx.set_npn_protocols(['http/1.1'])   # , 'h2'
ctx.set_alpn_protocols(['http/1.1'])  # , 'h2'


def cb(sock, domainname, ctx): print (sock, domainname, ctx)

ctx.set_servername_callback(cb)

# r.wrap_socket(ctx, accept_state=True) # acceptor

r.ssl_wrap(ctx, server_hostname=host)

print ('version', r.version())
print ('selected_npn_protocol', r.selected_alpn_protocol())
print ('selected_npn_protocol', r.selected_npn_protocol())


r.sendall(b'GET / HTTP/1.1\r\nHost: '+host+b'\r\n\r\n')
s = r.recv(64000)
len(s)
headers, body = s.split(b'\r\n\r\n', 1)
print (headers)
print (body[:20], body[-20:])

r.close()
###

Hope there are good things to use from the UltraGreenSocket, it is good with either py-versions

-- so far been working with the UltraGreenSocket class and It is all well. (considering, I could change from socket.socket to just the UltraGreenSocket)

My subject for making the UltraGreenSocket, It is the mem-leaks seen which are serious either from SSL/ssl/socket.socket

@kashirin-alex

This comment has been minimized.

Copy link

commented Apr 30, 2019

just catched up my self on the previous post,
yes, it is to minimize the *args and **kwargs, for py3.7+ new call
and for the case there is a _context kwargs, it is to set it directly to the self._context

@nat-goodspeed

This comment has been minimized.

Copy link
Contributor

commented Apr 30, 2019

But my use case is that I want to use the requests module with

eventlet.monkey_patch(socket=True)

I don't want to rewrite or manually patch requests, or the layers we've built on top of it.

I think it's worth trying to fix the existing mechanism.

@kashirin-alex

This comment has been minimized.

Copy link

commented Apr 30, 2019

Yes, I totally agree, I do as well still use requests pkg in some cases.
socket.socket - can be directly assigned to other class as done with the GreenSocket, but without Base or supers for SSL sockets.

just the minor patches to socket wraps would be still needed. - while the simplest form would be the sock.ssl_wrap - (in case of direct UltraGreenSocket use)

the _context in kwargs as an issue, It is just a matter of pop the ctx to self._context (for both py-version)

@nat-goodspeed

This comment has been minimized.

Copy link
Contributor

commented May 3, 2019

Would that work even in earlier Python library versions?

Interestingly, the Python 2.7 ssl.wrap_socket() literally does nothing except instantiate SSLSocket with exactly wrap_socket()'s parameters. Which is almost true of that version's SSLContext.wrap_socket() -- the only difference is that the latter passes _context=self (obviously where we got the eventlet usage) and that it presents (and passes) a different subset of SSLSocket.__init__() parameters.

(Since all the listed parameters in both wrap_socket() functions have the same defaults as in SSLSocket.__init__(), one wonders why the Python 2.7 ssl.py doesn't make both wrap_socket() functions pass (*args, **kwds) through. But that's not germane to this issue.)

The key point, I believe, is that Python 2.7's SSLContext.wrap_socket() has almost the same formal parameters -- with the same default values -- as Python 3.7's SSLContext.wrap_socket(). The difference is that the latter adds a session=None parameter -- which we should be able to ignore.

This suggests that changing GreenSSLContext.wrap_socket() to call its parent's SSLContext.wrap_socket(), as I suggested above, might well work with earlier Python versions too.

But how would I prove that?

That turns into the question: do we already have SSL tests for eventlet? The fact that this issue has been driven by user reports suggests not, though I can and will investigate.

If we don't already have SSL tests -- I just found an article describing a "remarkably simple" Python local HTTPS server for testing. I admit that I don't yet understand its usage of certfile='/tmp/cert-and-key.pem', but anyway that tactic is itself susceptible to testing before I try to embed it in the eventlet source code.

@nat-goodspeed nat-goodspeed reopened this May 3, 2019

@nat-goodspeed

This comment has been minimized.

Copy link
Contributor

commented May 3, 2019

Does super() work properly within __new__()?

Silly question: we already do this in GreenSSLSocket.__new__().

@Hawkzed

This comment has been minimized.

Copy link

commented Jun 4, 2019

Is there any news on a working 3.7.3 fork available? And if not does downgrading to 3.7.2 still work? I'm getting this issue trying to use Boto3 inside of a Celery worker to perform a DynamoDB batch write operation:

Traceback (most recent call last):
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\httpsession.py", line 262, in send
    chunked=self._chunked(request.headers),
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\urllib3\connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\urllib3\connectionpool.py", line 344, in _make_request
    self._validate_conn(conn)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\urllib3\connectionpool.py", line 843, in _validate_conn
    conn.connect()
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\urllib3\connection.py", line 370, in connect
    ssl_context=context)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\urllib3\util\ssl_.py", line 355, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\eventlet\green\ssl.py", line 438, in wrap_socket
    return GreenSSLSocket(sock, *a, _context=self, **kw)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\eventlet\green\ssl.py", line 76, in __new__
    *args, **kw
TypeError: wrap_socket() got an unexpected keyword argument '_context'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\app\trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\app\trace.py", line 648, in __protected_call__
    return self.run(*args, **kwargs)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\app\base.py", line 472, in run
    return task._orig_run(*args, **kwargs)
  File "C:\Users\Alex\PycharmProjects\Dynamight\ec2django\tasks.py", line 66, in queueBatch
    table: reqs
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\client.py", line 648, in _make_api_call
    operation_model, request_dict, request_context)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\client.py", line 667, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\endpoint.py", line 137, in _send_request
    success_response, exception):
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\endpoint.py", line 231, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\retryhandler.py", line 251, in __call__
    caught_exception)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\retryhandler.py", line 269, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\retryhandler.py", line 317, in __call__
    caught_exception)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\endpoint.py", line 200, in _do_get_response
    http_response = self._send(request)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\endpoint.py", line 244, in _send
    return self.http_session.send(request)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\httpsession.py", line 298, in send
    raise HTTPClientError(error=e)
botocore.exceptions.HTTPClientError: An HTTP Client raised and unhandled exception: wrap_socket() got an unexpected keyword argument '_context'
[2019-06-04 14:34:35,953: ERROR/MainProcess] Task ec2django.tasks.handleBatch[2471d1ff-13c0-4391-a685-a43bed4ee73d] raised unexpected: KeyError('error')
Traceback (most recent call last):
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\app\trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\app\trace.py", line 648, in __protected_call__
    return self.run(*args, **kwargs)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\app\base.py", line 472, in run
    return task._orig_run(*args, **kwargs)
  File "C:\Users\Alex\PycharmProjects\Dynamight\ec2django\tasks.py", line 51, in handleBatch
    print(req.status)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\result.py", line 473, in state
    return self._get_task_meta()['status']
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\result.py", line 412, in _get_task_meta
    return self._maybe_set_cache(self.backend.get_task_meta(self.id))
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\backends\rpc.py", line 259, in get_task_meta
    return self._set_cache_by_message(task_id, latest)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\backends\rpc.py", line 271, in _set_cache_by_message
    message.payload)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\backends\base.py", line 301, in meta_from_decoded
    meta['result'] = self.exception_to_python(meta['result'])
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\backends\base.py", line 281, in exception_to_python
    exc = cls(*exc_msg if isinstance(exc_msg, tuple) else exc_msg)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\exceptions.py", line 84, in __init__
    super(HTTPClientError, self).__init__(**kwargs)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\exceptions.py", line 40, in __init__
    msg = self.fmt.format(**kwargs)
KeyError: 'error'
@nat-goodspeed

This comment has been minimized.

Copy link
Contributor

commented Jun 4, 2019

I don't have news per se, only the fact that I'm still pursuing a solution (#565) ... but slowly, alas.

@daviskirk

This comment has been minimized.

Copy link

commented Jul 1, 2019

@sadakchap

This comment has been minimized.

Copy link

commented Jul 13, 2019

Fix Doesnt work on Py 3.7.3

[2019-04-12 21:25:29,814: INFO/MainProcess] mingle: searching for neighbors
[2019-04-12 21:25:30,831: INFO/MainProcess] mingle: all alone
[2019-04-12 21:25:30,845: INFO/MainProcess] pidbox: Connected to amqp://guest:**@127.0.0.1:5672//.
[2019-04-12 21:25:30,848: WARNING/MainProcess] c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\celery\fixups\django.py:202: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2019-04-12 21:25:30,848: INFO/MainProcess] celery@DESKTOP-D52B9F2 ready.
[2019-04-12 21:26:01,042: INFO/MainProcess] Received task: main.task.mul[3112d374-cdc6-4202-955f-e33f416914f2]
[2019-04-12 21:26:01,085: INFO/MainProcess] Task main.task.mul[3112d374-cdc6-4202-955f-e33f416914f2] succeeded in 0.031000000002677552s: True
[2019-04-12 21:26:01,086: INFO/MainProcess] Received task: main.task.ruun[98e6ddba-0b63-41cc-bb00-6d8868b715f3]
[2019-04-12 21:26:01,086: INFO/MainProcess] Received task: main.task.ruun[f45e4e2f-b58d-4d16-8d06-a1eba4d66d9a]
[2019-04-12 21:26:01,094: WARNING/MainProcess] https://www.hepsiburada.com/xiaomi-10000-mah-versiyon-3-tasinabilir-sarj-cihazi-gumus-ince-ve-hafif-kasa-p-HBV00000B8GOR
[2019-04-12 21:26:01,133: WARNING/MainProcess] https://www.hepsiburada.com/apple-iphone-6-32-gb-apple-turkiye-garantili-p-HBV000003XBYY
[2019-04-12 21:26:01,499: ERROR/MainProcess] Task main.task.ruun[f45e4e2f-b58d-4d16-8d06-a1eba4d66d9a] raised unexpected: TypeError("wrap_socket() got an unexpected keyword argument '_context'")
Traceback (most recent call last):
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\celery\app\trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\celery\app\trace.py", line 648, in __protected_call__
    return self.run(*args, **kwargs)
  File "E:\Github\pirays\main\task.py", line 10, in ruun
    management.call_command('crawler', '{}'.format(x))
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\django\core\management\__init__.py", line 148, in call_command
    return command.execute(*args, **defaults)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\django\core\management\base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "E:\Github\pirays\main\management\commands\crawler.py", line 17, in handle
    ProductHistory(**parser.main(x)).save()
  File "E:\Github\pirays\main\management\commands\parsers\hepsiburadacom.py", line 13, in main
    response = requests.request("GET", url, headers=headers)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\adapters.py", line 449, in send
    timeout=timeout
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 839, in _validate_conn
    conn.connect()
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connection.py", line 344, in connect
    ssl_context=context)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\util\ssl_.py", line 344, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\eventlet\green\ssl.py", line 438, in wrap_socket
    return GreenSSLSocket(sock, *a, _context=self, **kw)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\eventlet\green\ssl.py", line 76, in __new__
    *args, **kw
TypeError: wrap_socket() got an unexpected keyword argument '_context'
[2019-04-12 21:26:01,610: ERROR/MainProcess] Task main.task.ruun[98e6ddba-0b63-41cc-bb00-6d8868b715f3] raised unexpected: TypeError("wrap_socket() got an unexpected keyword argument '_context'")
Traceback (most recent call last):
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\celery\app\trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\celery\app\trace.py", line 648, in __protected_call__
    return self.run(*args, **kwargs)
  File "E:\Github\pirays\main\task.py", line 10, in ruun
    management.call_command('crawler', '{}'.format(x))
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\django\core\management\__init__.py", line 148, in call_command
    return command.execute(*args, **defaults)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\django\core\management\base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "E:\Github\pirays\main\management\commands\crawler.py", line 17, in handle
    ProductHistory(**parser.main(x)).save()
  File "E:\Github\pirays\main\management\commands\parsers\hepsiburadacom.py", line 13, in main
    response = requests.request("GET", url, headers=headers)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\adapters.py", line 449, in send
    timeout=timeout
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 839, in _validate_conn
    conn.connect()
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connection.py", line 344, in connect
    ssl_context=context)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\util\ssl_.py", line 344, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\eventlet\green\ssl.py", line 438, in wrap_socket
    return GreenSSLSocket(sock, *a, _context=self, **kw)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\eventlet\green\ssl.py", line 76, in __new__
    *args, **kw
TypeError: wrap_socket() got an unexpected keyword argument '_context'

worker: Hitting Ctrl+C again will terminate all running tasks!

worker: Warm shutdown (MainProcess)

(pirays_env) E:\Github\pirays>

did u find anything on how to work with celery4 + eventlet + web scrapping

@Mauro-Sbicego

This comment has been minimized.

Copy link

commented Jul 25, 2019

I had the same problem with 3.7.3 after moving to a fresh installation of Ubuntu 19.04. On Ubuntu 18.04 (one day before, everything up to date), everything worked fine.
I couldn't fix this issue on my 19.04 installation, but I deployed my program to an Ubuntu Server 18.04, and it worked fine.
So if you are on Ubuntu 19.04: maybe using Ubuntu 18.04 / repositories used by Ubuntu 18.04 helps?

nijel added a commit to WeblateOrg/docker that referenced this issue Aug 15, 2019

Remove eventlet usage
It breaks requests in some cases see
eventlet/eventlet#526

Fixes #340
Reverts #303

Signed-off-by: Michal Čihař <michal@cihar.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.