Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python 3.7 - wrap_socket() got an unexpected keyword argument '_context' #526

Closed
Nexodaru opened this issue Sep 29, 2018 · 74 comments
Closed

python 3.7 - wrap_socket() got an unexpected keyword argument '_context' #526

Nexodaru opened this issue Sep 29, 2018 · 74 comments

Comments

@Nexodaru
Copy link

@Nexodaru Nexodaru commented Sep 29, 2018

I develop an app with flask-socket.io and because of the new merge of #506 i thought i can upgrade the python version to 3.7. I used eventlet.monkey_patch. Not sure if it is an issue of eventlet, but on connection i get following Trace:

message handler error
Traceback (most recent call last):
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/engineio/server.py", line 411, in _trigger_event
    return self.handlers[event](*args)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/socketio/server.py", line 518, in _handle_eio_message
    self._handle_connect(sid, pkt.namespace)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/socketio/server.py", line 421, in _handle_connect
    self.environ[sid]) is False:
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/socketio/server.py", line 495, in _trigger_event
    event, *args)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/flask_socketio/namespace.py", line 26, in trigger_event
    *args)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/flask_socketio/__init__.py", line 632, in _handle_event
    ret = handler()
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/server/auth/__init__.py", line 10, in wrapped
    if not current_user.is_authenticated:
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/werkzeug/local.py", line 347, in __getattr__
    return getattr(self._get_current_object(), name)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/werkzeug/local.py", line 306, in _get_current_object
    return self.__local()
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/flask_login/utils.py", line 26, in <lambda>
    current_user = LocalProxy(lambda: _get_user())
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/flask_login/utils.py", line 335, in _get_user
    current_app.login_manager._load_user()
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/flask_login/login_manager.py", line 355, in _load_user
    return self._load_from_request(request)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/flask_login/login_manager.py", line 412, in _load_from_request
    user = self.request_callback(request)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/geocatch_persistence/login_manager.py", line 35, in load_user_from_request
    return user or validate_token(token)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/geocatch_persistence/login_manager.py", line 69, in validate_token
    current_app.config.get("GOOGLE_CLIENT_ID"))  # type: dict
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/google/oauth2/id_token.py", line 141, in verify_oauth2_token
    certs_url=_GOOGLE_OAUTH2_CERTS_URL)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/google/oauth2/id_token.py", line 120, in verify_token
    certs = _fetch_certs(request, certs_url)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/google/oauth2/id_token.py", line 94, in _fetch_certs
    response = request(certs_url, method='GET')
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/google/auth/transport/requests.py", line 120, in __call__
    **kwargs)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/requests/sessions.py", line 512, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/requests/sessions.py", line 622, in send
    r = adapter.send(request, **kwargs)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/requests/adapters.py", line 445, in send
    timeout=timeout
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 849, in _validate_conn
    conn.connect()
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/urllib3/connection.py", line 356, in connect
    ssl_context=context)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 359, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/eventlet/green/ssl.py", line 426, in wrap_socket
    return GreenSSLSocket(sock, *a, _context=self, **kw)
  File "/home/nexodaru/Workspace/Geocatch_2.0/code/socket.io/venv/lib/python3.7/site-packages/eventlet/green/ssl.py", line 79, in __init__
    *args, **kw
TypeError: wrap_socket() got an unexpected keyword argument '_context'

@nat-goodspeed
Copy link
Contributor

@nat-goodspeed nat-goodspeed commented Oct 2, 2018

Loading

@olii
Copy link

@olii olii commented Oct 11, 2018

Minimal example

with current eventlet master branch (cf47cb5) and Python interpreter version:

Python 3.7.0 (default, Jul 15 2018, 10:44:58) 
[GCC 8.1.1 20180531] on linux
Example # 1
import eventlet
eventlet.monkey_patch()

import socket
import ssl

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
wrappedSocket = ssl.wrap_socket(sock)
Example # 2 (with requests library from master branch):
import eventlet
eventlet.monkey_patch()

import requests
print(requests.get('https://www.google.com/').status_code)

Both examples fail with the exception: TypeError: wrap_socket() got an unexpected keyword argument '_context'

There were some breaking changes in Python 3.6 -> 3.7 in ssl module.
3.6 -> https://github.com/python/cpython/blob/3.6/Lib/ssl.py
3.7 -> https://github.com/python/cpython/blob/3.7/Lib/ssl.py

Loading

@Dormouse759
Copy link
Contributor

@Dormouse759 Dormouse759 commented Oct 11, 2018

I am aware of this. Unfortunately, I currently don't have the capacity to pay attention to this issue.
It could take some week or two until I become.

Let me at least explain some background on the issue:
SSL changed the API, the socket doesn't have public constructor and cannot be used by subclasses anymore.
This was worked around by #506 and the fix is not perfect yet.
This seems like argument handling failure at https://github.com/eventlet/eventlet/pull/506/files#diff-72b0a51127e4186c413957d1d5a59798R70

Loading

@luanjunyi
Copy link
Contributor

@luanjunyi luanjunyi commented Dec 9, 2018

I've a fix: #531. @Dormouse759 PTAL

Loading

@nat-goodspeed
Copy link
Contributor

@nat-goodspeed nat-goodspeed commented Mar 4, 2019

I'm closing this on the assumption that luanjunyi's fix works. Please reopen if it doesn't address the problem.

Loading

@mertcangokgoz
Copy link

@mertcangokgoz mertcangokgoz commented Apr 12, 2019

Fix Doesnt work on Py 3.7.3

[2019-04-12 21:25:29,814: INFO/MainProcess] mingle: searching for neighbors
[2019-04-12 21:25:30,831: INFO/MainProcess] mingle: all alone
[2019-04-12 21:25:30,845: INFO/MainProcess] pidbox: Connected to amqp://guest:**@127.0.0.1:5672//.
[2019-04-12 21:25:30,848: WARNING/MainProcess] c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\celery\fixups\django.py:202: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2019-04-12 21:25:30,848: INFO/MainProcess] celery@DESKTOP-D52B9F2 ready.
[2019-04-12 21:26:01,042: INFO/MainProcess] Received task: main.task.mul[3112d374-cdc6-4202-955f-e33f416914f2]
[2019-04-12 21:26:01,085: INFO/MainProcess] Task main.task.mul[3112d374-cdc6-4202-955f-e33f416914f2] succeeded in 0.031000000002677552s: True
[2019-04-12 21:26:01,086: INFO/MainProcess] Received task: main.task.ruun[98e6ddba-0b63-41cc-bb00-6d8868b715f3]
[2019-04-12 21:26:01,086: INFO/MainProcess] Received task: main.task.ruun[f45e4e2f-b58d-4d16-8d06-a1eba4d66d9a]
[2019-04-12 21:26:01,094: WARNING/MainProcess] https://www.hepsiburada.com/xiaomi-10000-mah-versiyon-3-tasinabilir-sarj-cihazi-gumus-ince-ve-hafif-kasa-p-HBV00000B8GOR
[2019-04-12 21:26:01,133: WARNING/MainProcess] https://www.hepsiburada.com/apple-iphone-6-32-gb-apple-turkiye-garantili-p-HBV000003XBYY
[2019-04-12 21:26:01,499: ERROR/MainProcess] Task main.task.ruun[f45e4e2f-b58d-4d16-8d06-a1eba4d66d9a] raised unexpected: TypeError("wrap_socket() got an unexpected keyword argument '_context'")
Traceback (most recent call last):
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\celery\app\trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\celery\app\trace.py", line 648, in __protected_call__
    return self.run(*args, **kwargs)
  File "E:\Github\pirays\main\task.py", line 10, in ruun
    management.call_command('crawler', '{}'.format(x))
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\django\core\management\__init__.py", line 148, in call_command
    return command.execute(*args, **defaults)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\django\core\management\base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "E:\Github\pirays\main\management\commands\crawler.py", line 17, in handle
    ProductHistory(**parser.main(x)).save()
  File "E:\Github\pirays\main\management\commands\parsers\hepsiburadacom.py", line 13, in main
    response = requests.request("GET", url, headers=headers)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\adapters.py", line 449, in send
    timeout=timeout
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 839, in _validate_conn
    conn.connect()
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connection.py", line 344, in connect
    ssl_context=context)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\util\ssl_.py", line 344, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\eventlet\green\ssl.py", line 438, in wrap_socket
    return GreenSSLSocket(sock, *a, _context=self, **kw)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\eventlet\green\ssl.py", line 76, in __new__
    *args, **kw
TypeError: wrap_socket() got an unexpected keyword argument '_context'
[2019-04-12 21:26:01,610: ERROR/MainProcess] Task main.task.ruun[98e6ddba-0b63-41cc-bb00-6d8868b715f3] raised unexpected: TypeError("wrap_socket() got an unexpected keyword argument '_context'")
Traceback (most recent call last):
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\celery\app\trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\celery\app\trace.py", line 648, in __protected_call__
    return self.run(*args, **kwargs)
  File "E:\Github\pirays\main\task.py", line 10, in ruun
    management.call_command('crawler', '{}'.format(x))
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\django\core\management\__init__.py", line 148, in call_command
    return command.execute(*args, **defaults)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\django\core\management\base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "E:\Github\pirays\main\management\commands\crawler.py", line 17, in handle
    ProductHistory(**parser.main(x)).save()
  File "E:\Github\pirays\main\management\commands\parsers\hepsiburadacom.py", line 13, in main
    response = requests.request("GET", url, headers=headers)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\adapters.py", line 449, in send
    timeout=timeout
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 839, in _validate_conn
    conn.connect()
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connection.py", line 344, in connect
    ssl_context=context)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\util\ssl_.py", line 344, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\eventlet\green\ssl.py", line 438, in wrap_socket
    return GreenSSLSocket(sock, *a, _context=self, **kw)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\eventlet\green\ssl.py", line 76, in __new__
    *args, **kw
TypeError: wrap_socket() got an unexpected keyword argument '_context'

worker: Hitting Ctrl+C again will terminate all running tasks!

worker: Warm shutdown (MainProcess)

(pirays_env) E:\Github\pirays>

Loading

@kashirin-alex
Copy link

@kashirin-alex kashirin-alex commented Apr 12, 2019

Looks like as wrap_socket does not accept anything but https://github.com/python/cpython/blob/42b1d6127bd8595522a78a75166ebb9fba74a6a2/Lib/ssl.py#L1199

and in-case there is a ready ctx assign it to the class(GreenSSLSocket) directly
https://github.com/python/cpython/blob/42b1d6127bd8595522a78a75166ebb9fba74a6a2/Lib/ssl.py#L821
at

if _is_under_py_3_7:

if '_context' in kw:
     self._context = kw['_context']

and use it to init(wrap) - as (https://github.com/python/cpython/blob/42b1d6127bd8595522a78a75166ebb9fba74a6a2/Lib/ssl.py#L1219)
for new -

ret = _original_wrap_socket(

to be, without *args or **kwargs:

  ret = context.wrap_socket(
        sock=sock, server_side=server_side,
        do_handshake_on_connect=do_handshake_on_connect,
        suppress_ragged_eofs=suppress_ragged_eofs
    )

predefined ready context must be allowed form of usage

I'm just yet using Py3.7+ to make a ready version for commit. (sorry)

A working Example of predefined Context should be remain well as well with 3.7+:

import eventlet
import ssl
import socket


cipher_list = "ECDH+ECDSA+AESGCM:" \
 \
              "ECDH+ECDSA+AES:" \
              "ECDH+AESGCM:" \
              "ECDH+AES:" \
 \
              "DH+ECDSA+AESGCM:" \
              "DH+ECDSA+AES:" \
              "DH+AESGCM:" \
              "DH+AES:" \
 \
              "RSA+AESGCM:" \
              "RSA+AESCBC:" \
 \
              "!aNULL:!MD5:!DSS:"

host = 'thither.direct'
port = 443

ips = socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM)
af, socktype, proto, canonname, sa = ips[0]
r = socket.socket(af, socktype, proto)

r.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 1024)
r.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 16384)
r.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
r.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
r.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)

r.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 10)
r.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 5)
r.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, 1)
r.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

r.connect(sa)

ctx = ssl.create_default_context(purpose=ssl.Purpose.SERVER_AUTH)
ctx.options |= (ssl.OP_NO_SSLv2 | ssl.OP_NO_SSLv3 | 
                ssl.OP_NO_TLSv1 | ssl.OP_NO_TLSv1_1 | ssl.OP_NO_COMPRESSION)
ctx.set_ciphers(cipher_list)

ctx.set_npn_protocols(['http/1.1', 'h2'])
ctx.set_alpn_protocols(['http/1.1', 'h2'])


def cb(sock, domainname, ctx): print (sock, domainname, ctx)


ctx.set_servername_callback(cb)

r = eventlet.wrap_ssl(r, server_hostname=host, _context=ctx)

print ('version', r.version())

print ('selected_npn_protocol', r.selected_alpn_protocol())
print ('selected_npn_protocol', r.selected_npn_protocol())

Loading

@kashirin-alex
Copy link

@kashirin-alex kashirin-alex commented Apr 14, 2019

or More correct, it is to use predefined-context as follow:

from eventlet.green import ssl

ctx = ssl.create_default_context(...... )
r = ctx.wrap_socket(r, server_hostname=host)

Loading

@nat-goodspeed
Copy link
Contributor

@nat-goodspeed nat-goodspeed commented Apr 30, 2019

Please be patient with my attempt to catch up...

  • urllib3 instantiates a context object, presumably using util.ssl_.create_urllib3_context(). Of course, with eventlet, this is a GreenSSLContext.
  • Eventually VerifiedHTTPSConnection.connect() calls ssl_wrap_socket() with that GreenSSLContext instance.
  • ssl_wrap_socket() calls context.wrap_socket(), which reaches GreenSSLContext.wrap_socket().
  • GreenSSLContext.wrap_socket() simply instantiates GreenSSLSocket(), passing _context=self to set the intended context.
  • GreenSSLSocket.__new__(), not having a _context argument itself, tries to pass through _context to ssl.wrap_socket() ...
  • but as ssl.wrap_socket() no longer has a _context argument, we get TypeError.

Thing is, Python 3.7.3's ssl.wrap_socket() is going to instantiate a whole new SSLContext anyway, and call its wrap_socket() method.

Specifically:

  • ssl_wrap_socket() calls GreenSSLContext.wrap_socket(sock, server_hostname=).
  • GreenSSLSocket.__new__() defines a lot of parameters, but most of them are defaulted. In fact, since server_hostname isn't one of them, the only explicit __new__() parameter with a non-default value is sock.
  • So when __new__() calls _original_wrap_socket() (aka ssl.wrap_socket()), it passes __new__()'s default values for most everything: keyfile=None, certfile=None, server_side=False, cert_reqs=CERT_NONE, ssl_version=PROTOCOL_SSLv23, ca_certs=None. It explicitly passes do_handshake_on_connect=False of course, along with server_hostname and _context courtesy of **kw.
  • That means that ssl.wrap_socket() would bypass its validation (server_side and keyfile are both None).
  • It would set the new SSLContext instance's verify_mode to CERT_NONE. (But we don't want a new SSLContext anyway, we want our existing GreenSSLContext instance.)
  • It would bypass the next several method calls because ca_certs, certfile and ciphers are all None.
  • In other words, all we really want from ssl.wrap_socket() is to call the (original) SSLContext.wrap_socket().

Would it be reasonable to change GreenSSLSocket.__new__() so that instead of calling _original_wrap_socket(), it calls super(GreenSSLSocket, self).wrap_socket() instead, adjusting parameters accordingly? (Does super() work properly within __new__()?)

Would that work even in earlier Python library versions?

Loading

@kashirin-alex
Copy link

@kashirin-alex kashirin-alex commented Apr 30, 2019

A suggestion, before changing/adjusting to new py version,

Instead wrapping and parenting, which is followed by memleaks, GreenSocket is seen intensively with objgraph.get_leaking_objects()

a One Class can handle it all, while some convenient-methods to patching can be useful
https://github.com/kashirin-alex/eventlet/blob/master/eventlet/greenio/ultra.py

the use case with the UltraGreenSocket,

from eventlet.greenio.ultra import UltraGreenSocket
# some of original pkg
from eventlet.greenio.ultra import socket  
from eventlet.greenio.ultra import ssl
from eventlet.greenio.ultra import SSL

r = UltraGreenSocket(socket.AF_INET, socket.SOCK_STREAM)
#    or
r =  UltraGreenSocket(fd=a_socket)
#    or, init with SSL socket
r =  UltraGreenSocket(fd=ctx.wrap_socket(r, **kw))
#    or
with UltraGreenSocket(af, socket.SOCK_STREAM) as r:
   r.conn ... ()
   r.ssl_wrap(ctx)
   r.ssl_unwrap()

What I mean patch/convenient methods, it is UltraGreenSocket to be the socket.socket
and convenient are the ssl.wrap_socket and SSL.wrap_socket to do a wrap around ultra_fd.wrap(ctx) returning ultra_fd
eg:

def pyopenssl_wrap(self, ultra_fd, **kw):
    self.socket_wrap(ultra_fd)
    ultra_fd.ssl_wrap(self, **kw)
    return ultra_fd
SSL.Context.socket_wrap = pyopenssl_wrap

# and another for ssl.wrap_socket with default or predefined ctx

considering the class all sums down at https://github.com/kashirin-alex/eventlet/blob/master/eventlet/greenio/ultra.py#L365

A simple test-check-example:

# BASIC TESTS:


# plain client-connection

###
import socket
from eventlet.greenio.ultra import UltraGreenSocket
host = b'www.thither.direct'
port = 80

ips = socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM)
af, socktype, proto, canonname, sa = ips[0]
r = UltraGreenSocket(af, socktype, proto)

r.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 1024)
r.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 16384)
r.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
r.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
r.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)

r.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 10)
r.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 5)
r.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, 1)
r.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

r.connect(sa)

r.sendall(b'GET / HTTP/1.1\r\nHost: '+host+b'\r\n\r\n')
s = r.recv(64000)
len(s)
headers, body = s.split(b'\r\n\r\n', 1)
print (headers)
print (body[:20], body[-20:])

r.close()
###

# ssl client-connection

###
from eventlet.greenio.ultra import UltraGreenSocket, socket, ssl

cipher_list = b"ECDH+ECDSA+AESGCM:ECDH+ECDSA+AES:ECDH+AESGCM:ECDH+AES:" \
              b"DH+ECDSA+AESGCM:DH+ECDSA+AES:DH+AESGCM:DH+AES:" \
              b"RSA+AESGCM:RSA+AESCBC:" \
              b"!aNULL:!MD5:!DSS:"
host = b'www.thither.direct'
port = 443

ips = socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM)
af, socktype, proto, canonname, sa = ips[0]
r = UltraGreenSocket(af, socktype, proto)

r.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 1024)
r.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 16384)
r.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
r.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
r.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)

r.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 10)
r.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 5)
r.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, 1)
r.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

r.connect(sa)


ctx = ssl.create_default_context(purpose=ssl.Purpose.SERVER_AUTH)
ctx.options |= (ssl.OP_NO_SSLv2 | ssl.OP_NO_SSLv3 | 
                ssl.OP_NO_TLSv1 | ssl.OP_NO_TLSv1_1 | ssl.OP_NO_COMPRESSION)
ctx.set_ciphers(cipher_list)

ctx.set_npn_protocols(['http/1.1'])   # , 'h2'
ctx.set_alpn_protocols(['http/1.1'])  # , 'h2'


def cb(sock, domainname, ctx): print (sock, domainname, ctx)

ctx.set_servername_callback(cb)

# r.wrap_socket(ctx, accept_state=True) # acceptor

r.ssl_wrap(ctx, server_hostname=host)

print ('version', r.version())
print ('selected_npn_protocol', r.selected_alpn_protocol())
print ('selected_npn_protocol', r.selected_npn_protocol())


r.sendall(b'GET / HTTP/1.1\r\nHost: '+host+b'\r\n\r\n')
s = r.recv(64000)
len(s)
headers, body = s.split(b'\r\n\r\n', 1)
print (headers)
print (body[:20], body[-20:])

r.close()
###

Hope there are good things to use from the UltraGreenSocket, it is good with either py-versions

-- so far been working with the UltraGreenSocket class and It is all well. (considering, I could change from socket.socket to just the UltraGreenSocket)

My subject for making the UltraGreenSocket, It is the mem-leaks seen which are serious either from SSL/ssl/socket.socket

Loading

@kashirin-alex
Copy link

@kashirin-alex kashirin-alex commented Apr 30, 2019

just catched up my self on the previous post,
yes, it is to minimize the *args and **kwargs, for py3.7+ new call
and for the case there is a _context kwargs, it is to set it directly to the self._context

Loading

@nat-goodspeed
Copy link
Contributor

@nat-goodspeed nat-goodspeed commented Apr 30, 2019

But my use case is that I want to use the requests module with

eventlet.monkey_patch(socket=True)

I don't want to rewrite or manually patch requests, or the layers we've built on top of it.

I think it's worth trying to fix the existing mechanism.

Loading

@kashirin-alex
Copy link

@kashirin-alex kashirin-alex commented Apr 30, 2019

Yes, I totally agree, I do as well still use requests pkg in some cases.
socket.socket - can be directly assigned to other class as done with the GreenSocket, but without Base or supers for SSL sockets.

just the minor patches to socket wraps would be still needed. - while the simplest form would be the sock.ssl_wrap - (in case of direct UltraGreenSocket use)

the _context in kwargs as an issue, It is just a matter of pop the ctx to self._context (for both py-version)

Loading

@nat-goodspeed
Copy link
Contributor

@nat-goodspeed nat-goodspeed commented May 3, 2019

Would that work even in earlier Python library versions?

Interestingly, the Python 2.7 ssl.wrap_socket() literally does nothing except instantiate SSLSocket with exactly wrap_socket()'s parameters. Which is almost true of that version's SSLContext.wrap_socket() -- the only difference is that the latter passes _context=self (obviously where we got the eventlet usage) and that it presents (and passes) a different subset of SSLSocket.__init__() parameters.

(Since all the listed parameters in both wrap_socket() functions have the same defaults as in SSLSocket.__init__(), one wonders why the Python 2.7 ssl.py doesn't make both wrap_socket() functions pass (*args, **kwds) through. But that's not germane to this issue.)

The key point, I believe, is that Python 2.7's SSLContext.wrap_socket() has almost the same formal parameters -- with the same default values -- as Python 3.7's SSLContext.wrap_socket(). The difference is that the latter adds a session=None parameter -- which we should be able to ignore.

This suggests that changing GreenSSLContext.wrap_socket() to call its parent's SSLContext.wrap_socket(), as I suggested above, might well work with earlier Python versions too.

But how would I prove that?

That turns into the question: do we already have SSL tests for eventlet? The fact that this issue has been driven by user reports suggests not, though I can and will investigate.

If we don't already have SSL tests -- I just found an article describing a "remarkably simple" Python local HTTPS server for testing. I admit that I don't yet understand its usage of certfile='/tmp/cert-and-key.pem', but anyway that tactic is itself susceptible to testing before I try to embed it in the eventlet source code.

Loading

@nat-goodspeed nat-goodspeed reopened this May 3, 2019
@nat-goodspeed
Copy link
Contributor

@nat-goodspeed nat-goodspeed commented May 3, 2019

Does super() work properly within __new__()?

Silly question: we already do this in GreenSSLSocket.__new__().

Loading

@Hawkzed
Copy link

@Hawkzed Hawkzed commented Jun 4, 2019

Is there any news on a working 3.7.3 fork available? And if not does downgrading to 3.7.2 still work? I'm getting this issue trying to use Boto3 inside of a Celery worker to perform a DynamoDB batch write operation:

Traceback (most recent call last):
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\httpsession.py", line 262, in send
    chunked=self._chunked(request.headers),
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\urllib3\connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\urllib3\connectionpool.py", line 344, in _make_request
    self._validate_conn(conn)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\urllib3\connectionpool.py", line 843, in _validate_conn
    conn.connect()
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\urllib3\connection.py", line 370, in connect
    ssl_context=context)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\urllib3\util\ssl_.py", line 355, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\eventlet\green\ssl.py", line 438, in wrap_socket
    return GreenSSLSocket(sock, *a, _context=self, **kw)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\eventlet\green\ssl.py", line 76, in __new__
    *args, **kw
TypeError: wrap_socket() got an unexpected keyword argument '_context'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\app\trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\app\trace.py", line 648, in __protected_call__
    return self.run(*args, **kwargs)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\app\base.py", line 472, in run
    return task._orig_run(*args, **kwargs)
  File "C:\Users\Alex\PycharmProjects\Dynamight\ec2django\tasks.py", line 66, in queueBatch
    table: reqs
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\client.py", line 648, in _make_api_call
    operation_model, request_dict, request_context)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\client.py", line 667, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\endpoint.py", line 137, in _send_request
    success_response, exception):
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\endpoint.py", line 231, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\retryhandler.py", line 251, in __call__
    caught_exception)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\retryhandler.py", line 269, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\retryhandler.py", line 317, in __call__
    caught_exception)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\endpoint.py", line 200, in _do_get_response
    http_response = self._send(request)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\endpoint.py", line 244, in _send
    return self.http_session.send(request)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\httpsession.py", line 298, in send
    raise HTTPClientError(error=e)
botocore.exceptions.HTTPClientError: An HTTP Client raised and unhandled exception: wrap_socket() got an unexpected keyword argument '_context'
[2019-06-04 14:34:35,953: ERROR/MainProcess] Task ec2django.tasks.handleBatch[2471d1ff-13c0-4391-a685-a43bed4ee73d] raised unexpected: KeyError('error')
Traceback (most recent call last):
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\app\trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\app\trace.py", line 648, in __protected_call__
    return self.run(*args, **kwargs)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\app\base.py", line 472, in run
    return task._orig_run(*args, **kwargs)
  File "C:\Users\Alex\PycharmProjects\Dynamight\ec2django\tasks.py", line 51, in handleBatch
    print(req.status)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\result.py", line 473, in state
    return self._get_task_meta()['status']
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\result.py", line 412, in _get_task_meta
    return self._maybe_set_cache(self.backend.get_task_meta(self.id))
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\backends\rpc.py", line 259, in get_task_meta
    return self._set_cache_by_message(task_id, latest)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\backends\rpc.py", line 271, in _set_cache_by_message
    message.payload)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\backends\base.py", line 301, in meta_from_decoded
    meta['result'] = self.exception_to_python(meta['result'])
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\celery\backends\base.py", line 281, in exception_to_python
    exc = cls(*exc_msg if isinstance(exc_msg, tuple) else exc_msg)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\exceptions.py", line 84, in __init__
    super(HTTPClientError, self).__init__(**kwargs)
  File "c:\users\alex\.virtualenvs\dynamight-ulxdqw9f\lib\site-packages\botocore\exceptions.py", line 40, in __init__
    msg = self.fmt.format(**kwargs)
KeyError: 'error'

Loading

@nat-goodspeed
Copy link
Contributor

@nat-goodspeed nat-goodspeed commented Jun 4, 2019

I don't have news per se, only the fact that I'm still pursuing a solution (#565) ... but slowly, alas.

Loading

@daviskirk
Copy link

@daviskirk daviskirk commented Jul 1, 2019

Loading

@sadakchap
Copy link

@sadakchap sadakchap commented Jul 13, 2019

Fix Doesnt work on Py 3.7.3

[2019-04-12 21:25:29,814: INFO/MainProcess] mingle: searching for neighbors
[2019-04-12 21:25:30,831: INFO/MainProcess] mingle: all alone
[2019-04-12 21:25:30,845: INFO/MainProcess] pidbox: Connected to amqp://guest:**@127.0.0.1:5672//.
[2019-04-12 21:25:30,848: WARNING/MainProcess] c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\celery\fixups\django.py:202: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2019-04-12 21:25:30,848: INFO/MainProcess] celery@DESKTOP-D52B9F2 ready.
[2019-04-12 21:26:01,042: INFO/MainProcess] Received task: main.task.mul[3112d374-cdc6-4202-955f-e33f416914f2]
[2019-04-12 21:26:01,085: INFO/MainProcess] Task main.task.mul[3112d374-cdc6-4202-955f-e33f416914f2] succeeded in 0.031000000002677552s: True
[2019-04-12 21:26:01,086: INFO/MainProcess] Received task: main.task.ruun[98e6ddba-0b63-41cc-bb00-6d8868b715f3]
[2019-04-12 21:26:01,086: INFO/MainProcess] Received task: main.task.ruun[f45e4e2f-b58d-4d16-8d06-a1eba4d66d9a]
[2019-04-12 21:26:01,094: WARNING/MainProcess] https://www.hepsiburada.com/xiaomi-10000-mah-versiyon-3-tasinabilir-sarj-cihazi-gumus-ince-ve-hafif-kasa-p-HBV00000B8GOR
[2019-04-12 21:26:01,133: WARNING/MainProcess] https://www.hepsiburada.com/apple-iphone-6-32-gb-apple-turkiye-garantili-p-HBV000003XBYY
[2019-04-12 21:26:01,499: ERROR/MainProcess] Task main.task.ruun[f45e4e2f-b58d-4d16-8d06-a1eba4d66d9a] raised unexpected: TypeError("wrap_socket() got an unexpected keyword argument '_context'")
Traceback (most recent call last):
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\celery\app\trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\celery\app\trace.py", line 648, in __protected_call__
    return self.run(*args, **kwargs)
  File "E:\Github\pirays\main\task.py", line 10, in ruun
    management.call_command('crawler', '{}'.format(x))
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\django\core\management\__init__.py", line 148, in call_command
    return command.execute(*args, **defaults)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\django\core\management\base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "E:\Github\pirays\main\management\commands\crawler.py", line 17, in handle
    ProductHistory(**parser.main(x)).save()
  File "E:\Github\pirays\main\management\commands\parsers\hepsiburadacom.py", line 13, in main
    response = requests.request("GET", url, headers=headers)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\adapters.py", line 449, in send
    timeout=timeout
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 839, in _validate_conn
    conn.connect()
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connection.py", line 344, in connect
    ssl_context=context)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\util\ssl_.py", line 344, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\eventlet\green\ssl.py", line 438, in wrap_socket
    return GreenSSLSocket(sock, *a, _context=self, **kw)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\eventlet\green\ssl.py", line 76, in __new__
    *args, **kw
TypeError: wrap_socket() got an unexpected keyword argument '_context'
[2019-04-12 21:26:01,610: ERROR/MainProcess] Task main.task.ruun[98e6ddba-0b63-41cc-bb00-6d8868b715f3] raised unexpected: TypeError("wrap_socket() got an unexpected keyword argument '_context'")
Traceback (most recent call last):
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\celery\app\trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\celery\app\trace.py", line 648, in __protected_call__
    return self.run(*args, **kwargs)
  File "E:\Github\pirays\main\task.py", line 10, in ruun
    management.call_command('crawler', '{}'.format(x))
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\django\core\management\__init__.py", line 148, in call_command
    return command.execute(*args, **defaults)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\django\core\management\base.py", line 364, in execute
    output = self.handle(*args, **options)
  File "E:\Github\pirays\main\management\commands\crawler.py", line 17, in handle
    ProductHistory(**parser.main(x)).save()
  File "E:\Github\pirays\main\management\commands\parsers\hepsiburadacom.py", line 13, in main
    response = requests.request("GET", url, headers=headers)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\requests\adapters.py", line 449, in send
    timeout=timeout
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 343, in _make_request
    self._validate_conn(conn)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connectionpool.py", line 839, in _validate_conn
    conn.connect()
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\connection.py", line 344, in connect
    ssl_context=context)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\urllib3\util\ssl_.py", line 344, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\eventlet\green\ssl.py", line 438, in wrap_socket
    return GreenSSLSocket(sock, *a, _context=self, **kw)
  File "c:\users\mertcan\anaconda3\envs\pirays_env\lib\site-packages\eventlet\green\ssl.py", line 76, in __new__
    *args, **kw
TypeError: wrap_socket() got an unexpected keyword argument '_context'

worker: Hitting Ctrl+C again will terminate all running tasks!

worker: Warm shutdown (MainProcess)

(pirays_env) E:\Github\pirays>

did u find anything on how to work with celery4 + eventlet + web scrapping

Loading

@maurosbicego
Copy link

@maurosbicego maurosbicego commented Jul 25, 2019

I had the same problem with 3.7.3 after moving to a fresh installation of Ubuntu 19.04. On Ubuntu 18.04 (one day before, everything up to date), everything worked fine.
I couldn't fix this issue on my 19.04 installation, but I deployed my program to an Ubuntu Server 18.04, and it worked fine.
So if you are on Ubuntu 19.04: maybe using Ubuntu 18.04 / repositories used by Ubuntu 18.04 helps?

Loading

nijel added a commit to WeblateOrg/docker that referenced this issue Aug 15, 2019
It breaks requests in some cases see
eventlet/eventlet#526

Fixes #340
Reverts #303

Signed-off-by: Michal Čihař <michal@cihar.com>
openstack-mirroring pushed a commit to openstack/openstack that referenced this issue Jun 30, 2020
* Update requirements from branch 'master'
  - Lower upper-constraints for requests
    
    Requests 2.24.0 has issues with eventlet related to monkey patching [0].
    Until those issues are resolved, let's lower our constraint to versions
    without this problem to help unbreak several projects.
    
    [0] eventlet/eventlet#526
    
    Change-Id: Ic475dc692971da7ad0c2dc4decb3a580620fdf98
    Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
@ShudelEV
Copy link

@ShudelEV ShudelEV commented Jul 1, 2020

I caught the same error while using celery worker (--pool=eventlet):

python 3.7.6
celery 4.4.6
eventlet 0.25.2
requests 2.24.0

[2020-06-29 06:54:57,318] ERROR/celery.app.trace: Task users.tasks.send_firebase_notification[a65cc49c-1cc6-4f94-ad8f-9fe862d9bc13] raised unexpected: 
TypeError("wrap_socket() got an unexpected keyword argument '_context'") 
Traceback (most recent call last): 
 File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 412, in trace_task R = retval = fun(*args, **kwargs)
 File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 704, in __protected_call__ return self.run(*args, **kwargs)
 File "/app/users/tasks.py", line 65, in send_firebase_notification **kwargs,
 File "/usr/local/lib/python3.7/site-packages/fcm_django/models.py", line 79, in send_message **kwargs
 File "/usr/local/lib/python3.7/site-packages/fcm_django/fcm.py", line 364, in fcm_send_bulk_message **kwargs
 File "/usr/local/lib/python3.7/site-packages/pyfcm/fcm.py", line 312, in notify_multiple_devices self.send_request(payloads, timeout)
 File "/usr/local/lib/python3.7/site-packages/pyfcm/baseapi.py", line 312, in send_request response = self.do_request(payload, timeout)
 File "/usr/local/lib/python3.7/site-packages/pyfcm/baseapi.py", line 302, in do_request response = self.requests_session.post(self.FCM_END_POINT, data=payload, timeout=timeout)
 File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 578, in post return self.request('POST', url, data=data, json=json, **kwargs)
 File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 530, in request resp = self.send(prep, **send_kwargs)
 File "/usr/local/lib/python3.7/site-packages/raven/breadcrumbs.py", line 341, in send resp = real_send(self, request, *args, **kwargs)
 File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 643, in send r = adapter.send(request, **kwargs)
 File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send timeout=timeout
 File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen chunked=chunked,
 File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 381, in _make_request self._validate_conn(conn)
 File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 976, in _validate_conn conn.connect()
 File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 370, in connect ssl_context=context,
 File "/usr/local/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 377, in ssl_wrap_socket return context.wrap_socket(sock, server_hostname=server_hostname)
 File "/usr/local/lib/python3.7/site-packages/eventlet/green/ssl.py", line 437, in wrap_socket return GreenSSLSocket(sock, *a, _context=self, **kw)
 File "/usr/local/lib/python3.7/site-packages/eventlet/green/ssl.py", line 76, in __new__ *args, **kw
TypeError: wrap_socket() got an unexpected keyword argument '_context'

It works fine with requests 2.23.0

Loading

javacruft added a commit to javacruft/eventlet that referenced this issue Jul 1, 2020
For SSL sockets created using the SSLContext class under Python >= 3.7,
eventlet incorrectly passes the context as '_context' to the top
level wrap_socket function in the native ssl module.

This causes:

  wrap_socket() got an unexpected keyword argument '_context'

as the context cannot be passed this way.

If a context is provided, use the underlying sslsocket_class to
wrap the socket, mirroring the implementation of the wrap_socket
method in the native SSLContext class.

Fixes issue eventlet#526
@javacruft
Copy link
Contributor

@javacruft javacruft commented Jul 1, 2020

I've put up a PR for what I think is the right fix for this issue - review appreciated!

Loading

@ba1dr
Copy link

@ba1dr ba1dr commented Jul 1, 2020

@ShudelEV you're my hero!
I confirm that for me it works with requests==2.23.0 on Python 3.8!

Loading

jstasiak added a commit that referenced this issue Jul 1, 2020
jstasiak added a commit that referenced this issue Jul 1, 2020
For SSL sockets created using the SSLContext class under Python >= 3.7,
eventlet incorrectly passes the context as '_context' to the top
level wrap_socket function in the native ssl module.

This causes:

  wrap_socket() got an unexpected keyword argument '_context'

as the context cannot be passed this way.

If a context is provided, use the underlying sslsocket_class to
wrap the socket, mirroring the implementation of the wrap_socket
method in the native SSLContext class.

Fixes issue #526
jstasiak added a commit that referenced this issue Jul 1, 2020
For SSL sockets created using the SSLContext class under Python >= 3.7,
eventlet incorrectly passes the context as '_context' to the top
level wrap_socket function in the native ssl module.

This causes:

  wrap_socket() got an unexpected keyword argument '_context'

as the context cannot be passed this way.

If a context is provided, use the underlying sslsocket_class to
wrap the socket, mirroring the implementation of the wrap_socket
method in the native SSLContext class.

Fixes issue #526

Co-authored-by: Tim Burke <tim.burke@gmail.com>
@thedrow
Copy link
Contributor

@thedrow thedrow commented Jul 9, 2020

Hi everyone.

We cannot merge celery/py-amqp#327 which will become a problem for Celery users soon if it isn't one already.

What can we do to help?

Loading

@jstasiak
Copy link
Contributor

@jstasiak jstasiak commented Jul 9, 2020

This should be resolved by #621 so it just needs to be released.

Loading

@lanmaster53
Copy link

@lanmaster53 lanmaster53 commented Jul 13, 2020

Any idea when we can expect to see this fix propagated to PyPi?

Loading

@yogevyuval
Copy link

@yogevyuval yogevyuval commented Jul 30, 2020

@temoto
Is this planned to be released? or should we start using eventlet from master? Any information you can share will be much appriciated

Loading

@temoto
Copy link
Member

@temoto temoto commented Jul 30, 2020

@yogevyuval @lanmaster53 I'm sorry, missed this. Releasing now.

Loading

@yogevyuval
Copy link

@yogevyuval yogevyuval commented Jul 30, 2020

@temoto Thanks!!
It might be a good idea to get in this version the dnspython<2.0.0 fix as this is a major breaking change introduced.... but your call :)

Loading

@temoto
Copy link
Member

@temoto temoto commented Jul 30, 2020

v0.26.0 just uploaded to PyPI, contains this fix and too many others. Sorry for release delay.

@yogevyuval my personal encouragement is to use eventlet master if you can. dnspython wasn't merged yet, so didn't go to release.

Loading

@yogevyuval
Copy link

@yogevyuval yogevyuval commented Jul 30, 2020

@temoto Thanks you very much. So I guess this historic issue can now be closed after confirmation :)

Loading

@thedrow
Copy link
Contributor

@thedrow thedrow commented Aug 2, 2020

@auvipy Please create an issue to bump our eventlet requirement since this bug is now resolved.

Loading

@auvipy
Copy link

@auvipy auvipy commented Aug 2, 2020

ok, thanks!

Loading

@ShaneHarvey
Copy link

@ShaneHarvey ShaneHarvey commented Aug 6, 2020

As the maintainer of pymongo, I can confirm that v0.26.1 fixes the TypeError("wrap_socket() got an unexpected keyword argument '_context'") incompatibility issue with Python 3.7 and 3.8: see PYTHON-1854. Thanks!

Loading

@temoto temoto closed this Aug 25, 2020
openstack-mirroring pushed a commit to openstack/requirements that referenced this issue Dec 6, 2020
The eventlet lib has a known bug with python 3.7 [0].
The fix was released as part of version 0.26.

[0] eventlet/eventlet#526

Change-Id: Iec87beaa799368790a7e3f73141b7e580a5aa493
openstack-mirroring pushed a commit to openstack/openstack that referenced this issue Nov 4, 2021
* Update os-ken from branch 'master'
  to 7911ff1820b83bf1657c322c36ca68dcb5c0a846
  - Merge "Bump min eventlet version to 0.26.1"
  - Bump min eventlet version to 0.26.1
    
    That version have fix for the issue with Python 3.7. See
    [1] for more details.
    
    It is done instead of backporting ryu workaround
    faucetsdn/ryu@ddb32f6
    
    [1] eventlet/eventlet#526
    
    Story: #2009283
    Task: #43563
    Change-Id: I5bb684c75bde1512f379127520b840f55f0e42ab
openstack-mirroring pushed a commit to openstack/os-ken that referenced this issue Nov 4, 2021
That version have fix for the issue with Python 3.7. See
[1] for more details.

It is done instead of backporting ryu workaround
faucetsdn/ryu@ddb32f6

[1] eventlet/eventlet#526

Story: #2009283
Task: #43563
Change-Id: I5bb684c75bde1512f379127520b840f55f0e42ab
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet