Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

need patch queue.SimpleQueue for py37 #1248

Closed
wwqgtxx opened this issue Jun 30, 2018 · 3 comments · Fixed by #1253
Closed

need patch queue.SimpleQueue for py37 #1248

wwqgtxx opened this issue Jun 30, 2018 · 3 comments · Fixed by #1253

Comments

@wwqgtxx
Copy link

wwqgtxx commented Jun 30, 2018

  • gevent version: gevent==1.3.4
  • Python version: cPython3.7.0 downloaded from python.org
  • Operating System: win10 1709

Description:

The python37 default SimpleQueue was implemented with C code, which will cause hang the main thread.
A simple way to solve this problem like that

from gevent import monkey
monkey.patch_all()
import queue
queue.SimpleQueue = queue._PySimpleQueue

but I think this code is so ugly and may be have better way to implements in gevent.

@wwqgtxx
Copy link
Author

wwqgtxx commented Oct 17, 2018

Another interesting question I want to know is why not patch the whole queue module by gevent.queue

@jamadden
Copy link
Member

There are native thread use cases for the original objects.

@wwqgtxx
Copy link
Author

wwqgtxx commented Oct 17, 2018

In my test, combine use the native thread and the monkey patched standard library sometimes will cause deadlock but I can't confirm when and where cause that. So I rewrite that code by use multiprocessing but gevent will block in some multiprocessing's native code too. Could gevent provide a document to says how to use native thread with gevent correctly and the compatibility with standard library to avoid the users writing the wrong code.

openstack-gerrit pushed a commit to openstack/cyborg that referenced this issue Mar 5, 2020
py3.7 deadlock with monkey patch of stdlib thread modules + use of
ThreadPoolExecutor

3.7, ThreadPoolExecuter was changed to use queue.SimpleQueue; on 3.6 it
uses queue.Queue. No issue in 3.6

The python37 default SimpleQueue was implemented with C code, which will
cause hang the main thread.

Commit 2a71b8e introduce eventlet for unittest, but I did not find
why we need it.
Check other project they also use ThreadPoolExecutor, but not all
of them introduce eventlet.

Maybe GreenThreadPoolExecutor is another solution.

Ref:
ThreadPoolExecutor
https://docs.python.org/3.7/library/concurrent.futures.html#threadpoolexecutor

Same isses about ThreadPoolExecutor and eventlet
https://bugs.launchpad.net/designate/+bug/1782647
eventlet/eventlet#508
gevent/gevent#1253
ClericPy/torequests#10
https://bugs.python.org/issue34173
gevent/gevent#1248
gevent/gevent#1251
http://lists.openstack.org/pipermail/openstack-dev/2018-July/132473.html
https://www.gitmemory.com/issue/eventlet/eventlet/508/511101004

Change-Id: I263f7222119d8d0c89cfc2a4758fe376ce1afd60
openstack-gerrit pushed a commit to openstack/openstack that referenced this issue Mar 5, 2020
* Update cyborg from branch 'master'
  - Merge "change default SimpleQueue to _PySimpleQueue for queue"
  - change default SimpleQueue to _PySimpleQueue for queue
    
    py3.7 deadlock with monkey patch of stdlib thread modules + use of
    ThreadPoolExecutor
    
    3.7, ThreadPoolExecuter was changed to use queue.SimpleQueue; on 3.6 it
    uses queue.Queue. No issue in 3.6
    
    The python37 default SimpleQueue was implemented with C code, which will
    cause hang the main thread.
    
    Commit 2a71b8e introduce eventlet for unittest, but I did not find
    why we need it.
    Check other project they also use ThreadPoolExecutor, but not all
    of them introduce eventlet.
    
    Maybe GreenThreadPoolExecutor is another solution.
    
    Ref:
    ThreadPoolExecutor
    https://docs.python.org/3.7/library/concurrent.futures.html#threadpoolexecutor
    
    Same isses about ThreadPoolExecutor and eventlet
    https://bugs.launchpad.net/designate/+bug/1782647
    eventlet/eventlet#508
    gevent/gevent#1253
    ClericPy/torequests#10
    https://bugs.python.org/issue34173
    gevent/gevent#1248
    gevent/gevent#1251
    http://lists.openstack.org/pipermail/openstack-dev/2018-July/132473.html
    https://www.gitmemory.com/issue/eventlet/eventlet/508/511101004
    
    Change-Id: I263f7222119d8d0c89cfc2a4758fe376ce1afd60
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants