Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Random crash on Windows when gevent.idle() and threadpool used #1489

Closed
HelloZeroNet opened this issue Dec 7, 2019 · 4 comments · Fixed by #1492
Closed

Random crash on Windows when gevent.idle() and threadpool used #1489

HelloZeroNet opened this issue Dec 7, 2019 · 4 comments · Fixed by #1492

Comments

@HelloZeroNet
Copy link

@HelloZeroNet HelloZeroNet commented Dec 7, 2019

gevent version: 1.4.0
Python version: cPython 3.7.3 downloaded from python.org
Operating System: Win10

Description:

The code bellow crashes on every second run.

Output:

Run #0 threadpool object num: 12
Run #1 threadpool object num: 13
Run #2 threadpool object num: 13
Run #3 threadpool object num: 14
Run #4 threadpool object num: 16
Run #5 threadpool object num: 16
Run #6 threadpool object num: 16
Run #7 threadpool object num: 18
Run #8 threadpool object num: 22
[Finished in 0.6s with exit code 3221226356]

What I've run:

import time
import gevent
import gevent.monkey
import gevent.threadpool
import sqlite3
import shutil
gevent.monkey.patch_all(thread=False, threading=False)
import threading

def worker():
    db = sqlite3.connect("test.db")
    db.close()
    return "worker done"

def test():
    pool = gevent.get_hub().threadpool
    gevent.idle()
    for i in range(20):
        pool.spawn(worker)
    assert pool.apply(worker) == "worker done"

time.sleep(0.1)
import gc
for i in range(100):
    test()
    pools = [obj for obj in gc.get_objects() if "threadpool" in str(type(obj))]
    print("Run #%s threadpool object num: %s" % (i, len(pools)))

print("Done.")

Update:
Adding hub = gevent.hub.Hub() into the test function makes it crash with similar success rate even without gevent.idle(). In this case I got an error message:

Fatal Python error: ffi.from_handle() detected that the address passed points to garbage. If it is really the result of ffi.new_handle(), then the Python object has already been garbage collected

Thread 0x00000e34 (most recent call first):
  File "E:\Web\Today\test2.py", line 11 in worker
  File "C:\python3\lib\site-packages\gevent\threadpool.py", line 281 in _worker

Thread 0x00004030 (most recent call first):
  File "C:\python3\lib\site-packages\gevent\_threading.py", line 84 in wait
  File "C:\python3\lib\site-packages\gevent\_threading.py", line 166 in get
  File "C:\python3\lib\site-packages\gevent\threadpool.py", line 270 in _worker

…

Current thread 0x00004b60 (most recent call first):
  File "C:\python3\lib\site-packages\gevent\libuv\loop.py", line 32 in _find_loop_from_c_watcher
  File "C:\python3\lib\site-packages\gevent\_ffi\loop.py", line 261 in python_prepare_callback
  File "C:\python3\lib\site-packages\gevent\libuv\loop.py", line 473 in run
  File "C:\python3\lib\site-packages\gevent\hub.py", line 582 in run
[Finished in 1.2s with exit code 3221226505]

Pretty edge case, but may have same connection with the original error.

@HelloZeroNet

This comment has been minimized.

Copy link
Author

@HelloZeroNet HelloZeroNet commented Dec 10, 2019

Running in pytest it displays the exception on crash:


$ py -3 -m pytest test_gevent_crash.py --capture=no
================================================================== test session starts ================================================================== 
platform win32 -- Python 3.7.3, pytest-5.0.0, py-1.7.0, pluggy-0.13.1                                                                                     
rootdir: F:\2019-12-10                                                                                                                                    
collecting ... Run #0 threadpool object num: 14                                                                                                           
Run #1 threadpool object num: 16                                                                                                                          
Run #2 threadpool object num: 16                                                                                                                          
Windows fatal exception: access violation                                                                                                                 
                                                                                                                                                          
Thread 0x000022d8 (most recent call first):                                                                                                               
  File "C:\Python3\lib\site-packages\gevent\_threading.py", line 84 in wait                                                                               
  File "C:\Python3\lib\site-packages\gevent\_threading.py", line 166 in get                                                                               
  File "C:\Python3\lib\site-packages\gevent\threadpool.py", line 270 in _worker                                                                           
                                                                                                                                                          
...                                                                           
                                                                                                                                                          
Thread 0x000017cc (most recent call first):                                                                                                               
  File "C:\Python3\lib\site-packages\gevent\_threading.py", line 84 in wait                                                                               
  File "C:\Python3\lib\site-packages\gevent\_threading.py", line 166 in get                                                                               
  File "C:\Python3\lib\site-packages\gevent\threadpool.py", line 270 in _worker                                                                           
                                                                                                                                                          
Current thread 0x000026d4 (most recent call first):                                                                                                       
  File "C:\Python3\lib\site-packages\gevent\libuv\loop.py", line 473 in run                                                                               
  File "C:\Python3\lib\site-packages\gevent\hub.py", line 582 in run                                                                                      

What could be the environmental differences when running it with pytest that makes it display the exception that is not visible when running it directly?

Changing the event loop to libev by adding gevent.config.loop = "libev-cext" after gevent import seem like fixes the crash.

@jamadden

This comment has been minimized.

Copy link
Member

@jamadden jamadden commented Dec 10, 2019

Thanks, I'm able to reproduce this on other platforms as well.

jamadden added a commit that referenced this issue Dec 11, 2019
Fixes #1489.
@HelloZeroNet

This comment has been minimized.

Copy link
Author

@HelloZeroNet HelloZeroNet commented Dec 12, 2019

Thanks for the quick fix, I the patch looks like fixed the error, but came with an assert error:

Traceback (most recent call last):
  File "F:\Today\test_assertfail.py", line 16, in <module>
    evt = pool1.spawn(getevt).get()
  File "src\gevent\event.py", line 279, in gevent._event.AsyncResult.get
  File "src\gevent\event.py", line 307, in gevent._event.AsyncResult.get
  File "src\gevent\event.py", line 297, in gevent._event.AsyncResult.get
  File "src\gevent\event.py", line 277, in gevent._event.AsyncResult._raise_exception
  File "F:\Today\gevent\_compat.py", line 62, in reraise
    raise value.with_traceback(tb)
  File "F:\Today\gevent\threadpool.py", line 285, in _worker
    value = func(*args, **kwargs)
  File "F:\Today\test_assertfail.py", line 12, in getevt
    return pool2.spawn(subtester)
  File "F:\Today\gevent\threadpool.py", line 225, in spawn
    assert self.hub == get_hub(), "%s != %s" % (self.hub, get_hub())
AssertionError: <Hub '' at 0x3b201b8 default default pending=0 ref=0 thread_ident=0x1888> != <Hub '' at 0x3b20270 default pending=0 ref=0 thread_ident=0x1e60>
import time
import gevent
import gevent.monkey
import gevent.threadpool
gevent.monkey.patch_all(thread=False, threading=False)
import threading

def subtester():
    return "hello"

def getevt():
    return pool2.spawn(subtester)

pool2 = gevent.threadpool.ThreadPool(5)
pool1 = gevent.threadpool.ThreadPool(5)
evt = pool1.spawn(getevt).get()
print(evt.get())

Update: Changed the script that more close to my actual use-case that worked before

@jamadden

This comment has been minimized.

Copy link
Member

@jamadden jamadden commented Dec 12, 2019

Sorry, it's never been allowed to create a ThreadPool using one hub and then call spawn() on it from another hub. Depending on the version of gevent, you'll get different errors, and there's even the possibility of crashing the process. The ThreadPool uses the hub it's created with for async watchers and semaphores, and both of those are both tied to the hub and not safe to use from multiple threads (with the exception of the one method of an async watcher that can be called from a different thread after the async watcher is started).

The original example:

def subtester():
    return "hello"

def tester():
    return pool2.spawn(subtester).get()

pool2 = gevent.threadpool.ThreadPool(5)
pool1 = gevent.threadpool.ThreadPool(5)
for _ in range(5000):
    print(pool1.spawn(tester).get())
    time.sleep(1)

Will raise LoopExit in gevent 1.2.2 and above.

The modified example:

def subtester():
    return "hello"

def getevt():
    return pool2.spawn(subtester)

pool2 = gevent.threadpool.ThreadPool(5)
pool1 = gevent.threadpool.ThreadPool(5)
evt = pool1.spawn(getevt).get()
print(evt.get())

will raise greenlet.error: cannot switch to a different thread in gevent 1.2 and 1.3. It will appear to work in gevent 1.4, but that's just because we're staying well below the limits. If we tweak it slightly to hit those limits:

def getevt():
    pool2.spawn(subtester)
    pool2.spawn(subtester)
    pool2.spawn(subtester)

pool2 = gevent.threadpool.ThreadPool(1)

We find LoopExit being raised.

I agree this could use clarification in the documentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.