Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Considering acquiring a lock by blocking or nonblocking, call greenlet switch once. #1464
Recently, I found the following code doesn't run well with gevent monkey patch.
import gevent.monkey; gevent.monkey.patch_all() import time import threading mutex = threading.Lock() g = 0 def f(): global g if g != 0: return while True: if not mutex.acquire(blocking=False): # the greenlet never yield at here. if g != 0: return else: continue if g != 0: return time.sleep(0.0001) g += 1 mutex.release() return tasks =  for i in range(2): tasks.append(threading.Thread(target=f)) for t in tasks: t.start() for t in tasks: t.join()
The process never exits. I am wondering why this happended and found the answer at
# _simaphore.py:125 if not blocking: return False # do not switch greenlet if not blocking success = self._wait(timeout)
If acquiring a lock by nonblocking mode, the function just return, so the loop never stops.
if not blocking: success = self._wait(0) else: success = self._wait(timeout)
The problem solved. So I wondering if we could switch greenlet even if took
I've been thinking about this quite a bit.
The example code is using busy waiting (a spin lock) --- constantly testing whether it can acquire the lock in a tight loop. Spin locks are useful for avoiding the overhead of context switching if the lock is mostly un-contended and/or the lock is held for very short periods of time (less than the time it takes to context switch).
If the lock is contended, they only work if the holder of the lock can make forward progress and release the lock so that the waiter can acquire it. That's simplest with true native threads running in parallel at the operating system level, but it can be accomplished in other ways (as we'll see).
In CPython 3 (and I think PyPy3), acquiring a lock with
On CPython 2 (and I think PyPy2), acquiring a lock with
It looks a bit less silly, especially on Python 3, if the loop does more than just spin tightly. Imagine that it wants to publish results as it computes them, but it can do so in batches and can buffer the results between iterations:
def worker(work_items, result_lock, result_destination): results =  for item in work_items: results.append(process_item(item)) if result_lock.acquire(blocking=False): result_destination.publish(results) result_lock.release() del results[:] # Anything left goes now if results: with result_lock: result_destination.publish(results)
In this way, it gets to put its whole quantum to beneficial use.
Under (monkey-patched) gevent, as noted here, the tight spin loop spins forever because the holder never has a chance to run to release the lock. The more complex example could easily wind up with one worker in
One of gevent's benefits (when not monkey-patched) is that without arbitrary preemption (the purely cooperative model) it's easier to reason about code and concurrency. Monkey-patching does give that up to an extent, though.
So with all of that said, I think a good path forward that keeps the best of both worlds is:
I'll work on that.