gevent breaks try-finally semantics where greenlet doesn't

[erikkaplun]

Greenlets in greenlet always get their finally block invoked for example when a KeyboardInterrupt occurs. To reproduce:

import time, greenlet
def bar():
    try: greenlet.getcurrent().parent.switch()
    finally: print "CLEANUP BAR"
def foo():
    try: time.sleep(1.0)
    finally: print "CLEANUP FOO"
b1 = greenlet.greenlet(bar)
b1.switch()
b2 = greenlet.greenlet(bar)
b2.switch()
greenlet.greenlet(foo).switch()

...2 x CLEANUP BAR + 1 x CLEANUP FOO gets printed when Ctrl-C is pressed during that time.sleep(1.0) call.

However, when using gevent (unless time.sleep is used, for obvious reasons), no cleanup code gets invoked whatsoever when Ctrl-C is pressed:

from gevent import spawn, sleep
from gevent.event import Event
def bar():
    try: Event().wait()  # just block and switch back to main
    finally: print "CLEANUP BAR"
def foo():
    try: sleep(1.0)
    finally: print "CLEANUP FOO"  # this does get printed for obvious reasons if time.sleep is used instead
gevent.joinall([spawn(bar), spawn(bar), spawn(foo)])

...neither CLEANUP FOO nor CLEANUP BAR gets printed.

This is both weird, because greenlet already handles this automatically, as well as inconsistent with standard Python semantics, which I suppose should always be sticked to. Violating fundamental semantic guarantees of the language will eventually cause confusion and problems, or at least inconvenience, such as having to manually keep track of all spawned greenlets and to kill them in a global finally cleanup block, or in a signal handler—i.e. stuff that should be expected from gevent.

(I've run the above code on OS X 10.7 and 64bit Python 2.7)

P.S. Also, an except KeyboardInterrupt block never gets invoked either.

[denik]

If I run your script with 1.0 I get:

~/work/gevent $ PYTHONPATH=. python issue250.py 
CLEANUP FOO
Traceback (most recent call last):
  File "issue250.py", line 10, in <module>
    gevent.joinall([spawn(bar), spawn(bar), spawn(foo)])
  File "/home/denis/work/gevent/gevent/greenlet.py", line 401, in joinall
    wait(greenlets, timeout=timeout)
  File "/home/denis/work/gevent/gevent/hub.py", line 616, in wait
    return list(iwait(objects, timeout))
  File "/home/denis/work/gevent/gevent/hub.py", line 594, in iwait
    item = waiter.get()
  File "/home/denis/work/gevent/gevent/hub.py", line 568, in get
    return self.hub.switch()
  File "/home/denis/work/gevent/gevent/hub.py", line 331, in switch
    return greenlet.switch(self)
gevent.hub.LoopExit: This operation would block forever

which is what I'd expect.

[erikkaplun]

I've tried looking into the current source code to see if gevent overrides the SIGINT handler, but at least grepping reveals nothing useful.

[erikkaplun]

@denik did you press Ctrl-C while the program was still running, or you just let the program die a natural death?

...also, even then my point holds—CLEANUP BAR never gets printed.

[denik]

I did not press Ctrl-C. If I did, no CLEANUPs are printed, but that's also expected, because KeyboardInterrupt is only raised in the current greenlet, not in all that are currently running.

[erikkaplun]

Why did you close the issue? I've experimentally demonstrated with reproduction code that the behaviour of gevent differs from greenlet and regular Python.

Can you at least explain why one should actually expect or at least accept this behavior? Or why it shouldn't or can't be fixed to match the behavior of greenlet and pure Python?

[denik]

Think of Greenlet as a daemon thread. The KeyboardInterrupt only kills the current greenlet/thread/generator. If the process is sleeping (polling), then KeyboardInterrupt is raised in the main greenlet.

If you have other greenlet spawned and you want them to clean up, you have to catch KeyboardInterrupt in your main greenlet and kill all those greenlets yourself.

This is consistent with how threads behave. Try interrupting this program with Ctrl-C:

~/work/gevent $ cat t.py 
import sys
import thread
import time


def bar():
    sys.stderr.write('starting bar\n')
    try:
        time.sleep(5)
    finally:
        sys.stderr.write('CLEANUP BAR\n')


t = thread.start_new_thread(bar, ())
time.sleep(7)

~/work/gevent $ python t.py 
starting bar
^CTraceback (most recent call last):
  File "t.py", line 15, in <module>
    time.sleep(7)
KeyboardInterrupt

[erikkaplun]

But greenlet already does do proper cleanup—why couldn't gevent do the same if it's implemented on top of greenlet? Unless the behavior of threads (and the current behavior of gevent) is to be considered as advantageous.

The way I see it, even though the threading argument makes sense, at least from a factual point of view, having finally blocks called should be a trait that is wanted of a concurrency library. In the case of threads, extra care needs to be taken to ensure proper clean up of resources—this cannot in any way be viewed as something to be admired.

I understand and perfectly accept that you personally are not willing or motivated to look into this, but if somebody else is, I don't see why you should have anything against this being improved—i.e. why just close the issue immediately...

[denik]

On Tue, Feb 12, 2013 at 8:16 PM, Erik Allik notifications@github.comwrote:

But greenlet already does do proper cleanup—why couldn't gevent do the
same if it's implemented on top of greenlet? Unless the behavior of threads
is to be considered as advantageous.

Your greenlet example only works because your explicitly switch into them.
If you don't switch() into greenlet it won't do any cleanup.

The way I see it, even though the threading argument makes sense, at least
from a factual point of view, having finally blocks called should be a
trait that is wanted of a concurrency library. In the case of threads,
extra care needs to be taken to ensure proper clean up of resources—this
cannot in any way be viewed as something to be liked.

You can always handle KeyboardInterrupt yourself and then call
gevent.wait() repeatedly.

[erikkaplun]

In the pure Greenlet example, I explictly switch into the greenlets because otherwise the try block is not reached—this is not in any way related to the question at hand. And at the time Ctrl-C is pressed, the other 2 greenlets are "suspended". When the program exits, their finally-block gets invoked, even though the KeyboardInterrupt happened inside another greenlet.

The fact that I .switch() into them is thus in no way relevant—although from a purely factual point of view, of course the finally block never gets invoked if switch() is never called and try is never even reached in the first place.

Also, in gevent, that switch() to bar() greenlets is made behind the scenes just as well, otherwise bar() greenlets wouldn't even start, which they factually indeed do (and wouldn't otherwise make any sense anyway). It's just that in gevent this switch() is done by the gevent scheduler/hub. And of course since you wrote gevent, you know this better than myself.

So at the time of the Ctrl-C, both in the case of Greenlet as well as in the case of gevent, there is a bunch of suspended greenlets, and one active greenlet (Hub in gevent, and foo in the greenlet example). So the program states we are comparing here are equivalent. The only difference is that in Greenlet, Ctrl-C triggers a kill/cleanup of all alive greenlets, whereas the internal logic of gevent has somehow caused this behavior to have diverged from that of pure Greenlet.

[denik]

The only difference is that in Greenlet, Ctrl-C triggers a kill/cleanup of all alive greenlets, whereas the internal logic of gevent has somehow caused this behavior to have diverged from that of pure Greenlet.

this is false. KeyboardInterrupt (like any other exception) is propagated to greenlet's parent. RTFM.

[erikkaplun]

In the Greenlet example, the exception happens inside of foo, which is NOT the parent of bar. However, bar gets cleaned up just as well. So considering this, what TFM says about exception propagation has nothing to do with the clean up of greenlets elsewhere in the hierarchy.

[erikkaplun]

To clarify, when I said "Ctrl-C triggers", it should have been kind of obvious that what was meant wasn't that "Ctrl-C is directly handled by a routine that kills all greenlets" but that as a result of Ctrl-C, greenlets are garbage collected and get a change to do final clean up (or in the reverse order).

Talking of which—is it possible that this is a garbage collection issue instead? finally blocks in greenlets are invoked in Greenlet whenever the greenlet goes out of scope and is garbage collected. In gevent on the other hand, all spawned greenlets are referenced by the Hub, potentially keeping them from being garbage collected—is this a possible explanation?

[erikkaplun]

@denik I understand that you are probably too busy with work and other duties, so I'm not going to waste your time to convince you if you'd rather prefer to stay true to your opinion. However, can I just ask your expertise on what might be causing this discrepancy between Greenlet and gevent. Hopefully my last 2 comments were sufficient to demonstrate that there factually is a discrepancy in the semantics/behavior. (I also attempted to confirm my GC theory but I'm not sure if the experiment was set up badly or the result was that the theory doesn't hold)

Perhaps I could also kindly request you to at least re-open the issue so that if I myself or somebody else finds a way to improve this while in no way affecting stability and performance, it would still be possible to do so. Although of course pull requests can be sent at any time, but having the issue as "closed" feels like "no I will not accept a patch even if you provide one".