"async stop" seems to conflict with the "respawn" feature

[thefab]

During an "async stop" with a big graceful timeout, depending on the context, it seems that some processes can be respawned between the "kill -TERM" and the "kill -KILL".

I'm not completly sure about that but I have some suspicious behaviours with this kind of watcher:

#!/usr/bin/env python

import signal
import random
import sys
import time


def handler(signum, frame):
    value = random.randint(3, 10)
    print "waiting %i seconds..." % value
    time.sleep(value)
    print "exiting"
    sys.exit(0)

signal.signal(signal.SIGTERM, handler)


print "starting"
while True:
    print "start sleep(1)..."
    time.sleep(1)
    print "end sleep(1)"

and with this config file:

[circus]
check_delay = 5
endpoint = tcp://127.0.0.1:5555
pubsub_endpoint = tcp://127.0.0.1:5556
stats_endpoint = tcp://127.0.0.1:5557
httpd = False
debug = False
loglevel = DEBUG

[watcher:dummy]
cmd = dummy.py
args = foo
warmup_delay = 0
numprocesses = 5
graceful_timeout = 30
stdout_stream.class = FileStream
stdout_stream.filename = test1.log
stderr_stream.class = FileStream
stderr_stream.filename = test2.log

I'm not really sure because I am still not comfortable with the circus source but a kind of "stopping flag" seems to be missing between the kill -TERM and the kill -KILL.

This issue blocks #515 for us.

And of course, this issue is not solved by the #528 fix.

Thanks

[almet]

Can you tell me if 7e4e3cf fixes your issue? https://github.com/mozilla-services/circus/tree/fix-529

[thefab]

It sounds good @ametaireau ! We are going to do a "real life" test in a few hours to be sure.

[almet]

Cool, anyway, this will need tests as well :)

[ghost]

It looks like _stopping should be initialized in Watcher.__init__:

Traceback (most recent call last):
  File "/home/synext/lib/python2.7/site-packages/zmq/eventloop/minitornado/ioloop.py", line 799, in _run
    self.callback()
  File "/home/synext/lib/python2.7/site-packages/circus-0.9.3-py2.7.egg/circus/arbiter.py", line 533, in manage_watchers
    watcher.manage_processes()
  File "/home/synext/lib/python2.7/site-packages/circus-0.9.3-py2.7.egg/circus/util.py", line 319, in _log
    return func(self, *args, **kw)
  File "/home/synext/lib/python2.7/site-packages/circus-0.9.3-py2.7.egg/circus/watcher.py", line 438, in manage_processes
    and not self._stopping):
AttributeError: 'Watcher' object has no attribute '_stopping'

[tarekziade]

Good catch, thanks. We will add a test that covers this

[almet]

Fixed in the branch. https://github.com/mozilla-services/circus/tree/fix-529

[tarekziade]

added PR at #536 - LGTM. @ametaireau I will let you merge

[almet]

Okay, adding a test before that.

[almet]

should be fixed.