Circusctl hangs. #962

Open
kramarz opened this Issue Feb 12, 2016 · 18 comments

Projects

None yet

5 participants

@kramarz
kramarz commented Feb 12, 2016

We have updated circus from 0.12.1 to 0.13.0
In our deploy script we use comand "circusctl stop <watcher_name>" Sometimes it works, but sometimes it displays "ok", and than hangs forever.
It never happened before.

@k4nar
Contributor
k4nar commented Feb 12, 2016

Thanks for the report. I just took a look at all the changes and I don't see anything obvious.

I'll try to reproduce this when I have some time. Do you have a simple failing scenario in order to help narrowing this down?

@kramarz
kramarz commented Feb 12, 2016

OK. I managed to reproduce it in small setup and tested it few times with circus 0.13.0 and 0.12.1 It never hanged on older version and hangs in 50% cases in new. I believe it has nothing to do with this bug but during tests I was using docker 1.8.3
https://www.dropbox.com/s/5n8j3sz9gxcrg67/minimal_circus.tar.gz?dl=0

@k4nar
Contributor
k4nar commented Feb 12, 2016

Thank your for your testcase. I'm reproducing using your script, but not using the same ini file on my system. I don't know what's going on.

@k4nar
Contributor
k4nar commented Feb 12, 2016

Same behavior (works in my env, not in docker) with a .ini as simple as:

[watcher:test]
cmd = sleep
args = 100

Maybe it has something to do with docker, idk.

@k4nar
Contributor
k4nar commented Feb 12, 2016

Still reproducing in Docker with the following script:

#!/bin/bash                                                                                                                             
docker build -t minimal_circus .
docker run -d --name=test_daemon minimal_circus
sleep 1
docker exec test_daemon circusctl stop test
docker rm -f test_daemon
@kramarz
kramarz commented Feb 15, 2016

When I replaced in requirements circus==0.13.0 with
-e git+https://github.com/circus-tent/circus.git@af14fc91c2212a2300d86144d075e3219358f6a7#egg=circus
or even
-e git+https://github.com/circus-tent/circus.git#egg=circus
everything works. Maybe you released some development code to pypi?

@k4nar
Contributor
k4nar commented Feb 15, 2016

Hum that's weird, I just checked the package on PyPI and it seems to be ok.

@kramarz
kramarz commented Feb 15, 2016

Ok, I have checked it to and it is the same. Only conclusion is that installing circus from wheel causes this problem.

@k4nar
Contributor
k4nar commented Feb 15, 2016

IIRC I just ran "python setup.py sdist bdist_wheel upload" to push the 0.13.0 on PyPI. I don't really see what could have gone wrong…

@kramarz
kramarz commented Feb 15, 2016

I have found workaround. Executing

docker run --rm --net container:test_daemon minimal_circus python -m circus.circusctl stop test

works every time. I don't really know what is the problem, but wheel adds different wrapper than instaling from source and it causes the circusctl to hang.
Using wheel (pip install circus==0.13.0):

$ docker run --rm minimal_circus which circusctl
/usr/local/bin/circusctl
$ docker run --rm minimal_circus cat /usr/local/bin/circusctl
#!/usr/local/bin/python3.4

# -*- coding: utf-8 -*-
import re
import sys

from circus.circusctl import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

Using source (pip install -e git+https://github.com/circus-tent/circus.git#egg=circus):

$ docker run --rm minimal_circus which circusctl
/usr/local/bin/circusctl
$ docker run --rm minimal_circus cat /usr/local/bin/circusctl
#!/usr/local/bin/python3.4
# EASY-INSTALL-ENTRY-SCRIPT: 'circus==0.13.1','console_scripts','circusctl'
__requires__ = 'circus==0.13.1'
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.exit(
        load_entry_point('circus==0.13.1', 'console_scripts', 'circusctl')()
    )
@kramarz
kramarz commented Mar 25, 2016

I have reproduced it without docker as well - using virtualenv. I have run:

mkvirtualenv --python=/usr/bin/python3 circusctl-test
pip install circus==0.13.0
echo "[watcher:test]
cmd = sleep
args = 100
[circus]
endpoint = ipc:///tmp/circus-endpoint
" > circus.ini
circusd circus.ini &

And then run few times.

circusctl --endpoint ipc:///tmp/circus-endpoint stop test
@takeontom

Hello,

I'm having the same issue with 0.13.0 circus installed into a virtualenv using pip. Identical behaviour in both our development and live environments, both running Ubuntu 14.04.

Is there any information I can give to help?

I'm new to circus, is it pretty safe to downgrade to 0.12.1 until this problems sorted?

@takeontom

If it helps, I've just downgraded to 0.12.1 via pip and am still seeing the same issue. Looks like either a break in a dependency or the way PyPi is installing?

Again, tell me what information I can give to help.

For now I've got 0.13.0 working fine by following the suggestion from @kramarz to install circus from GitHub rather than PyPi:

-e git+https://github.com/circus-tent/circus.git@af14fc91c2212a2300d86144d075e3219358f6a7#egg=circus
@takeontom

Is there any movement on this?

Other than using a "-e" entry in requirements.txt, is there any other workaround?

@kramarz
kramarz commented May 24, 2016

Maybe something like this will work for you:

cat <<_END >`which circusctl`
#!/bin/bash
python -m circus.circusctl \$@
_END
@takeontom

Works a treat, thanks.

@Telofy
Telofy commented May 26, 2016

In my Buildout the Python and the paths depend on the Buildout environment, so I’m using this wrapper now:

#!/bin/bash
endpoint=$(grep "^endpoint" circus.conf | sed "s/^.* = \(.*\)$/\1/g")
timeout 10s bin/circusctl --endpoint $endpoint $*

It also extracts the endpoint from the config file and adds it on the command line, so circusctl can connect to it. (Btw., it’d be great if circusctl could itself read the config file.)

@k4nar k4nar added the True Bug™ label Jun 11, 2016
@j00bar
j00bar commented Nov 7, 2016

Under docker, circus would be running as PID 1, which Linux expects to respond differently to various signals. Have you tried running circus using something like Yelp's dumb-init as PID 1?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment