Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

supervisord Starts All Processes at the Same Time #122

Open
fredpalmer opened this issue May 30, 2012 · 216 comments
Open

supervisord Starts All Processes at the Same Time #122

fredpalmer opened this issue May 30, 2012 · 216 comments

Comments

@fredpalmer
Copy link

@fredpalmer fredpalmer commented May 30, 2012

My config:

[program:redis-testapp]
command=/opt/bcs/bin/redis-server /apps/testapp/releases/current/environments/all/redis.conf
stdout_logfile=/var/log/redis_testapp_log
stderr_logfile=/var/log/redis_testapp_log
startsecs=30
priority=1
autostart=true
autorestart=true

[program:celerybeat-testapp]
command=python -O manage.py celerybeat --loglevel=INFO --schedule=/apps/testapp/db/celerybeat_schedule_db
stdout_logfile=/var/log/celerybeat_testapp_log
stderr_logfile=/var/log/celerybeat_testapp_log
priority=999
startsecs=5
autostart=true

[program:celery-testapp]
command=python -O manage.py celeryd --loglevel=INFO --events
stdout_logfile=/var/log/celeryd_testapp_log
stderr_logfile=/var/log/celeryd_testapp_log
priority=100
startsecs=10
autostart=true

[program:gunicorn-testapp]
command=gunicorn_django --workers=10 --log-level info --timeout 500 --bind=127.0.0.1:8004
stdout_logfile=/var/log/gunicorn_testapp_log
stderr_logfile=/var/log/gunicorn_testapp_log
priority=999
startsecs=10
autostart=true

[program:memcached-testapp]
command=/opt/bcs/bin/memcached -m 128 -l 127.0.0.1 -p 11212 -u nobody -P /apps/testapp/run/memcached.pid
stdout_logfile=/var/log/memcached_testapp_log
stderr_logfile=/var/log/memcached_testapp_log
priority=11
autostart=true
autorestart=true


My output =>

2012-05-30 22:37:33,181 INFO daemonizing the supervisord process
2012-05-30 22:37:33,182 INFO supervisord started with pid 16230
2012-05-30 22:37:34,195 INFO spawned: 'redis-testapp' with pid 16232
2012-05-30 22:37:34,206 INFO spawned: 'memcached-testapp' with pid 16233
2012-05-30 22:37:34,214 INFO spawned: 'celery-testapp' with pid 16234
2012-05-30 22:37:34,238 INFO spawned: 'celerybeat-testapp' with pid 16235
2012-05-30 22:37:34,477 INFO spawned: 'gunicorn-testapp' with pid 16241
2012-05-30 22:37:35,240 INFO success: memcached-testapp entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2012-05-30 22:37:39,434 INFO success: celerybeat-testapp entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2012-05-30 22:37:44,434 INFO success: celery-testapp entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2012-05-30 22:37:44,435 INFO success: gunicorn-testapp entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2012-05-30 22:38:04,197 INFO success: redis-testapp entered RUNNING state, process has stayed up for > than 30 seconds (startsecs)

What I expected to happen =>

redis would start and supervisord would wait 30 seconds before starting any lower priority processes.

@mnaberez mnaberez closed this May 30, 2012
@disposable-ksa98
Copy link

@disposable-ksa98 disposable-ksa98 commented Jul 2, 2013

I don't see what's the point of using supervisor if it does not get this right.

@mnaberez If there is an open issue for this, you should post the link so we can find it.

@mnaberez mnaberez reopened this Jul 2, 2013
@rca
Copy link

@rca rca commented Jul 9, 2013

I also need the ability to start processes in a particular order.

An event-based approach could work. For example, if I have a [program:agent] and a [program:client] I could subscribe the client to start up when the agent emits a started event. By default, if a program does not subscribe to any event, it will be started when supervisord starts.

@sevastos
Copy link

@sevastos sevastos commented Mar 3, 2014

+1

2 similar comments
@maximilize
Copy link

@maximilize maximilize commented Mar 6, 2014

+1

@tomislacker
Copy link

@tomislacker tomislacker commented Apr 9, 2014

+1

@dmitriy-kiriyenko
Copy link

@dmitriy-kiriyenko dmitriy-kiriyenko commented Apr 21, 2014

+1, this major issue.

@jefferai
Copy link

@jefferai jefferai commented Apr 22, 2014

+1

8 similar comments
@hunterloftis
Copy link

@hunterloftis hunterloftis commented Apr 22, 2014

+1

@orarbel
Copy link

@orarbel orarbel commented May 1, 2014

+1

@abierbaum
Copy link

@abierbaum abierbaum commented May 2, 2014

+1

@gigaroby
Copy link

@gigaroby gigaroby commented May 13, 2014

+1

@KenjiTakahashi
Copy link

@KenjiTakahashi KenjiTakahashi commented May 14, 2014

👍

@karanlyons
Copy link

@karanlyons karanlyons commented May 30, 2014

+1

@voleg
Copy link

@voleg voleg commented Jun 10, 2014

+1

@toxsick
Copy link

@toxsick toxsick commented Jun 10, 2014

+1

@harmy
Copy link

@harmy harmy commented Jun 13, 2014

must have

@xbeta
Copy link

@xbeta xbeta commented Jun 23, 2014

is there a dependency in supervisor? if not, why not?

@analytically
Copy link

@analytically analytically commented Jun 27, 2014

+1

4 similar comments
@devrim
Copy link

@devrim devrim commented Jul 8, 2014

+1

@alp82
Copy link

@alp82 alp82 commented Jul 10, 2014

+1

@mtrienis
Copy link

@mtrienis mtrienis commented Aug 12, 2014

+1

@Pindar
Copy link

@Pindar Pindar commented Sep 2, 2014

+1

@miguno
Copy link

@miguno miguno commented Sep 3, 2014

Not wanting to add to any perceived pressure, but I'd be +1 on the very same feature (process dependencies). :-)

@nand0p
Copy link

@nand0p nand0p commented Sep 4, 2014

this would be very useful and mean less ugly hacks like sleep

@jowenn
Copy link

@jowenn jowenn commented Sep 6, 2014

+1

1 similar comment
@pangon
Copy link

@pangon pangon commented Sep 14, 2014

+1

@ricardosasilva
Copy link

@ricardosasilva ricardosasilva commented Sep 15, 2014

+1. Maybe an option to turn the process loading synchronous.

@pickledgator
Copy link

@pickledgator pickledgator commented Oct 24, 2017

+1

@chezbut
Copy link

@chezbut chezbut commented Dec 6, 2017

+1

@alecbz
Copy link

@alecbz alecbz commented Dec 6, 2017

FYI: replying to an issue just to say '+1' sends a notification to everyone subscribed to the issue. You can just react to the issue's initial comment with the 👍 emoji instead.

plus_1_github

@AndrewSav
Copy link

@AndrewSav AndrewSav commented Dec 6, 2017

@AlecBenzer

FYI: replying to an issue just to say '+1' sends a notification to everyone subscribed to the issue.

But that's kind of the point, is not it? I'm subscribed to this issue, and I'm very interested to see that other people are interested is well. This issue is more than 5 years old. It is a good thing to actually ping the developers to gently remind them that people are still interested. It is also good for those who subscribed, to confirm that there are also others who care, not just that person ;)

FYI: There is an "Unsubscribe" button to the right of the thread, if you do not like to receive the notification from this thread, this is perfectly fine, you can use this button to opt-out.

@LMCom
Copy link

@LMCom LMCom commented Dec 12, 2017

I can confirm, that starting processes in a specific order is an important requirement. Sorry, I was not able to confirm it earlier. Will it be implemented now?

@jkemp101
Copy link

@jkemp101 jkemp101 commented Dec 12, 2017

Sharing my solution to this problem, your mileage may vary. While waiting for this to be fixed I transitioned most of the pieces of our infrastructure relying on this feature to Docker/Kubernetes. The various Docker orchestration frameworks (docker-compose, swarm, Kubernetes, etc) typically have mechanisms to control startup ordering (often with their own set of challenges). So if you are waiting for this to be fixed maybe use it as an excuse to play around with a simple docker-compose project to see if it provides an alternative approach to this problem. I recognize this may only apply to a small fraction of the users waiting for this to be fixed.

@bendikro
Copy link

@bendikro bendikro commented Mar 2, 2018

Hi everyone

I've implemented an event plugin with support for starting services in order specified by dependent services.
Take a look at this example.

https://github.com/bendikro/supervisord-dependent-startup

@CoryPulm
Copy link

@CoryPulm CoryPulm commented Mar 26, 2018

An alternative to @bendikro's awesome plugin could also be the Docker community's wait-for-it.sh script. It's a bit easier to set up if you have a complex automation system managing it, though it isn't as pretty as a native plugin:
https://github.com/vishnubob/wait-for-it

@mtdeguzis
Copy link

@mtdeguzis mtdeguzis commented Nov 5, 2018

Also, you can do a simple wait for certain processes like this:

# Wait until PHP FPM is up to start, since supervisord like to start everything at once...
# See
while [[ $(ps -aux | grep "[p]hp-fpm: master process") == "" ]];
do
    echo "Waiting for PHP FPM to come up..."
    sleep 2s
    if [[ $(ps -aux | grep "[p]hp-fpm: master process") != "" ]]; then
        echo "PHP FPM looks to be started, continuing with Icinga daemon initialization"
        ps -aux | grep "[p]hp-fpm: master process"
    fi
done

I only have a few, so I settled for this method

@DenLilleMand
Copy link

@DenLilleMand DenLilleMand commented Dec 29, 2018

Something i just realized is that, let's say i use supervisor to host my database and my server. The server service has autorestart=true so as long as it fails connecting to the database it is just going to retry right. So it works out.

@CBEPX
Copy link

@CBEPX CBEPX commented Feb 13, 2019

+1

@zgdgod
Copy link

@zgdgod zgdgod commented Aug 5, 2019

+1

@eugene-kulak
Copy link

@eugene-kulak eugene-kulak commented Oct 8, 2019

+1

@mtdeguzis
Copy link

@mtdeguzis mtdeguzis commented Oct 8, 2019

Please click the thumbs up on the original post instead of just commenting "+1". Gets my hopes up when I see a notification :D

@AdiBrucker
Copy link

@AdiBrucker AdiBrucker commented Feb 2, 2020

On option to solve this problem is to define in the supervisor.conf that the program will get autostart value of false for example:

[program:test]
Autostart:” false”
Command=“/home/user/test”

Second step is to use bash script to start the supervisorctl example:

Gnome-terminal -e “supervisord”
Gnome-terminal -e “supervisorctl”
Sleep 10
Gnome-terminal -e “supervisorctl start test”

*Thanks to Avner gidron for giving me the idea

@kumarshorav11
Copy link

@kumarshorav11 kumarshorav11 commented Feb 28, 2020

Hi,
I have below script in the supervisor.cnf file :

[program:zookeeper]
startsecs=60
directory= /app
command=/bin/bash -c "java -jar zoo.jar"
priority=1
autostart = true
autorestart = true

[program:kafka]
startsecs=60
directory= /app
command=/bin/bash -c "java -jar kaf.jar"
stdout_logfile=/var/log/supervisor/%(program_name)s.log
stderr_logfile=/var/log/supervisor/%(program_name)s.log
priority=999
autostart = true
autorestart = true

I want to start first Zookeeper and then Kafka when Zookeeper is up. This script is some times not working as expected.
What would be the way to handle this through supervisor.
Please suggest.

@mtdeguzis
Copy link

@mtdeguzis mtdeguzis commented Mar 2, 2020

@kumarshorav11 it's not pretty, but what I do is monitor for the process to come up first (whatever needs to wait)/. This is an example only.

httpd.conf

[program:httpd]
command=/opt/supervisor/httpd_supervisor
autorestart=true
startretries=3
# Start only after PHP FPM
priority=2
# redirect output to stdout/stderr and do not use a regular logfile
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

/opt/supervisor/httpd_supervisor:

#!/bin/bash

# Wait until PHP FPM is up to start, since supervisord likes to start everything at once...
# See https://github.com/Supervisor/supervisor/issues/122

echo -e "\n==> Starting httpd"
# PHP FPM
while [[ $(ps -aux | grep "[p]hp-fpm: master process") == "" ]];
do
    echo "Waiting for PHP FPM to come up to start httpd..."
    sleep 2s
    if [[ $(ps -aux | grep "[p]hp-fpm: master process") != "" ]]; then
        echo "PHP FPM looks to be started, continuing with Icinga daemon initialization"
        ps -aux | grep "[p]hp-fpm: master process"
    fi
done

# Another project had this, but we won't be using systemd
# service httpd start
/usr/sbin/httpd

# Allow any signal which would kill a process to stop server
trap "service httpd stop" HUP INT QUIT ABRT ALRM TERM TSTP

while pgrep -u apache httpd > /dev/null; do sleep 5; done
@danielmotaleite
Copy link

@danielmotaleite danielmotaleite commented Mar 16, 2020

Another team with this problem, we need to start a apps only after dnsmasq correctly started, if not the app will resolve and cache the wrong DNS

So there is already the https://github.com/jasoncorbett/ordered-startup-supervisord, why this is not merged in the main supervisor, is there any reason for this not to be fixed?

edit: another similar fork https://github.com/bendikro/supervisord-dependent-startup

@mnaberez mnaberez mentioned this issue Apr 9, 2020
0 of 3 tasks complete
@OrangePJ
Copy link

@OrangePJ OrangePJ commented Jun 11, 2020

Any elegant solution?

@mrkeyiano
Copy link

@mrkeyiano mrkeyiano commented Aug 26, 2020

i'm all for this feature

@benma
Copy link

@benma benma commented Aug 26, 2020

It has been fun tracking this issue for all these years, but I finally unsubscribed 😂

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.