Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is there a way to delay container startup to support dependant services with a longer startup time #374

dancrumb opened this issue Aug 4, 2014 · 316 comments


Copy link

dancrumb commented Aug 4, 2014

I have a MySQL container that takes a little time to start up as it needs to import data.

I have an Alfresco container that depends upon the MySQL container.

At the moment, when I use fig, the Alfresco service inside the Alfresco container fails when it attempts to connect to the MySQL container... ostensibly because the MySQL service is not yet listening.

Is there a way to handle this kind of issue in Fig?

Copy link

d11wtq commented Aug 4, 2014

At work we wrap our dependent services in a script that check if the link is up yet. I know one of my colleagues would be interested in this too! Personally I feel it's a container-level concern to wait for services to be available, but I may be wrong :)

Copy link

nubs commented Aug 4, 2014

We do the same thing with wrapping. You can see an example here:

Copy link

bfirsh commented Aug 4, 2014

It'd be handy to have an entrypoint script that loops over all of the links and waits until they're working before starting the command passed to it.

This should be built in to Docker itself, but the solution is a way off. A container shouldn't be considered started until the link it exposes has opened.

Copy link

dancrumb commented Aug 4, 2014

@bfirsh that's more than I was imagining, but would be excellent.

A container shouldn't be considered started until the link it exposes has opened.

I think that's exactly what people need.

For now, I'll be using a variation on

Copy link

silarsis commented Aug 4, 2014

Yeah, I'd be interested in something like this - meant to post about it earlier.

The smallest impact pattern I can think of that would fix this usecase for us would be to be the following:

Add "wait" as a new key in fig.yml, with similar value semantics as link. Docker would treat this as a pre-requisite and wait until this container has exited prior to carrying on.

So, my docker file would look something like:

  image: tutum/mysql:5.6

  build: /path/to/db
    - db:db
  command: /usr/local/bin/init_db

    - db:db
    - initdb

On running app, it will start up all the link containers, then run the wait container and only progress to the actual app container once the wait container (initdb) has exited. initdb would run a script that waits for the database to be available, then runs any initialisations/migrations/whatever, then exits.

That's my thoughts, anyway.

Copy link

dnephin commented Aug 5, 2014

(revised, see below)

Copy link

dsyer commented Aug 14, 2014

+1 here too. It's not very appealing to have to do this in the commands themselves.

Copy link

jcalazan commented Aug 15, 2014

+1 as well. Just ran into this issue. Great tool btw, makes my life so much easier!

Copy link

arruda commented Aug 16, 2014

+1 would be great to have this.

Copy link

prologic commented Aug 19, 2014

+1 also. Recently run into the same set of problems

Copy link

chymian commented Aug 19, 2014

+1 also. any statement from dockerguys?

Copy link

codeitagile commented Aug 22, 2014

I am writing wrapper scripts as entrypoints to synchronise at the moment, not sure if having a mechanism in fig is wise if you have other targets for your containers that perform orchestration a different way. Seems very application specific to me, as such the responsibility of the containers doing the work.

Copy link

prologic commented Aug 22, 2014

After some thought and experimentation I do kind of agree with this.

As such an application I'm building basically has a synchronous
waitfor(host, port) function that lets me waits for services the application
is depending on (either detected via environment or explicitly
configuration via cli options).


James Mills / prologic


On Fri, Aug 22, 2014 at 6:34 PM, Mark Stuart

I am writing wrapper scripts as entrypoints to synchronise at the moment,
not sure if having a mechanism in fig is wise if you have other targets for
your containers that perform orchestration a different way. Seems very
application specific to me as such the responsibility of the containers
doing the work.

Reply to this email directly or view it on GitHub
#374 (comment).

Copy link

shuron commented Aug 31, 2014

Yes some basic "depend's on" neeeded here...
so if you have 20 container, you just wan't to run fig up and everything starts with correct order...
However it also have some timeout option or other failure catching mechanisms

Copy link

ahknight commented Oct 23, 2014

Another +1 here. I have Postgres taking longer than Django to start so the DB isn't there for the migration command without hackery.

Copy link

dnephin commented Oct 23, 2014

@ahknight interesting, why is migration running during run ?

Don't you want to actually run migrate during the build phase? That way you can startup fresh images much faster.

Copy link

ahknight commented Oct 23, 2014

There's a larger startup script for the application in question, alas. For now, we're doing non-DB work first, using nc -w 1 in a loop to wait for the DB, then doing DB actions. It works, but it makes me feel dirty(er).

Copy link

dnephin commented Oct 23, 2014

I've had a lot of success doing this work during the fig build phase. I have one example of this with a django project (still a work in progress through):

No need to poll for startup. Although I've done something similar with mysql, where I did have to poll for startup because the mysqld init script wasn't doing it already. This postgres init script seems to be much better.

Copy link

arruda commented Oct 24, 2014

Here is what I was thinking:

Using the idea of moby/moby#7445 we could implement this "wait_for_helth_check" attribute in fig?
So it would be a fig not a Docker issue?

is there anyway of making fig check the tcp status on the linked container, if so then I think this is the way to go. =)

Copy link

docteurklein commented Nov 10, 2014

@dnephin can you explain a bit more what you're doing in Dockerfiles to help this ?
Isn't the build phase unable to influence the runtime?

Copy link

dnephin commented Nov 10, 2014

@docteurklein I can. I fixed the link from above (

The idea is that you do all the slower "setup" operations during the build, so you don't have to wait for anything during container startup. In the case of a database or search index, you would:

  1. start the service
  2. create the users, databases, tables, and fixture data
  3. shutdown the service

all as a single build step. Later when you fig up the database container it's ready to go basically immediately, and you also get to take advantage of the docker build cache for these slower operations.

Copy link

docteurklein commented Nov 10, 2014

nice! thanks :)

Copy link

arruda commented Nov 11, 2014

@dnephin nice, hadn't thought of that .

Copy link

oskarhane commented Dec 5, 2014

+1 This is definitely needed.
An ugly time delay hack would be enough in most cases, but a real solution would be welcome.

Copy link

dnephin commented Dec 5, 2014

Could you give an example of why/when it's needed?

Copy link

dacort commented Dec 5, 2014

In the use case I have, I have an Elasticsearch server and then an application server that's connecting to Elasticsearch. Elasticsearch takes a few seconds to spin up, so I can't simply do a fig up -d because the application server will fail immediately when connecting to the Elasticsearch server.

Copy link

ddossot commented Dec 5, 2014

Say one container starts MySQL and the other starts an app that needs MySQL and it turns out the other app starts faster. We have transient fig up failures because of that.

Copy link

oskarhane commented Dec 5, 2014

crane has a way around this by letting you create groups that can be started individually. So you can start the MySQL group, wait 5 secs and then start the other stuff that depends on it.
Works in a small scale, but not a real solution.

Copy link

arruda commented Dec 6, 2014

@oskarhane not sure if this "wait 5 secs" helps, in some cases in might need to wait more (or just can't be sure it won't go over the 5 secs)... it's isn't much safe to rely on time waiting.
Also you would have to manually do this waiting and loading the other group, and that's kind of lame, fig should do that for you =/

Copy link

Silex commented Mar 8, 2017

@vladikoff: more info about version 3 at #4305

Basically, it won't be supported, you have to make your containers fault-tolerant instead of relying on docker-compose.

Copy link

shin- commented Mar 21, 2017

I believe this can be closed now.

Copy link

slava-nikulin commented May 11, 2017

Unfortunatelly, condition is not supported anymore in v3. Here is workaround, that I've found:

      - 'postgres'
    build: .
      - '3000'
      - '.:/news_app'
      - 'bundle_data:/bundle'
    entrypoint: ./ postgres 5432

    image: 'postgres:9.6.2'
      - '5432'


shift 2

# wait for the postgres docker to be running
while ! pg_isready -h $postgres_host -p $postgres_port -q -U postgres; do
  >&2 echo "Postgres is unavailable - sleeping"
  sleep 1

>&2 echo "Postgres is up - executing command"

# run the command
exec $cmd

Copy link

riuvshin commented May 11, 2017

@slava-nikulin custom entrypoint is a common practice, it is almost the only (docker native) way how you can define and check all conditions you need before staring your app in a container.

Copy link

mbdas commented May 11, 2017

Copy link

patrickml commented Jun 22, 2017

I was able to do something like this

set -eu

docker volume create --name=gql-sync
echo "Building docker containers"
docker-compose build
echo "Running tests inside docker container"
docker-compose up -d pubsub
docker-compose up -d mongo
docker-compose up -d botms
docker-compose up -d events
docker-compose up -d identity
docker-compose up -d importer
docker-compose run status
docker-compose run testing

exit $?



set -eu pipefail

echo "Attempting to connect to bots"
until $(nc -zv botms 3000); do
    printf '.'
    sleep 5
echo "Attempting to connect to events"
until $(nc -zv events 3000); do
    printf '.'
    sleep 5
echo "Attempting to connect to identity"
until $(nc -zv identity 3000); do
    printf '.'
    sleep 5
echo "Attempting to connect to importer"
until $(nc -zv importer 8080); do
    printf '.'
    sleep 5
echo "Was able to connect to all"

exit 0

// in my docker compose file

    image: yikaus/alpine-bash
      - "./internals/scripts:/scripts"
    command: "sh /scripts/"
      - "mongo"
      - "importer"
      - "events"
      - "identity"
      - "botms"

Copy link

usamaB commented Oct 30, 2017

I have a similar problem but a bit different. I have to wait for MongoDB to start and initialize a replica set.
Im doing all of the procedure in docker. i.e. creating and authentication replica set. But I have another python script in which I have to connect to the primary node of the replica set. I'm getting an error there.

and in the python script im trying to do something like this
for x in range(1, 4): client = MongoClient(host='node' + str(x), port=27017, username='admin', password='password') if client.is_primary: print('the client.address is: ' + str(client.address)) print(dbName) print(collectionName) break

Am having difficulty in doing so, anyone has any idea?

Copy link

chaicode88 commented Apr 26, 2018

@patrickml If I don't use docker compose, How I you do it with Dockerfile?
I need 'cqlsh' to execute my build_all.cql. However, 'cqlsh' is not ready...have to wait for 60 seconds to be ready.

cat Dockerfile

FROM store/datastax/dse-server:5.1.8

USER root

RUN apt-get update
RUN apt-get install -y vim

ADD db-scripts- /docker/cms/

WORKDIR /docker/cms/db-scripts-
RUN cqlsh -f build_all.cql

USER dse


Step 8/9 : RUN cqlsh -f build_all.cql
---> Running in 08c8a854ebf4
Connection error: ('Unable to connect to any servers', {'': error(111, "Tried connecting to [('', 9042)]. Last error: Connection refused")})
The command '/bin/sh -c cqlsh -f build_all.cql' returned a non-zero code: 1

vijayvepa pushed a commit to vijayvepa/AnsibleExamples that referenced this issue Dec 20, 2018
rpdelaney added a commit to CMSgov/easi-app that referenced this issue Apr 9, 2020
A race condition between these two containers was causing the database
to sometimes get cleaned after migrations had been run. Rather than hack
together scripts to track this state I'm simply removing the clean
service from the docker-compose configuration.

See also:
  * docker/compose#374
abitrolly added a commit to abitrolly/ that referenced this issue Jul 29, 2020
Because `docker-compose` is not capable of checking open ports
Copy link

henroFall commented Oct 27, 2020

Requires= var-lib-libvirt.mount var-lib-libvirt-images-ram.mount

Copy link

2kCarlos commented May 24, 2022

In case anyone comes back to this years later, read this wonderful page from the Docker Compose docs for updated info!

Copy link

skorokithakis commented May 24, 2022

There are also health checks built in now, which solve the problem better.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet

No branches or pull requests