introduce up --wait condition #8777

ndeloof · 2021-10-11T15:54:45Z

What I did
introduced compose up --wait to run detached BUT wait for services to reach state running or healthy (for those with a Healthcheck defined).

Typical usage is to have some backend services (let's say postgres database) ran by compose, and user need to wait for service to be actually up so he can apply database migrations and start an happy coding day.

services:
  db:
    image: postgres:alpine
    ports: ...
    healthcheck:
      test: ['CMD', 'pg_isready']

$ docker compose up -d --wait
(...)
$ rake db:migrate

Related issue
#8351

(not mandatory) A picture of a cute animal, if possible in relation with what you did

cmd/compose/up.go

thaJeztah · 2021-10-12T07:55:10Z

Wondering why this should not be the default for -d / --detach; -d would mean that you want the stack/project to be running in the background, which shouldn't be mutually exclusive with "keep the cli running until it reaches that state", similar to how docker service create waits for the service to reconcile its desired state. One thing that may changing in that case is a way to detach from the client (so that deploying the stack can continue).

ndeloof · 2021-10-12T08:03:43Z

Wondering why this should not be the default for -d

we hardly can change the existing behavior

thaJeztah · 2021-10-12T08:07:00Z

So, it currently already waits for things to be created, and started. Are there any cases where the up action to be considered "complete" until services have become healthy? (curious)

Looking at available options; what does ServiceConditionStarted do? (isn't that the same as the default?)

ndeloof · 2021-10-12T08:12:38Z

"service_started" indeed is the default condition.

Are there any cases where the up action to be considered "complete" until services have become healthy? (curious)

First, service might not define a healthcheck. Other than that, this is just how things used to be, aka "Legacy"

mik-laj · 2021-10-13T18:14:56Z

Do we need any unit tests, integration tests, E2E tests?

thaJeztah · 2021-10-14T21:15:11Z

service might not define a healthcheck

For that case, we should consider "running" to be the equivalent of "healthcheck==ok". Healthcheck during startup is just a more customized "readiness" check. If there's no customized one, then "container running" means it's ready.

Curious; what's the current behavior in this PR if I would pass a --wait on a service that doesn't have a healthcheck?

ndeloof · 2021-10-15T07:43:46Z

I don't think we should assume healthy == running, better detect container has no healthcheck configuration (by ContainerInspect.Config.Healthcheck) and report an error

ndeloof · 2021-10-22T07:47:05Z

we actually already return error when service/docker image doesn't define a healthcheck

ndeloof · 2021-10-26T17:41:00Z

@ulyssessouza any thoughts?

cmd/compose/up.go

pkg/compose/start.go

cmd/compose/up.go

pkg/api/api.go

pkg/compose/start.go

thaJeztah · 2021-10-27T07:27:23Z

I don't think we should assume healthy == running

The point is that the general design for healthchecks was to (during startup) be a readiness check; in that design "no healthcheck" implicitly means "no readiness checck, so running == "healthy"

better detect container has no healthcheck configuration (by ContainerInspect.Config.Healthcheck) and report an error

So, what are the other options?

started is the default
healthy will error if the service doesn't have a healthcheck
completed_successfully ??? not sure what this does; wait for the container to exit?

I'm personally still in favor of making this part of the standard behavior; we'e already printing the steps taken to bring up the stack, so it would be one more output to that list

creating network
creating volumes
start services
wait for services to be running
(*) waiting for services to be healthy

The UX would be more logical for docker compose up to stay attached until the project is deployed (similar to docker service create). After all; the project won't be functional until it's healthy?

ndeloof · 2021-10-27T07:35:44Z

The point is that the general design for healthchecks was to (during startup) be a readiness check; in that design "no healthcheck" implicitly means "no readiness checck, so running == "healthy"

There's indeed no distinction in docker between "ready" and "healthy", but I think we can assume actual usages based on this. Better reject a --wait when no healthcheck is set than pretending to be clever.

So, what are the other options?
started is the default

Then this flag is not needed. compose up -d already exit after all containers have been started

healthy will error if the service doesn't have a healthcheck

this is what's implemented here

completed_successfully ??? not sure what this does; wait for the container to exit?

yep. Basically, init container.

I'm personally still in favor of making this part of the standard behavio

This is all about backward compatibilty, can't just change the behavior for a command used by thousands (millions?) because is looks niceer

thaJeztah · 2021-10-27T07:57:03Z

we can assume actual usages based on this.

I'm not sure what you mean with this "assume actual usages"

than pretending to be clever.
..
yep. Basically, init container.

For both cases, the --wait would be for the service to reach/reconcile with it's expected state, which could be "running" (no healthcheck), "healthy" (with healthcheck), or "exited succesfully" (init containers)

This is all about backward compatibilty, can't just change the behavior for a command used by thousands (millions?) because is looks niceer

I think we're waaaay past "backward compatiblity" when we did #8655. We could still use the --compatibility flag for that if it's a real concern.

because is looks niceer

It's not "looking nicer"; I think the current behavior could be considered a bug; the expectation of a compose up would be to deploy the project (as described in the compose file), and detach once it succesfully did so. not waiting for that, means the project deployment didn't finish.

ndeloof · 2021-10-29T12:55:38Z

#8655 changes container names, but doesn't prevent most users to compose up and keep running the same commands they're used to
changing the behavior for up to wait for healthcheck status would make many user suddenly get up to get stuck for minutes without any extra logs. I can't imagine the number of issues we would receive about this.

ndeloof · 2021-10-30T12:34:20Z

seems we hardly can find a consensus here.

Here is my last proposal, reducing the scope:
Introduce --wait flag to wait for all services to reach either the running state or healthy for those who define a healthcheck test (either by compose file or image)

ndeloof · 2021-11-03T08:13:28Z

implemented the proposed "simpler" --wait boolean flag. Please reconsider

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>

thaJeztah · 2021-11-03T15:26:12Z

Here is my last proposal, reducing the scope:
Introduce --wait flag to wait for all services to reach either the running state or healthy for those who define a healthcheck test (either by compose file or image)

Looks like I missed your comment. Yes, I think that's a good middle-ground for now.

thaJeztah

LGTM

thaJeztah · 2021-11-03T16:04:09Z

pkg/compose/convergence.go

 		dep, config := dep, config
 		eg.Go(func() error {
 			ticker := time.NewTicker(500 * time.Millisecond)
 			defer ticker.Stop()
 			for {
 				<-ticker.C
 				switch config.Condition {
+				case ServiceConditionRuningOrHealthy:
+					healthy, err := s.isServiceHealthy(ctx, project, dep, true)


(just thinking out loud); perhaps we need to have a look at the events API, and see if we can improve it enough so that compose can (more easily) use that. We already have a health_status event, but perhaps that alone is not sufficient, but we could look at enhancing it. (that way compose wouldn't have to poll docker inspect calls)

would ne a nice addition to the event stream indeed

mitasov-ra · 2021-11-10T18:55:46Z

Sorry, but is there any timeout for waiting?

I've set my health check to "exit 1" with 5 retries every 1 second, but

docker compose up -d --wait

waits for several minutes already and seems like it will go forever.

I've thought it should fail if any of services is unhealthy, isn't it?

The previous code would wait for dependencies to become healthy forever, even if they'd become unhealthy in the meantime. I can't find an issue report for this bug, but it was described in a comment on the PR that introduced the `--wait` flag [0]. [0]: docker#8777 (comment)

The previous code would wait for dependencies to become healthy forever, even if they'd become unhealthy in the meantime. I can't find an issue report for this bug, but it was described in a comment on the PR that introduced the `--wait` flag [0]. [0]: docker#8777 (comment) Signed-off-by: Nikhil Benesch <nikhil.benesch@gmail.com>

The previous code would wait for dependencies to become healthy forever, even if they'd become unhealthy in the meantime. I can't find an issue report for this bug, but it was described in a comment on the PR that introduced the `--wait` flag [0]. [0]: #8777 (comment) Signed-off-by: Nikhil Benesch <nikhil.benesch@gmail.com>

…r.yml. Checks - Add Django checks - Add PYTHONWARNINGS check - Remove diff-cover check Move build step to Docker workflow - Remove permissions section - Remove CI=True POSTGRES_HOST=127.0.0.1 DOMAIN_URL=http://127.0.0.1/api environment variables - Use --wait instead of sleep 60 docker/compose#8777 - Use default values for API_PREFIX, POSTGRES_DB, POSTGRES_USER, JOB_FILES_TIMEOUT - Move if-statement for SKIP_TEST Refactor test workflow - Move Transifex upload to new job - Remove coverage thresholds (defer to coveralls) - Remove CI=True DEBUG=True environment variables - Remove fetch-depth: 0 - Use default values for API_PREFIX, POSTGRES_DB, POSTGRES_USER - Change service ports - Change PostgreSQL credentials Remove pytest configuration from setup.cfg - Add --cov spoonbill_web - Allow auto-discovery of DJANGO_SETTINGS_MODULE - Use default for python_files (test_*.py) and testpaths (all) - Remove norecursedirs = .git Other changes - Delete .envrc - Remove ALLOWED_HOSTS from docker-compose.test.yaml - Change settings defaults to require fewer overrides: - API_PREFIX - CELERY_BACKEND - CELERY_BROKER - DB_HOST - POSTGRES_DB - POSTGRES_PASSWORD - POSTGRES_USER

The previous code would wait for dependencies to become healthy forever, even if they'd become unhealthy in the meantime. I can't find an issue report for this bug, but it was described in a comment on the PR that introduced the `--wait` flag [0]. [0]: docker#8777 (comment) Signed-off-by: Nikhil Benesch <nikhil.benesch@gmail.com>

noorul · 2022-08-04T07:22:02Z

How long does this wait?

ndeloof · 2022-08-04T07:25:00Z

@noorul wait until dependent service reports a healthy state, as configured by HEALTHCHECK

noorul · 2022-08-04T07:57:47Z

@ndeloof I think a timeout option will add value here instead of waiting forever.

This enables starting lms services and the dev server by running `make services && make dev`, without encountering errors due to the DB not being ready. nb. The `--wait` flag is missing from the docs but see docker/compose#8777.

This enables starting apps with `make services dev` without potentially running into errors from the app due to services not being ready when the web server tries to connect. Services must define a healthcheck [1] in `docker-compose.yml` for this to work. The `--wait` flag is missing from the Docker Compose docs, but see docker/compose#8777. [1] https://docs.docker.com/compose/compose-file/05-services/#healthcheck

nawa · 2023-08-04T16:00:07Z

@ndeloof Thanks for the useful feature

Do you have a proposal how to avoid the issue with init containers that exit with 0 status as a result and it is totally ok. But --wait expects that all containers described in config are in running status

services:
    db:
        image: mysql:8.0
    
    init-db:
        image: mysql:8.0
        command: ./init-db.sh
        environment:
            <<: *cenv
        volumes:
            - ./init-db.sh:./init-db.sh
        depends_on:
            db:
                condition: service_started

docker compose up -d --wait

ndeloof commented Oct 11, 2021

View reviewed changes

cmd/compose/up.go Outdated Show resolved Hide resolved

ndeloof requested review from lorenrh and ulyssessouza October 11, 2021 15:56

ndeloof force-pushed the up_wait branch from 89d3616 to 5ce5bed Compare October 12, 2021 05:44

ulyssessouza reviewed Oct 12, 2021

View reviewed changes

cmd/compose/up.go Outdated Show resolved Hide resolved

branchvincent mentioned this pull request Oct 24, 2021

being implemented into docker compose! ufoscout/docker-compose-wait#62

Closed

ndeloof force-pushed the up_wait branch 2 times, most recently from 8f0605c to e1e2d06 Compare October 26, 2021 17:40

mat007 reviewed Oct 27, 2021

View reviewed changes

ndeloof force-pushed the up_wait branch 2 times, most recently from 4bc573f to 5df5f77 Compare October 27, 2021 07:54

ndeloof requested a review from ulyssessouza October 27, 2021 10:15

ndeloof force-pushed the up_wait branch from 23342f0 to 62ceb6a Compare November 3, 2021 08:13

ndeloof force-pushed the up_wait branch 2 times, most recently from 09976f7 to 61d63cd Compare November 3, 2021 08:25

introduce up --wait condition

b1b4ea6

Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>

ndeloof force-pushed the up_wait branch from 61d63cd to b1b4ea6 Compare November 3, 2021 09:22

thaJeztah approved these changes Nov 3, 2021

View reviewed changes

thaJeztah reviewed Nov 3, 2021

View reviewed changes

ndeloof merged commit 72e4519 into docker:v2 Nov 3, 2021

ndeloof deleted the up_wait branch November 3, 2021 17:22

PierreAntoineGuillaume mentioned this pull request Nov 5, 2021

Fix typo in --wait option mechanism #8888

Merged

benesch mentioned this pull request Jan 7, 2022

--await for healthcheck #8003

Closed

benesch mentioned this pull request Jan 7, 2022

Don't wait forever for unhealthy dependencies #9092

Merged

felipecrs mentioned this pull request Jan 25, 2022

Stream logs during --wait #9122

Open

max-tet mentioned this pull request Feb 13, 2022

Feature: docker-compose up -d with waiting for healthy statuses #8351

Closed

fetinin mentioned this pull request Jun 28, 2022

Aerospike fixtures support lamoda/gonkey#168

Merged

robertknight mentioned this pull request Jul 10, 2023

Make make services wait for healthchecks to pass using --wait hypothesis/lms#5556

Closed

robertknight mentioned this pull request Jul 10, 2023

Wait for healthchecks in make services hypothesis/cookiecutters#141

Merged

Huluti mentioned this pull request Sep 10, 2023

Health check not working with --wait param of docker compose maildev/maildev#470

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

introduce up --wait condition #8777

introduce up --wait condition #8777

ndeloof commented Oct 11, 2021 •

edited

Loading

thaJeztah commented Oct 12, 2021

ndeloof commented Oct 12, 2021

thaJeztah commented Oct 12, 2021

ndeloof commented Oct 12, 2021

mik-laj commented Oct 13, 2021

thaJeztah commented Oct 14, 2021

ndeloof commented Oct 15, 2021

ndeloof commented Oct 22, 2021

ndeloof commented Oct 26, 2021

thaJeztah commented Oct 27, 2021

ndeloof commented Oct 27, 2021

thaJeztah commented Oct 27, 2021

ndeloof commented Oct 29, 2021

ndeloof commented Oct 30, 2021

ndeloof commented Nov 3, 2021

thaJeztah commented Nov 3, 2021

thaJeztah left a comment

thaJeztah Nov 3, 2021

ndeloof Nov 3, 2021

mitasov-ra commented Nov 10, 2021

noorul commented Aug 4, 2022

ndeloof commented Aug 4, 2022

noorul commented Aug 4, 2022

nawa commented Aug 4, 2023 •

edited

Loading

introduce up --wait condition #8777

introduce up --wait condition #8777

Conversation

ndeloof commented Oct 11, 2021 • edited Loading

thaJeztah commented Oct 12, 2021

ndeloof commented Oct 12, 2021

thaJeztah commented Oct 12, 2021

ndeloof commented Oct 12, 2021

mik-laj commented Oct 13, 2021

thaJeztah commented Oct 14, 2021

ndeloof commented Oct 15, 2021

ndeloof commented Oct 22, 2021

ndeloof commented Oct 26, 2021

thaJeztah commented Oct 27, 2021

ndeloof commented Oct 27, 2021

thaJeztah commented Oct 27, 2021

ndeloof commented Oct 29, 2021

ndeloof commented Oct 30, 2021

ndeloof commented Nov 3, 2021

thaJeztah commented Nov 3, 2021

thaJeztah left a comment

Choose a reason for hiding this comment

thaJeztah Nov 3, 2021

Choose a reason for hiding this comment

ndeloof Nov 3, 2021

Choose a reason for hiding this comment

mitasov-ra commented Nov 10, 2021

noorul commented Aug 4, 2022

ndeloof commented Aug 4, 2022

noorul commented Aug 4, 2022

nawa commented Aug 4, 2023 • edited Loading

ndeloof commented Oct 11, 2021 •

edited

Loading

nawa commented Aug 4, 2023 •

edited

Loading