depends_on Should Obey {{ State.Healthcheck.Status }} Before Launching Services

[midnightconman]

Is it possible for the depends_on parameter to wait for a service to be in a "Healthy" state before starting services, if a healthcheck exists for the container? At the moment in 1.8.0-rc2 the depends_on parameter only verifies that the container is in a running state, via {{ State.Status.running }}. This would allow for better dependency management in docker-compose.

[aanand]

Yes, we should probably do this. It will involve:

• Adding support for both setting the health check options for a container and querying its health state to docker-py
• Adding support for those options (command, interval, retries, timeout) to the Compose file, probably under a health_check key
• Updating the logic of depends_on to query containers' health state before considering a dependency to be ready

There are some important questions to answer, e.g.

• What qualifies a service (i.e. multiple containers) as healthy? Is it healthy if and only if all its containers are healthy?
• Should containers not join networks until they are healthy?
• etc

[midnightconman]

I like the idea of not joining a network until healthy also, that would allow for auto joining of containers via the service dns record. Is it possible to also have a mechanism where containers would leave the service dns record if the healthcheck is failing? I think this portion might be a question for docker engine.

[aanand]

Yes, that's the problem with implementing it in Compose - its state can't be monitored. I think that suggests we shouldn't look at a container's health before connecting it to the network.

[gittycat]

I think that the health state of a service is separate from whether it is connected to the network. A service might need to connect to other services first before it becomes healthy.

The health check should also be done at the service level, not container level. It makes more sense now to think in terms of Services than containers, specially with Docker 1.12 (Service has become a 1st class citizen). A consumer of the service should not have to care that the service is made up of 1,2 or dozens of containers. What counts is that the service as a whole is considered ready to accept requests.
This means that each service would need some centralised way for its containers to report their status to. The DNS resolver is already used to report the IP of each service. It could probably be used to keep the health status as well.

[aanand]

Quoting @dnephin in #3815 (comment):

I haven't tried out healthchecks yet. Is there some api call we can use to do "wait on container state == HEALTHY"?

If that exits, I think it would make sense to use it on depends_on. I'd rather not build polling into Compose to handle it.

I agree, so next step here would be to open a PR against Engine to implement that functionality if it doesn't already exist. It's important to get it into Engine first and as early as possible, because we have a policy of supporting at least two versions of Engine. If we can get the API endpoint into Engine 1.13, then we can get healthcheck support into Compose 1.10.

[ColinHebert]

What about using the event system to do that?

You can filter based on event="health_status: healthy"
https://docs.docker.com/engine/reference/api/docker_remote_api_v1.24/#/monitor-docker-s-events

[earthquakesan]

I tweaked service.py and parallel.py and was able to make one container wait for another until it is healthy.

Basically every container, which has dependencies on other containers (as I see it dependencies are inferred from links, volumes_from, depends_on... --> get_dependency_names() method -- line 519 in service.py) will wait until those containers are healthy.

Regarding API, docker-compose uses docker-py and health check can be performed as follows (in service.py):

    def is_healthy(self):
        try:
            status = self.client.inspect_container(self.name)["State"]["Health"]["Status"]
            if status == "healthy":
                return True
            else:
                return False
        except docker.error.NotFound as e:
            return False
        #didn't test --> this bit is necessary for backward compatibility, otherwise it will not work with previous docker API --> need more specific exception class too
        #except BaseException as e:
            #If API does not support healthchecks, just return true
            #return True

Then in parallel.py I just added one more check for firing the producer for the object. Iteration for the pending loop looks like this now:

    for obj in pending:
        deps = get_deps(obj)

        if any(dep in state.failed for dep in deps):
            log.debug('{} has upstream errors - not processing'.format(obj))
            results.put((obj, None, UpstreamError()))
            state.failed.add(obj)
        elif all(
            dep not in objects or dep in state.finished
            for dep in deps
        ) and all(
            #This is a new case checking for healthy status --> is dep always an instance of Service class?
            dep.is_healthy() for dep in deps
        ):
            log.debug('Starting producer thread for {}'.format(obj))
            t = Thread(target=producer, args=(obj, func, results))
            t.daemon = True
            t.start()
            state.started.add(obj)

Can anyone from the maintainers comment on it? It works with up command, down command also no issue. However, it might break something or?

[earthquakesan]

tests/acceptance/cli_test.py::CLITestCase::test_down <- tests/integration/testcases.py

This test case hungs =\

[dnephin]

I don't think we need to use the healthcheck for volumes_from. It only requires the container to be running.

I think it would be good to only use healthchecks for depends_on, and leave links with the old method (only waiting on the container to start). That way there is a way to add dependencies without incurring the extra time waiting on healthchecks. Some applications may handle dependencies more gracefully, and we shouldn't make their startup time slower.

[mortensteenrasmussen]

Sounds like a good idea @dnephin