Make ContainerWait and healthchecks smarter, fixes #1287

[rfay]

The Problem/Issue/Bug:

1. ContainerWait() was sometimes waiting for a really long time when one of the containers (usually db) had already exited. (It does seem like the code already handles this. I think the actual problem is waiting too long for an unhealthy container; in other words, the container does not display as unhealthy soon enough. )

It turns out that this behavior was mostly because we were polling health of the web container first and waiting for its healthcheck to complete, then tried the db container, which may have long since exited.

2. People have mentioned (and I've noticed) that ddev projects use quite a lot of CPU and battery. And it's mostly the healthcheck. (I experimented with a number of solutions to this, but all of them result in ddev start taking longer, which I think is probably unacceptable. So I backed out my experiments.)

How this PR Solves The Problem:

• Check db container first.
• Add healthcheck timeouts and retries to the docker-compose.yaml (and related) so we control them at docker-compose level.
• back off how long we'll wait for db container. It was set up for 30 retries, trying every 2 seconds, so 60 seconds before it became "unhealthy".

Manual Testing Instructions:

• Try ddev rm && ddev start.
• Try breaking a container deliberately. My technique for this was to do a docker-compose.breakit.yaml with contents like this:

version: '3.6'
services:
  web:
    command: "bash -c 'sleep 2 && exit '"

• Do things to break the db container:
  • in mysql DROP DATABASE db; and then watch behavior as it changes (ddev list) then ddev rm && ddev start and see how long it takes to fail.
  • In the db container, rm /var/lib/mysql/mysql/*.MYD. Watch behavior after you do it. Then try ddev rm && ddev start

Automated Testing Overview:

The overall behavior has not changed, I don't think it needs new tests.

This doesn't make a major change in behavior, so no test changes yet.

Related Issue Link(s):

OP #1287

Release/Deployment notes:

It may be worth

• Writing a stack overflow to tell people how to use longer healthchecks to save battery. Or perhaps adding a ddev start --slow-healthcheck or something.

[andrewfrench]

Code changes look good to me, and testing showed healthy and unhealthy containers being reported reliably.

• ddev rm && ddev start reported healthy containers,
• ddev rm && ddev start with .ddev/compose.breakit.yaml reported an unhealthy web container within a few seconds
• After dropping the db database, ddev list reported healthy containers for a few seconds, then reported an unhealthy db container; the container was still reported as unhealthy after ddev rm && ddev start after just a few seconds (time ddev start reports 11s, the second execution, after other containers are running, takes 3.5s)
• After deleting mysql files, ddev rm && ddev start reported unhealthy containers in around the same time as above