Increase retries for container health checks

[codegain]

Version

21.8.0

Steps to Reproduce

1. Use some plugin that requires some additional apt programs to be installed (like sentry-ldap-auth)
2. Add the line to sentry/entrypoint.sh after checksum_file=... like apt-get update && apt-get install -y gcc libsasl2-dev python-dev libldap2-dev libssl-dev
3. Run ./install.sh
4. Run docker-compose up -d

Expected Result

Sentry booting up normally - in docker-compose logs -f you can see the packages getting installed.

Actual Result

# docker-compose up -d
Creating network "sentry_onpremise_default" with the default driver
Creating sentry_onpremise_postgres_1             ... done
Creating sentry_onpremise_clickhouse_1           ... done
Creating sentry_onpremise_symbolicator_1         ... done
Creating sentry_onpremise_symbolicator-cleanup_1 ... done
Creating sentry_onpremise_memcached_1            ... done
Creating sentry_onpremise_geoipupdate_1          ... done
Creating sentry_onpremise_redis_1                ... done
Creating sentry_onpremise_zookeeper_1            ... done
Creating sentry_onpremise_smtp_1                 ... done
Creating sentry_onpremise_kafka_1                ... done
Creating sentry_onpremise_snuba-api_1                                ... done
Creating sentry_onpremise_snuba-sessions-consumer_1                  ... done
Creating sentry_onpremise_snuba-consumer_1                           ... done
Creating sentry_onpremise_snuba-cleanup_1                            ... done
Creating sentry_onpremise_snuba-transactions-cleanup_1               ... done
Creating sentry_onpremise_snuba-transactions-consumer_1              ... done
Creating sentry_onpremise_snuba-subscription-consumer-transactions_1 ... done
Creating sentry_onpremise_snuba-outcomes-consumer_1                  ... done
Creating sentry_onpremise_snuba-replacer_1                           ... done
Creating sentry_onpremise_snuba-subscription-consumer-events_1       ... done
Creating sentry_onpremise_web_1                                      ... done
Creating sentry_onpremise_cron_1                                     ... done
Creating sentry_onpremise_subscription-consumer-transactions_1       ... done
Creating sentry_onpremise_subscription-consumer-events_1             ... done
Creating sentry_onpremise_worker_1                                   ... done
Creating sentry_onpremise_ingest-consumer_1                          ... done
Creating sentry_onpremise_post-process-forwarder_1                   ... done
Creating sentry_onpremise_sentry-cleanup_1                           ... done

ERROR: for relay  Container "f7106d52f474" is unhealthy.
ERROR: Encountered errors while bringing up the project.

This is because the packages take a few moments to install and the default health check is set to 30s intervals with only 3 retries. Sentry thinks that the containers are unhealthy and will stop booting up, but in the logs I can see that the packages are installing without any problems. They just would need more time.

I just increased the number of retries in x-healthcheck-defaults to 10 and it worked for me.

[BYK]

Thanks a lot for the report. I've observed this on our CI tests too. I'll increase it to 5 and then add a "run sentry web container once" step to the install as the installation cost should only be paid once anyway.

Realized this is happening in a modified entrypoint and there's no need for a separate run as we pay the plugin installation price in web upgrade part.

I recommend using a custom image with those dependencies already installed (to avoid installing them at every container start) or using docker-compose.override.yml file to adjust the web service's healthcheck retry.