Retry Loop never resolving Race Condition

[jnovack]

Describe the bug
If the server container starts before the redis container (read: before the dns entry of the redis container hits the DNS server), the server container will never resolve the IP of the redis container and will result in an infinite loop and never resolving the DNS entry.

To Reproduce
Steps to reproduce the behavior:

1. docker stack deploy checkmate.yml within Docker Swarm.
2. See error

This occurs naturally as Docker Swarm does not respect depends-on from docker-compose as you intend.

Expected behavior
I am expecting the server container to down itself after a pre-defined limit of failed attempts contacting either the redis or mongo end-points.

If 5 backed-off retries (immediately, 0.5 sec, 1 sec, 2 sec, 4 sec) fail, the container should exit with error. It is the responsibility of the orchestration engine (swarm, kubernetes, etc) to respawn the container, which gives the other containers plenty of time to be functional and registered.

Screenshots

checkmate_server.1.fa0rb3gwhxyj@swarm-master-04    | Error: getaddrinfo ENOTFOUND redis
checkmate_server.1.fa0rb3gwhxyj@swarm-master-04    |     at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:109:26) {
checkmate_server.1.fa0rb3gwhxyj@swarm-master-04    |   errno: -3008,
checkmate_server.1.fa0rb3gwhxyj@swarm-master-04    |   code: 'ENOTFOUND',
checkmate_server.1.fa0rb3gwhxyj@swarm-master-04    |   syscall: 'getaddrinfo',
checkmate_server.1.fa0rb3gwhxyj@swarm-master-04    |   hostname: 'redis'
checkmate_server.1.fa0rb3gwhxyj@swarm-master-04    | }

Work Around

1. docker service scale checkmate_server=0 && docker service scale checkmate_server=1