Registrator joins Consul client prematurely

[botond-veress]

• What version of docker are you running?
  Docker version 17.06.2-ce, build cec0b72
• What version of registrator are you running?
  gliderlabs/registrator:latest
• Did you build a custom version of registrator? If so, what is that image?
  No
• What is the exact command you are running registrator with?
  Given below
• What is the exact command you are running your container with?
  Given below
• A log capture of all the docker events before, during, and after the issue.
  Given below
• If relevant, Dockerfile for application that is having issues.
  Given below

Description of the problem:
I've searched through several SO posts and the documentation but couldn't find anything related to my problem. I am unable to prevent registrator to connect the consul client prematurely. I have a cluster of e.g. 3 consul servers and I want to have 1 consul client + 1 registrator running in every container. They are started up in the same time with docker compose but unfortunately the registrator tries to pull information from the consul client before it joins the cluster so the consul client returns Internal Server Error (no cluster leader). The registrator exists instantly after the error received.

Here is my docker-compose.yml file:

version: '3'
services:
  consul:
    image: "consul:latest"
    container_name: consul-agent
    hostname: consul-agent
    ports:
    - "8500:8500"
    environment:
    - "CONSUL_LOCAL_CONFIG={\"leave_on_terminate\": true}"
    entrypoint:
    - consul
    - agent
    - -ui
    - -client=0.0.0.0
    - -bind=0.0.0.0
    - -advertise={{ GetInterfaceIP "eth0" }}
    - -retry-join=${CONSUL_CLUSTER_ADDRESS}
    - -encrypt=${CONSUL_ENCRYPT_SECRET}
    - -datacenter=${CONSUL_DATACENTER}
    - -node=${CONSUL_NODE}
    - -data-dir=/tmp/consul

  registrator:
    image: gliderlabs/registrator:latest
    container_name: registrator
    hostname: registrator
    depends_on:
    - consul
    links:
    - consul
    volumes:
    - /var/run/docker.sock:/tmp/docker.sock
    command: "-ip consul consul://consul:8500

Here are the logs:

registrator    | 2017/09/24 22:57:24 Starting registrator v7 ...
consul-agent   | ==> Consul agent running!
consul-agent   |            Version: 'v0.9.3'
consul-agent   |            Node ID: 'dd6bc068-eb29-2678-e857-dfeb2be387c5'
registrator    | 2017/09/24 22:57:24 Forcing host IP to consul
registrator    | 2017/09/24 22:57:24 Using consul adapter: consul://consul:8500
registrator    | 2017/09/24 22:57:24 Connecting to backend (0/0)
registrator    | 2017/09/24 22:57:24 Unexpected response code: 500 (No known Consul servers)
consul-agent   |          Node name: '<consul-node-name>'
consul-agent   |         Datacenter: '<consul-data-center>' (Segment: '')
consul-agent   |             Server: false (Bootstrap: false)
consul-agent   |        Client Addr: 0.0.0.0 (HTTP: 8500, HTTPS: -1, DNS: 8600)
consul-agent   |       Cluster Addr: 172.18.0.2 (LAN: 8301, WAN: 8302)
consul-agent   |            Encrypt: Gossip: true, TLS-Outgoing: false, TLS-Incoming: false
consul-agent   | 
consul-agent   | ==> Log data will now stream in as it occurs:
consul-agent   | 
consul-agent   |     2017/09/24 22:57:23 [INFO] serf: EventMemberJoin: <consul-node-name> 172.18.0.2
consul-agent   |     2017/09/24 22:57:23 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp)
consul-agent   |     2017/09/24 22:57:23 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp)
consul-agent   |     2017/09/24 22:57:23 [INFO] agent: Started HTTP server on [::]:8500
consul-agent   |     2017/09/24 22:57:23 [INFO] agent: Retry join LAN is supported for: aws azure gce softlayer
consul-agent   |     2017/09/24 22:57:23 [INFO] agent: Joining LAN cluster...
consul-agent   |     2017/09/24 22:57:23 [INFO] agent: (LAN) joining: [<consul-host-address>]
consul-agent   |     2017/09/24 22:57:23 [WARN] manager: No servers available
consul-agent   |     2017/09/24 22:57:23 [ERR] agent: failed to sync remote state: No known Consul servers
consul-agent   |     2017/09/24 22:57:24 [WARN] manager: No servers available
consul-agent   |     2017/09/24 22:57:24 [ERR] http: Request GET /v1/status/leader, error: No known Consul servers from=172.18.0.3:46796
consul-agent   |     2017/09/24 22:57:26 [INFO] serf: EventMemberJoin: <consul-server-1> <ip-address-1>
consul-agent   |     2017/09/24 22:57:26 [INFO] consul: adding server <consul-server-1> (Addr: tcp/<ip-address-1>:8300) (DC: <consul-data-center>)
registrator exited with code 1
consul-agent   |     2017/09/24 22:57:27 [INFO] agent: (LAN) joined: 2 Err: <nil>
consul-agent   |     2017/09/24 22:57:27 [INFO] agent: Join LAN completed. Synced with 2 initial agents
consul-agent   |     2017/09/24 22:57:28 [INFO] agent: Synced node info

You can note that registrator exists right after it receives internal server error from the local consul client. Any ideas how to fix this?

Thanks

[jhmartin]

Take a look at the retry option:

  -retry-attempts=0: Max retry attempts to establish a connection with the backend. Use -1 for infinite retries
  -retry-interval=2000: Interval (in millisecond) between retry-attempts.