Prometheus should try to create consul discovery forever or fail on startup

[discordianfish]

If Prometheus is configured to use consul sd but consul is down when Prometheus starts, it tries to connect to consul and fails but still comes up.
The result is a up and running Prometheus server which doesn't scrape the configured targets.

Instead Prometheus should try to connect to consul forever or fail on startup.

[discordianfish]

This applies to all other SD providers as well. First I tended towards failing on startup but the nice thing about using promtool to check config is to know that if it's okay, Prometheus will start. Which makes me prefer trying to reconnect forever and have a error counters for the SD providers.

[roman-vynar]

According to the code, when you set datacenter option in your config (consul_sd_configs section), it won't be a problem on startup. Prometheus will retry to get service list as it normally does.

The problem is when datacenter is not set, Prometheus tries to get it once on startup from Consul which is down and discovery fails to run and didn't restart.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.