Add Prometheus service discovery

[cgrinds]

Harvest should use one of Prometheus's existing service discovery options to make it easier for customers to add hundreds of pollers to Harvest without having to specify the exact port for each poller and each Prometheus target.

Reference

https://prometheus.io/docs/guides/file-sd/
https://prometheus.io/blog/2018/07/05/implementing-custom-sd/
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#dns_sd_config
https://yetiops.net/posts/prometheus-srv-discovery/

[cgrinds]

Comment from Vachagan:

• file-based SD would be very easy to implement, unfortunately it only works if Prometheus runs on the same machine as Harvest (usually not the case)
• HTTP-based SD is discussed by the Prometheus devs, but not clear if it will be implemented anytime soon.
• Using the Adapter mentioned in your second link may be the most realistic - needs more investigation

[cgrinds]

Consul

Use consul for service discovery. The Prometheus exporter will be extended to optionally support Consul.

To use Consul support, add the consul section to the Prometheus exporter. The consul section allows pollers to export to Prometheus without having to specify the Prometheus address and port for each poller individually. Exporters that include the consul block create an HTTP endpoint to serve the ONTAP metrics data, just like the standard Prometheus exporter, with three important differences.

1. A random free port will be picked for the open-metrics endpoint
2. Harvest will register that endpoint with Consul
3. Harvest will monitor and update Consul on the health of the poller

We will tell Prometheus to ask Consul for the list of targets to scrape by using Prometheus' consul_sd_configs support. Let's take a look at an example.

Add an exporter to harvest.yml like so:

Exporters:
  pluto:                   # name of your exporter, can be any valid yaml string
    exporter: Prometheus   # type of exporter - Prometheus in this case
    consul:                # enable service discovery via Consul
      host: 1.2.3.4        # ip or hostname of Consul cluster
      port: 8500           # port of Consul cluster, defaults to Consul's default 8500

See consul configuration options for the full list of configuration options.

Configure Prometheus to find targets from Consul

Continuing with the example, we tell Prometheus to scrape our Consul registered pollers by adding a new scrape config to the prometheus.yml and using the consul_sd_configs section like so:

scrape_configs:
  - job_name: 'harvest'
    scrape_interval:     60s
    consul_sd_configs:
    - server: 'localhost:8500'
      services:
        - 'harvest-poller'   # this name matches the service name defined in harvest.yml

This tells Prometheus to ask Consul (running on localhost:8500) for a list of targets associated with the harvest-poller service. The service name matches the service named defined in harvest.yml, which defaults to harvest-poller.

See Prometheus Consul_sd_config for the full list of ways to configure consul_sd_configs.

Consul Config Options

NOTE When consul is enabled, the port defined in the Prometheus section will be ignored. That's because service discovery will assign the port. You can omit the port parameter altogether when using consul.

An overview of Consul parameters defined as children of the consul block:

parameter	type	description	default
host	string, required	ip or hostname of Consul cluster
port	int, optional	port of Consul cluster	8500
service_name	string, optional	service name, should be valid DNS label	harvest-poller
ttl	string, Go duration	Consul TTL check - amount of time before Consul considers poller down	5m (five minutes)
tags	list of strings, optional	list of tags to assign to this service

Example:

Exporters:
  pluto:                     # name of your exporter, can be any valid yaml string
    exporter: Prometheus
    port: 12990              # ignored since this exporter contains a consul block
    consul:                  # consul block added because we want service discovery
      host: 1.2.3.4          # ip or hostname of Consul cluster
      service_name: monitor  # all pollers exporting to pluto will be registered with the service name of 'monitor'
      tags:
        - netapp             # each of the pollers exporting to pluto will be tagged with two tags: 'netapp' and 'harvest'
        - harvest

References

• Consul Services
• Finding Consul services to monitor with Prometheus (2016)

[vgratian]

this looks great, thanks for the detailed breakdown.

the only downside is that users have to install an additional software where Prometheus is running.

[cgrinds]

Closed by #575

[Dinesvl]

Hi Cris,,

Prometheus services stopped after few mins of start up. i see starting up replaying WAL prompt and services stopped after some time. May i help get fix this issue? Thanks