Custom health check exporter for the Operations & Reliability Prometheus system.
We use the Prometheus file-based service discovery, and a custom process writing to the NFS volume to generate the following configuration from health endpoints stored in the BizOp API.
Prometheus asks this exporter for metrics, one health endpoint at a time.
For exported metrics, view the /scrape endpoint of the running exporter, for a given check. These are up-to-date with the code-ified prometheus descriptions - rather than keeping a static list in the README.
observe label is added as part of the service discovery.
- job_name: health_check_exporter scheme: https static_configs: - targets: - prometheus-health-check-exporter-eu-west-1.in.ft.com - prometheus-health-check-exporter-us-east-1.in.ft.com - job_name: health_check scheme: https metrics_path: /scrape scrape_interval: 60s file_sd_configs: - files: - /prometheus/service-discovery/health-check-service-discovery.json relabel_configs: - source_labels: [__address__] target_label: __param_endpoint - source_labels: [__address__] target_label: instance - target_label: __address__ replacement: prometheus-health-check-exporter.in.ft.com
||The port which the service listens to HTTP connections over|
||Whether to enable verbose logging|
||The FT API key for the healthcheck validator service.|
||The base URL for the healthcheck validator service.|
Ensure the following variables are set in the CircleCI project:
Other variables are pulled from a shared CircleCI context.
Use the Makefile to locally test your changes. The 'make build' command will create a new local docker image, then the 'make run' command will execute these changes locally.
http://localhost:8080/ on your browser to see the index page for the exporter.