Health check exporter for Prometheus.
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.

README.md

prometheus-health-check-exporter

CircleCI

Custom health check exporter for the Operations & Reliability Prometheus system.

We use the Prometheus file-based service discovery, and a custom process writing to the NFS volume to generate the following configuration from health endpoints stored in the BizOp API.

Prometheus asks this exporter for metrics, one health endpoint at a time.

Exported Metrics

For exported metrics, view the /scrape endpoint of the running exporter, for a given check. These are up-to-date with the code-ified prometheus descriptions - rather than keeping a static list in the README.

e.g. https://prometheus-healthcheck-exporter.in.ft.com/scrape?endpoint=https://vault.in.ft.com/__health

Note: The observe label is added as part of the service discovery.

Prometheus Configuration

- job_name: health_check_exporter
  scheme: https
  static_configs:
    - targets:
      - prometheus-health-check-exporter-eu-west-1.in.ft.com
      - prometheus-health-check-exporter-us-east-1.in.ft.com

- job_name: health_check
  scheme: https
  metrics_path: /scrape
  scrape_interval: 60s
  file_sd_configs:
    - files:
      - /prometheus/service-discovery/health-check-service-discovery.json
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_endpoint
    - source_labels: [__address__]
      target_label: instance
    - target_label: __address__
      replacement: prometheus-health-check-exporter.in.ft.com

Development

Environment variables

Variable Default Description
PORT 8080 The port which the service listens to HTTP connections over
VERBOSE false Whether to enable verbose logging
HEALTHCHECK_VALIDATOR_API_KEY https://healthcheck-validator.in.ft.com The FT API key for the healthcheck validator service.
HEALTHCHECK_VALIDATOR_BASE_URL https://api.ft.com/monitoring/healthcheck The base URL for the healthcheck validator service.

CircleCI

Ensure the following variables are set in the CircleCI project:

  • HEALTHCHECK_VALIDATOR_API_KEY

Other variables are pulled from a shared CircleCI context.

Local Development

Use the Makefile to locally test your changes. The 'make build' command will create a new local docker image, then the 'make run' command will execute these changes locally.

Go to http://localhost:8080/ on your browser to see the index page for the exporter.