Skip to content

How to setup a monitoring environment(docker compose)

Jump to bottom
Chenyu Zhang edited this page Jun 2, 2022 · 1 revision

Background

Currently, the perf repository provides the ability of performance testing, includes prepare different size data on a fresh installed harbor and run API tests, finally a performance report can be generated after running the tests. In addition to focusing on performance test reports, the runtime metrics are also very important for us to help analyze performance problems. For example, we also need resource usage and distributed tracing capabilities to help locate performance bottlenecks. So here is an example to show how to set up a monitoring environment.

Installation

Deploy harbor

Harbor v2.4: https://github.com/goharbor/harbor/releases/tag/v2.4.0

We can download offline installer to install harbor, harbor.yml configuration should be modified before install to enable monitor related features.

Uncomment metric related fields to enable harbor metrics.

metric:
  enabled: true
  port: 9090
  path: /metrics

Uncomment trace related fields to enable harbor tracing.

trace:
  enabled: true
  jaeger:
    # use service name `jaeger` as endpoint host
    endpoint: http://jaeger:14268/api/traces

Monitoring components

cAdvisor & postgresql exporter

  • cadvisor is a tool which can analyze resource usage and performance characteristics of running containers.
  • postgresql exporter can expose postgresql database metrics.

prometheus & grafana & jaeger

  • prometheus is a systems and service monitoring system.
  • grafana is an open and composable observability and data visualization platform which can visualize metrics, logs, and traces from multiple sources.
  • jaeger is a distributed tracing platform.

To simplify the deployment of these components, we have integrated a simple docker-compose YAML for one-click deploy. Just copy below two YAML files to the harbor install folder.

prometheus.yml

global:
  scrape_interval: 5s

scrape_configs:
  - job_name: "cadvisor"
    static_configs:
    - targets: ["cadvisor:8080"]

  - job_name: "harbor-db-exporter"
    static_configs:
    - targets: ["harbor-db-exporter:9187"]

  - job_name: 'harbor-exporter'
    static_configs:
      # Scrape metrics from the Harbor exporter component
    - targets: ['nginx:9090']

  - job_name: 'harbor-core'
    params:
      # Scrape metrics from the Harbor core component
      comp: ['core']
    static_configs:
    - targets: ['nginx:9090']

  - job_name: 'harbor-registry'
    params:
      # Scrape metrics from the Harbor registry component
      comp: ['registry']
    static_configs:
    - targets: ['nginx:9090']

  - job_name: 'harbor-jobservice'
    params:
      # Scrape metrics from the Harbor jobservice component
      comp: ['jobservice']
    static_configs:
    - targets: ['nginx:9090']

monitor.yml

version: '2.3'

volumes:
  prometheus_data: {}
  grafana_data: {}


services:
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.37.5
    container_name: cadvisor
    restart: unless-stopped
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    ports:
      - 8080:8080
    privileged: true
    devices:
      - /dev/kmsg
    networks:
      - harbor

  harbor-db-exporter:
    image: quay.io/prometheuscommunity/postgres-exporter:v0.10.0
    container_name: harbor-db-exporter
    restart: unless-stopped
    environment:
      DATA_SOURCE_NAME: "postgresql://postgres:root123@harbor-db:5432/postgres?sslmode=disable"
    ports:
      - 9187:9187
    networks:
      - harbor

  # prometheus & grafana & jaeger
  prometheus:
    image: prom/prometheus:v2.31.1
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--web.enable-lifecycle'
    ports:
      - 9091:9090
    networks:
      - harbor

  grafana:
    image: grafana/grafana:8.2.3
    container_name: grafana
    restart: unless-stopped
    volumes:
      - grafana_data:/var/lib/grafana
    ports:
      - 3000:3000
    networks:
      - harbor

  jaeger:
    image: jaegertracing/all-in-one:1.26
    container_name: jaeger
    restart: unless-stopped
    ports:
      - 16686:16686
      - 14268:14268
      - 14250:14250
    networks:
      - harbor

Deployment has become very easy, we only need to append compose CLI option to up harbor and monitor together by one command.

$ docker-compose -f docker-compose.yml -f monitor.yml up -d

After these components' container be ready, we can open the browser with http://nodeIP:3000 to access the grafana and http://nodeIP:16686 to access the jaeger, for grafana we require adding prometheus as data source(because of the prometheus is running in the same network with grafana, so we can configure the data source endpoint just by service name like http://prometheus:9090), and then import dashboard to grafana manually. Here are some useful dashboards from the community.

Clone this wiki locally