Monitoring Stack (Grafana + Prometheus + Loki + Tempo + Pyroscope + Mimir + OTel Collector)

All-in-one repo to spin up a monitoring & logging stack on Docker. Target: Proxmox LXC (unprivileged) running Debian 12 + Docker.

Quickstart

# 1) Copy repo to your LXC host (monstack), then:
cp .env.example .env 2>/dev/null || true    # if you keep an example
# Edit .env and set GRAFANA_ADMIN_PASSWORD

docker compose pull
docker compose up -d

Services:

• Grafana: http://:${GRAFANA_PORT}
• Prometheus: http://:${PROMETHEUS_PORT}
• Loki: http://:${LOKI_PORT}
• Alertmanager: http://:${ALERTMANAGER_PORT}
• Tempo UI / Trace API: http://:${TEMPO_PORT}
• Pyroscope: http://:${PYROSCOPE_PORT} (view flamegraphs directly in Pyroscope UI)
• Mimir (Prometheus-compatible API): http://:${MIMIR_PORT}
• OTLP Collector (gRPC ${OTELCOL_OTLP_GRPC_PORT}, HTTP ${OTELCOL_OTLP_HTTP_PORT}, metrics ${OTELCOL_METRICS_PORT})
• Promtail (container + host logs → Loki)
• Demo service (optional overlay): http://:18000
• Demo Nginx gateway + exporter (optional overlay): http://:${NGINX_PORT:-18080}
• NPMplus (demo reverse proxy UI): http://:${NPMPLUS_UI_PORT:-18081} (HTTP ingress on ${NPMPLUS_HTTP_PORT:-18090}, HTTPS on ${NPMPLUS_HTTPS_PORT:-18443})

Repo layout

• docker-compose.yml — all services
• prometheus/prometheus.yml — scrape targets + alerting -> Alertmanager
• prometheus/alert_rules/*.yml — example alerting rules
• loki/loki-config.yaml — filesystem backend with boltdb-shipper + compactor retention
• otel-collector/config.yaml — OpenTelemetry Collector pipelines (OTLP ingest → Tempo/Mimir/Prometheus/Loki)
• tempo/tempo.yaml — Tempo single-binary config + local filesystem storage
• mimir/mimir.yaml — Grafana Mimir single-binary config + local filesystem storage
• pyroscope/config.yaml — Pyroscope single-binary config + local filesystem storage
• promtail/promtail-config.yaml — ships Docker and system logs to Loki
• grafana/provisioning/* — auto-provision Prometheus & Loki datasources and dashboards
• alertmanager/alertmanager.yml — minimal route/receiver (extend as needed)
• scripts/deploy_stack.sh — SSH deploy helper
• mongo-exporter/ — lightweight MongoDB → Prometheus exporter used by the demo overlay
• telegraf/telegraf.conf — Telegraf collector (nginx stub_status + host stats + access log tail for the Grafana Nginx board)
• TODO.md — live checklist/report for the actively requested fixes (linked from AGENTS.md)

Note: Promtail now ships system & container logs directly to Loki; application telemetry uses the OpenTelemetry Collector.

Grafana dashboards

Core (folder Root):

• Stack Overview (grafana/dashboards/stack-overview.json) — infrastructure + service SLO view with logs/traces links.
• Demo Service Deep Dive (grafana/dashboards/demo-service.json) — queue depth, latency, error insight for the Node.js workload (backed by /metrics on demo-app).

Imported (folder imported/ — automatically provisioned):

• Pino HTTP Logs (grafana/dashboards/imported/pino-http-logs.json, Grafana ID 21900) — live Loki queries over pino JSON logs with filters for service / level / status codes.
• Node Exporter Full (grafana/dashboards/imported/node-exporter-full.json, ID 1860) — detailed host metrics from node-exporter + cadvisor.
• Redis Exporter (grafana/dashboards/imported/redis-dashboard.json, ID 11835) — Redis cache activity via redis_exporter.
• MongoDB Overview (grafana/dashboards/imported/mongodb-overview.json, ID 2583) — MongoDB storage + ops from mongodb-exporter.
• NodeJS Application (grafana/dashboards/imported/nodejs-application.json, ID 11159) — process/runtime stats from Prometheus default metrics (via prom-client).
• Nginx (grafana/dashboards/imported/nginx-metrics.json, ID 14900) — request/latency metrics from the demo Nginx gateway hitting demo-app.

Manual verification checklist

1. Ensure services are running

   docker compose -f docker-compose.yml -f docker-compose.demo.yml ps

   Confirm grafana, otel_collector, prometheus, tempo, loki, mimir, pyroscope, and the demo services are all Up.

2. Generate telemetry load (optional but recommended)

   DEMO_BASE_URL=http://localhost:18000 \
   DEMO_SMOKE_ITERATIONS=25 \
   DEMO_SMOKE_PAUSE_MS=250 \
   npm --prefix demo-app run simulate

3. Grafana UI — open http://localhost:${GRAFANA_PORT} (default 3000) and check:

   • Dashboard Stack Overview shows host CPU/memory, request rate, job depth, logs pane, and Tempo trace search.
   • Dashboard Demo Service Deep Dive shows request latency histograms, error counts, job durations, queue states, log stream, and slow traces.
   • Folder imported/ contains: Pino HTTP Logs (Loki), Node Exporter Full, Loki Stack Monitoring, Redis, MongoDB, NodeJS Application, and Nginx dashboards — all should show live data once the demo overlay + load generator are running.

4. Pyroscope UI — open http://localhost:${PYROSCOPE_PORT} (default 4040) to inspect live flamegraphs from demo-node-app.

5. Traces & logs — from Grafana Explore you can:

   • Query Tempo datasource for service.name="demo-node-app".
   • Query Loki datasource {service="demo-node-app"} to view structured logs.

Local smoke test (optional)

Spin up data generators before deploying to Proxmox:

# Build the demo app image once
docker compose -f docker-compose.yml -f docker-compose.demo.yml build demo-app

# Start the full stack plus demo generators
docker compose -f docker-compose.yml -f docker-compose.demo.yml up -d

# Optional: run targeted smoke scenario (ensure stack is up)
docker compose -f docker-compose.yml -f docker-compose.demo.yml run --rm \
  -e DEMO_SMOKE_ITERATIONS=30 \
  demo-app npm run simulate
# Or from host with services exposed locally:
#   DEMO_BASE_URL=http://localhost:18000 DEMO_SMOKE_ITERATIONS=30 npm --prefix demo-app run simulate
#   # Optional tunables: DEMO_SMOKE_PAUSE_MS, DEMO_JOB_TIMEOUT_MS

Included demo components (docker-compose.demo.yml):

• demo-app — Node.js (Express) service with MongoDB + Redis + BullMQ, instrumented for traces, metrics, logs (exposed on port 18000)
• nginx + nginx-exporter — reverse proxy in front of demo-app (port ${NGINX_PORT:-18080}) with metrics scraped by Prometheus.
• redis-exporter + mongodb-exporter — Redis exporter plus the custom Node-based Mongo Prometheus bridge in mongo-exporter/.
• telegraf_nginx — Telegraf agent that scrapes nginx stub_status + host metrics and tails the access logs (nginxlog_resp_bytes) for both the built-in nginx and NPMplus instances.
• npmplus — instance NPMplus lokal untuk mencoba konfigurasi proxy host dan menghasilkan log akses/galat yang sama dengan lingkungan produksi.
• demo-load — curl-based traffic generator that now targets nginx to exercise proxy + exporter metrics.
• Pyroscope data is produced directly by demo-app via the Pyroscope SDK
• mongo & redis — backing data stores used by the demo service (ephemeral volumes)
• By default NGINX_PORT maps to 18080; override it in .env if the port is taken.

Grafana dashboards should populate within ~1 minute; Pyroscope and Tempo will display synthetic profiles and traces. Tear down with:

docker compose -f docker-compose.yml -f docker-compose.demo.yml down

NPMplus reverse proxy (demo mode)

The demo overlay also ships NPMplus so you can:

• exercise the proxy-host UI without a second server,
• relay traffic to demo-app through NPMplus, and
• generate the same access/error logs that Telegraf tails for the Grafana dashboard (agent, client_ip, verb, etc.).

Default port mapping:

Component	Default
Public HTTP	http://localhost:${NPMPLUS_HTTP_PORT:-18090}
Public HTTPS	https://localhost:${NPMPLUS_HTTPS_PORT:-18443}
Admin UI	http://localhost:${NPMPLUS_UI_PORT:-18081} (login using NPMPLUS_ADMIN_EMAIL / NPMPLUS_ADMIN_PASSWORD, default admin@demo.local / Demo1234!)
GoAccess UI	http://localhost:${NPMPLUS_GOACCESS_PORT:-18091}

Quick tour:

1. docker compose -f docker-compose.yml -f docker-compose.demo.yml up -d npmplus (already included when you bring up the overlay).
2. Open the UI (default http://localhost:18081) and follow the upstream onboarding wizard to create the first admin user.
3. Add a Proxy Host:
   • Domain: demo.local (or any test domain),
   • Scheme: http,
   • Forward Hostname/IP: demo_app,
   • Forward Port: 8000,
   • Enable “Block Common Exploits” and “Cache Assets”.
4. Run the smoke test (npm --prefix demo-app run simulate) so logs keep flowing.

Telegraf reads both the local nginx log (/var/log/nginx/proxy_access.log) and the NPMplus log directory (/npmplus-data/logs/...). No extra agent is required—Prometheus scrapes a single nginxlog_resp_bytes metric enriched with all the labels your dashboard expects.

Environment overrides: NPMPLUS_HTTP_PORT, NPMPLUS_HTTPS_PORT, NPMPLUS_UI_PORT, NPMPLUS_GOACCESS_PORT, NPMPLUS_ACME_EMAIL, NPMPLUS_ADMIN_EMAIL, NPMPLUS_ADMIN_PASSWORD, and TZ can be set via .env. Set ACME email to a real address when requesting certificates; the default npmplus@local.test is only meant for local testing.

Remote management (Komodo / CI)

Prefer SSH context for remote Docker access instead of exposing 0.0.0.0:2375. Example: DOCKER_HOST=ssh://user@monstack docker compose up -d.

If you must enable TCP API, use TLS on port 2376 and firewall it strictly.

Security notes

• Docker Remote API: use SSH or TLS; avoid 0.0.0.0:2375
• Limit who can reach Grafana/Prometheus/Loki/Tempo/Pyroscope/OTel Collector ports
• Secure OTLP ports if ingesting telemetry from remote hosts
• Change default Grafana admin password in .env
• Tune Loki retention in .env and loki-config.yaml; adjust Tempo/Pyroscope retention and Mimir limits in their configs as needed