GenericLabMonitor is a public reference architecture for whole-homelab monitoring: a fleet of small static Go daemons with Tailscale-style enrollment, server-owned configuration with drift detection, and checksum-verified self-update from git-forge releases, streaming metrics through Kafka into a Java processor and a real-time dashboard — alongside a firewall analytics plane that correlates NetFlow v9, Suricata, and DNS intelligence in ClickHouse into a single server-computed verdict per time range.
Reference architecture, not a supported product. This repository is a scrubbed port of a private production system. It exists so you (and your AI agents) can read it, learn from it, and lift code out of it — not so you can file a ticket. There is no support SLA, no roadmap, and no guarantee any given commit deploys cleanly in your environment. Every hostname is
example.com, every IP is from the documentation range203.0.113.0/24, and every credential is<REPLACE_ME>.
All screenshots in this README come from the compose quickstart
below — the bundled demo-host container plus a second daemon built on the host and
enrolled as host-01.
git clone https://github.com/Comzee/GenericLabMonitor.git
cd GenericLabMonitor/deploy/compose
docker compose up --build -dOpen http://localhost:3000: a single-broker Kafka, Postgres, the processor, the
ingest gateway, and the dashboard build from source, and one containerized daemon
starts reporting the machine it runs on as demo-host — live charts within seconds.
Adding your real machine as a second host is a go build away, and the full
enrollment flow works against the stack too. Details, optional steps, and teardown:
deploy/compose/README.md.
Host detail — live one-second tiles and a per-core heat row:
The daemon fleet page — every daemon's version and liveness, with per-host server-owned configs and one-click self-update behind the buttons:
Most of this stack is well-trodden (Kafka, Spring Boot, React, ClickHouse). Three parts are not, and they are the reason this repo is public.
A ~12 MB static Go agent (daemon/) that you install once and then never
SSH to again:
- Tailscale-style enrollment — the daemon presents a one-time token, and the
server answers with its entire identity: Kafka credentials, the cluster CA, its
metric schedule, and a per-daemon API key (
daemon/enroll.go, processor side inprocessor/). - Server-owned config with drift detection — the fleet is managed from the
dashboard's
/daemonspage, not from a shell on every host. Config is hashed; a daemon whose running config no longer matches the server's shows a drift badge and can be refreshed with one click. - Checksum-verified self-update from git-forge releases — tagging
v*builds all six platforms; each daemon downloads its artifact from the release, verifies the checksum, swaps its own binary, and lets its service manager restart it (daemon/update.go). The flow speaks the Gitea release API but is forge-agnostic in spirit.
Firewall dashboards usually export the interpretation problem to a human staring at
22 panels. Here, NetFlow v9 (inline collector in
firewall-api/server.ts), Suricata IDS events, firewall
syslog, and Unbound DNS telemetry all land in ClickHouse, and the server computes
one verdict — ALL CLEAR / REVIEW / ACTION — from threat-intel matches, RITA-style
beacon scores, DNS entropy/DGA anomalies, and IDS severity, with explicit
benign-context suppression rules (computeVerdict in
firewall-api/server.ts). We are not aware of a named OSS
project that combines these sources into a single computed verdict; if you build one,
start here. Design notes: docs/FIREWALL-REDESIGN.md.
Small, hard-won fixes that most monitoring agents get wrong:
| Detail | Where | Why it matters |
|---|---|---|
| ZFS ARC memory correction | daemon/collect_memext.go, daemon/zfs_linux.go |
ARC is reclaimable cache counted as "used"; without subtracting it, every ZFS host reads perpetually near-full and memory alerts are useless. |
SMART -n standby |
daemon/collect_smart.go |
Polling SMART wakes sleeping NAS disks. -n standby plus a served-from-cache row keeps health visible without ruining spin-down (and smartctl's exit code is a bitmask, not a status). |
| Wake-from-sleep delta resets | daemon/power_watcher_darwin.go, daemon/collect_diskio.go |
Rate metrics are counter deltas; after a laptop sleeps, the first delta spans hours and graphs an absurd spike. Counters re-baseline on wake and on implausible rates. |
| Drop-oldest publish decoupling | the publish queue in runDaemon, daemon/main.go |
Collection never blocks on a stalled broker or POST: on backpressure the oldest buffered sample is dropped and the newest kept, so the daemon degrades to lower resolution instead of falling behind real time. |
| CA-rotation-self-healing Kafka TLS | daemon/enroll.go, kafkaTLSOpts in daemon/main.go |
The cluster CA rides the enrollment config channel and is part of the config hash — rotating the CA propagates to the whole fleet through the ordinary refresh-and-restart path, with zero manual redeployment. |
The daemon's metric-collection layer is derived from Beszel (MIT) — collection patterns for network rates, per-core CPU, memory, Docker containers, systemd units, and SMART began as adaptations of the Beszel agent, and the derived files carry attribution comments. Everything above the collector layer — enrollment, server-owned config, self-update, Kafka publishing, the processor, and the dashboard — is original to this project. Thanks to henrygd for an excellent, readable agent.
╔═══════════════ Kubernetes cluster · namespace glm ═══════════════╗
LAN host daemons ────► ║ ║
(Go, Kafka mode, ║ Kafka (Strimzi ×3, RF=3) ──► processor ×2 (Spring Boot) ║
SASL_SSL :9095) ║ ▲ │ ║
node-collector ──────► ║ │ ├──► PostgreSQL (CNPG) ║
(DaemonSet) ║ │ ├──► WebSocket (STOMP) ──► dashboard ×2 (React)
║ │ daemon-commands └──► REST API ║
remote daemons ──────► ║ gateway ×2 (Go) ◄──────────────────┘ ║
(HTTPS + OAuth2 JWT) ║ ║
║ vector-ingest ×2 ──► ClickHouse ◄── firewall-api (Bun, ║
firewall appliance ──► ║ Loki ◄── vector-agent DaemonSet NetFlow v9 collector) ║
(syslog · Telegraf · ╚══════════════════════════════════════════════════════════════════╝
NetFlow v9)
Off-cluster host (systemd): network-api :3007 · apps-api :3006 · api-meter
Front door: reverse proxy / tunnel + SSO forward-auth ──► Gateway VIP (203.0.113.20)
Four loosely-coupled planes share one dashboard:
- Metrics plane — daemons → Kafka → processor → PostgreSQL → WebSocket → dashboard. The real-time heart of the system.
- Security analytics plane — firewall telemetry (syslog, Telegraf, NetFlow v9,
Suricata, Unbound DNS) → ClickHouse, queried by
firewall-apifor the/firewallpage's verdict, threat intel, beacon detection, and DNS anomaly hunting. - Logging plane — cluster pods and off-cluster services ship structured logs to
Loki, surfaced on the
/logspage. - Auxiliary collectors — LAN device inventory (
network-api), Kubernetes app health (apps-api), external-API traffic metering (api-meter), environmental sensors (/observatory).
| Component | Source | Runs as | Role |
|---|---|---|---|
| daemon | daemon/ |
systemd / rc.d / launchd / SCM on every fleet host | Metrics agent: 1 s headline metrics + staggered deep families; self-updating; server-owned config |
| node-collector | daemon/ (container build) |
k8s DaemonSet | The same agent, giving host-true metrics for the cluster nodes themselves |
| gateway | gateway/ |
k8s Deployment ×2 | HTTPS ingest for off-LAN daemons → Kafka; piggybacks queued commands onto ingest responses |
| processor | processor/ |
k8s Deployment ×2 | Kafka consumer, PostgreSQL persistence (raw SQL + Flyway), REST API, STOMP WebSocket broadcaster, alert engine, retention/aggregation jobs |
| dashboard | dashboard/ |
k8s Deployment ×2 | Vite + React SPA: live canvas sparklines, ECharts history, WebSocket-fed pages |
| firewall-api | firewall-api/ |
k8s Deployment (LB 203.0.113.50) |
Bun server: ClickHouse named-query API, inline NetFlow v9 collector, verdict engine, KPI publisher to Kafka, threat-intel/beacon/DNS-intel jobs |
| network-api | network-api/ |
systemd on an off-cluster host (:3007) |
LAN device inventory: liveness sweeps, periodic deep scans (nmap), enrichment, SQLite, event feed |
| apps-api | apps-api/ |
systemd (:3006) |
Read-only health collector for your k8s apps: registry-driven probe battery, per-app capabilities |
| api-meter | api-meter/ |
systemd | iptables accounting-chain byte meter for any external API's traffic → Kafka |
| vector-ingest | deploy/k8s/ |
k8s Deployment ×2 (LB 203.0.113.40) |
Receives firewall syslog + Telegraf, parses into ClickHouse with disk buffers |
| Loki + vector-agent | deploy/k8s/ |
k8s (LB 203.0.113.60) + DaemonSet |
Centralized logs, 30-day retention, backing the /logs page |
| Kafka | deploy/k8s/kafka-strimzi.yaml |
Strimzi CRs | 3-broker KRaft cluster; per-client users and ACLs in kafka-users.yaml |
| PostgreSQL | deploy/k8s/cnpg-cluster.yaml |
CNPG cluster glm-db |
System of record: metrics, state, alerts, daemon configs; in-cluster only |
| ClickHouse | deploy/k8s/clickhouse.yaml |
StatefulSet | Analytics store: firewall logs, NetFlow, Suricata, DNS, threat intel |
Deployment manifests, the placeholder IP plan, and apply order live in
deploy/k8s/README.md.
One static Go binary, six platforms. It enrolls once with a token and from then on
the server owns its configuration. Every second it publishes headline CPU / memory /
disk; on staggered beats it publishes deep families: systemd services, SMART and
mdraid disk health, Docker containers, per-disk usage and iostat, temperature
sensors, per-core CPU, uptime, and a 60-second network census (listening ports with
owning process, connection counts, per-interface rates, peers). LAN daemons talk
SASL/TLS Kafka directly; remote daemons POST over HTTPS with OAuth2 client-credential
JWTs and receive queued commands on the response. Internals:
docs/howthedaemonswork.md.
/ overview · /stats per-host live cards · /hosts/:hostId tabbed host detail
(sparklines, per-core heatmap, sensors, storage + SMART, services, containers) ·
/alerts · /daemons fleet management (versions, configs, enrollment tokens, drift
badges, update triggers) · /infra Kafka + WebSocket health · /network LAN
inventory · /firewall verdict + explorers · /logs Loki explorer · /api-usage
external-API meter · /observatory environmental sensors · /apps k8s app monitor ·
/cicd pipeline health.
Live rendering is deliberately two-tier: a pure-canvas sparkline component (shared
requestAnimationFrame loop; data flows through closures, bypassing React re-renders)
handles the 1 Hz charts, while ECharts (with animation: false, always) handles
historical time-series with median-based gap detection.
Metrics (LAN): daemon collects (gopsutil) → JSON to Kafka topic host-metrics,
keyed by host → processor validates, persists, updates host status, evaluates alert
thresholds → broadcasts STOMP /topic/metrics|status|alerts after the DB transaction
commits → dashboard merges into pre-loaded chart buffers.
Metrics (remote): daemon POSTs to the gateway's /api/ingest over HTTPS (OAuth2
JWT minted by your SSO provider) → gateway validates shape and republishes to Kafka →
identical pipeline from there. Queued commands ride back on the ingest response.
Commands: dashboard/REST → processor. Kafka-mode daemons consume the
daemon-commands topic instantly; HTTP-mode daemons drain a server-side queue on
their next poll.
Security analytics: the firewall appliance streams syslog + Telegraf to
vector-ingest and NetFlow v9 to the firewall-api collector, all landing in
ClickHouse. KPI snapshots additionally flow through Kafka → PostgreSQL → WebSocket
like any host metric — a hybrid path that keeps live tiles cheap while deep analytics
stay ClickHouse-strength. Scheduled jobs maintain threat-intel feeds, beacon scores,
and DNS anomaly detection.
Logs: cluster pods (vector-agent DaemonSet) and off-cluster services (native
vector) ship structured JSON to Loki, browsable on /logs.
| Data | Store | Retention |
|---|---|---|
| Raw 1 s metrics | PostgreSQL host_metrics |
7 days (also the raw→aggregate switchover for the history API) |
| Hourly/daily aggregates | PostgreSQL host_aggregates |
365 days |
| Metric families (services, disks, containers…) | PostgreSQL host_family_state |
latest state; reaped when idle > 24 h |
| Alerts | PostgreSQL | resolved alerts pruned after 7 days |
| Firewall logs / NetFlow / DNS / Suricata | ClickHouse | 30 days |
| Telegraf system metrics | ClickHouse | 7 days |
| Logs (all services) | Loki | 30 days (720 h) |
| LAN device inventory + events | SQLite | indefinite |
The history API returns ~500 points regardless of range: PostgreSQL date_bin()
downsampling over raw rows inside the retention window, pre-computed aggregates
beyond it. Default alert thresholds — CPU 80 %, memory 85 %, disk 90 %, host offline
after 10 s of silence — are tunable per host from the dashboard.
- One SSO front door. Every browser-facing path sits behind forward-auth at the reverse proxy (any OIDC provider works; the reference deployment used Authentik). API and WebSocket paths return JSON 401 on session expiry so the SPA can re-authenticate cleanly instead of receiving an HTML login page.
- Deliberate carve-outs only. Daemon enrollment and command polling are token-authenticated by the processor itself: rate-limited, hashed at rest, with the enrollment token doubling as the per-daemon API key.
- Kafka zero-plaintext. Every listener is SCRAM-SHA-512 over TLS with per-client
users and enforced ACLs. Each daemon host gets its own credential, delivered
through enrollment along with the cluster CA; only the processor may write the
command topic. Roadmap and rationale:
docs/KAFKA-SECURITY-ROADMAP.md. - Secrets stay out of git. Kubernetes Secrets (some created imperatively — see
the table in
deploy/k8s/README.md), root-only env files on hosts, and enrollment-delivered daemon credentials. This public repo contains placeholders only.
The design goal is that every tier survives the loss of one node:
- Stateless services ×2, zone-spread (processor, gateway, dashboard, vector-ingest) with PodDisruptionBudgets. The processor pair coordinates through ShedLock (single-leader scheduled jobs) plus a Kafka fan-out relay so both pods' WebSocket clients receive every broadcast.
- Kafka: three brokers, RF=3, min-ISR=2, one broker per fault domain
(
topology.example.com/rack). - PostgreSQL: CNPG cluster with automated failover. ClickHouse: replicated block storage across zones; vector's disk buffers cover the failover gap so no telemetry is lost.
- Front door: run two reverse-proxy legs on separate hosts (the manifests assume
203.0.113.100and203.0.113.101) so either can carry all traffic. - Fleet resilience: daemons buffer through broker outages with drop-oldest semantics, and every platform's service manager restarts them unconditionally.
- Go for the agent — zero-dependency static binary, trivial cross-compilation, the same artifact from a Raspberry Pi to a Windows desktop.
- Kafka as the spine — decouples collection from processing, absorbs outages, and gives every new producer (firewall KPIs, the API meter) a uniform on-ramp.
- 1-second resolution with aggressive retention tiers — responsive live charts without unbounded storage; server-side downsampling keeps payload size constant.
- Server-owned daemon config — manage the fleet from the dashboard, not by SSH-ing to every machine.
- Right store for the job — PostgreSQL for stateful truth, ClickHouse for analytical scans, Loki for logs, SQLite for a single-writer inventory.
- Native WebSocket + canvas rendering — STOMP over native WS (no SockJS) and React-bypassing canvas sparklines keep dozens of 1 Hz charts smooth.
- Verdict-first security UX — compute one server-side answer instead of asking a human to interpret 22 panels of perimeter noise.
- SSO everywhere, carve-outs on purpose — one door, plus explicitly enumerated, token-authenticated machine paths.
docs/BUILDING.md— toolchains and build commands for every componentdocs/howthedaemonswork.md— enrollment, fleet management, self-update internalsdocs/ADDING-FIELDS.md— the end-to-end recipe for adding a metric fielddocs/FIREWALL-REDESIGN.md— the verdict plane's designdocs/APPS-MONITOR.md— the k8s app monitordocs/KAFKA-SECURITY-ROADMAP.md— how the bus went zero-plaintextdocs/PREEMPTION-FORENSICS.md— recovering k8s preemption evidence past the apiserver's event horizondocs/GOTCHAS.md— landmines to avoid when modifying the codedeploy/k8s/README.md— manifests, IP plan, apply orderdeploy/compose/README.md— the one-command local quickstart
Small, focused PRs are welcome — bug fixes, doc-drift corrections, and
correctness details in the spirit of the ones above. Large features won't
merge, and there is no support SLA. Expectations and ground rules:
CONTRIBUTING.md.
MIT — see LICENSE. Beszel-derived portions of the daemon retain their
original MIT license and attribution.