Repository navigation
WatchLog v0.5.0
WatchLog 0.5.0
The launch release. WatchLog 0.5 reads four more of the services people run next to their workloads (MySQL and MariaDB, MongoDB, Traefik and Kafka), each with its own page, default alerts and the probable cause it can name: "inventory's writes wait on a row lock in shop.stock: shop/inventory-lock holds it." A container's last log lines are one click away and never stored, requests and limits are compared with what each container really uses, and ⌘K finds anything by name. Everything on Kubernetes clusters and Docker hosts alike.
What's new
- MySQL and MariaDB (ADR-163, guide): one integration for both, named by what the server reports ("MySQL 8.4", "MariaDB 11.4"); MySQL 8.0+ and MariaDB 10.6+.
- Connections and who holds them, statements by digest with their latency (performance_schema), lock waits with the table and the session holding the lock, the buffer pool, replication and the largest tables, on six tabs. Never table data.
- Default alerts: connections above 90% of max_connections, a query running over 5 minutes, replication stopped or lagging, deadlocks.
- Probable cause follows a lock wait to the workload or container holding the lock, and connections near the limit to the client holding most of them.
- MariaDB without performance_schema (its default) still works; the page says what that leaves empty and the one line to turn it on.
- MongoDB (ADR-164, guide): 6.0 and later, standalone and replica sets, with a
clusterMonitoruser.- Operations and their latency, connections by workload, replica set members, replication lag and the oplog window, slow queries kept as their shape (field names and operators, never values), collections and indexes.
- Default alerts: no primary, a member down, replication lag, the oplog nearly full, slow operations.
- Probable cause names the client sending slow collection scans, and the client holding most connections near the limit.
- Traefik (ADR-165, guide): v2.10+ and v3, k3s's bundled Traefik included, from its own Prometheus metrics.
- Requests and errors per entrypoint and service, latency, the servers behind each service, certificates.
- On the service map, what eBPF can't see: the requests Traefik answers itself (503 when a service has no server up), and its per-service stats on the edges to the backends.
- Default alerts: a service with no server up, high latency, a failed configuration reload, a certificate about to expire.
- Probable cause follows "no server up" to the backend that was scaled to 0, crashed or was OOM-killed.
- Kafka (ADR-166, guide): Kafka 3.0+, ZooKeeper or KRaft, read-only through the Kafka protocol (no JMX); SASL and TLS when the listener wants them.
- Brokers and the controller, offline and under-replicated partitions, topics by size and messages in, consumer-group lag per topic and its trend.
- The consumer group that lost its members: lag growing with no member left, and probable cause follows it to the consumer that was OOM-killed, crashed or changed. A Prometheus endpoint (JMX exporter, Strimzi, Bitnami) on its usual port is read too.
- Default alerts: a broker down, offline or under-replicated partitions, lag growing, a group that lost its consumers.
- Container logs on demand (ADR-167, guide): the last 100 to 1,000 lines of a container on the pod and Docker container pages, the previous run's too, and followed live for up to 5 minutes. Streamed through the backend and never stored (no database, no metrics store, no agent buffer); redacted in the agent before they leave the cluster or host. Not log management: no search, no history. Every role may read them by default, Viewers included; Settings → General → Container logs narrows it to Owners and Admins or turns it off, and each read is in the audit log. MCP gains
get_container_logs. - Right-sizing hints (ADR-168, guide): each container's requests and limits against the last 7 days of use, in plain sentences, no scores: "web asks for 500m CPU; its busiest pod used at most 60m (p95) over 7 days. About 80m would cover it." OOM risk, CPU throttling at the limit, requests too low or too high, and what isn't set; on Docker, limits, memory reservations and containers with no memory limit. On the workload, Docker container and Compose service pages, an overview card,
GET /clusters/{id}/right-sizingand MCPget_right_sizing(16 tools). - ⌘K (ADR-169): ⌘K or Ctrl+K (or
/outside a field, or the top bar's search button) finds pages, clusters and Docker fleets, and the selected cluster's workloads, pods, nodes, hosts, containers, Compose projects and applications. Letters in order are enough (pay7d9findspayments-api-7d9f…), one typo is forgiven, and the last picks show before you type. Keyboard-first, a screen reader hears the count, full screen on phones.GET /clusters/{id}/searchbehind it. - For Docker-in-Docker and shared hosts (ADR-170, configuration):
WATCHLOG_EBPF_CONTAINER_FILTERkeeps the Docker agent's eBPF to the containers of its own daemon, so nothing else on the machine is recorded. The seed file'sclustersregisters a Docker fleet by its token's SHA-256 and prefix, never the token.
Fixed
- Discovery offered clients as services (ADR-172): a pod or container that runs
mysql,mongo,postgresorredisonly for its client (a job runningpsql, a pod runningmongosh) showed under "Found N services". One that connects out on the type's port, that nothing connects to there, and that declares or publishes no such port, isn't offered now. Without network flows nothing changes. - The Docker service map drew servers as standalone: containers that never connect out (
api,db,redis) sat under "Standalone" by their container name instead of in their Compose project by their service's name; so did a replica-set member reached by the others. - PostgreSQL's connection holder: probable cause named the last client listed, not the one holding most connections.
- "90%", not "90 %": the metric rules' alert summaries, the default rules' names and the MCP sentences write percentages without the space.
- A probable-cause step under a MariaDB or Valkey service names MariaDB or Valkey, not MySQL or Redis.
- MariaDB without performance_schema shows the one hint that says how to turn it on, not two "needs permission" lines beside it.
- MongoDB slow scans name the client sending them in the last 10 minutes, so an hourly job's new pod is named, not the one before.
- MongoDB connections near the limit on a replica set: a full primary beside an idle secondary read as half full (100 of 190), so "connections near max" never showed; each member is now measured against its own limit.
- An idle PostgreSQL server (nobody but WatchLog connected) showed an empty Overview and no connections in the list; it shows 0 now. (A 0.4 regression.)
Upgrading
No manual steps (ADR-107). Kubernetes: helm upgrade … --version 0.5.0 --reuse-values. Docker: set WATCHLOG_VERSION=0.5.0 and docker compose pull && docker compose up -d for the server; run the agent's install command again on each host, and the integrations collector's where it runs, for logs on demand, the reservation metric and the new integrations.
- Migrations run by themselves: the new integration types, and the logs setting with its audit rows. The new Helm values are read nil-safe, so
--reuse-valuesfrom 0.4 works. - An upgrade turns logs on demand on, and grants the cluster-agent
getonpods/log. To keep it off:logs.enabled: falsein thewatchlog-agentchart (watchlog-agent.logs.enabled: falsein thewatchlogchart; the grant goes too),-e WATCHLOG_LOGS=falseon the Docker agent, or Settings → General → Container logs → Off for the whole installation. Every role, Viewers included, may read logs until you change it (who can read logs). - Docker memory-reservation hints need 0.5.0 agents: older agents don't send the reservation, so those containers simply get no reservation hint. Everything else in right-sizing works with 0.4 agents.
- Default rules renamed without the space ("Redis memory above 90% of maxmemory", "PostgreSQL connections above 90% of max_connections", "Nginx 5xx above 5%"): only rules whose name is still the shipped one. A rule you renamed keeps your name.
- The new integrations need the 0.5.0 integrations collector; existing ones keep working with 0.4's.
Known limits
- MongoDB: a member that's down is named, but probable cause doesn't yet follow it to its pod's or container's own cause (OOM, crash, node). A
mongosreads as a member; shards behind it aren't discovered. - Kafka: no JMX, so bytes in and out and request latency only come from a Prometheus endpoint when you have one. Rates are offset growth ("offsets/s").
- Logs: Docker's
json-file,localandjournalddrivers only; the link from a crash loop's probable cause to its last lines comes later. - Right-sizing needs 24 hours of data per container and judges at most the last 7 days.
- Elasticsearch, MinIO and Google sign-in come after the launch, in the order users ask. English only.
Questions and problems: GitHub Discussions or an issue.
Changes since v0.4.0
Features
- seeded clusters, registered by their token's SHA-256 (ADR-170) (backend)
- WATCHLOG_EBPF_CONTAINER_FILTER, the Docker-mode eBPF filter (ADR-170) (agent)
- the ⌘K command palette (ADR-169) (web)
- GET /clusters/:id/search, the ⌘K palette's name search (ADR-169) (backend)
- right-sizing on the workload, Docker container and Compose service pages, and an overview card (ADR-168) (web)
- right-sizing hints: GET /clusters/:id/right-sizing and MCP get_right_sizing (ADR-168) (backend)
- the Docker container's memory reservation, for right-sizing (ADR-168) (agent)
- a Logs section on the pod and Docker container pages, and Settings → Container logs (ADR-167) (web)
- logs on demand: the relay, the routes, the Settings switch, the audit row and MCP get_container_logs (ADR-167) (backend)
- container logs on demand: pods/log on the cluster-agent, the Docker logs endpoint, redacted in the agent (ADR-167) (agent)
- Kafka: the page's strings, reasons, logo, rule metrics and cause words (web)
- the Kafka page, its reasons, rules, probable cause and MCP (ADR-166) (backend)
- Kafka integration through the admin API (franz-go kgo/kadm, ADR-166) (agent)
- Traefik: the page's strings, reasons, logo, rule metrics, and its stats on the service map (web)
- the Traefik page, its reasons, rules, probable cause, MCP, and its stats on the service map (ADR-165) (backend)
- Traefik integration, its Prometheus metrics read with promtext (ADR-165) (agent)
- MongoDB: the page's strings, reasons, rule metrics, the query-text switch (web)
- the MongoDB page, its reasons, rules, probable cause and MCP (ADR-164) (backend)
- MongoDB's oplog fullness, and the replica set's name on each member row (agent)
- MongoDB integration (official driver v2, ADR-164) (agent)
- MySQL and MariaDB: the page's strings, reasons, the logo, rule metrics, "90%" (web)
- the MySQL and MariaDB page, its reasons, rules, probable cause and MCP (ADR-163) (backend)
- MySQL and MariaDB integration (go-sql-driver, ADR-163) (agent)
Fixes
- MongoDB connections near the limit read each member's own limit (backend)
- an idle PostgreSQL server has 0 connections, not none (backend)
- the Docker map draws servers that never connect out in their Compose project (backend)
- a cause step names MariaDB or Valkey when that's what runs (web)
- a hint replaces the needs-permission lines it covers (backend)
- a cause step under an application carries the product the collector found (backend)
- discovery leaves out client pods and containers that run a server's image (backend)
- a slow scan's client is the one sending it in the last 10 minutes (agent)
- the 5xx reason without a share, when the 5xx are Traefik's own 503s for a backend (web)
- Traefik's backends as measured on the e2e stack: a Service scaled to 0, a label-named Docker service, edges with requests (backend)
- MongoDB's member_down carries the member's state as its error, as the web renders it (backend)
- the connection holder probable cause follows is the first client, not the last (backend)
Install
Kubernetes (backend, web, agents in one chart):
helm install watchlog oci://ghcr.io/watchlog-light/charts/watchlog --version 0.5.0 \
--namespace watchlog --create-namespaceDocker hosts: see the install guide.
Verify
Images and charts are signed with cosign (keyless, GitHub Actions OIDC):
cosign verify ghcr.io/watchlog-light/agent:0.5.0 \
--certificate-identity-regexp '^https://github.com/watchlog-light/watchlog/\.github/workflows/release\.yml@refs/tags/v' \
--certificate-oidc-issuer https://token.actions.githubusercontent.com