Repository navigation
WatchLog v0.4.0
WatchLog 0.4.0
Why is this broken? WatchLog 0.4 names the probable cause of every alert and every problem, on Kubernetes clusters and Docker hosts, and shows the evidence: "web fails because api was scaled to 0: nothing answers its calls", with the steps behind it. Fixed rules over the data WatchLog already collects; no AI in the product, no scores. Community now holds any number of clusters, Docker fleets and nodes.
What's new
- Probable cause (ADR-146 to ADR-150, guide):
- Likely, Possible or No clear cause, in words. A likely cause has a complete chain of evidence that began before the symptom; a possible one is a lead to check; with no clear cause, "What we checked" lists every check and what it found.
- Twelve rules, each with its Kubernetes and Docker form: a dependency that's down, a bad change (a rollout, image change or re-create, or a dependency that turned slow right after its own), out of memory at the limit, crashes, a node or host (NotReady, cordoned, under pressure, silent, rebooted), an application finding (a queue whose consumer was OOM-killed, PostgreSQL out of connections and who holds them), connections that fail, a name that doesn't exist (NXDOMAIN), CPU throttling, and, on Kubernetes only, an image that can't be pulled, a pod that can't be scheduled and a failed Job.
- Across the service map: the walk follows failing, slow or slower-than-usual calls up to two hops, so a frontend's alert can end at the dependency that was scaled to 0.
- The panel on the alert detail (with the cause's moments on the chart), the pod and workload pages, the Docker container and Compose service pages, the service map's edge panel and the application page beside a finding. Copy as text gives the sentence and the chain for a ticket or a chat.
- Needs attention: one line under each problem, a Likely or Possible chip with its sentence.
- Notifications (ADR-157): Slack, Telegram and email get a "Likely cause: …" line when the cause is likely, worked out as the alert fires within 2 seconds; the resolve says "Was likely: …". The webhook gains
likelyCause(alerts and the webhook payload). - API and MCP (ADR-155):
GET /api/v1/causesreturns the whole answer; the alert detail, the problems and the applications carry an optionallikelyCause. MCP gainsexplain_problem(14 tools),explain_alertreturns the causes, andget_problemseach problem's likely cause.
- Community is unlimited (ADR-151): any number of Kubernetes clusters, Docker fleets and nodes in one installation, with the cluster switcher between them.
- CPU throttling alerts on the share of the time (ADR-160): "Container CPU throttling" fires when a container is throttled at its CPU limit 25 % of the time for 15 minutes ("throttled 40% of the time"). Docker gets the same rule, "Docker container CPU throttling". A container that touches its limit for a moment no longer fires.
- The overview (ADR-159): two columns that stack on their own, Needs attention with the first six problems worst first and "+N more in Alerts", a chart of fixed height. Charts break their line where there is no data instead of drawing across an agent's silence. The traffic tile keeps one line.
- The service map fits the desktop (ADR-158): the map fills the height and the page doesn't scroll; the panel shows the numbers and the probable cause first, then its charts in sections you open.
Fixed
- PostgreSQL connections were counted twice: 37 of 40 read as 64 (160 %), so the connections-near-max finding and rule fired early. The totals count each connection once.
- A Docker re-create went unmarked when the new container was stored before the old one was seen gone (two of five Compose re-creates on the test host). Re-creates and their image changes are marked.
- A Docker crash loop whose process runs a while before it exits was never in Needs attention. Restarted by its policy 3 times or more in 5 minutes, it is "keeps restarting".
- Host and Docker restarts aren't crashes: after a node reboot every pod's container ends "Unknown" (exit 255), and pods were "crash-looping" for 10 minutes; containers stopped with their Docker host's reboot or their daemon's restart stayed "exited" problems for a day. Now a pod restarted by its node's reboot isn't crash-looping, and a container its host or daemon stopped, and its restart policy left stopped, isn't a problem.
- Restarts in the last hour no longer count an old pod's or container's whole lifetime.
- Failed connections never read above 100 % ("102% of connections failed": the two counters are scraped apart).
- Demo mode's query budget: the reads the server makes for itself counted against each visitor's 120 queries a minute.
- Service map destinations right after an agent starts (ADR-158): for about 35 s after a node agent started, before it had the cluster's address map, pods, Services and nodes showed as external services. Private addresses now wait for the map. A stopped container or a deleted pod keeps its address for an hour, so a client still dialling it fails to reach that dependency instead of drawing a new external.
- A keep-alive connection that got slower read as gone ("stopped since …") when no new connections were opened; bytes and requests count as traffic now.
- Times in cause sentences on another day carry the date ("restarted at Nov 10, 10:33").
- Go 1.27.4 for the agent (GO-2026-6859 in
net/http).
Upgrading
No manual steps. Kubernetes: helm upgrade … --version 0.4.0 --reuse-values. Docker: set WATCHLOG_VERSION=0.4.0 and docker compose pull && docker compose up -d for the server; run the agent's install command again on each host (and the integrations collector's, where it runs) for the agent's fixes.
- Migrations run by themselves: alerts keep the likely cause they fired with (
alert_states.cause), and the throttling rule moves to its new value. - If you changed the "Container CPU throttling" threshold, it goes back to 25 %, now a share of the time: the old value (throttled ÷ used CPU) can't be translated. Its switch, channels, severity and duration stay. The old metric stays for custom rules.
- "Docker container CPU throttling" is a new default rule on Docker fleets, on like the other defaults.
Known limits
- A cause comes from the last 60 minutes and at most two hops along the service map; the rules that follow the map need network flows (eBPF).
- MongoDB comes after the launch, Elasticsearch in 0.5. English only.
Questions and problems: GitHub Discussions or an issue.
Changes since v0.3.1
Features
- the Docker throttling rule and the share metrics; the Docker status strip says container and host (web)
- CPU throttling alerts on the share of the time, on Kubernetes and Docker (backend)
- the overview below the tiles is two columns that stack on their own (web)
- the overview's cards keep their height; Needs attention shows six, unclipped (web)
- reboots aren't crashes, a cause adds a fact, Needs attention worst first (backend)
- the service map fits the desktop; the panel folds its charts into sections (web)
- the headlines from @watchlog/cause-words; the timeline says which message carried the likely cause (web)
- the likely cause in Slack, Telegram, email and the webhook (v0.4 slice 7) (backend)
- @watchlog/cause-words, the probable-cause headlines the web and the backend share
- the probable-cause panel in six places and the Needs-attention line (v0.4 slice 6) (web)
- causeState and possibleCauses on the Needs-attention problems (v0.4 slice 6) (backend)
- MCP explain_problem, explain_alert's causes, get_problems' likelyCause (v0.4 slice 5) (backend)
- likelyCause on the problems, the alert detail and the applications (v0.4 slice 5) (backend)
- the sentences of the rules under the object (web)
- probable-cause rules under the object (v0.4 slice 4) (backend)
- the sentences of the rules across the map (web)
- probable-cause rules across the map (v0.4 slice 3) (backend)
- the sentences of the rules on the object (web)
- probable-cause rules on the object (v0.4 slice 2) (backend)
- probable-cause sentence codes and their helper (web)
- probable-cause engine and GET /causes (v0.4 slice 1) (backend)
- drop the Community one-cluster message (ADR-151) (web)
- no cluster limit in Community (ADR-151) (backend)
Fixes
- the MongoDB service's note names no release (web)
- the release notes print their highlights (release)
- a headline moment on another day carries its date
- a keep-alive edge that got slower is in use, not gone or quiet (backend)
- the traffic tile's sub-line is the failures and the latency, on one line (web)
- charts break their line where there is no data (web)
- a stopped Docker container's cause reads its own exit code (backend)
- restarts in the last hour no longer count an old object's lifetime (backend)
- in-cluster addresses before the first IP map, and addresses that leave the map (agent)
- the failed-connections share never reads above 100% (backend)
- a container the node restarted is no crash_exit (backend)
- Go 1.27.4 for GO-2026-6859 in net/http (agent)
- a Docker edge alert's message names its container, not "undefined/…" (backend)
- the webhook's likelyCause keeps its documented key order (backend)
- a dipped minute doesn't restart cpu_throttled's run (backend)
- a resolved alert's steps start when its trouble did, not after it (backend)
- the Needs-attention Possible line reads where each cause starts (web)
- each possible cause on a problem says where it starts (backend)
- no space before % in the probable-cause and finding sentences (web)
- a step's logo stays with its object's name, which wraps at spaces (web)
- "What we checked" keeps its label on one line on phones (web)
- 44 px targets for the edge panel's close button and links on touch screens (web)
- in-process reads count once against the demo's query limit (backend)
- a Docker OOM kill's start ignores an OOMKilled event first seen before the window (backend)
- a Docker crash loop whose process runs a while is a restarting problem (backend)
- a Docker re-create is marked when a delta stored the new container first (backend)
- PostgreSQL connection counts leave out the per-database backends (backend)
Install
Kubernetes (backend, web, agents in one chart):
helm install watchlog oci://ghcr.io/watchlog-light/charts/watchlog --version 0.4.0 \
--namespace watchlog --create-namespaceDocker hosts: see the install guide.
Verify
Images and charts are signed with cosign (keyless, GitHub Actions OIDC):
cosign verify ghcr.io/watchlog-light/agent:0.4.0 \
--certificate-identity-regexp '^https://github.com/watchlog-light/watchlog/\.github/workflows/release\.yml@refs/tags/v' \
--certificate-oidc-issuer https://token.actions.githubusercontent.com