Report authoritative provider window failures

Author: bitprecipice

monitor/SIGNALS.md

@@ -6096,7 +6096,7 @@ error CLASS, not the volume. Classes, causes, and the action each implies:
 | `LOADING` / `READONLY` | Node restarting (rdb load) / replica mid-failover. Transient; retried in-client. | Only alert if sustained > 2 min. |
 | `[redis][ttl]` (server-side guard, server/redis_ttl_warn.go) | A redis write carried an effective ttl beyond its family limit, or a raw Go `time.Duration` command/eval arg. Raw Durations serialize as int64 NANOSECONDS, so an 8h ttl can become `EXPIRE <key> 28800000000000` (~913,000 years); alternatively, a correct `EXPIREAT` can expose an unbounded durable deadline. The 2026-07-20 signature was ~1.1M immortal legacy `s_sk_*` stream keys. | The warning names the command + redacted key family. For raw Duration, pass seconds/ms ints and clean the affected family. For a long `EXPIREAT`, preserve authoritative data and bound only the Redis mirror horizon; see §5.11. |
 | `providertunnel: tun read error: Done` (`provider-tunnel-read-done`) | `Tun.Read` returned terminal `Done`; the line alone proves neither outer context state nor active artifact ancestry. On an artifact proven to predate `20e289bd`, it is consistent with ordinary canceled teardown reaching the unconditional legacy logger. On a proven descendant, the fix would suppress only a canceled-context read error, so recurrence is an affirmative unexpected Tun/context close-order fault. The locally inspected `v2026.9.3-1036806790` tag lacks the fix, but tag ancestry is not runtime provenance. | Prove the active Taskworker artifact first. Deploy a containing Taskworker only if it predates `20e289bd`; otherwise diagnose the close-order/context fault. Require zero exact lines for 10 minutes through comparable ProviderEgress churn. Never suppress another TUN read error, infer cancellation from `Done`, or restart an unproven release. |
-| `[rel] event=window_stall ... failed=0` (`window-stall`), `[rel] event=window_failed ... after=<milliseconds>` (`window-stall-terminal`), or compatibility `window_stall ... failed=1` | Connect emits `window_stall failed=0` when the bounded reason changes while a provider window is still trying. `failOutcome` instead emits one authoritative `window_failed` after the second zero-provider deadline and then calls `SetStallStatus` directly; that dispatch does not normally produce `window_stall failed=1`, which remains accepted only as a compatibility shape. The terminal event warns on its first line. The 2026-09-08 watcher initially mislabeled a 26/min `failed=0` shape as `novel`, and the 2026-09-09 watcher missed real `window_failed` lines while waiting for the normally absent `failed=1`; the exact classes now preserve both states, while malformed fields remain novel schema drift. | Branch on the bounded reason and correlate the same window with provider progress plus explicit transport/framer/reachability, provider-response, rate-limit, or authentication evidence. Do not infer terminal impact, restart Taskworker, or deploy a transport change from `failed=0`; do not infer a root cause from `window_failed` or compatible `failed=1` alone. Require nonterminal churn below 20/min and no terminal event for ten minutes under comparable traffic, with provider windows reaching their configured minimum. |
+| `[rel] event=window_stall ... failed=0` (`window-stall`), `[rel] event=window_failed ... after=<milliseconds>` (`window-stall-terminal`), or compatibility `window_stall ... failed=1` | Connect emits `window_stall failed=0` when the bounded reason changes while a provider window is still trying. `failOutcome` instead emits one authoritative `window_failed` after the second zero-provider deadline and then calls `SetStallStatus` directly. That dispatch does not itself produce `window_stall failed=1`, but a later reason change can publish that compatibility transition while the failed latch remains set. The class rate intentionally counts both diagnostic lines; `failed_window_events` separately counts exact-replay-deduplicated authoritative events. Compatibility-only evidence keeps the alert but renders that cardinality unknown. The 2026-09-08 watcher initially mislabeled a 26/min `failed=0` shape as `novel`, and the 2026-09-09 watcher missed real `window_failed` lines while waiting for `failed=1`; the exact classes now preserve both states, while malformed fields remain novel schema drift. | Branch on the bounded reason and correlate the same window with provider progress plus explicit transport/framer/reachability, provider-response, rate-limit, or authentication evidence. Do not infer incident size from the diagnostic line rate, terminal impact from `failed=0`, or a root cause from `window_failed` or compatible `failed=1` alone. Do not restart Taskworker or deploy a transport change from these lines. Require nonterminal churn below 20/min and no terminal event for ten minutes under comparable traffic, with provider windows reaching their configured minimum. |
 | `[multi]window enumerate error timeout = generator call canceled` or `[multi]create client args error = generator call canceled` (`window-generator-canceled`) | The exact text is artifact- and context-dependent; it does not prove that the owning window was canceled. On Connect with legacy log-before-context ordering, a population paired with nonterminal `platform-unreachable` stalls is consistent with ordinary teardown being falsely recorded as a platform error. Fixed Connect suppresses only an error observed after authoritative outer cancellation, so recurrence on a proved fixed artifact establishes that an inner generator returned the identical text while the outer context was live at the guard. Exact `generator call abandoned after ...` and every other suffix remain separate hung-call/live-error evidence. | At 20/min WARN, prove the emitting artifact's recorded Connect build input under §8.12. Deploy the context-ordering fix only to a proved pre-fix Taskworker; on a proved fixed artifact, diagnose the preserved live inner error. Treat a paired `window-stall` as the same causal boundary, not a second failure. Never infer ancestry from a release label/module tag or restart from the line alone. For a pre-fix rollout, require zero cancellation-correlated exact lines and paired stalls for ten minutes through comparable teardown, while deterministic live-context exact errors and other genuine errors remain visible. See §14.6. |
 | Panic stack traces (`trace.go` "Unexpected error") | The STACK identifies the load-bearing call path (e.g. AddNetworkPeer → NominateLocalResident = connection-killing). | Rate per unique innermost app frame; a new frame appearing at rate = new incident. |
 | `dohRouteForConn.func1` with `runtime error: invalid memory address or nil pointer dereference` | HTTP/2 reused or retired a live connection wrapper whose `LocalAddr()` or `RemoteAddr()` was nil. The optional route-observation callback dereferenced that endpoint, so `HandleError` recovered the resolver goroutine but the in-flight DNS result was lost; the proxy process and public listener remain healthy while a request can time out. This is not provider unresponsiveness. | Any occurrence identifies a pre-fix Connect module. Current code treats nil and typed-nil endpoints as absent diagnostic metadata and preserves the DoH response. Deploy the fixed proxy generation, then require zero new occurrences while sustained HTTP/SOCKS/WireGuard acceptance runs. See §14.6. |
@@ -13365,9 +13365,23 @@ pass through the separate transition logger that emits `event=window_stall
 ... failed=1`. The standing tailer previously recognized only that normally
 absent compatibility shape and left the real terminal event unclassified.
 `window-stall-terminal` now accepts both exact structured forms at 1/min
-without changing ticket identity; malformed durations, reasons, or field order
-remain in the generic novelty safety net rather than being silently treated as
-terminal.
+without changing ticket identity. Its raw rate remains the §1.5 diagnostic-line
+count. `failed_window_events` is the separate exact-replay-deduplicated count of
+authoritative `window_failed` records; a compatibility-only observation reports
+that count as unknown rather than zero. Malformed durations, reasons, or field
+order remain in the generic novelty safety net rather than being silently
+treated as terminal.
+
+The 2026-09-10 recurrence demonstrates why those counts differ. One Taskworker
+emitter produced two `window_failed` records 2.140 milliseconds apart. About
+eight milliseconds later, two client-args cancellation diagnostics changed the
+latched reason and produced two distinct `window_stall ... failed=1`
+transitions. The watcher therefore correctly saw four terminal-class lines,
+while the authoritative failed-window cardinality was two. This was neither
+standing-tail replay nor classifier double matching, and the class rate must
+not be read as failed-window or user-incident size. Two near-simultaneous
+authoritative records remain two events; only their paired compatibility
+transitions are excluded from the canonical count.
 
 The bounded same-emitter cohort distinguishes the terminal fact from its
 reason: it had zero exits, proven providers, or flows, repeated candidate

monitor/tailer.go

@@ -53,6 +53,13 @@ type logBurst struct {
 type logCanonical struct {
 	eventRe *regexp.Regexp
 	name    string
+	// sourceToken and sourceLabel describe the selected canonical event. The
+	// defaults retain the task-evaluator wording used by payout classes.
+	sourceToken string
+	sourceLabel string
+	// Some legacy compatibility lines carry terminal evidence without the
+	// canonical source. Render that cardinality as unknown, never zero.
+	unknownWhenAbsent bool
 	// correlation optionally joins each canonical source second to canonical
 	// attempts already counted by another class's burst detector. This proves
 	// incident co-residency without treating a minute-wide rate as ordering or
@@ -119,11 +126,13 @@ var framerRejectRe = regexp.MustCompile(`\[framer\]\[reject\](?:read|write(?: ba
 
 // Connect's relEvent grammar renders booleans as 0/1 and durations as integer
 // milliseconds. Keep nonterminal transitions separate from terminal outcomes:
-// failOutcome logs window_failed and then calls SetStallStatus directly, so it
-// does not normally emit a window_stall failed=1 line. Retain failed=1 as a
-// compatibility shape, and keep both out of the generic novel detector.
+// failOutcome logs window_failed and then calls SetStallStatus directly, so
+// that dispatch does not itself emit window_stall failed=1. A later reason
+// change can publish that compatibility transition while the failed latch is
+// still set. Retain both shapes and keep them out of the generic novel detector.
 var (
 	windowStallNonterminalRe  = regexp.MustCompile(`\[rel\][[:space:]]+event=window_stall[[:space:]]+window=[a-z-]+[[:space:]]+reason=[a-z-]+[[:space:]]+failed=0(?:[[:space:]]|$)`)
+	windowFailedEventRe       = regexp.MustCompile(`\[rel\][[:space:]]+event=window_failed[[:space:]]+window=[a-z-]+[[:space:]]+reason=[a-z-]+[[:space:]]+after=[0-9]+(?:[[:space:]]|$)`)
 	windowTerminalRe          = regexp.MustCompile(`(?:\[rel\][[:space:]]+event=window_stall[[:space:]]+window=[a-z-]+[[:space:]]+reason=[a-z-]+[[:space:]]+failed=1|\[rel\][[:space:]]+event=window_failed[[:space:]]+window=[a-z-]+[[:space:]]+reason=[a-z-]+[[:space:]]+after=[0-9]+)(?:[[:space:]]|$)`)
 	windowStallEventRe        = regexp.MustCompile(`\[rel\][[:space:]]+event=window_stall[[:space:]]+window=[a-z-]+[[:space:]]+reason=[a-z-]+[[:space:]]+failed=[01](?:[[:space:]]|$)`)
 	windowGeneratorCanceledRe = regexp.MustCompile(
@@ -451,11 +460,18 @@ var logClasses = []logClass{
 	{name: "window-stall-terminal", re: windowTerminalRe,
 		sample:        windowTerminalLogSample,
 		rateThreshold: 1, tier: tierWarn, playbook: "SIGNALS.md §4 and §14.6",
-		meaning:   "a provider window crossed both bounded outcome deadlines with no provider added; window_failed is authoritative terminal state for that window but does not identify the underlying failure branch",
-		mechanism: "Connect failOutcome logs one structured window_failed event and then calls SetStallStatus directly. That dispatch updates the UI-facing terminal latch but does not normally emit window_stall failed=1; the latter remains an accepted compatibility shape. The reason names the current diagnostic branch, not a proven transport root cause.",
-		context:   "Keep this distinct from window_stall failed=0, which means the window is still trying. A taskworker log locates the embedded Connect observer, not a customer or provider identity. The event alone cannot choose among platform reachability, provider response, rate limiting, or authentication causes.",
+		meaning:   "a provider window crossed both bounded outcome deadlines with no provider added; window_failed is authoritative terminal state and failed_window_events is its logical-event cardinality, while the diagnostic line rate can also include a later compatibility transition",
+		mechanism: "Connect failOutcome logs one structured window_failed event and then calls SetStallStatus directly. That dispatch does not itself emit window_stall failed=1, but a later reason change can publish that compatibility transition while the failed latch remains set. Both lines retain fail-safe class visibility; only exact-replay-deduplicated window_failed lines count as authoritative failed-window events. The reason names the current diagnostic branch, not a proven transport root cause.",
+		context:   "Keep this distinct from window_stall failed=0, which means the window is still trying. Read diagnostic_lines as terminal-class telemetry, not incident size. When only a compatibility failed=1 line is present, failed_window_events is unknown rather than zero. A taskworker log locates the embedded Connect observer, not a customer or provider identity. The event alone cannot choose among platform reachability, provider response, rate limiting, or authentication causes.",
 		action:    "Correlate the exact reason and window with provider-window progress, explicit transport/framer/auth/rate-limit classes, peer availability, and the emitting artifact identity. Preserve the event and natural retry state; do not restart or deploy from the terminal bit alone.",
 		verify:    "No window_failed event or compatible failed=1 transition recurs for ten minutes under comparable provider-window traffic, affected windows add providers or emit their ordinary recovery transition, and the independently identified causal control remains healthy.",
+		canonical: &logCanonical{
+			eventRe:           windowFailedEventRe,
+			name:              "failed_window_events",
+			sourceToken:       "authoritative-window-failed-event",
+			sourceLabel:       "authoritative window_failed event",
+			unknownWhenAbsent: true,
+		},
 	},
 	{name: "window-stall", re: windowStallNonterminalRe,
 		sample:        windowStallLogSample,
@@ -1592,17 +1608,41 @@ func (self *logTailer) drainWindow() []finding {
 			}
 			canonicalEvidence := ""
 			if c.canonical != nil {
-				observed += fmt.Sprintf(
-					" %s=%d diagnostic_lines=%d canonical_source=exact-replay-deduplicated-task-evaluator",
-					c.canonical.name,
-					self.canonicalCounts[key],
-					count,
-				)
-				canonicalEvidence = fmt.Sprintf(
-					"\nlogical event count: %d exact-replay-deduplicated task evaluator line(s) from %d diagnostic line(s)",
-					self.canonicalCounts[key],
-					count,
-				)
+				canonicalCount := self.canonicalCounts[key]
+				sourceToken := c.canonical.sourceToken
+				if sourceToken == "" {
+					sourceToken = "task-evaluator"
+				}
+				sourceLabel := c.canonical.sourceLabel
+				if sourceLabel == "" {
+					sourceLabel = "task evaluator"
+				}
+				if c.canonical.unknownWhenAbsent && canonicalCount == 0 {
+					observed += fmt.Sprintf(
+						" %s=unknown diagnostic_lines=%d canonical_source=absent",
+						c.canonical.name,
+						count,
+					)
+					canonicalEvidence = fmt.Sprintf(
+						"\nlogical event count: unknown; no %s line was present among %d diagnostic line(s), so the diagnostic class retains fail-safe visibility",
+						sourceLabel,
+						count,
+					)
+				} else {
+					observed += fmt.Sprintf(
+						" %s=%d diagnostic_lines=%d canonical_source=exact-replay-deduplicated-%s",
+						c.canonical.name,
+						canonicalCount,
+						count,
+						sourceToken,
+					)
+					canonicalEvidence = fmt.Sprintf(
+						"\nlogical event count: %d exact-replay-deduplicated %s line(s) from %d diagnostic line(s)",
+						canonicalCount,
+						sourceLabel,
+						count,
+					)
+				}
 				correlationObserved, correlationEvidence := self.canonicalCorrelationLocked(c, key)
 				observed += correlationObserved
 				canonicalEvidence += correlationEvidence