Darling tier-1 field-incident response: file log + Event Log source, store conf headroom, honest bootstrap status, PG18 docs - #1552
Merged
Conversation
…onest bootstrap status From the 24-server report (sequential first sweep crawling under store pressure looked like servers being permanently dropped, with no log anywhere to say otherwise): - DarlingFileLoggerProvider (new): rolling file log under %ProgramData%\PerformanceMonitorDarling\logs, buffered, daily-rotated, 14-day retention, never-throwing - the PRIMARY diagnostic surface. Every existing ILogger call site lands there unchanged. - Event Log: source pinned to "PerformanceMonitor Darling", best-effort registration at startup, and the README install steps gain the one elevated New-EventLog line - the provider was always wired but the NT SERVICE account can't register a source, so a by-the-book install has silently dropped every event since day one. - Managed store v4 conf block (append-and-heal like v2/v3): max_connections = 200, max_wal_size = 4GB. PG defaults (100/1GB) are toy-sized; the fleet bootstrap burst forced back-to-back spread checkpoints and backend-spawn churn (Windows error 487). - "Awaiting first collection": never-collected is a distinct freshness band (amber card + fleet Warning + grey sidebar dot) instead of the red Offline overlay - "queued, not reached yet" is the truth during bootstrap. MCP get_server_status gains the additive AwaitingFirstCollection value. - PostgreSQL 18 doc sweep: the bundled runtime has been PG 18.4 + TimescaleDB 2.28.1 since the runtime bump; README/sample/comments said 17. Tests: Darling suite 2151 passed / 0 failed (Release); new pins for the v4 block, the NeverCollected band (cards, fleet, sidebar), and updated freshness pins. Tier 2 (bounded-parallel sweep + cadence jitter) follows separately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 17, 2026
pull Bot
pushed a commit
to ehtick/PerformanceMonitor
that referenced
this pull request
Jul 22, 2026
…itter (erikdarlingdata#1553) Tier 2 of the 24-server field-incident response (tier 1 was erikdarlingdata#1552). The service swept the fleet strictly sequentially -- one foreach over every server, every step awaited inline -- so under store distress one server's 3+ minute first-connect work left the alphabetical tail unreached for tens of minutes (indistinguishable from a permanent drop), and every 15s cadence boundary re-herded all servers at once. Fire-and-track sweep (D2/D3): the loop now LAUNCHES each server's collection body (new ProcessServerSweepAsync, tracked per-server in InFlightSweep) without awaiting it, falling straight through to the fleet-level reload/purge/disk-check steps, so one slow or hung server can no longer head-of-line-block the fleet. Concurrency is bounded to N=4 (MaxConcurrentServerSweeps, hardcoded) by a shared SemaphoreSlim acquired inside each body; collectors stay sequential within a server (Lite's RemoteCollectorService shape). Deliberately not await-all-per-sweep -- that recreates the incident in miniature. Cadence jitter: CadencePhaseOffset ((uint)serverId % periodSeconds; the serverId is already an FNV-1a hash) gives a deterministic, restart-stable per-server phase applied only at initial cadence stamps -- each collector's first post-connect due time, a reload's newly-enabled collectors, and the first scheduled analysis (fixed 2.5-minute window, not the full interval). The steady-state advance is never jittered. Per-server exception containment (D3, Design Goal 4): the extracted body has a catch-all (the gap the sweep loop previously had no cover for), so one server's throw is logged and isolated and a faulted fire-and-track task never surfaces unobserved. Concurrency-supporting invariants: per-server NextDue becomes a ConcurrentDictionary (D1) since a reload's recompute can now touch it while a body reads it (indexed only by static catalog keys -- a drop-in); disabling a server marks its state Retired (checked as the first statement inside each body after it acquires the gate, plus a connect-path re-check before any durable side-effect) so an in-flight/queued body for a removed server no-ops (never connects, never runs XE DDL, never re-writes self-alert edge state after Forget); and shutdown drains in-flight bodies (up to 15s, inside the host's 30s ShutdownTimeout) before the command loop, with the gate deliberately never disposed. Hang observability (D2b): because the incident was hangs, not throws, the skip branch logs Debug each sweep and one Warning per episode once a body crosses 60s continuous (queue time counts), with an Info on eventual completion; all skip-log state is outer-thread-owned. Contracts unchanged: collection_log write semantics, MCP surfaces, and alert dedup/edge semantics preserved; the per-server CollectionGate behavior is unchanged. New DarlingSweepSchedulingTests pins the jitter offset (deterministic, unsigned-modulo including negative FNV ids, bounded [0, period), not-all-identical across ids, non-positive period -> zero) and the N=4 cap. Full Darling.Tests (2157 passed / 134 gated-live skipped) and Lite.Tests (1352 passed) green. Darling service only -- no store schema change, no Lite/Dashboard/viewer changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Tier 1 of the 24-server field-incident response (full diagnosis in the incident thread): the sequential first sweep crawling under store pressure was indistinguishable from servers being permanently dropped, because (a) there was no service log anywhere on a by-the-book install, and (b) the viewer rendered "hasn't been reached yet" as red Offline.
Changes
DarlingFileLoggerProvider(new) — rolling file log under%ProgramData%\PerformanceMonitorDarling\logs\darling-service_yyyyMMdd.log; buffered (5s flush), daily rotation, 14-day retention, never-throwing; anILoggerProvider, so every existing log site lands there with zero call-site changes. PRIMARY diagnostic surface.PerformanceMonitor Darling(matches service name + docs), best-effortCreateEventSourceat startup, and the README install steps gain the one elevatedNew-EventLogline. The provider was always wired, but the recommendedNT SERVICEaccount cannot register a source → every event silently dropped since day one (confirmed in the field report:SourceExistsreturned false).max_connections = 200,max_wal_size = 4GB. PG defaults (100 / 1GB) forced back-to-back spread checkpoints under the fleet-bootstrap write burst while Windows backend-spawn churn (error 487) surfaced as transient store write failures. Existing clusters heal on next service-owned start.ServerFreshness.NeverCollectedband: amber card status + fleet Needs-Attention Warning ("Awaiting first collection") + grey sidebar dot, never the red Offline overlay. MCPget_server_statusgains the additiveAwaitingFirstCollectionstatus value (existing values unchanged — contract note).darling.sample.json/workflow comments said 17; BYO guidance now "16 or newer, validated against 18".Contracts
get_server_status: additive status value only.Testing
NeverCollectedclassification/apply/status pins (cards, fleet band + reason, sidebar dot), and the updated no-collection pins.Tier 2 (bounded-parallel sweep + cadence jitter) follows as its own PR.
🤖 Generated with Claude Code