Darling Web: an "AG Health" seed notebook, and a covered template drift guard (#991)

CHANGELOG.md

@@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **Darling Web: an "AG Health" seed notebook** ([#1699]) - the fifth Custom Views v2 seed template, built on the Availability Group measures from [#1688] and [#1695]. Five panels in diagnostic order: secondary lag over time grouped by replica; a **dual-axis** line putting log send rate against redo rate on their own axes (the one panel that answers "which side is the bottleneck" - send outpacing redo means the secondary is receiving faster than it can replay, and failover time is growing); a **stacked** redo-queue series by database; a **stat tile** for the worst estimated redo drain; and a top-10 bar of send-queue backlog by database. The prose between panels carries the two things that make the numbers readable rather than merely present: point the view at the PRIMARY (a secondary only ever sees its own replica in `sys.dm_hadr_*`, so a secondary-scoped view is a one-row self-view), and a blank drain estimate means there is no drain rate - idle, caught up, or suspended - not that it drains instantly. Also adds the two template helpers the existing seeds never needed, `overlay` support on the time-series builder and a scalar `statPanel`, so this is the first seed exercising dual-axis, stacked and stat modes end to end. **The templates drift-guard got a real hole closed along the way**: it hand-mirrors each template's panels for validation, but nothing checked the mirror COVERED every template, so a sixth template added without a mirror entry would have gone silently unvalidated until it 400'd in someone's browser. It now reads the template keys out of `notebook.js` and requires the mirror to match exactly, verified by renaming a key and watching it go red.
+- **AG latency: commit-time columns, drain-time estimates, and the primary-side perfmon counters** ([#1695]) - completes the Availability Group latency picture [#1688] started. `ag_database_replica_states` gains **four DMV timestamps the reference query skips** (`last_commit_time`, `last_hardened_time`, `last_redone_time`, `last_received_time`) - unlike `secondary_lag_seconds` these are directly comparable across replicas, which is what the canonical primary-vs-secondary commit-time lag math needs - plus **two server-computed drain-time estimates**, `est_redo_completion_time_min` and `est_send_drain_time_min` (queue / rate / 60). Both estimates carry the two guards the raw expression needs: `* 1.0` stops BIGINT-over-BIGINT integer division flooring a sub-minute drain to zero, and `NULLIF(rate, 0)` stops the divide-by-zero an idle or suspended replica would otherwise raise - which would fail the whole cycle, not one column. A NULL estimate honestly means "no drain rate" and is never coerced to 0, which would read as "drains instantly". They are computed per row at the sample's own instant rather than composed later as a ratio of two window averages, because avg(queue)/avg(rate) is not the average of the per-sample ratios and the two diverge worst exactly when rates swing. Both are Gauge compose measures (duration family, native minutes). **Two perfmon counters** join the existing whitelist - `Transaction Delay` and `Mirrored Write Transactions/sec` - carrying the PRIMARY side of commit latency (their ratio is the average delay per mirrored transaction); zero new schema, they ride `perfmon_stats`. Verified on SQL2022 that both exist even with no AGs configured, live on `SQLServer:Database Replica`, and occur exactly once server-wide, so the collector's counter_name-only filter cannot collide. Together with `HADR_SYNC_COMMIT`, which already flows through `wait_stats`, sync-commit pressure is now composable end to end: primary-side delay, secondary-side queues and rates, drain estimates, and the wait itself. **A shipped doc claim is corrected against a live AG.** [#1688] restated MS Learn's assertion that `secondary_lag_seconds` reads 0 while data movement is suspended; a Docker AG fixture measured the inverse on SQL Server 2022 in a `CLUSTER_TYPE = NONE` group - it reads 0 while movement is ACTIVE and caught up, and accrues monotonically once suspended (0 to 62 s across a 60 s `SUSPEND_FROM_USER`, back to 0 on resume). So a suspended replica does not hide as zero lag and a lag threshold fires on its own; reading `is_suspended` alongside explains WHY lag is climbing rather than catching lag that is masked. Two further measured quirks are now documented: `log_send_queue_size` goes NULL while suspended while `redo_queue_size` FREEZES at its last value (so a redo-queue reading on a suspended replica is stale, not current), and collecting from a SECONDARY yields a one-row self-view because `sys.dm_hadr_*` carries only the local replica there - a complete AG picture requires collecting from the primary. Store migration V36 appends the six columns additively rather than widening V34 in place, because V34's `CREATE TABLE IF NOT EXISTS` is a no-op on an already-migrated store and editing it would silently leave every existing store short the new columns while fresh installs got them; the schema pin now reconstructs the current shape from V34 + V36 and compares it to the generator, with both failure modes confirmed by planted defects. Hannah Vernon's SqlServerAgMonitor is now credited in `THIRD_PARTY_NOTICES.md` alongside the existing collector-header attribution.
 - **AG collection: document the second grant it needs, and make the lag trap filterable** ([#1691]) - review follow-ups to [#1688]. **The grant is the one that matters in the field.** Both AG collectors join the `sys.availability_groups` / `sys.availability_replicas` CATALOG VIEWS to the `sys.dm_hadr_*` DMVs, and while the DMVs are covered by the `VIEW SERVER STATE` the product asks for, [the catalog views require `VIEW ANY DEFINITION`](https://learn.microsoft.com/en-us/sql/database-engine/availability-groups/windows/monitor-availability-groups-transact-sql) - which catalog views enforce by HIDING ROWS, not by raising an error. So on a fully configured AG cluster a monitoring login with only the documented grant returned zero rows, which is exactly what an AG-less server returns: the collectors would have looked healthy forever while collecting nothing, with no error anywhere to notice. Now called out in both READMEs (the Lite/Darling grant script, the Darling permission table's If-missing column) and in both collector headers, along with the fingerprint that identifies it if it is ever worth detecting automatically - the DMV returning rows while the catalog view returns none is unambiguous, and `SERVERPROPERTY('IsHadrEnabled')` is readable by every login. **The documented lag trap is now actionable instead of just documented**: `secondary_lag_seconds` reads 0 rather than NULL while data movement is suspended, so a suspended replica charts as perfectly healthy - but nothing exposed the suspension state to a panel, so the misread the code comment warned about was unavoidable. `synchronization_state_desc` and `suspend_reason_desc` are now compose dimensions, so a lag panel can filter suspended replicas out or group by suspend reason. (`is_suspended` itself cannot be a dimension: the compiler binds filter values as text, which would not match a boolean column.) **And the V34 migration is now genuinely pinned.** Its test asserted only that each column NAME appeared somewhere in the DDL - a V34 with the columns reordered, `is_local` typed `text`, or a spurious `NOT NULL` passed every test, while a comment claimed the shape was pinned elsewhere. It now compares the whole generated `CREATE TABLE` via the existing `Migrations_JobHistoryAndAgentStatus_MatchGeneratedFreshShape` idiom; detection power confirmed by planting `is_local text` and watching it go red. That test's long-standing rationale was corrected too - it blamed a "positional binary COPY", but `PgCollectorRowWriter.CopyCommandFor` emits a named column list so Postgres binds by name; names and types are the real hazard on that side, and the positional appender is Lite's.
 - **Darling: Availability Group alerts - failover, replica disconnected, sync fell behind, database suspended** ([#1692]) - the alert half of #991, over the collectors [#1688] landed. Four conditions on the headless service's existing per-server self-alert sweep: **`AG Failover`** (Warning - a replica's `role_desc` changed since the previous sweep), **`AG Replica Disconnected`** (Critical - `connected_state_desc` crossed into `DISCONNECTED`, with an informational **`AG Replica Reconnected`** on recovery), **`AG Sync Fell Behind`** (Warning - a secondary past `ag_lag_alert_seconds` or `ag_redo_queue_alert_kb`), and **`AG Database Suspended`** (Warning - `is_suspended` false to true, carrying `suspend_reason_desc`). The metric names are webhook automation keys and are consts rather than inline literals, and all five are registered in the shared `AlertSeverity` map so an alert-history replay that reaches the map without an explicit severity override does not render INFO-blue. State is keyed per AG grain (ag+replica, ag+database+replica) rather than per server, so two lagging databases on one host track and recover independently, but every alert still FIRES under the real `server_id`, keeping the per-server delivery-mode override, mute rules and history correlation intact - the grain lives in the alert text. **First sighting of any replica or database is a silent baseline** and **a NULL state string is never a transition**: under WSFC quorum loss the AG catalog views serve only locally cached metadata, so any column can read NULL, and treating that as an edge would spray alerts across the whole fleet at the moment the cluster is already in trouble. The sync decision returns THREE states rather than a bool, which is the load-bearing detail: MS Learn documents `secondary_lag_seconds` reading `0` (not NULL) while data movement is SUSPENDED, so the seconds trigger has to abstain on a suspended row - and if abstaining were a plain "not behind", the caller would read it as recovery and announce *"AG Sync Recovered - has caught up with the primary"* in the same sweep that reported the database suspended. Only a database a sweep actually MEASURED as caught up resolves a standing alert, which also makes cross-server resolution structurally impossible instead of something a scoping check has to catch. The redo-queue trigger keeps judging while suspended, because that backlog is real and still growing. Store-backed settings on `config_alert_settings` (Darling store migration **V35**, every column `NOT NULL DEFAULT`): `notify_ag_health` (default on - a fleet with no AGs collects no AG rows and is silent anyway, and an operator who does run AGs should not have to find a switch to be told about a failover), `ag_lag_alert_seconds` (default 300, clamped 0-86400) and `ag_redo_queue_alert_kb` (default 0 = off, clamped 0-1073741824 - a healthy redo queue size is entirely workload-specific, so a shipped guess would page half the fleet on day one). All three are read live through the same by-reference settings seam a store reload hot-swaps, editable from the viewer's Settings window, and the whole AG read is skipped when the master switch is off, so an AG-free fleet pays nothing on the sweep. Deliberate limits, stated rather than hidden: Lite collects both grains but has no AG alerts yet; every replica is visible from every node, so a 3-node AG with all 3 nodes monitored reports a role change once per monitored server; and `AG Replica Disconnected` is a pure edge with no [#1674]-style re-fire.
 - **Availability Group health collection in Lite + Darling** ([#1688]) - closes #991 and the AG item of #1606, the one real coverage gap the Datadog DBM comparison turned up. Two new shared collectors, both server-scope and both zero-cost on a server without Always On: **`ag_replica_states`** (replica grain - role, operational/connected state, recovery and synchronization health, availability and failover mode, endpoint URL) from `sys.availability_replicas` joined to `sys.availability_groups` and `sys.dm_hadr_availability_replica_states`, and **`ag_database_replica_states`** (database grain - synchronization state, log send and redo queue sizes, send and redo rates, secondary lag, suspension state and reason, and both LSNs) from `sys.dm_hadr_database_replica_states`. The metric surface follows Hannah Vernon's [SqlServerAgMonitor](https://github.com/HannahVernon/SqlServerAgMonitor) (MIT), attributed in both collector headers. `WHERE COALESCE(is_distributed, 0) = 0` keeps distributed-AG container rows out while member AGs still appear; drilling into a DAG's remote members needs a connection per member and is out of scope. Runs everywhere except Azure SQL DB, which has no AG surface - an AG-less on-prem server is deliberately NOT gated off, it collects and stores zero rows, so turning Always On on later starts producing data with no configuration change. Three details that are easy to get wrong and are pinned by test: the queues, rates and lag are **instantaneous gauges, not counters**, so neither collector touches the delta framework and the compose measures are Gauge (avg/min/max, never SUM - summing a backlog over a window is a category error); `last_hardened_lsn` / `last_commit_lsn` are `numeric(25, 0)`, **wider than BIGINT**, so they are converted server-side and stored as text rather than silently overflowing; and every column is read null-tolerantly, because under WSFC quorum loss `sys.availability_replicas` serves only locally cached metadata and `endpoint_url` is documented NULL. Both apps schedule it per minute with 30-day retention (the grain at which a lag or queue spike is still visible), and it is in all three cadence presets (per-minute on Aggressive and Balanced, 5 min on Low-Impact). **Custom Views v2** gets five Gauge measures on the database grain - send queue, redo queue, send rate, redo rate and secondary lag - under an Availability Groups category, sliceable and groupable by AG, database and replica plus the synchronization/suspend state (and the universal server dimension), which lights up send-rate-vs-redo-rate on a dual axis and worst-lag-per-replica out of the box. Only the database-grain table carries measures - the replica-grain table is all state strings with nothing numeric to aggregate, so it is stored for the coming viewer tab rather than queryable from Custom Views today. Darling store migration V34; Lite's storage registers itself off the collector catalog. Collection only in this first cut: no viewer tab, and no alerts (the alert family landed straight after in [#1692]), and both apps' viewer-coverage ratchets carry the two tables as explicitly tracked debt so the tab cannot be forgotten. That ratchet also got a real fix along the way - it text-scans the viewer's reader layer for a table name, and naming a collector table as a schema-version probe sentinel (as V34 does) made the table read as "already covered", silently exempting it; the scan now strips the probe's `information_schema` lines, which retroactively hardens the same pin for `long_query_completions`.
@@ -1667,6 +1669,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 [#1688]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1688
 [#1691]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1691
 [#1692]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1692
+[#1695]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1695
+[#1699]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1699
 [#1697]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1697
 [#1690]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1690
 [#1693]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1693

Darling/Darling.Tests/DarlingAnalysisStoreTests.cs

@@ -8,6 +8,7 @@
 
 using System;
 using System.Collections.Generic;
+using System.Globalization;
 using System.Linq;
 using System.Threading.Tasks;
 using Npgsql;
@@ -239,12 +240,22 @@ public async Task EndToEnd_FindingStoreRoundTrip_AgainstDevPostgres()
         /* Migrations are idempotent — an older store comes up to current, a current store no-ops. */
         await PgMigrations.MigrateAsync(connection, TestContext.Current.CancellationToken);
 
+        /* Two separate invariants, because the single COUNT(*) == SchemaVersion check they replace conflated
+           them: it only holds while migration versions are DENSE from 1, so the first concurrently-developed
+           pair of migrations (V35 alerts / V36 AG latency, built on separate branches) broke it with a
+           temporary gap that is completely inert to the applier — MigrateAsync applies every script whose
+           version exceeds MAX(version) and never assumes contiguity. Assert what actually matters instead,
+           which is also strictly stronger than the old proxy. */
+        using (var maxVersion = new NpgsqlCommand("SELECT COALESCE(MAX(version), 0) FROM darling_schema_version", connection))
+        {
+            /* The store reached the version this build knows — the same expression MigrateAsync reads. */
+            Assert.Equal(StorageVersion.SchemaVersion, Convert.ToInt32(await maxVersion.ExecuteScalarAsync(TestContext.Current.CancellationToken), CultureInfo.InvariantCulture));
+        }
+
         using (var versions = new NpgsqlCommand("SELECT COUNT(*) FROM darling_schema_version", connection))
         {
-            /* One row per applied migration, so the count tracks the current schema version. Reference the
-               constant rather than a literal — this was hardcoded to a since-superseded number (15) and, as a
-               live-only assertion CI skips, silently rotted until the schema reached V17. */
-            Assert.Equal((long)StorageVersion.SchemaVersion, await versions.ExecuteScalarAsync(TestContext.Current.CancellationToken));
+            /* And EVERY script ran, one stamped row apiece — so a script silently skipped still fails here. */
+            Assert.Equal((long)PgMigrations.Scripts.Count, await versions.ExecuteScalarAsync(TestContext.Current.CancellationToken));
         }
 
         /* Clear leftovers from an earlier aborted run so the assertions below are deterministic. */