Availability Group health collection in Lite + Darling (#991)

[erikdarlingdata]

Closes #991, and the AG item of #1606 (the one real coverage gap the Datadog DBM comparison turned up).

Scope is collection only, per the design: two tables, both apps, compose measures. No viewer tab and no alerts in this cut.

What lands

Two shared collectors in PerformanceMonitor.Collectors, both server-scope, one query per sweep:

Collector	Grain	Source
ag_replica_states	one row per replica per AG	sys.availability_replicas + sys.availability_groups + sys.dm_hadr_availability_replica_states
ag_database_replica_states	one row per database per replica	sys.dm_hadr_database_replica_states + the two catalog views + sys.databases

The metric surface follows Hannah Vernon's SqlServerAgMonitor (MIT, (c) 2026 Hannah Vernon), attributed in both collector header comments. WHERE COALESCE(ag.is_distributed, 0) = 0 keeps distributed-AG container rows out while member AGs still appear; DAG member drill-through needs a connection per member and is out of scope.

Two collector classes rather than one, because the catalog idiom is strictly one table per collector (TargetTable is a single string, PayloadColumns a single list). Git history backs that reading: latch_stats/spinlock_stats and cpu_scheduler_stats/plan_cache_stats each shipped as a related pair of classes in one commit.

Gating

AppliesTo is !target.IsAzureSqlDb -- the same idiom server_config and trace_flags use -- which gives on-prem + Managed Instance + RDS and excludes Azure SQL DB. An AG-less server is deliberately not gated off: it collects and stores zero rows, so turning Always On on later starts producing data with no configuration change.

One thing worth flagging: MS Learn stamps sys.dm_hadr_database_replica_states as SQL Server + Managed Instance, but stamps the two catalog views as SQL Server only. I followed the instructed gate (MI included) since MI does expose the AG objects; if an MI target ever errors on the catalog views, that gate is the place to narrow.

Three traps, each pinned by test

• The queues, rates and lag are instantaneous gauges, not counters. The DMV recomputes each from the current backlog, so neither collector touches the delta framework, and the compose measures are Gauge (avg/min/max, never SUM). WritePayload_EmitsPayloadOrder_AndTakesNoDeltas asserts the delta calculator is never called.
• last_hardened_lsn / last_commit_lsn are numeric(25, 0) -- wider than BIGINT -- so they convert server-side and store as text. Storing them numerically would silently overflow. (Only last_commit_lsn is a real LSN; the other is a zero-padded log-block id, so neither is safe for arithmetic. Deriving a byte distance is analysis, left to a reader.)
• Every column is read null-tolerantly. Under WSFC quorum loss sys.availability_replicas serves only locally cached metadata, and endpoint_url is documented NULL in that state. Both ToleratesNulls tests cover an all-NULL row.

A fourth worth recording for whoever builds the alerts: secondary_lag_seconds reads 0, not NULL, while data movement is suspended, so a suspended replica charts as zero lag. is_suspended is collected alongside it precisely so a reader can tell the two apart. Documented in the collector.

Registrations

• Catalog: both in CollectorCatalog.All + CollectorScheduleDefaults (1 min / 30 days, default ON, in all three cadence presets in both apps).
• Lite: dispatch case + a RemoteCollectorService.AvailabilityGroups.cs partial, ScheduleManager defaults, bundled collection_schedule.json. The "three registrations" (Schema.cs / DuckDbInitializer / ArchiveService) no longer exist as separate steps -- all three now enumerate CollectorCatalog.All, so catalog registration is the whole storage wiring. Verified against long_query_completions, the newest collector, which touches none of those files.
• Darling: DarlingWorker dispatch, store migration V34 + StorageVersion bump, viewer schedule presets.

The V34 migration needed a viewer change too

ViewerDataService.MapProbedSchemaVersion is a newest-first sentinel ladder, and the connect-time gate refuses a store below RequiredStoreSchemaVersion. Bumping StorageVersion without adding a matching sentinel + arm would have made a fully-migrated V34 store probe as v33 and permanently refuse to open the viewer. Added both (table-existence sentinel on the database-grain table); the existing pinning test caught it, which is what it is there for.

Compose

Five Gauge measures on ag_database_replica_states under a new Availability Groups category: send queue and redo queue (bytes family, native KB, default Max -- the worst backlog in the bucket is the signal), send rate and redo rate (bytes family, native KB, default Avg), and secondary lag (duration family, native s, default Max). Dimensions: ag_name, database_name, replica_server_name, plus the universal server.

The rates are KB/second but the catalog has no per-second unit family, so per the instructions they ride the bytes family -- correct under conversion, since kb/s to mb/s scales by the same factor -- with the per-second nature stated in the display name rather than implied by a unit the picker renders as a plain size.

The replica-grain table contributes no measures: it is all state strings with nothing numeric to aggregate.

A coverage-pin bug found and fixed along the way

Both apps' collector-coverage ratchets text-scan the viewer's reader layer for a table name. Naming a collector table as a schema-version probe sentinel (as V34 does, and as V29 already did for long_query_completions) made that table read as "already covered" and silently exempted it from the ratchet. The Darling scan now strips the probe's information_schema lines before matching -- those are DDL existence checks, never data reads, and information_schema appears nowhere else in that layer. That retroactively hardens the pin for long_query_completions too.

Both tables are listed as tracked debt in both ratchets, with the follow-up named.

Tests

Extended every pin the change touches (catalog counts, golden DuckDB schema + indexes, Lite/Darling preset sets, migration ladder, PG generator counts, schema-probe map) and added:

• Lite.Tests/AgCollectorDefinitionTests.cs -- 13 tests: verbatim query text for both, DAG filter, target gate across all four platforms, no per-database run, payload order + declared types, ReadAsync mapping, null tolerance, zero-rows-is-not-an-error, payload emission with no delta calls, catalog + schedule registration, and by-name gate agreement.
• DarlingComposeTests -- 4 new tests (all five measures Gauge/non-summable/right category, native units + default aggregates, the three dimensions, the lag panel compiling and grouping by replica) plus an InlineData row proving SUM on an AG gauge is rejected.

Lite.Tests      1494 passed, 0 failed
Darling.Tests   3152 passed, 0 failed (149 skipped - the live-Postgres gated set)
Solution build  0 warnings, 0 errors

Installer.Tests was not run (live-DB categories clobber real Agent jobs on SQL2022).

Live smoke -- SQL2022, no AGs

Ran both collector queries verbatim against sql2022 (SQL2022, 16.0.4255.1, IsHadrEnabled = 0). Both parse, execute, and return zero rows with the full expected column set -- the normal AG-less path, proving it is a clean empty read and not an error:

=== ag_replica_states ===
ag_name  replica_server_name  role_desc  operational_state_desc  connected_state_desc  recovery_health_desc
synchronization_health_desc  availability_mode_desc  failover_mode_desc  endpoint_url
(0 rows affected)
EXIT=0

=== ag_database_replica_states ===
ag_name  database_name  replica_server_name  is_local  synchronization_state_desc  last_hardened_lsn
last_commit_lsn  log_send_queue_size  redo_queue_size  log_send_rate  redo_rate  is_suspended
suspend_reason_desc  availability_mode_desc  secondary_lag_seconds
(0 rows affected)
EXIT=0

The smoked SQL was diffed against the QueryText constants in both collector sources and is byte-identical, so this exercised exactly what ships -- not a hand-retyped approximation.

Follow-ups (deliberately not here)

An Availability Groups viewer tab, and the failover-detected / sync-fell-behind alerts. Both ratchets will go red when the tab ships, forcing the allow-list entries out.

🤖 Generated with Claude Code