Darling Web: an "AG Health" seed notebook, and a covered template drift guard (#991) - #1699
Merged
Conversation
# Conflicts: # CHANGELOG.md
…erfmon counters (#991) ag_database_replica_states gains six appended columns: - last_commit_time / last_hardened_time / last_redone_time / last_received_time, the four DMV timestamps the reference query skips. Unlike secondary_lag_seconds these are directly comparable across replicas, which the primary-vs-secondary commit-time lag math needs. - est_redo_completion_time_min / est_send_drain_time_min, computed server-side as queue / rate / 60 with both guards the raw expression needs: `* 1.0` stops BIGINT/BIGINT integer division flooring a sub-minute drain to zero, NULLIF(rate, 0) stops the divide-by-zero an idle or suspended replica raises (which fails the whole cycle, not one column). NULL means "no drain rate" and is never coerced to 0, which would read as "drains instantly". The estimates are computed per row at the sample's own instant rather than composed later, because a ratio of two window aggregates is not the average of the per-sample ratios and the two diverge worst exactly when rates swing. Both are Gauge compose measures, duration family, minutes. PerfmonStatsCollector's whitelist gains Transaction Delay and Mirrored Write Transactions/sec - the primary side of commit latency, their ratio being the average delay per mirrored transaction. Zero new schema. Both verified live on SQL2022: they exist with no AGs configured, sit on SQLServer:Database Replica, and occur exactly once server-wide, so the counter_name-only filter cannot collide. Corrects a doc claim shipped in #1688. It restated MS Learn's assertion that secondary_lag_seconds reads 0 while data movement is suspended; a Docker AG fixture measured the inverse on SQL Server 2022 in a CLUSTER_TYPE = NONE group - 0 while movement is ACTIVE and caught up, accruing monotonically once suspended (0 to 62s across a 60s SUSPEND_FROM_USER, back to 0 on resume). A suspended replica does not hide as zero lag. Also documents two further measured quirks: log_send_queue_size goes NULL while suspended while redo_queue_size freezes at its last value, and collecting from a secondary yields a one-row self-view because sys.dm_hadr_* carries only the local replica there. Store migration V36 appends the columns additively instead of widening V34, because V34's CREATE TABLE IF NOT EXISTS is a no-op on an already-migrated store - editing it would leave every existing store six columns short while fresh installs got them. The schema pin now reconstructs the current shape from V34 + V36 and compares it to the generator; both failure modes confirmed by planted defects. Version 36 because 35 is claimed by the concurrent AG-alerts work. THIRD_PARTY_NOTICES.md now credits Hannah Vernon's SqlServerAgMonitor alongside the existing collector-header attribution. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The end-to-end Postgres test asserted COUNT(*) FROM darling_schema_version == StorageVersion.SchemaVersion. That identity only holds while migration versions are DENSE from 1, so the first concurrently-developed pair of migrations broke it: V35 (AG alerts) and V36 (AG latency) are being built on separate branches, and V36 alone leaves a temporary gap. The gap is inert to the applier - MigrateAsync applies every script whose version exceeds MAX(version) and never assumes contiguity - so the proxy was the only thing that cared. Replaced with the two invariants it was conflating, which together are strictly stronger: MAX(version) == SchemaVersion (the store reached this build's version, the same expression MigrateAsync itself reads) and COUNT(*) == PgMigrations.Scripts.Count (every script ran, one stamped row apiece, so a silently skipped script still fails). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ver every template (#991) The fifth Custom Views v2 seed, built on the AG measures from #1688/#1695. Five panels in diagnostic order: 1. Secondary lag over time, grouped by replica. 2. Dual-axis line: log send rate vs redo rate on their own axes - the panel that answers which side is the bottleneck. 3. Stacked redo-queue series by database. 4. Stat tile: worst estimated redo drain. 5. Top-10 bar of send-queue backlog by database. The prose between panels carries the two things that make the numbers readable: point the view at the PRIMARY (sys.dm_hadr_* on a secondary only carries the local replica, so a secondary-scoped view is a one-row self-view), and a blank drain estimate means no drain rate - idle, caught up, or suspended - not that it drains instantly. Adds the two template helpers the existing seeds never needed: overlay support on tsPanel (dual-axis, legal only on an ungrouped line/area) and a scalar statPanel. This is the first seed exercising dual-axis, stacked and stat modes end to end. The brief asked for span 2 on the lag panel; notebooks are single-column documents and the renderer strips span, so it is omitted rather than stored as a field nothing honors. Also closes a hole in the templates drift guard. It hand-mirrors each template's panels so a MeasureCatalog change that breaks a seed fails in CI, but nothing checked the mirror COVERED every template - a sixth template added without a mirror entry would have gone silently unvalidated until it 400'd in a browser. It now reads the template keys out of notebook.js and requires the mirror to match exactly. Verified by renaming a key and watching it go red, and the new mirror was checked against the real module output by executing notebook.js under node and diffing the emitted panel JSON. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
erikdarlingdata
enabled auto-merge
July 26, 2026 18:45
# Conflicts: # CHANGELOG.md # Darling/Darling.Tests/DarlingObservabilityTests.cs # Darling/Darling.Tests/DarlingServerTagsTests.cs # Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs # Darling/PerformanceMonitor.Darling.Storage/StorageVersion.cs # Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs
# Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md
…ngdata/PerformanceMonitor into feature/991-ag-latency
# Conflicts: # CHANGELOG.md
# Conflicts: # CHANGELOG.md
…ingdata/PerformanceMonitor into feature/991-ag-template
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The seeded "AG Health" Custom Views v2 template, built on the Availability Group measures from #1688 and #1695.
Stacked on #1695 (still open), so the diff here includes that PR's commits until it merges.
The template
Fifth entry in
NOTEBOOK_TEMPLATES, five panels in diagnostic order — lag says how bad, rates and queues say why, the drain estimate says how long until it clears:groupBy: replica_server_nameoverlay: ag_redo_rategroupBy: database_nametopN: 10Panel 2 is the one that earns its place: send outpacing redo means the secondary is receiving faster than it can replay, so the redo queue — and failover time — is growing. Two separate axes is the only way that reads, since the two series routinely differ by an order of magnitude.
The markdown between panels carries the two things that make the numbers readable rather than merely present:
sys.dm_hadr_*carries only the local replica's rows on a secondary, so a secondary-scoped view is a one-row self-view that says nothing about its peers. Someone building an AG dashboard against a secondary would otherwise conclude their AG has one replica.The lag note also reflects the measured behavior from the AG fixture rather than the docs: a suspended replica shows lag accruing, so a spike is not automatically a performance problem and the synchronization state is worth checking before chasing it.
Two helper gaps
The existing four seeds only ever needed
tsPanel/rankedPanel, so neither dual-axis nor scalar mode was expressible:tsPanelgainedoverlaypassthrough (the Darling/PG vs Datadog DBM: close the in-scope cutover gaps (client) #1606 second measure — legal only on an ungrouped line/area, which the server enforces).statPanel: notimeBucketand notopNis what puts a panel in scalar mode, so it needed its own builder rather than a flag.This is the first seed exercising dual-axis, stacked and stat end to end, which makes its drift-guard entry the regression test for those three modes staying valid.
One deviation from the brief
The brief asked for span 2 on the lag panel. Notebooks are single-column documents and the renderer explicitly strips
span(notebook.js: "a notebook is a single-column document, so a panel cell is always full width").NOTEBOOK_TEMPLATESis also the only template mechanism — the "New from template" menu is notebook-only, there is no plain-view template array — so there was no non-notebook shape to build this as instead. I omittedspanrather than storing a field nothing honors. Everything else in the brief is as specified.A hole in the drift guard, closed
ValidateDefinition_AcceptsEverySeedNotebookTemplatehand-mirrors each template's panels so that aMeasureCatalogrename breaks CI instead of the template silently 400ing in the UI. But nothing checked the mirror covered every template — a sixth template added without a mirror entry would have gone completely unvalidated, which is the same class of silent-exemption bug as the coverage-ratchet issue fixed in #1691.It now reads the template keys out of
notebook.js(source-scanned;key:appears nowhere else in that file, and Darling.Tests has no JS runtime) and requires the mirror to cover exactly that set.Verified both directions rather than assumed:
ag-healthtoag-health-renamedinnotebook.js→ test red. Restored → green.notebook.jsunder node, dumped the emitted panel JSON, and diffed it against the mirror parsed back out of the C# string literals.mirror panels: 5, actual panels: 5 — MIRROR IS FAITHFUL. That closes the gap a hand-mirror always has, which is that it can be written wrong from the start and still pass its own validation.Tests
Lite.Tests is untouched by this change (frontend JS + one Darling test file).
Flaky test worth flagging
ViewerWave3DisplayTests.TimeLocal_TreatsTheStoredValueAsUtcfailed once in a full-suite run and passed both in isolation and on an immediate re-run. That is the second*TimeLocal*failure of this kind today —ViewerActiveQueriesDisplayTests.QuerySnapshotRow_CollectionTimeLocal_ConvertsNaiveUtc_EmptyForMinValuedid the same earlier and also passed isolated. Neither is related to any change in these PRs (both are local-time conversion tests). Two independent flakes in the same family suggests a shared time-zone or clock assumption that is order- or load-sensitive, and it is worth someone looking at properly rather than re-running past.🤖 Generated with Claude Code