Docker Availability Group test fixture, and AG collector validation (#991)

[erikdarlingdata]

Adds tools/ag-fixture, a two-node SQL Server 2022 Availability Group in Docker, and uses it to validate the AG collector queries from #991.

The AG DMVs return nothing at all on a standalone instance, so AG collection has never had anything to be validated against. This is that thing. It belongs in the release checklist next to the other install validation: any change to AG collection, AG alerting, or the AG topology views should be run against it before it ships.

What it is

A clusterless AG (CLUSTER_TYPE = NONE, read-scale) - real replicas, real synchronization state, real send/redo queues, real suspend/resume, no WSFC or Pacemaker. What it does not get is a listener or automatic failover, so it is not the fixture for those.

• docker-compose.yml - ag1/ag2, HADR enabled, pinned hostnames, errorlog healthchecks copied from the shape in sql-validation.yml. Primary on localhost,14331, secondary on localhost,14332.
• setup.ps1 - one idempotent orchestrator. Waits for health, then for logins, runs the numbered sql/ scripts through each container's own sqlcmd, waits for the seed to land, prints the state. A full re-run against a live fixture is a no-op.
• sql/01-07 - master key and shared endpoint certificate, certificate transport to ag2, mirroring endpoints on 5022, CREATE AVAILABILITY GROUP, join plus GRANT CREATE ANY DATABASE, seeded AgFixtureDb, verification.
• write-load.ps1 / sql/90_write_load.sql - insert loop so the queues and rates move off zero.
• teardown.ps1 - compose down -v.
• README.md - purpose, start/stop, port map, pointing Lite/Darling at it, the suspend/resume fault recipe, a troubleshooting section, and the warning that sa and the passwords are fixture-local.

Passwords come from a gitignored .env; .env.example ships placeholders, so no working password lands in the repo.

Validation

Both collector queries from feature/991-ag-health-collector @ 530b3b66 were run verbatim against both replicas, under write load, before and after a suspend fault. Full evidence with row output is in tools/ag-fixture/VALIDATION.md.

Check	Result
AG reaches PRIMARY + SECONDARY, CONNECTED, HEALTHY	Pass
setup.ps1 idempotent (full re-run against live fixture)	Pass
Replica-grain query on primary	Pass, 2 rows
Database-grain query on primary	Pass, 2 rows
Non-NULL queues and rates under load	Pass
Suspend sets is_suspended + SUSPEND_FROM_USER, resume clears	Pass

The collector queries are correct and need no changes. But the fixture turned up three behaviors worth knowing about, and one of them contradicts the documentation.

1. A secondary reports only itself

                                      ag1   ag2
sys.availability_replicas               2     2
sys.dm_hadr_availability_replica_states 2     1
sys.dm_hadr_database_replica_states     2     1

The catalog view knows both replicas on both nodes; the two DMVs carry only the local replica on a secondary, so the collectors' inner join narrows a secondary to a one-row self-view. A clusterless AG gives a secondary no cluster metadata store from which to learn its partners' runtime state.

So a complete AG picture requires collecting from the primary, the replica that returns the full set changes after a failover, and anything assuming "one row per replica per AG from any monitored node" is wrong.

2. log_send_queue_size reads NULL while suspended

Sampled on the primary's remote row across a 60-second suspend, write load running:

sample            is_suspended  log_send_queue_size  redo_queue_size
active/caught up  0             60                   3728
+15s suspended    1             NULL                 4036
+30s suspended    1             NULL                 4036
+45s suspended    1             NULL                 4036
+60s suspended    1             NULL                 4036
after resume      0             60                   388620

The send queue does not grow while movement is suspended - it reads NULL - and redo_queue_size freezes rather than climbing. A "send queue over N KB" rule is blind to a suspended secondary, which is one of the most common ways a secondary falls behind; is_suspended / suspend_reason_desc are the signal for that. NULL must not compare as zero-and-healthy. The 388,620 KB in the last row is legitimate post-resume catch-up that a single-sample redo threshold would fire on.

3. secondary_lag_seconds accrues while suspended, contradicting MS Learn

MS Learn: "This value shows as 0 if the data movement is suspended. The data movement needs to be in a non-suspended state in order for this value to show active lag."

Observed, same suspend:

sample            is_suspended  sync_state         secondary_lag_seconds
active/caught up  0             SYNCHRONIZED       0
+15s suspended    1             NOT SYNCHRONIZING  15
+30s suspended    1             NOT SYNCHRONIZING  31
+45s suspended    1             NOT SYNCHRONIZING  46
+60s suspended    1             NOT SYNCHRONIZING  62
after resume      0             SYNCHRONIZED       0

It accrues monotonically with suspension time - the inverse of the documented behavior, reading 0 when movement is active and caught up. The blind spot the docs imply does not exist: a suspended secondary reports growing lag and a lag threshold fires on it unaided.

This one matters beyond the fixture: the AgDatabaseReplicaStatesCollector doc comment currently restates the MS Learn sentence as fact and reasons from it ("a suspended replica presents as zero lag ... or it will under-report the worst case"). Faithful to the docs, but contradicted by the hardware. Passed to the collector and alerts authors rather than edited here, since those files belong to other branches in flight.

Scope of the claim: one build (16.0.4265.3), clusterless AG only. A WSFC AG was not tested and this fixture cannot test one.

Resource footprint

The fixture is capped so it can share a machine with a busy VM fleet: 1 CPU and 2 GB per container, engine held to 1536 MB via MSSQL_MEMORY_LIMIT_MB, no SQL Agent, 64 MB database files.

mem_limit cannot go below 2 GB - SQL Server on Linux refuses to start under ~2000 MB - so MSSQL_MEMORY_LIMIT_MB is the knob that actually lowers the engine's appetite inside it. 1536 MB proved stable through a clean build, seeding, write load, and a suspend/resume fault, so it was not stepped up to 2048. Measured, not asserted:

state                     ag1                    ag2
idle, just after setup    3.82%  1.345GiB/2GiB   4.06%   1.325GiB/2GiB
under write load          71.69% 1.718GiB/2GiB   99.38%  1.689GiB/2GiB
settled, after load       2.75%  1.78GiB/2GiB    4.33%   1.798GiB/2GiB

No OOM kill or restart at any point. Memory settles near 90% of the cap and stays there, which is expected - SQL Server does not release the buffer pool, and MSSQL_MEMORY_LIMIT_MB governs the engine's budget rather than the container's total RSS, so ~250 MB of process overhead sits on top of the 1536 MB.

One thing this surfaced that is worth calling out: an AG database must stay in RECOVERY FULL, so it cannot truncate its own log, and the write-load script was taking the log to 1632 MB used (and the data file to 1376 MB) in a single 90-second run. The script now backs the log up periodically to one overwritten file, and the log holds at its initial 64 MB across the same workload. That was not in the brief; unbounded log growth on a fixture meant to be left running defeats the point of capping anything else.

Everything was re-validated at the new limits: AG builds clean to HEALTHY/SYNCHRONIZED, both collector queries still return real rows with non-NULL queues and rates, and findings 1-3 reproduce exactly (they are DMV semantics, not resource effects). teardown.ps1 is also now exercised - it was the one untested path before - and removes both containers, the network, and both volumes.

Follow-on: finding 3 reconciled, and one comment corrected

The lag finding got sharper. @ag-alerts-builder reproduced the accrual independently on this fixture but measured it starting near 3993 where I measured 0. Their redo_queue_size was 0 throughout - an idle AG, where mine was under write load. Re-testing the idle case explicitly reconciles both:

idle, NOT suspended       lag=0    (373s since last hardening - still 0)
idle, suspended +15s      lag=0    NOT SYNCHRONIZING
idle, suspended +30s      lag=450  (secs_since_hardened 424)
idle, suspended +45s      lag=465  (secs_since_hardened 440)
after resume              lag=0

While movement is active it reads 0 no matter how long the group has been idle. Once suspended it latches onto roughly how stale the secondary's last hardened log is (now - last_hardened_time), not time since suspension - so it starts near zero under load and at thousands of seconds when idle. Same behavior, different starting point. It also does not latch immediately: at +15s suspended it still read 0 while already NOT SYNCHRONIZING. VALIDATION.md now carries all of this.

This was not academic. The alert family (#1692) had shipped a lag rule that abstained on a suspended row, written specifically to compensate for the MS Learn sentence - which, the docs being inverted, silenced the lag alert on suspended data movement. That is fixed in #1700.

Since #1688 merged to dev with the collector's doc comment unchanged, that comment was going to ship this week still asserting the false behavior and reasoning from it. I corrected it here (5f7a20b8): it now states the measured behavior, the idle-vs-loaded base, the non-immediate latch, and the operative rule - a suspended row may raise a lag alarm but must never clear one, and the magnitude is staleness, not volume at risk. Comment only, no behavior change; the query and column set were already correct and validated verbatim. Builds clean, 0 warnings. @ag-collector-builder has been told so nobody duplicates it, and the two additive doc points from my review (the one-row self-view, the NULL send queue) were deliberately left to them.

Erik merged origin/dev into this branch mid-flight; CHANGELOG entries and link-refs from both sides survived intact, no conflict markers.

Judgment calls

• One shared endpoint certificate, not one per replica. It is owned by dbo on both sides and dbo in master maps to sa, so no separate endpoint login or GRANT CONNECT ON ENDPOINT is needed. This is the form Microsoft documents for read-scale AGs on Linux, and the extra login the brief mentioned would have been dead weight. Validated working end to end.
• SYNCHRONOUS_COMMIT. It is what drives SYNCHRONIZED / HEALTHY, which is the state the collectors are being checked against. REQUIRED_SYNCHRONIZED_SECONDARIES_TO_COMMIT stays at its default of 0 so stopping ag2 does not freeze writes on ag1.
• Backups and certificates go to /var/opt/mssql/data. SQL Server on Linux has no /var/opt/mssql/backup by default, and adding a volume for it would only be one more thing to clean up.
• setup.ps1 and teardown.ps1 prepend Docker's bin directory to PATH. A default Docker Desktop install puts none of it on PATH, and docker shells out to docker-credential-desktop.exe by name - without this, compose up dies on "error getting credentials" before pulling anything. Hit this on the first run.
• docker cp output is chowned to mssql. Files landed by docker cp are root-owned and SQL Server cannot read them; CREATE CERTIFICATE then fails with a misleading "cannot find the file".
• No "stop but keep the data" verb. A half-torn-down AG is worse than no AG.

Test plan

• setup.ps1 from nothing: containers healthy, AG built, AgFixtureDb seeded to ag2, verification shows PRIMARY + SECONDARY / CONNECTED / HEALTHY.
• setup.ps1 -SkipCompose re-run against the live fixture: no errors, same output - idempotency guards hold.
• write-load.ps1: 5,270 batches / 2,635,000 rows in 130s; queues and rates non-NULL during it.
• Both collector queries verbatim on ag1 and ag2, before and during load.
• Suspend on ag2: is_suspended = 1, suspend_reason_desc = SUSPEND_FROM_USER, synchronization_state_desc = NOT SYNCHRONIZING from both vantage points; replica grain flips HEALTHY -> NOT_HEALTHY. Resume clears all of it.
• secondary_lag_seconds sampled across a 60s suspend to confirm finding 3 is monotonic accrual and not a stale read.
• .env confirmed gitignored and absent from the commit.
• teardown.ps1: removes both containers, the network, and both volumes, and nothing else - the compose project scope is exactly ag-fixture/ag1/ag2.
• Full teardown + rebuild at the capped limits: AG clean to HEALTHY/SYNCHRONIZED, collector queries still return non-NULL queues and rates at 1 CPU, suspend/resume findings reproduce, no OOM kill.
• Log trim verified: 64 MB held across a 70s load, against 1632 MB for the same workload without it.

Containers are up now: ag1 localhost,14331, ag2 localhost,14332, sa password in the local .env.

Closes nothing on its own; supports #991.

🤖 Generated with Claude Code