[INF-1722] Avoid full status scans in CDC metadata reads - #20
Merged
Conversation
thesyncim
marked this pull request as ready for review
August 27, 2026 12:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ticket
Summary
The CDC receiver checks the cutover boundary before every replication message.
Store.Migrationrebuilt the entire status snapshot for that check: a read transaction with ten SELECTs, including inventory aggregates and verification progress. Read the same nine singleton metadata fields directly instead.One production file changes (29 added / 4 removed lines). No caching, new settings, dependencies, schema changes, WAL/replay-format changes, or changes to cutover/acknowledgement/transaction logic. The full dashboard
Snapshotis unchanged.Evidence
Synthetic c3-sized inventory: 113 tables, 276 parts, 509 indexes, 12 constraints. Median of three 1-second runs, Go 1.26.0, Apple M4 Max:
The metadata read is 15.4x faster; the unchanged full Snapshot remains approximately 146 microseconds. This is not an end-to-end capture-throughput multiplier.
Validation
go test ./... -count=1,go vet ./..., andgo test -race ./internal/state ./internal/cdc ./internal/cutover -count=1.-race. TheirTestPG17...names are historical: the actual runtime was PostgreSQL 18.6, matching production's major version. Docker storage was broken, so a temporary Go-overlayreplaced only the test server launcher with fresh loopback PostgreSQL instances. No test assertions or production code were overlaid; the adapter is outside this PR. PostgreSQL 16/17 integration execution is not claimed.Rollout and risk
The user authorized a progress-preserving c3 deployment after validation. Image v68 adds only the tested Linux binary to the exact running v67 image; the base layers and runtime configuration are unchanged. A credential-free, PVC-free pod smoke test passed and the binary checksum matched. The existing PVC, source slot, capture checkpoints, and target progress/receipts must be preserved; no recopy, index rebuild, source-data changes, or cutover is part of this rollout.
Deployed on 2026-08-27, resumed at 12:16:37 UTC, pinned to
sha256:d8876cff3c7a42471ba0af75cae3fc45f887dba8e2f41ce668945f369f34cce2. Saved configuration and snapshot metadata hashes matched before/after. The pre-stop partial claim resumed and finalized at its exact end LSN, adding exactly its 371 transactions and 1,283 changes. The 113 tables, 276 parts, 509 indexes and 12 constraints stayed complete. By 12:22:29 UTC another 1,500,603 changes had been committed beyond the pre-stop checkpoint; no new operation error. Local UI/port-forward verified live.Measured WAL-position rates: capture averaged 20.0 MiB/s over the 5.5-minute pre-update sample and 130.3 MiB/s over the 5.3-minute post-update sample. In the latter window actual production generated 17.0 MiB/s, while committed target replay advanced about 42.6 MiB/s. These are live observations, not a controlled workload benchmark or a guarantee. Around 192 GiB of end-to-end lag remained; faster capture moves some backlog into the local queue. The UI's “source” rate is capture, not production. Existing retention warnings remain; no whole-dataset parity or cutover-readiness claim.
Checklist