E4 replica-correctness topology proof + serving audit fixes (#184-186)#187
Merged
Conversation
…6-07-11) Completes the cutover-plan Phase 3 LIVE-verify item (attempted-but-blocked 2026-07-06 by shared-host resource contention). On a freed VM, scripts/k8s_replica_correctness_verify.sh Checks 1-2 pass: replicaCount=2 with both pods on the postgres control-plane store, and a webhook registered via the Service visible on all 8 round-robin reads across the two pods (no split-brain, ADR 0010 class 5). Exactly-one delivery / one-alert-per-incident already store-level verified 31/31 (slice-5 PG probe); this adds the two-real-pods topology layer. Canon: docs/perf/e4-replica-topology-2026-07-11.md; cutover-plan checkbox flipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- webhook_dispatcher: drop dead bare-event_id membership test (audit #184); dedup is strictly on tenant:event_id, the only shape the seen-set ever stores. - webhook_dispatcher: seed the startup scan cursor from the newest row whose processed_at parses, not blindly the newest row (audit #185); a malformed newest timestamp left the cursor None, which made the first scan re-deliver the whole seeded batch from the oldest journal row. - cache_invalidation: raise the metric-cache scan window 200 -> 2000 rows (audit #186); 200 left no margin for a non-pushing writer at the >=100 eps target (pre-merge ReplacingMergeTree emits ~2 rows/event), and the comment claimed headroom it did not have. - tests: per-tenant dedup guard (#184) + malformed-newest seed-edge (#185). - docs/serving-bridge: sync the window size and its sizing rationale. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DORA Metrics
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two independent changes on this branch (bundled into one PR).
545c2b1— E4 delivery topology proof (kind)Closes the Phase-3 cutover checkbox that 07-06 failed on resource pressure. Two real pods on the Postgres control-plane store, webhook visible 8/8 cross-pod (no split-brain, ADR-0010 class 5).
scripts/k8s_replica_correctness_verify.shChecks 1–2 exit 0. Proof doc:docs/perf/e4-replica-topology-2026-07-11.md. Doc-only commit — no runtime code changes.b5b950d— serving audit fixes (#184-186)Adversarial audit of #184/#185/#186 surfaced three non-blocker nits; fixed:
event_idmembership test inwebhook_dispatcher. The seen-set only ever holdstenant:event_id, so the bare check never matched. Dedup is strictly per-tenant now (regression test added).mark_existing_events_seenseeds the scan cursor from the newest row whoseprocessed_atparses, not blindly the newest row. A malformed newest timestamp left the cursorNone, which made the first scan fetch from the oldest journal row and re-deliver the whole seeded batch.DEFAULT_SCAN_WINDOW_ROWS200 → 2000 incache_invalidation. At the ≥100 eps target the pre-merge ReplacingMergeTree journal emits ~2 rows/event (~400 tail rows per 2 s pass); 200 left no margin for a non-pushing writer, and the comment claimed headroom it did not have. Push feeds still cover writers that publish.Test plan
pytest tests/unit/test_webhook_dispatcher_unit.py tests/unit/test_cache_invalidation.py— 41 passedmypy src/ --ignore-missing-imports— clean, 130 filesruff check+ruff format --check— clean🤖 Generated with Claude Code