feat(dv2): event-driven OLTP→vault freshness via LISTEN/NOTIFY#93
Merged
Conversation
With both the OLTP hot tier and the raw vault on PostgreSQL, freshness no longer needs a replication slot, a WAL consumer, or a second engine. An AFTER INSERT/UPDATE trigger on each ops_<branch> table issues pg_notify on the dv2_vault_refresh channel; freshness_listener.py LISTENs and runs the idempotent promotion the moment a change lands -- push, not poll. This is the PostgreSQL-native equivalent of the ClickHouse MaterializedPostgreSQL push-CDC: the same property with no replication slot and no second engine, because the vault is in the same instance. - freshness_listen_notify.sql: rv.notify_oltp_change() + one idempotent trigger per OLTP table; payload carries branch/table/op and a clock_timestamp() emit time. - freshness_listener.py: guarded-psycopg listener with a driver-agnostic pure core (parse_notification / lag_ms / process_notifications). The observation clock is the PostgreSQL server clock (db_now) so the measured lag is skew-free across a containerised host. - tests: 18 no-Docker tests (trigger SQL structure + listener logic via fake notifications + db_now contract). Verified: ruff/format/mypy clean; 18 no-Docker tests pass. Single-node Mac smoke (throwaway postgres:16) ran the full INSERT -> trigger -> NOTIFY -> promote -> bv_order_canonical round-trip, lag 62 ms (DB-clock measured). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DORA Metrics
|
brownjuly2003-code
added a commit
that referenced
this pull request
Jun 27, 2026
…nts.txt header (#95) * fix(cache): use redis set(ex=) instead of deprecated setex `Redis.setex` is `@deprecated_function` in redis-py 8.0.0 ("Use 'set' instead") and emits a DeprecationWarning from the query-cache hot path on every cached write. Switch `QueryCache.set` to `set(key, value, ex=ttl)`, which is behaviorally identical (int seconds), and drop the now-unused `timedelta` import. All six in-repo Redis test doubles (unit cache/entity_cache/versioning, integration tenant-isolation, chaos RESP client) implemented `setex`; they move to `set(self, key, value, ex=None)` with the matching argument order, and the chaos RESP client now issues `SET … EX` over the wire. The two `set_calls` ttl assertions compare the integer `ex` directly. Verified no-Docker: ruff + mypy clean, full unit suite 1096 passed / 1 skipped (the redis.setex DeprecationWarning is gone). The integration and chaos doubles change symmetrically and are validated by their CI jobs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: sync version story to v1.5.0 and clarify requirements.txt The README Status section and release badge still described v1.4.0 as the current line even though v1.5.0 is tagged and published, and the CHANGELOG [Unreleased] section did not record the DV2 re-architecture already on main. - README: badge v1.4 -> v1.5, "current release line" -> v1.5.0, extend the release arc to five increments with a v1.5.0 bullet (argon2id O(1) key hashing, NL->SQL guard bypass fix, strict-mypy expansion), and add a note that main carries post-v1.5.0 work pending the next tag. - CHANGELOG [Unreleased]: document the DV2 raw vault migration ClickHouse -> PostgreSQL (#91), the PyIceberg sink backed by real MinIO (#92), the LISTEN/NOTIFY OLTP->vault freshness (#93), and the dependency batch (#94). - requirements.txt: add a header explaining it is a supplemental OTel pin set installed on top of the pyproject package by the e2e/mutation/staging workflows and the security Safety scan, not the full dependency set (pyproject.toml is the source of truth). load_requirements() skips comment lines and `pip -r` ignores them, so the header is non-breaking. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: JuliaEdom <uedomskikh@gmail.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Completes the OLTP→raw-vault path on PostgreSQL (after #91) with push-based freshness. With both OLTP and the vault on Postgres there's no need for a replication slot / WAL reader / second engine: an
AFTER INSERT/UPDATEtrigger on eachops_<branch>table firespg_notify('dv2_vault_refresh', …), and a listener runs the idempotent promote on each event (push, not polling). This is the PG-native equivalent of ClickHouseMaterializedPostgreSQLpush-CDC, but the whole mechanism isNOTIFY+ in-DBINSERT…SELECT.Changes
postgres_oltp/freshness_listen_notify.sql—rv.notify_oltp_change()+ 4 idempotent triggers (DROP IF EXISTS + CREATE); payload carries branch/table/op +clock_timestamp()emit time.postgres_oltp/freshness_listener.py— guarded-psycopg listener with a driver-agnostic pure core (parse_notification/lag_ms/process_notifications), no-Docker testable.tests/unit/test_dv2_freshness_listen_notify.py— 17 no-Docker tests (trigger-SQL structure + listener logic on fake notifications + db-clock contract).postgres_oltp/README.md— documents the push-freshness path.Verification
INSERTinto OLTP → trigger →NOTIFY→ listener → idempotent promote → row visible inbv_order_canonical, lag 62 ms. The observation clock must be the server-side PG clock (db_now), not the client wall clock, to avoid host↔container skew (a first run mismeasured 14774 ms before the fix). This branch is content-identical to that validated commit, rebased onto the merged feat(dv2): migrate raw vault to PostgreSQL + cloud supplier reference #91 vault it targets.