Conversation
There was a problem hiding this comment.
Pull request overview
Adds a reactive “live query” capability to Typegres (db.live(query)) by extracting equality predicates from queries, emitting in-transaction mutation events into a shadow table, and using an in-process polling bus to re-run queries when relevant commits become visible.
Changes:
- Introduces live-query pipeline: predicate extraction/closure, snapshot visibility logic, event bus polling + subscriptions, and
Database.startLive()/live()/stopLive(). - Adds mutation SQL transformation hooks on tables and wraps INSERT/UPDATE/DELETE in event-emitting CTE chains.
- Updates type/runtime + codegen to support typed parameters and safer primitive-overload matching; refactors tests to use new shared DB helpers.
Reviewed changes
Copilot reviewed 38 out of 69 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tsconfig.json | Bumps TS target to ES2024 (supports new runtime usage like Promise.withResolvers). |
| src/types/runtime.ts | Narrows PgOp to accept Raw operators; aligns runtime types with SQL node changes. |
| src/types/overrides/any.ts | Uses TypedParam for primitive literals; improves column-vs-expression validation for mutations. |
| src/types/generated/xid.ts | Regenerated overloads / primitive matching for xid operators. |
| src/types/generated/tsvector.ts | Regenerated overloads / primitive matching for text search methods. |
| src/types/generated/tsquery.ts | Regenerated overloads / primitive matching for tsquery methods. |
| src/types/generated/timetz.ts | Regenerated overloads / primitive matching changes (notably primitive allowances). |
| src/types/generated/timestamptz.ts | Regenerated overloads / primitive matching changes. |
| src/types/generated/time.ts | Regenerated overloads / primitive matching changes. |
| src/types/generated/regconfig.ts | Regenerated overloads / primitive matching changes. |
| src/types/generated/polygon.ts | Regenerated overloads / primitive matching changes. |
| src/types/generated/pg_lsn.ts | Regenerated overloads / primitive matching changes. |
| src/types/generated/numeric.ts | Regenerated overloads / primitive matching changes. |
| src/types/generated/money.ts | Regenerated overloads / primitive matching changes. |
| src/types/generated/jsonb.ts | Regenerated overloads / primitive matching changes. |
| src/types/generated/json.ts | Regenerated overloads / primitive matching changes. |
| src/types/generated/interval.ts | Regenerated overloads / primitive matching changes. |
| src/types/generated/int8.ts | Regenerated overloads / primitive matching changes. |
| src/types/generated/inet.ts | Regenerated overloads / primitive matching changes. |
| src/types/generated/circle.ts | Regenerated overloads / primitive matching changes. |
| src/types/generated/bytea.ts | Regenerated overloads / primitive matching changes. |
| src/types/generated/bit.ts | Regenerated overloads / primitive matching changes. |
| src/types/generated/anyrange.ts | Regenerated overloads / primitive matching changes for range types. |
| src/types/generated/anymultirange.ts | Regenerated overloads / primitive matching changes for multirange types. |
| src/types/generated/anycompatiblearray.ts | Regenerated overloads / primitive matching changes for compatible arrays. |
| src/types/generated/anyarray.ts | Regenerated overloads / primitive matching changes for arrays. |
| src/types/generate.ts | Reworks “allowPrimitive” codegen logic to avoid ambiguous overload resolution across types sharing TS primitives. |
| src/test-helpers.ts | New shared Vitest DB lifecycle + transaction helpers; adds SQL structural equality helper. |
| src/table.ts | Adds per-table mutation transformer hook and isTableClass helper; updates Table() factory signature. |
| src/table.test.ts | Migrates tests to new shared setupDb() helpers. |
| src/rpc.test.ts | Migrates tests to new shared setupDb() helpers. |
| src/live/test-helpers.ts | Adds helper to create/truncate _typegres_live_events for live tests. |
| src/live/snapshot.ts | Implements snapshot parsing + JS visibility predicate for xid checks. |
| src/live/snapshot.test.ts | Tests snapshot parsing + visibility rules. |
| src/live/run.ts | Implements a single live iteration (repeatable read txn + extractor + query). |
| src/live/run.test.ts | Tests runLiveIteration outputs (rows + cursor + predicate set). |
| src/live/extractor.ts | Implements predicate extraction, alias ordering, extractor SQL generation, and predicate closure. |
| src/live/extractor.test.ts | Tests traversal ordering, SQL shape, and predicate closure behavior. |
| src/live/events.ts | Implements mutation wrappers that append before/after images into _typegres_live_events. |
| src/live/events.test.ts | Tests insert/update/delete event emission and RETURNING passthrough. |
| src/live/db-live.test.ts | End-to-end tests for db.live() reruns on matching commits. |
| src/live/bus.ts | Implements polling bus, reverse-indexed subscription matching, and backfill window handling. |
| src/live/bus.test.ts | Tests bus signaling, cursor visibility skipping, backfill, and CursorTooOldError. |
| src/live/ISSUES.md | Tracks known gaps and follow-ups for live queries. |
| src/live/AGENTS.md | Documents live-query design, scope, and semantics. |
| src/index.ts | Exposes transaction option types publicly. |
| src/exoeval/tool.ts | Removes per-file eslint no-redeclare suppression (handled globally now). |
| src/exoeval/index.ts | Removes per-file eslint no-redeclare suppression (handled globally now). |
| src/exoeval/expr.ts | Removes per-file eslint no-redeclare suppression (handled globally now). |
| src/database.ts | Adds transaction isolation options + nesting rules; adds startLive/live/stopLive APIs. |
| src/database.test.ts | Adds tests for isolation nesting rules; migrates to shared DB helpers. |
| src/builder/update.ts | Introduces FinalizedUpdate, returningMerge, transformer hook support, and shared SET compilation. |
| src/builder/update.test.ts | Migrates to shared DB helpers. |
| src/builder/test-helper.ts | Removes old always-on DB test helper (replaced by src/test-helpers.ts). |
| src/builder/sql.ts | Refines SQL node types (Op uses Raw, Column fields), adds TypedParam, and overloads `sql`` for Raw literals. |
| src/builder/query.ts | Adds FinalizedQuery, exposes opts, adds mergeReturning, and uses isTableClass. |
| src/builder/query.test.ts | Migrates to shared DB helpers. |
| src/builder/insert.ts | Introduces FinalizedInsert, returningMerge, and transformer hook support. |
| src/builder/insert.test.ts | Migrates to shared DB helpers. |
| src/builder/delete.ts | Introduces FinalizedDelete, returningMerge, and transformer hook support. |
| src/builder/delete.test.ts | Migrates to shared DB helpers. |
| eslint.config.js | Disables no-redeclare globally (TS overloads). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+115
to
+134
| const wrapInsertOrDelete = ( | ||
| builder: InsertBuilder<any, any, any> | DeleteBuilder<any, any, any>, | ||
| side: "before" | "after", | ||
| ): Sql => { | ||
| const tableName = builder.tableName; | ||
| const liveKey = `__live_${side}` as const; | ||
|
|
||
| // returningMerge() has the same shape on both InsertBuilder and | ||
| // DeleteBuilder; we cast to one of them to pick a single overload | ||
| // rather than writing a generic structural type. | ||
| const innerFinalized = (builder as InsertBuilder<any, any, any>) | ||
| .returningMerge(liveAfterReturning(tableName, side)) | ||
| .finalize(); | ||
|
|
||
| const beforeRef = side === "before" ? sql.ident(liveKey) : sql`NULL::jsonb`; | ||
| const afterRef = side === "after" ? sql.ident(liveKey) : sql`NULL::jsonb`; | ||
| const events = eventsInsertCte(tableName, beforeRef, afterRef); | ||
| const projection = userReturningProjection(innerFinalized.opts.returning ?? {}, liveKey); | ||
| return sql`WITH cte AS (${innerFinalized}), _events AS (${events}) SELECT ${projection} FROM cte`; | ||
| }; |
There was a problem hiding this comment.
wrapInsertOrDelete() always emits WITH cte AS (...), _events AS (...) ... FROM cte using unscoped, generic CTE identifiers. A user table named cte (or a query that already uses those identifiers in a larger composed SQL fragment) can conflict because CTE names shadow relations within the WITH statement. Recommend switching to uniquely prefixed internal CTE names (and quoting them via sql.ident) to avoid accidental name shadowing.
Comment on lines
+175
to
+187
| // FinalizedUpdate.bind() can't be reused: the live wrap needs `FROM "before" | ||
| // WHERE foos.ctid = "before".ctid` instead of the user's WHERE inline. | ||
| const beforeCte = sql`SELECT *, ctid FROM ${sql.ident(tableName)} AS ${alias} WHERE ${where.toSql()} FOR UPDATE`; | ||
| const ctidJoin = sql`${sql.column(alias, sql.ident("ctid"))} = ${sql.ident("before")}.${sql.ident("ctid")}`; | ||
| const updateCte = sql`UPDATE ${sql.ident(tableName)} AS ${alias} SET ${sql.join(compileSetClauses(instance, setRow))} FROM ${sql.ident("before")} WHERE ${ctidJoin} RETURNING ${compileSelectList(returning)}`; | ||
|
|
||
| const events = eventsInsertCte(tableName, sql.ident("__live_before"), sql.ident("__live_after")); | ||
| const projection = userReturningProjection(returning ?? {}, "__live_before", "__live_after"); | ||
|
|
||
| return sql.withScope( | ||
| [alias], | ||
| sql`WITH ${sql.ident("before")} AS (${beforeCte}), cte AS (${updateCte}), _events AS (${events}) SELECT ${projection} FROM cte`, | ||
| ); |
There was a problem hiding this comment.
The UPDATE live-events wrapper hard-codes a CTE name "before" and also uses generic CTE names like "cte"/"_events". If the target table is named "before" (or another chosen CTE name), Postgres will resolve references inside the CTE body against the CTE name instead of the base table and error (non-recursive CTE self-reference) or behave ambiguously. Consider using a Typegres-prefixed, collision-resistant CTE name (e.g. "__typegres_before") or generating a fresh Alias/Scope-registered name for these internal CTEs so user table names can’t conflict.
ryanrasti
force-pushed
the
ryan_live
branch
2 times, most recently
from
990169e to
0e20519
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Typegres sits between the client and pg, so mutations and subscribers are both in-process -- no triggers, WAL, or external CDC needed.
This means we see all mutations and have a direct line back to client -- the perfect place for a reactive setup.
How it works:
col = literal(anchor) orcol = other.col(edge).users.id=1flows tonotes.user_ideven when no matching notes exist yet).transformer: TypegresLiveEvents.makeTransformer()is wrapped in a CTE that atomically appends to_typegres_live_events.