fix: update trigger for runs status and update schema for consistency - #4426
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
| parent_task_external_id UUID, | ||
|
|
||
| PRIMARY KEY (inserted_at, id, readable_status, kind) | ||
| PRIMARY KEY (inserted_at, id) |
There was a problem hiding this comment.
drive-by: cleanup
There was a problem hiding this comment.
Pull request overview
This PR optimizes the v1_runs_olap_status_update_function trigger path by switching from an UPDATE ... FROM to an INSERT ... ON CONFLICT DO UPDATE, aiming to avoid TimescaleDB hypertable-wide scans and reduce contention/latency. It also updates the OLAP schema file to stay consistent with the current table keying.
Changes:
- Rewrites
v1_runs_olap_status_update_functionto use an upsert intov1_statuses_olap(instead ofUPDATE ... FROM). - Adds a new migration to apply the trigger-function rewrite to existing databases.
- Updates
sql/schema/v1-olap.sqlto use the(inserted_at, id)primary key forv1_runs_olapfor schema consistency.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| sql/schema/v1-olap.sql | Updates v1_runs_olap primary key for consistency and rewrites the status-update trigger function to an upsert. |
| cmd/hatchet-migrate/migrate/migrations/20260714120000_v1_0_126.sql | Migration to apply the upsert-based trigger function change (with a down migration restoring the prior update behavior). |
| SELECT | ||
| external_id, | ||
| inserted_at, | ||
| tenant_id, | ||
| workflow_id, | ||
| kind, | ||
| readable_status | ||
| FROM new_rows | ||
| ON CONFLICT (external_id, inserted_at) DO UPDATE |
| SELECT | ||
| external_id, | ||
| inserted_at, | ||
| tenant_id, | ||
| workflow_id, | ||
| kind, | ||
| readable_status | ||
| FROM new_rows | ||
| ON CONFLICT (external_id, inserted_at) DO UPDATE |
|
|
Description
Updates the trigger on
v1_runs_statusso that it updates the correspondingv1_statuses_olaptable using anINSERT INTO...ON CONFLICTinstead of anUPDATE.Note the following p99 reduction in latency (milliseconds) on
UpdateTaskStatusesFromMQafter this fix was applied:This particular database runs TimescaleDB and
v1_statuses_olapis a hypertable. When running query plans simulating this trigger, we saw that every chunk of the hypertable was being index scanned when calling theUPDATE. This seems internal to Timescale's partitioning implementation; we don't see the same thing on updates of our partitioned tables.This was ultimately the cause of extremely heavy
LWLock:BufferContent(calledlwlock:buffer_contentin some systems) due to the index scan touching every partition across 30 days of retention.Type of change
What's Changed
v1_runs_olap_status_update_functionto useINSERT INTO...ON CONFLICTinstead ofUPDATEChecklist
Changes have been:
🤖 AI Disclosure