Skip to content

fix: GetWorkerActionsByWorkerActionHash query optimization - #4138

Merged
grutt merged 2 commits into
mainfrom
fix--remove-distinct-from-worker-action-hash
Jul 15, 2026
Merged

fix: GetWorkerActionsByWorkerActionHash query optimization#4138
grutt merged 2 commits into
mainfrom
fix--remove-distinct-from-worker-action-hash

Conversation

@grutt

@grutt grutt commented Jun 10, 2026

Copy link
Copy Markdown
Member

Description

DISTINCT in GetWorkerActionsByWorkerActionHash seems to cause a sequence scan on _ActionToWorker leading to high CPU on high worker churn systems.

Old query

Unique  (actual time=821.6..832.2 rows=157)
  Buffers: shared hit=91,263
  -> Gather Merge (2 parallel workers)
     -> Sort -> HashAggregate (DISTINCT collapses 1.9M rows -> 157)
        -> Hash Join (aw."A" = a.id)  rows=631,923 x 3 loops = ~1.9M
           -> Parallel Hash Join (aw."B" = w.id)
              -> Parallel Seq Scan on "_ActionToWorker"   <- reads ENTIRE 5.7M-row table
                    Buffers: 78,900
              -> Parallel Hash
                 -> Parallel Seq Scan on "Worker"          <- reads ENTIRE Worker table
                       Filter: tenantId + actionHash = ANY(...)
                       rows=14,226 x 3 = ~42k workers matched
                       Buffers: 11,832
           -> Index Scan on "Action"
Execution Time: 832.324 ms

New query

Nested Loop  (actual time=0.10..5.0 rows=157)
  Buffers: shared hit=512
  -> Nested Loop
     -> Function Scan on unnest  (rows=5)                  <- the 5 input hashes
     -> Index Scan on "_ActionToWorker_B_index"            <- ~31 rows per representative
           Index Cond: ("B" = (SubPlan 1))
           SubPlan 1
             -> Limit (rows=0.80 per loop)                 <- stops at FIRST match
                -> Index Scan on "Worker_tenantId_actionHash_idx"
                      (planner expected 10,663 rows available; LIMIT reads 1)
  -> Memoize -> Index Scan on "Action_id_key"              <- cached action lookups
Execution Time: 5.097 ms

Type of change

  • Bug fix (non-breaking change which fixes an issue)

What's Changed

  • Improved

Checklist

Changes have been:

  • Tested (unit, integration, or manually with steps specified)
  • Linted and formatted
  • Documented (where applicable)
  • Added to CHANGELOG (where applicable) -- see Keep a Changelog

🤖 AI Disclosure
  • I acknowledge that an LLM was used in the creation of this Pull Request, in accordance with Hatchet's AI_POLICY.md.
  • Details: [e.g. generating tests, writing docs]

Copilot AI review requested due to automatic review settings June 10, 2026 11:44
@vercel

vercel Bot commented Jun 10, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hatchet-docs Ready Ready Preview, Comment Jul 15, 2026 12:08am

Request Review

@github-actions github-actions Bot added the engine Related to the core Hatchet engine label Jun 10, 2026
@grutt
grutt requested a review from abelanger5 June 10, 2026 11:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Optimizes the GetWorkerActionsByWorkerActionHash SQLC query to avoid expensive scans on _ActionToWorker in high worker-churn environments by selecting a single “representative” worker per action hash and joining from that worker to actions.

Changes:

  • Replaced SELECT DISTINCT ... FROM Worker ... JOIN _ActionToWorker ... with a representative-worker approach using unnest(actionHashes) + indexed lookup of one worker ID per hash.
  • Added an explanatory note documenting the correctness assumption (same actionHash ⇒ same action set) and the performance motivation.

Reviewed changes

Copilot reviewed 1 out of 2 changed files in this pull request and generated no comments.

File Description
pkg/repository/sqlcv1/workers.sql Rewrites GetWorkerActionsByWorkerActionHash to use a representative worker per hash (avoids _ActionToWorker seq scans).
pkg/repository/sqlcv1/workers.sql.go Regenerated SQLC output reflecting the updated query and emitted note.
Files not reviewed (1)
  • pkg/repository/sqlcv1/workers.sql.go: Language not supported

@mrkaye97 mrkaye97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice idea!

@grutt
grutt merged commit 7916f4f into main Jul 15, 2026
65 of 66 checks passed
@grutt
grutt deleted the fix--remove-distinct-from-worker-action-hash branch July 15, 2026 12:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

engine Related to the core Hatchet engine

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants