Skip to content

workflow run list times out on some organizations #3128

@migmartri

Description

@migmartri

Problem

Listing workflow runs (/workflow-runs) times out for some organizations. The slow query is the org-scoped workflow_runs list:

SELECT workflow_runs.* FROM workflow_runs
WHERE EXISTS (
  SELECT workflows.id FROM workflows
  WHERE workflow_runs.workflow_id = workflows.id
    AND workflows.deleted_at IS NULL
    AND workflows.organization_id = $1
)
ORDER BY workflow_runs.created_at DESC
LIMIT 26;

Observed runtime: ~10s (request times out) on affected organizations. Other orgs are fast.

Root cause

workflow_runs has no organization_id column, so org scoping has to traverse the workflows edge. With the correlated EXISTS form combined with ORDER BY created_at DESC LIMIT 26, the planner has two main options:

  1. Backward index scan on workflow_runs.created_at, checking the EXISTS subquery for every row.
  2. Nested loop through workflow IDs of the org, then merge by created_at.

For organizations with sparse recent activity but many workflow_runs rows from other orgs in front of theirs by created_at, the planner picks option 1 and walks most of the table before finding 26 matches. Hence the timeout.

A prior fix (#1635) addressed a related shape by reordering composite indexes. That fix only helps when the planner picks the workflow-driven plan; the current EXISTS shape lets it regress to the backward index scan.

Fix direction

Denormalize organization_id onto workflow_runs and add an (organization_id, created_at DESC) index so the org-scoped list becomes a single sargable range scan, independent of how many workflows the org has.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions