UN-3972 [DEV] Cut dashboard cron DB time by indexing workflow_file_execution on (status, created_at) - #2264
Open
kirtimanmishrazipstack wants to merge 5 commits into
Conversation
The dashboard metrics cron's documents_processed and failed_pages queries filter this table on status + a created_at window, but all four existing indexes lead with workflow_execution_id. With no entry point here the planner drives top-down from the org and sequentially scans all 1.28M rows of workflow_execution — 83% of the cron's DB time on production. Built CONCURRENTLY with atomic = False; a plain AddIndex would hold a SHARE lock over a 3.4GB table taking live inserts. Guarded against a leftover INVALID index from an interrupted build, which IF NOT EXISTS would otherwise keep silently. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KGZBF68CShem3pbUJM2tBc
Contributor
|
| Filename | Overview |
|---|---|
| backend/workflow_manager/file_execution/migrations/0007_wfe_status_created_idx.py | Adds an independently ordered, non-atomic migration that creates and drops the composite index concurrently and rejects an existing invalid index. |
| backend/workflow_manager/file_execution/models.py | Adds the matching (status, created_at) index declaration to Django model state. |
| backend/workflow_manager/file_execution/tests/test_wfe_status_created_idx.py | Verifies concurrent migration SQL, non-atomic execution, invalid-index handling, and agreement between migration and model state. |
Reviews (7): Last reviewed commit: "Merge remote-tracking branch 'origin/UN-..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
kirtimanmishrazipstack
changed the base branch from
UN-3883-Optimize-DB-cron-queries-causing-high-DB-load
to
main
August 31, 2026 14:55
The docstring restated the prod plan, deployment runbook and recovery steps. That detail belongs in the PR, not in a file every future agent scans. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KGZBF68CShem3pbUJM2tBc
Contributor
Unstract test resultsPer-group results
Critical paths
|
Contributor
Author
|
@greptile-apps please re-review. The Confidence Score block above is stale — it still cites For a human reader: this PR is |
kirtimanmishrazipstack
changed the base branch from
main
to
UN-3883-Optimize-DB-cron-queries-causing-high-DB-load
August 31, 2026 15:37
…into UN-3972-index-file-execution-status-created-at
…ape with tests The suite runs with --no-migrations, so 0007 is never executed in CI. Regenerating it with makemigrations, or dropping atomic = False / CONCURRENTLY while tidying, would land a plain AddIndex — a SHARE lock held for the whole build on a 3.4 GB table that takes live inserts — with every test still green. Five DB-free assertions on the migration module and the model's Meta.indexes: non-atomic, concurrent in both directions, the INVALID-index guard present, AddIndex confined to state_operations, and model/migration agreement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KGZBF68CShem3pbUJM2tBc
…-causing-high-DB-load' into UN-3972-index-file-execution-status-created-at
|
Contributor
Author
|
@greptile-apps migration order is revised. Review again. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
One database index on processed files, covering status and date together.
The second index the ticket also asks for is not here — dropped in comment 45015.
Why
The dashboard cron costs ~3,580 s of database time per 6 h on prod. Two queries are 83% of
that, and both look up processed files by date and status.
Nothing organises the data that way, so the database reads all 1,278,885 execution rows on every
call. This index lets it start from the files in the date range instead.
How
SeparateDatabaseAndState:RunSQLbuilds the indexCONCURRENTLY IF NOT EXISTS, a state-onlyAddIndexkeeps Django's model state in step, andatomic = FalsebecauseCONCURRENTLYcannotrun in a transaction. A
RAISE EXCEPTIONguard fails loudly on a leftover INVALID index, whichIF NOT EXISTSwould otherwise keep while Django recorded the migration as applied.Can this PR break any existing features. If yes, please list possible items. If no, please explain why.
No. Nothing but an index is added — no behaviour changes — and it is built without blocking
writes. The cost is slightly slower writes on that table, against 1,679 ms of query time saved
per 6 h.
Database Migrations
file_execution/0007_wfe_status_created_idx.py— builds the index without locking writes, andreverses cleanly. Optionally build it by hand first; the deploy then skips it:
Migration Order
The three UN-3883 PRs stack on the same integration branch. Merge in this order.
dashboard_metrics/0005_add_reconciliation_task0004_pg_periodic_tasks(UN-3445, already onmain)file_execution/0007_wfe_status_created_idxfile_execution/0006_…dashboard_metrics/0006_split_aggregation_schedule0005_add_reconciliation_task(#2255)#2264's migration is in a different app and has no interaction with the other two. The one
hard dependency is #2265 on #2255 — merged out of order, #2265 fails at graph build:
Its tests are unaffected — the backend suite runs with
--no-migrations.Verified on a throwaway Postgres:
0004→0005→0006applies from an empty database,reverses, and re-applies, with
makemigrations --checkclean at each step.Env Config
None.
Relevant Docs
UN-3883 analysis §6.4.
Related Issues or PRs
Parent UN-3883. #2255 (UN-3973) — AC-3 depends on it. #2265 (UN-3974). See Migration Order above.
Dependencies Versions
None.
Notes on Testing
Nine scenarios and five unit tests. Detail in Jira comment 45381.
indisvalid = tCONCURRENTLY, no write-blocking lockget_documents_processedfree of a seq scan onworkflow_file_executionget_failed_pagesfree of a seq scan onworkflow_executionget_recent_activityunder 1 sAfter deploy: confirm the index is valid, then re-pull the 6-hour Query Insights window against
the 870 / 809 ms baseline.
Screenshots
n/a — schema-only change.
Checklist
I have read and understood the Contribution Guidelines.