UN-3883 [FEAT] Cut dashboard metrics cron DB load: narrower source windows, monthly from daily, two new indexes

[kirtimanmishrazipstack]

What

Three changes to the dashboard metrics cron, already reviewed and merged individually as #2255, #2264 and #2265:

• Monthly figures are now added up from the daily figures instead of being recalculated from the raw tables every time.
• Each run looks back 2 days instead of ~2 months. A separate pass at 04:40 UTC looks back 7 days once a day, so a short outage still repairs itself.
• The expensive half of the work runs hourly instead of every 15 minutes. Hourly figures keep their 15-minute cadence; daily and monthly move to hourly at :20.
• Two new database indexes, on the two tables the cron reads most.

Why

The cron was using roughly 55 minutes of database time every 6 hours on production. It re-read between 32 and 62 days of raw data for all 38 organisations, 96 times a day, to produce figures that only change once a month.

Nothing a customer sees changes. Hourly dashboard numbers are still at most 15 minutes old; daily and monthly numbers are now at most an hour old instead of 15 minutes, which is the one deliberate trade.

How

• _rollup_monthly_from_daily replaces the per-org monthly source queries with one INSERT ... ON CONFLICT DO UPDATE over event_metrics_daily for every org. Upsert-only, per the design agreed on UN-3973: a monthly row the daily tier no longer produces is left in place, because a stale total is recoverable with backfill_metrics and a deleted one is not.
• aggregate_metrics_from_sources(tier, source_window_days) is called by three schedule rows. Both kwargs travel on both transports — Beat calls the Django task directly, the PG scheduler goes worker proxy → internal endpoint → the same function.
• Lock keys are one per granularity written, namespaced by source window. ALL takes both, so it genuinely excludes a concurrent hourly run; distinct windows are distinct jobs, so the once-daily reconciliation pass can never be starved by the 15-minute schedule it is never retried after.
• Both indexes are built CONCURRENTLY under atomic = False, with AddIndex confined to state_operations. Each migration asserts the index is valid and has the expected definition before recording itself applied.

Can this PR break any existing features?

The realistic risks, and what bounds each:

• Monthly figures go stale if the daily tier has a gap. Monthly is now only as good as event_metrics_daily. Gaps shorter than 7 days repair themselves on the next reconciliation pass; anything older needs backfill_metrics. Bounded to the current and previous month.
• Rolling the code back past this release breaks aggregation until migration 0006 is reversed — the schedule rows carry a tier kwarg the previous release's signature rejects. Documented in the migration docstring; migrate dashboard_metrics 0005 restores it.
• A pod still on the old image during the rollout receives tier and raises TypeError until it rolls. Self-healing, and no aggregation is lost because the next tick succeeds.
• Both index builds are non-blocking, and both migrations are reversible.

Database Migrations

Four, in three apps. No schema changes to any metrics table.

Migration	What it does
dashboard_metrics/0005_add_reconciliation_task	Adds the 04:40 UTC reconciliation schedule, on both Beat and the PG scheduler
dashboard_metrics/0006_split_aggregation_schedule	Splits the aggregation into two rows by tier
file_execution/0007_wfe_status_created_idx	Index on workflow_file_execution (status, created_at)
workflow_v2/0029_we_created_at_idx	Index on workflow_execution (created_at)

Both index migrations no-op via IF NOT EXISTS if the index was built out of band first, which is the preferred production path — the exact statement is in each migration's docstring.

Env Config

None.

Deploy Steps

Run once, before the first aggregation after deploy:

python manage.py backfill_metrics --days 60 --skip-hourly --skip-monthly

Monthly is now derived from the daily tier, so that tier has to be complete across the rollup window first. --skip-monthly is deliberate: repair daily and let the rollup derive monthly.

Relevant Docs

backend/dashboard_metrics/README.md is updated — schedules, windows, staleness bounds, and the ownership overlap with backfill_metrics.

Related Issues or PRs

Merged into this branch: #2255 (UN-3973), #2264 (UN-3972), #2265 (UN-3974). Parent: UN-3883.

Dependencies Versions

No dependency changes.

Notes on Testing

Ticket	Acceptance criterion	State
UN-3973	Monthly derived from event_metrics_daily, not source tables	Met
UN-3973	Per-run daily source window is 2 days	Met
UN-3973	A once-daily 7-day reconciliation pass exists and is scheduled	Met
UN-3973	Daily and monthly match pre-change values, incl. across a month boundary	Met
UN-3973	Tests cover the month boundary and the reconciliation pass	Met
UN-3972	Index present, indisvalid = t	Met
UN-3972	Non-atomic + CONCURRENTLY, no write-blocking lock	Met
UN-3972	get_documents_processed free of a seq scan on workflow_file_execution	Confirm on prod
UN-3972	get_failed_pages free of a seq scan on workflow_execution	Met
UN-3972	get_recent_activity under 1 s	Flagged — belongs to the (created_at DESC) index descoped in comment 45015
UN-3974	Hourly stays 15 min, daily/monthly go hourly	Met
UN-3974	Hourly figures unchanged, daily/monthly lag ≤ 1 h	Met
UN-3974	Prefilter leaves the Query Insights top 10	Confirm on prod

The two Confirm on prod rows are post-deploy readings against Query Insights, not outstanding work.

Screenshots

Not applicable — no UI change.

Checklist

I have read and understood the Contribution Guidelines.

[greptile-apps]

Greptile Summary

The PR reduces dashboard aggregation load by splitting tier schedules, narrowing routine source windows, rolling monthly metrics up from daily data, and adding supporting database indexes.

• Keeps hourly aggregation on a 15-minute schedule while moving daily and monthly aggregation to hourly execution.
• Adds a daily seven-day reconciliation pass across both scheduler transports.
• Builds monthly metrics from persisted daily aggregates.
• Adds concurrent indexes for workflow and file-execution metric queries.

Confidence Score: 4/5

The PR does not appear safe to merge until the outstanding active-organization prefilter defect is fixed, because valid recent source activity can still be omitted from aggregation.

The current prefilter selects organizations only through recent WorkflowExecution creation, while pages, LLM usage, and HITL metrics can have recent activity in independent source tables; affected organizations are skipped before those queries execute, leaving dashboard values absent or stale.

Files Needing Attention: backend/dashboard_metrics/tasks.py

Important Files Changed

Filename	Overview
backend/dashboard_metrics/tasks.py	Splits aggregation by tier, narrows source windows, introduces per-tier locking and active-organization filtering, and derives monthly metrics from daily rows.
backend/dashboard_metrics/migrations/0005_add_reconciliation_task.py	Adds the daily seven-day reconciliation schedule for both Celery Beat and PostgreSQL scheduling.
backend/dashboard_metrics/migrations/0006_split_aggregation_schedule.py	Splits the existing aggregation schedule into frequent hourly-tier and hourly daily/monthly-tier jobs while preserving scheduler ownership.
backend/dashboard_metrics/internal_views.py	Validates and forwards aggregation tier and source-window arguments through the internal task endpoint.
workers/scheduler/dashboard_metrics_tasks.py	Carries scheduler task arguments through the worker proxy to the backend aggregation endpoint.
backend/workflow_manager/file_execution/migrations/0007_wfe_status_created_idx.py	Adds the concurrent status-and-created-at index used by file-execution metric queries.
backend/workflow_manager/workflow_v2/migrations/0029_we_created_at_idx.py	Adds the concurrent created-at index intended to support active-organization filtering.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  S[Source metric tables] -->|Every 15 minutes| H[Hourly aggregation]
  S -->|Hourly at :20| D[Daily aggregation]
  S -->|Daily 7-day reconciliation| R[Hourly and daily repair]
  D --> M[Monthly rollup from daily]
  H --> API[Dashboard APIs]
  D --> API
  M --> API

Loading

_{Reviews (4): Last reviewed commit: "UN-3883 [FIX] Pin the second lock suite ..." | Re-trigger Greptile}

[greptile-apps]

Prefilter drops valid metric activity

When an organization has recent PageUsage, Usage, or HITLQueue activity but no WorkflowExecution created within the lookback, _active_org_ids excludes it from every metric query, causing its pages-processed, LLM, or HITL dashboard rows to remain stale or absent even after reconciliation.

Knowledge Base Used: Restore dashboard metrics and navigation

Prompt To Fix With AI

This is a comment left during a code review.
Path: backend/dashboard_metrics/tasks.py
Line: 631-634

Comment:
**Prefilter drops valid metric activity**

When an organization has recent `PageUsage`, `Usage`, or `HITLQueue` activity but no `WorkflowExecution` created within the lookback, `_active_org_ids` excludes it from every metric query, causing its pages-processed, LLM, or HITL dashboard rows to remain stale or absent even after reconciliation.

**Knowledge Base Used:** [Restore dashboard metrics and navigation](https://app.greptile.com/zipstack/-/custom-context/knowledge-base/zipstack/unstract/-/reverts/incident-mitigation_1813-20260302-dashboard-metrics-sidebar-440ae49.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[kirtimanmishrazipstack]

@greptile-apps The finding is real, and the concrete case is Prompt Studio: an LLM call there writes a Usage row without ever creating a WorkflowExecution, so an org doing only Prompt Studio work in the lookback is excluded from llm_calls, challenges, summarization_calls, llm_usage, pages_processed and both HITL metrics.

It is pre-existing rather than introduced here, and this PR does not widen or narrow it.

At the merge base the prefilter is the same single table:

# origin/main, dashboard_metrics/tasks.py
active_org_ids = set(
    WorkflowExecution.objects.filter(created_at__gte=daily_start)
    .values_list("workflow__organization_id", flat=True).distinct()
)

daily_start there was 7 days. This PR narrows the metric window to 2 days, which would have dragged the prefilter down with it — so _active_org_ids takes min(window_start, end_date - DASHBOARD_ACTIVE_ORG_LOOKBACK_DAYS) to floor it. Measured on the current head:

source_window_days= 2  ->  cutoff 7 days back   (floored at LOOKBACK)
source_window_days= 7  ->  cutoff 7 days back   (widened by the caller)
source_window_days=60  ->  cutoff 60 days back  (widened by the caller)

So the org set is identical to before the change, and widens with the reconciliation pass rather than being pinned.

Not fixing it in this PR, for two reasons. Widening the prefilter means unioning Usage, PageUsage and HITLQueue — three more queries per run on the hot path, on a change whose entire purpose is removing DB load from this cron (UN-3883: ~55 min of DB time per 6 h). And UN-4045 deletes the prefilter outright by grouping every metric query by organization, at which point the org set comes from the metric queries themselves and the gap closes without a union.

Tracking it there rather than papering over it here.

[greptile-apps]

Agreed. This is a real limitation, but it is pre-existing and not introduced by this PR. The min(...) floor preserves the previous 7-day prefilter for the normal 2-day run and correctly widens it for reconciliation, so this change does not regress the affected organizations. Given the hot-path load objective and the planned UN-4045 fix that removes the prefilter, I would not block this PR on the finding; it should be tracked as follow-up work there.

[sonarqubecloud]

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.7% Duplication on New Code

See analysis details on SonarQube Cloud

[github-actions]

Unstract test results

Per-group results

Status	Group	Tier	Passed	Failed	Errors	Skipped	Duration (s)
✅	e2e-api-deployment	e2e	3	0	0	0	16.0
✅	e2e-coowners	e2e	1	0	0	0	1.1
✅	e2e-etl	e2e	1	0	0	0	11.8
✅	e2e-login	e2e	2	0	0	0	0.9
✅	e2e-prompt-studio	e2e	1	0	0	0	9.2
✅	e2e-smoke	e2e	2	0	0	0	0.8
✅	e2e-workflow	e2e	1	0	0	0	19.8
❌	frontend	unit	0	1	0	0	0.0
✅	integration-backend	integration	358	0	0	26	47.8
✅	integration-connectors	integration	1	0	0	7	7.9
✅	integration-workers	integration	157	0	0	1	51.5
❌	ui	e2e	0	1	0	0	0.0
✅	unit-backend	unit	1194	0	0	1	43.0
✅	unit-connectors	unit	63	0	0	0	10.2
✅	unit-core	unit	33	0	0	0	1.4
✅	unit-platform-service	unit	15	0	0	0	2.7
✅	unit-rig	unit	120	0	0	0	4.8
✅	unit-runner	unit	5	0	0	0	3.0
✅	unit-sdk1	unit	563	0	0	0	29.1
✅	unit-workers	unit	1376	0	0	1	126.2
	TOTAL		3896	2	0	36	387.4

Critical paths

⚠️ Critical paths not yet covered

• workflow-execution-fan-out — Multi-file workflow execution fans out to file-processing workers and rejoins. (declared coverage: no groups declared)

✅ Covered critical paths

• auth-login — covered by e2e-login
• adapter-register-llm — covered by integration-backend
• workflow-author — covered by integration-backend
• co-owner-manage — covered by integration-backend, e2e-coowners
• workflow-create-execute — covered by e2e-workflow
• api-deployment-provision — covered by integration-backend
• api-deployment-auth — covered by integration-backend
• api-deployment-run — covered by e2e-api-deployment
• mcp-server-auth — covered by integration-backend
• mcp-platform-auth — covered by integration-backend
• prompt-studio-author — covered by integration-backend
• prompt-studio-fetch-response — covered by e2e-prompt-studio
• connector-register-test — covered by integration-backend
• pipeline-etl-execute — covered by e2e-etl
• usage-aggregate-read — covered by integration-backend
• usage-token-tracking — covered by e2e-api-deployment
• callback-result-delivery — covered by e2e-api-deployment