Skip to content

fix: Add missing indices to PilotAgentsDB and JobDB#8463

Merged
fstagni merged 2 commits intoDIRACGrid:integrationfrom
chrisburr:fix/pilotdb-jobdb-index-optimization
Feb 23, 2026
Merged

fix: Add missing indices to PilotAgentsDB and JobDB#8463
fstagni merged 2 commits intoDIRACGrid:integrationfrom
chrisburr:fix/pilotdb-jobdb-index-optimization

Conversation

@chrisburr
Copy link
Member

@chrisburr chrisburr commented Feb 19, 2026

Two hot-path queries are doing full table scans due to missing indices:

  1. PilotAgentsDBSiteDirector.countPilots() filters on (DestinationSite, Queue, Status) every scheduling cycle. The existing Statuskey index is (GridSite, DestinationSite, Status) which doesn't cover Queue and has the wrong leading column for this query. A composite idx_dest_queue_status index is added.

  2. JobDBWorkflowTasks.updateTransformationReservedTasks() calls getJobs({"JobName": jobNames}), which does SELECT JobID FROM Jobs WHERE JobName IN (...). JobName had no index at all, causing a full table scan. Also hit by dstat.py and the REST API. A JobName index is added.

Both are pure schema changes to the .sql definition files. For existing deployments the schema change only takes effect on newly created tables (__initializeDB skips tables that already exist). Production DBs need a one-off ALTER TABLE to add the indices.

BEGINRELEASENOTES
*WorkloadManagement
FIX: Add composite (DestinationSite, Queue, Status) index to PilotAgents table for SiteDirector.countPilots()
FIX: Add JobName index to Jobs table for TransformationSystem and monitoring lookups

*Deployment
CHANGE: USE JobDB; ALTER TABLE Jobs ADD INDEX JobName (JobName), ALGORITHM=INPLACE, LOCK=NONE;
CHANGE: USE PilotAgentsDB; ALTER Table PilotAgents ADD INDEX idx_dest_queue_status (DestinationSite,Queue,Status), ALGORITHM=INPLACE, LOCK=NONE;

ENDRELEASENOTES

…Agents

SiteDirector.countPilots() filters on these three columns every
scheduling cycle. The existing Statuskey index has GridSite as the
leading column and does not cover Queue, resulting in a full table scan.
WorkflowTasks.updateTransformationReservedTasks() looks up jobs by
JobName via SELECT JobID FROM Jobs WHERE JobName IN (...). Without an
index this requires a full table scan. Also benefits dstat.py and
the REST API job-name lookups.
@chrisburr chrisburr force-pushed the fix/pilotdb-jobdb-index-optimization branch from 4196a30 to 0ad5ac6 Compare February 19, 2026 14:49
@fstagni fstagni merged commit c42c992 into DIRACGrid:integration Feb 23, 2026
23 checks passed
@DIRACGridBot DIRACGridBot added the sweep:ignore Prevent sweeping from being ran for this PR label Feb 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sweep:ignore Prevent sweeping from being ran for this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants