perf(reingest): batch dataset loading and remove N+1 queries

[lewisjared]

Description

ref executions reingest reconstructs each completed execution's ExecutionDefinition from the database before re-running build_execution_result(). Reconstruction walked execution.datasets and then dataset.files lazily, and _extract_dataset_attributes triggered a per-dataset load of the polymorphic subclass table, so a bulk reingest issued O(M × D) queries (M executions, D datasets each). The dataset links were also copied with one INSERT per row, and the eligibility scan lazy-loaded each group's full executions collection just to read [0].

This makes reingest scale with the number of executions rather than the number of files, with no behaviour change:

• reconstruct_execution_definition now loads the linked datasets in a single query using with_polymorphic(Dataset, "*") (eager-loads the polymorphic subclass columns) plus selectinload(.files), so neither the for file in dataset.files loop nor _extract_dataset_attributes issues a per-dataset query.
• The dataset-link copy is now a single batched session.execute(insert(), rows), mirroring Execution.register_datasets.
• get_executions_for_reingest warms the ExecutionGroup.executions identity map once before the loop instead of lazy-loading each group's collection.

On a synthetic execution with 8 datasets and 3 files each, reconstruction drops from 18 SELECTs to a constant 3, independent of dataset count. The reconstructed definition, the set of (execution_id, dataset_id) links, and the reingest failure-handling path are all unchanged. A new regression test asserts the reconstruction SELECT count is constant in the dataset count (it grows, and fails, if the eager load is reverted).

Checklist

Please confirm that this pull request has done the following:

• Tests added
• Documentation added (where applicable)
• Changelog item added to changelog/

Summary by CodeRabbit

Performance Improvements

• Reference execution reingestion is now substantially faster through operational optimisations. Bulk reingests now scale more efficiently based on the number of executions rather than the total number of data files, delivering notably improved performance for users conducting large-scale bulk reingestion operations and workflows without any degradation in quality or reliability.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: e3d6ba08-fdf4-44d5-8b9c-4f23208f468b

📥 Commits

Reviewing files that changed from the base of the PR and between 522a43a and 69860ae.

📒 Files selected for processing (3)

• changelog/750.improvement.md
• packages/climate-ref/src/climate_ref/executor/reingest.py
• packages/climate-ref/tests/unit/executor/test_reingest.py

---

📝 Walkthrough

Walkthrough

reingest.py is updated to eliminate N+1 query patterns by eagerly loading datasets and files via a single polymorphic SQLAlchemy query in reconstruct_execution_definition, bulk-inserting execution_datasets link rows, and preloading ExecutionGroup.executions with selectinload. A query-count regression test is added to verify constant SELECT behaviour.

Changes

Reingest N+1 Query Optimisation

Layer / File(s)	Summary
Session-bound polymorphic eager-loading in reconstruct_execution_definition packages/climate-ref/src/climate_ref/executor/reingest.py	Adds object_session, selectinload, and with_polymorphic imports; removes the conditional Dataset import; rewrites reconstruct_execution_definition to fetch datasets and their files in a single polymorphic query via the session rather than iterating execution.datasets.
Bulk dataset-link insert, execution-group warm-up, and rollback cleanup packages/climate-ref/src/climate_ref/executor/reingest.py, changelog/750.improvement.md	Replaces per-row execution_datasets inserts with a single bulk insert; adds a selectinload(ExecutionGroup.executions) prefetch to prevent per-group SELECTs; ensures rollback cleanup removes both scratch and results directories; records the improvement in the changelog.
Query-count regression test packages/climate-ref/tests/unit/executor/test_reingest.py	Adds _count_selects (SQLAlchemy before_cursor_execute listener) and _seed_execution_with_datasets (seeds CMIP6Dataset + DatasetFile rows); adds test_query_count_is_constant_in_dataset_count to assert SELECT count is equal for 1 vs 8 datasets and bounded to a small non-zero constant.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately and specifically summarises the primary changes: performance improvements to reingest via dataset batching and N+1 query removal.
Description check	✅ Passed	The description fully addresses all required template sections and provides comprehensive explanation of changes, rationale, and testing approach.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch perf/batch-reingest

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

Flag	Coverage Δ
core	92.68% <100.00%> (+<0.01%)	⬆️
providers	91.82% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...s/climate-ref/src/climate_ref/executor/reingest.py	97.26% <100.00%> (+0.17%)	⬆️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.