feat: parallel slice aggregation with n_jobs

[FBumann]

Summary

Add n_jobs parameter to aggregate() for parallel slice processing:

result = tsam_xarray.aggregate(
    da,
    time_dim="time",
    cluster_dim=["variable", "region"],
    n_clusters=8,
    n_jobs=-1,  # all CPUs
)

Implementation

Uses ThreadPoolExecutor — sklearn releases the GIL during C-level clustering, giving real parallelism with zero overhead (no process spawn, no pickling, no file IO).

Benchmarked: 2x speedup on 8760h × 50 vars × 5 scenarios.

We evaluated ProcessPoolExecutor (like tsam's tuning module) but the ~1.5s process spawn overhead never amortizes for our use case (5-50 slices at ~30-150ms each).

API

• None or 1 = sequential (default)
• -1 = all CPUs, -2 = all minus one (joblib convention)
• N = exactly N workers
• Ignored when there are no slice dims

Closes #12

Test plan

• Parallel matches sequential (values + assignments)
• n_jobs=None, n_jobs=-1, n_jobs ignored without slices
• 224 tests, mypy strict
• Benchmark script in scripts/benchmark_parallel.py
• CI green

🤖 Generated with Claude Code

Summary by CodeRabbit

Release Notes

• New Features

  • Aggregation now supports parallel execution with a configurable workers parameter, enabling faster processing of large datasets.

• Tests

  • Added comprehensive tests verifying parallel execution produces correct results matching sequential processing.

• Chores

  • Added benchmarking script for comparing sequential versus parallel aggregation performance across dataset sizes.

[coderabbitai]

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

This PR introduces parallel execution support to the aggregate() function via an optional n_jobs parameter, enabling concurrent processing of independent slice dimensions using ThreadPoolExecutor. A new benchmark script and comprehensive test suite validate the parallel behavior across various configurations.

Changes

Cohort / File(s)	Summary
Parallelization Core src/tsam_xarray/_core.py	Added n_jobs parameter to aggregate() function; introduced _get_n_workers() helper to convert n_jobs values to worker counts, and _run_slice() helper to execute per-slice aggregation; slice processing now dispatches to ThreadPoolExecutor when worker count > 1.
Parallel Execution Tests test/test_aggregate.py	New TestParallel test class verifying that parallel aggregation (n_jobs=2) produces results matching sequential execution; covers n_jobs=None, n_jobs=-1, and n_jobs with non-slice dimensions.
Benchmarking Utilities scripts/benchmark_parallel.py	New script with make_data() to generate synthetic time-series data and benchmark() to compare sequential vs parallel aggregation performance across multiple dataset scales.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant aggregate as aggregate()
    participant helper as _get_n_workers()
    participant executor as ThreadPoolExecutor
    participant slice as _run_slice()
    participant single as _aggregate_single()

    User->>aggregate: call with n_jobs=-1
    aggregate->>helper: _get_n_workers(n_jobs=-1)
    helper-->>aggregate: worker_count=4 (example)
    alt worker_count <= 1
        aggregate->>single: sequential execution
    else worker_count > 1
        aggregate->>executor: create ThreadPoolExecutor
        executor->>slice: map _run_slice over slices
        slice->>single: _aggregate_single(slice_key)
        single-->>slice: aggregated result
        slice-->>executor: return result
        executor-->>aggregate: all results
    end
    aggregate-->>User: final aggregated DataArray

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐰 In parallel paths they run so fast,
No sequencing needed, threads amassed!
Each slice soars free, no locks to bind,
Speedup comes swift to those who grind—
n_jobs=-1 makes workflows align! ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 72.73% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'feat: parallel slice aggregation with n_jobs' directly and clearly describes the main change: adding parallel slice aggregation via the n_jobs parameter.
Linked Issues check	✅ Passed	The PR implementation meets all core coding requirements from issue #12: adds n_jobs parameter with correct semantics (None/1 sequential, negative for all CPUs), uses ThreadPoolExecutor to avoid pickling, correctly ignores n_jobs without slice dimensions, and includes comprehensive tests validating parallel and sequential result equivalence.
Out of Scope Changes check	✅ Passed	All changes are directly related to implementing parallel slice aggregation: the new n_jobs parameter in aggregate(), helper functions for worker management, parallel execution logic, a benchmarking script for testing parallelization, and comprehensive test coverage for the feature.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/parallel-slices

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines	Coverage Δ
src/tsam_xarray/_core.py	98.37% <100.00%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[FBumann]

Wontfix, as minimal value.