WIP: perf(waterdata): Optional parallel chunk processing in multi_value_chunked (draft)

[thodson-usgs]

Draft on top of #276. Adds an opt-in max_workers kwarg to @multi_value_chunked; when set > 1, sub-requests run concurrently in a ThreadPoolExecutor. Default (None) keeps the existing sequential behavior, including the mid-call quota guard and QuotaExhausted.partial_frame resumability.

Opening draft to capture the benchmark and the truncation issue surfaced under load; not ready to merge.

Benchmark — 671 CAMELS sites × get_field_measurements, 337,808 rows, ~14 paginated chunks

config	mean ± stdev	notes
sync	82.1s ± 12.1s	reference
workers=2	100.2s	unstable; one run truncated by 429
workers=4	51.5s ± 2.8s	~1.6x; one run lightly truncated
workers=8	crashed	429 mid-flight, unusable

The ceiling is the per-IP rate budget USGS enforces, not local CPU. Parallelism helps modestly on pagination-bound workloads and not at all on the common metadata case (already <1.5s).

Tradeoffs documented in the docstring

Parallel mode forfeits the mid-call quota guard (workers race past the floor before any one observes the crossing) and QuotaExhausted.partial_frame resumability (executor.map discards completed-but-uncollected results when any chunk raises). Sequential remains the default.

Open issue blocking merge: silent truncation under parallel load

Two of the parallel runs above returned fewer rows than the sync baseline (257,634 and 330,174 vs 337,808) with no exception raised. _walk_pages returns whatever rows it accumulated when a paginated request hits 429 mid-stream; the chunker then combines those partial frames into a full-looking result. This pre-existed serially but is much easier to trigger under parallelism.

Before this can merge we need:

• _walk_pages to surface mid-pagination 429s instead of returning partial rows silently, or
• The parallel branch to detect partial-pagination responses and raise.

I lean toward fixing _walk_pages — it's the same class of silent-truncation bug PR #273 fixed for the non-429 case.

Refs #276

[thodson-usgs]

Design note: partial-frame propagation from _walk_pages

#279 makes _walk_pages and get_stats_data raise on any mid-pagination failure instead of silently returning truncated data. That closes the silent-data-loss bug at the source, but it also means any mid-pagination 429 (or 5xx, or network error) hit by a chunk in this PR's parallel branch will lose all pages collected within that chunk, and the chunker's existing QuotaExhausted.partial_frame will only carry data from chunks that completed cleanly before the failure.

For most workloads that's fine — a chunk is at most a few thousand rows × a handful of pages, and the user reruns the failed chunk on retry. But for the heavy pagination cases this PR's benchmark targets (671 sites × get_field_measurements, ~24k rows per chunk paginating across 5-10 pages), losing a chunk's near-complete paginated walk on a final-page 429 is a non-trivial regression in recoverable work.

Suggested follow-up before this PR lands

Promote _walk_pages's plain RuntimeError to a richer exception that carries the partial state, then have the chunker translate it:

# utils.py
class PaginatedRequestError(RuntimeError):
    def __init__(self, msg, *, partial_frame, last_response):
        super().__init__(msg)
        self.partial_frame = partial_frame
        self.last_response = last_response

# in _walk_pages' except block:
raise PaginatedRequestError(
    _paginated_failure_message(len(dfs), e),
    partial_frame=pd.concat(dfs, ignore_index=True),
    last_response=initial_response,
) from e

In this PR's parallel wrapper, catch PaginatedRequestError per chunk and feed its partial_frame into the QuotaExhausted(partial_frame=...) payload — same shape the existing quota guard produces between chunks, so callers downstream have one place to look for partial results.

This is purely additive: existing except RuntimeError handlers still catch it (it inherits RuntimeError), and the message + __cause__ chain are unchanged. The .partial_frame attribute is just available for callers who want to use it.

This is the kind of "design when there's a concrete consumer" extension that #279 deliberately left out. This PR is that consumer.