Skip to content

feat(ogc): add opt-in parallel chunk fan-out#341

Merged
thodson-usgs merged 2 commits into
DOI-USGS:mainfrom
thodson-usgs:feat/chunk-granularity
Jul 23, 2026
Merged

feat(ogc): add opt-in parallel chunk fan-out#341
thodson-usgs merged 2 commits into
DOI-USGS:mainfrom
thodson-usgs:feat/chunk-granularity

Conversation

@thodson-usgs

@thodson-usgs thodson-usgs commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add parallel_chunks(n) for opt-in fan-out of large Water Data and NGWMN requests.
  • Preserve conservative byte-limit chunking by default and cap optional fan-out across all chunk axes.
  • Document quota, max_rows, retry/resume tradeoffs and add planner/integration coverage.

Basic usage

from dataretrieval import waterdata

sites, _ = waterdata.get_monitoring_locations(
    state="Ohio",
    site_type_code="ST",
)

with waterdata.parallel_chunks(32):
    df, metadata = waterdata.get_daily(
        monitoring_location_id=sites["monitoring_location_id"],
        parameter_code="00060",
        time="2004-01-01/2023-12-31",
    )

Use this only for large, paginated pulls with a multi-value argument. Each sub-request consumes quota; concurrent execution remains bounded separately by API_USGS_CONCURRENT (default 32).

Timing

Fixed 271-site Ohio stream-gage subset, get_daily(parameter_code="00060"), limit=250; each run used a cold one-year window.

n Parallelism Pages Wall-clock Speedup
off 1 ~30 9.5 s / 9.1 s
8 8 ~32 2.2 s / 1.9 s ~4.5×
32 32 54 1.2 s ~8×

Validation

  • pytest -q tests/waterdata_chunking_test.py tests/utils_test.py
  • ruff check on changed Python files
  • 89,600 generated multi-axis plan checks

@thodson-usgs
thodson-usgs force-pushed the feat/chunk-granularity branch 3 times, most recently from b47e5cc to 0195113 Compare July 1, 2026 16:25
@thodson-usgs thodson-usgs changed the title feat(waterdata): add chunk_granularity to control OGC chunk fan-out feat(waterdata): add parallel_chunks to control OGC chunk fan-out Jul 8, 2026
@thodson-usgs
thodson-usgs marked this pull request as ready for review July 17, 2026 17:53
Add parallel_chunks(n) so large Water Data and NGWMN queries can split
into more independently paginated sub-requests while preserving the
conservative byte-limit-only default.

Cap optional fan-out across all chunk axes, validate positive integer
inputs, export the helper publicly, and document usage, tradeoffs, and
benchmark results. Add planner, context, and integration coverage.
@thodson-usgs
thodson-usgs force-pushed the feat/chunk-granularity branch from 8ae1b9a to a2c5627 Compare July 22, 2026 14:09
@thodson-usgs thodson-usgs changed the title feat(waterdata): add parallel_chunks to control OGC chunk fan-out feat(ogc): add opt-in parallel chunk fan-out Jul 22, 2026
@thodson-usgs
thodson-usgs merged commit d292021 into DOI-USGS:main Jul 23, 2026
9 checks passed
@thodson-usgs
thodson-usgs deleted the feat/chunk-granularity branch July 23, 2026 14:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant