Skip to content

[GH-3283] Honor max_items in STAC client searches with bbox/datetime filters - #3284

Merged
jiayuasu merged 3 commits into
apache:masterfrom
jiayuasu:stac-client-max-items
Aug 26, 2026
Merged

[GH-3283] Honor max_items in STAC client searches with bbox/datetime filters#3284
jiayuasu merged 3 commits into
apache:masterfrom
jiayuasu:stac-client-max-items

Conversation

@jiayuasu

Copy link
Copy Markdown
Member

Did you read the Contributor Guide?

Is this PR related to a ticket?

What changes were proposed in this PR?

CollectionClient.load_items_df only passed max_items to the reader as itemsLimitMax when no filters were provided. With a bbox or datetime filter it fell back to df.limit(max_items) behind the filters — which cannot bound the scan — so client.search(collection_id="sentinel-2-c1-l2a", datetime="2025", max_items=100) sequentially enumerated every matching result page (hundreds of thousands of items at 10 per request) before Spark trimmed to 100. In our integration environment this ran to a 1-hour timeout.

Searches the STAC API can represent directly now follow pystac-client semantics:

  • When a search against a named collection uses at most one bbox and one datetime interval, no ID or geometry filter, and a positive max_items, the Python client forwards the constraints through internal reader options.
  • The datasource applies them as parameters of the collection's advertised JSON rel=items endpoint (preserving custom, cross-origin, and query-bearing hrefs), trusts the conforming API's spatial and temporal matching, and stops enumeration after max_items server results via the existing counting walk, paging at min(200, max_items). The motivating search becomes ~1 request.
  • Endpoints that already own a same-kind constraint (bbox/bbox-crs/intersects/datetime, matched on percent-decoded parameter names) are rejected rather than widened or duplicated, and this mode refuses to combine with Spark-pushed predicates, whose retained Filter would discard rows the cap already counted.
  • Multiple bounding boxes or datetime intervals, ID filters, and geometry filters are Sedona extensions evaluated by Spark; those shapes keep the current behavior (full enumeration, then Spark filters and the final limit) because a raw fetch cap before Spark-side predicates could return fewer matching items than requested.

Note this makes the representable searches API-owned in semantics: e.g. an interval-valued Item a conforming server returns for a datetime search is kept rather than re-filtered against its nominal datetime — which matches what pystac-client users expect.

Also fixes a timezone bug in _move_attributes_to_properties: PySpark materializes timestamps in the local timezone, so item round-trips previously produced naive local-time strings; datetime values are now normalized to UTC RFC 3339 (Z) strings.

How was this patch tested?

  • New Scala tests: the API-mode request chain against a local HTTP server (cursor pagination stops at the cap, child links never fetched, custom query-bearing href preserved) in StacBatchTest, plus applyClientApiSearchParameters unit tests (encoding, fragment placement, conflicting-constraint rejection incl. percent-encoded names) in StacBatchUrlTest. All STAC suites pass under -Dspark=3.5: 73 passed.
  • New Python tests: API-search option matrix (single vs multiple bbox/datetime, geometry/ids fallback, URL validation incl. antimeridian bbox), production reader-option wiring, Spark-filtered fallback, and UTC round-trip of interval item datetimes. 31 passed.

Did this PR include necessary documentation updates?

  • Yes, I have updated the documentation. The STAC tutorial (EN/ZH) now documents the pystac-client max_items semantics and which search shapes are API-owned vs Spark-evaluated.

…etime filters

CollectionClient.load_items_df only passed max_items to the reader as
itemsLimitMax when no filters were provided. With a bbox or datetime
filter it fell back to a DataFrame limit applied behind the filters,
which cannot bound the scan, so a search asking for 100 items of a
year-long window sequentially enumerated every matching result page at
the default 10 items per request before Spark trimmed the rows.

Searches that the STAC API can represent directly now follow
pystac-client semantics. When a search against a named collection uses
at most one bbox and one datetime interval, no ID or geometry filter,
and a positive max_items, the client forwards the constraints through
reader options; the datasource applies them as parameters of the
collection's advertised JSON rel=items endpoint, trusts the API's
spatial and temporal matching, and stops enumeration after max_items
server results using the existing counting walk, paging at
min(200, max_items). Endpoints that already own a same-kind constraint
are rejected instead of being widened or duplicated, this mode never
combines with Spark-pushed predicates, and multiple ranges, ID filters,
and geometry filters keep the current Spark-evaluated behavior.

Also normalizes datetime values to UTC RFC 3339 strings in
_move_attributes_to_properties: PySpark materializes timestamps in the
local timezone, so item round-trips previously produced naive
local-time strings.
@jiayuasu jiayuasu added this to the sedona-2.0.0 milestone Aug 25, 2026
@jiayuasu
jiayuasu merged commit 672bf1b into apache:master Aug 26, 2026
45 checks passed
@jiayuasu
jiayuasu deleted the stac-client-max-items branch August 26, 2026 21:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

STAC Python client: search with bbox/datetime filters enumerates the entire collection instead of honoring max_items

1 participant