[GH-3283] Honor max_items in STAC client searches with bbox/datetime filters - #3284
Merged
Conversation
…etime filters CollectionClient.load_items_df only passed max_items to the reader as itemsLimitMax when no filters were provided. With a bbox or datetime filter it fell back to a DataFrame limit applied behind the filters, which cannot bound the scan, so a search asking for 100 items of a year-long window sequentially enumerated every matching result page at the default 10 items per request before Spark trimmed the rows. Searches that the STAC API can represent directly now follow pystac-client semantics. When a search against a named collection uses at most one bbox and one datetime interval, no ID or geometry filter, and a positive max_items, the client forwards the constraints through reader options; the datasource applies them as parameters of the collection's advertised JSON rel=items endpoint, trusts the API's spatial and temporal matching, and stops enumeration after max_items server results using the existing counting walk, paging at min(200, max_items). Endpoints that already own a same-kind constraint are rejected instead of being widened or duplicated, this mode never combines with Spark-pushed predicates, and multiple ranges, ID filters, and geometry filters keep the current Spark-evaluated behavior. Also normalizes datetime values to UTC RFC 3339 strings in _move_attributes_to_properties: PySpark materializes timestamps in the local timezone, so item round-trips previously produced naive local-time strings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Did you read the Contributor Guide?
Is this PR related to a ticket?
[GH-XXX] my subject. Closes STAC Python client: search with bbox/datetime filters enumerates the entire collection instead of honoring max_items #3283What changes were proposed in this PR?
CollectionClient.load_items_dfonly passedmax_itemsto the reader asitemsLimitMaxwhen no filters were provided. With abboxordatetimefilter it fell back todf.limit(max_items)behind the filters — which cannot bound the scan — soclient.search(collection_id="sentinel-2-c1-l2a", datetime="2025", max_items=100)sequentially enumerated every matching result page (hundreds of thousands of items at 10 per request) before Spark trimmed to 100. In our integration environment this ran to a 1-hour timeout.Searches the STAC API can represent directly now follow pystac-client semantics:
bboxand onedatetimeinterval, no ID or geometry filter, and a positivemax_items, the Python client forwards the constraints through internal reader options.rel=itemsendpoint (preserving custom, cross-origin, and query-bearing hrefs), trusts the conforming API's spatial and temporal matching, and stops enumeration aftermax_itemsserver results via the existing counting walk, paging atmin(200, max_items). The motivating search becomes ~1 request.bbox/bbox-crs/intersects/datetime, matched on percent-decoded parameter names) are rejected rather than widened or duplicated, and this mode refuses to combine with Spark-pushed predicates, whose retained Filter would discard rows the cap already counted.Note this makes the representable searches API-owned in semantics: e.g. an interval-valued Item a conforming server returns for a datetime search is kept rather than re-filtered against its nominal
datetime— which matches what pystac-client users expect.Also fixes a timezone bug in
_move_attributes_to_properties: PySpark materializes timestamps in the local timezone, so item round-trips previously produced naive local-time strings; datetime values are now normalized to UTC RFC 3339 (Z) strings.How was this patch tested?
StacBatchTest, plusapplyClientApiSearchParametersunit tests (encoding, fragment placement, conflicting-constraint rejection incl. percent-encoded names) inStacBatchUrlTest. All STAC suites pass under-Dspark=3.5: 73 passed.Did this PR include necessary documentation updates?
max_itemssemantics and which search shapes are API-owned vs Spark-evaluated.