[GH-3110] Fix STAC Client.search(datetime="YYYY-mm") raising TypeError

[jiayuasu]

Did you read the Contributor Guide?

• Yes, I have read the Contributor Rules and Contributor Development Guide

Is this PR related to a ticket?

• Yes, and the PR name follows the format [GH-XXX] my subject. Closes STAC Client.search(datetime="YYYY-mm") raises TypeError: NoneType object is not callable #3110

What changes were proposed in this PR?

Calling the STAC Python Client.search() (or CollectionClient.get_dataframe() / get_items()) with a datetime argument in YYYY-mm form failed with RuntimeError: Failed to get filtered dataframe wrapping TypeError: 'NoneType' object is not callable.

Root cause: python/sedona/spark/stac/collection_client.py imported from pyspark.sql.types import dt. pyspark.sql.types exposes no dt, so the name resolved to None, and the YYYY-mm branch of _expand_date then called dt(...), raising the TypeError. The old arithmetic also overflowed for December (int(month) + 1 == 13).

This PR:

• Removes the erroneous from pyspark.sql.types import dt import.
• Computes the last day of the month with calendar.monthrange, which handles December and leap years correctly, and zero-pads the day so the emitted timestamp stays valid ISO 8601.
• Makes whole-period upper bounds inclusive of the final fractional second. Sedona stores STAC datetimes as timestamps and filters them with an inclusive datetime <= end bound, so a rounded 23:59:59Z end silently drops items in the last fraction of a second. The YYYY, YYYY-mm, and YYYY-mm-dd forms now expand to end at 23:59:59.999999Z (Spark's microsecond timestamp precision), which is the bound the residual Spark filter uses.
• Preserves that precision through the remote filter push-down. SpatialTemporalFilterPushDownForStacScan previously converted the Spark TimestampType literal with Instant.ofEpochMilli(v / 1000), discarding sub-millisecond digits; it now keeps full microseconds via a floorDiv/floorMod helper.
• Makes the pushed remote bound a superset of the residual Spark filter for sub-microsecond timestamps. STAC allows up to nine fractional digits, but Spark keeps only microseconds, so it truncates a legal item at ...999999500Z down to ...999999, which the residual datetime <= end filter retains. StacUtils.getFilterTemporal now serializes nine fractional digits and widens the inclusive upper bound (from LessThan/LessThanOrEqual and the upper side of an equality) by 999 ns to the last nanosecond of its microsecond, so the remote catalog no longer drops items Spark would keep. The lower bound is left exact; a slightly wider remote window is always safe because the residual filter re-checks each row at microsecond precision.

_expand_date("2020-05") now returns ["2020-05-01T00:00:00Z", "2020-05-31T23:59:59.999999Z"], consistent with the docstring and the STAC tutorial.

How was this patch tested?

Python (python/tests/stac/test_collection_client.py):

• test_expand_date covers all supported forms plus the previously broken edge cases: December (2020-12 -> 31), leap-year February (2020-02 -> 29), and non-leap February (2021-02 -> 28).
• test_expand_date_filter_includes_final_fractional_second runs the real Spark temporal filter over sub-second timestamps and asserts rows at 23:59:59.5Z on the last day/month/year of a period are retained while the first instant of the next period is excluded.

Scala (spark/common), all run via the two suites below (38 tests, all passing locally):

• SpatialTemporalFilterPushDownForStacScanTest (new) feeds Catalyst <=/>= predicates with microsecond TimestampType literals through the push-down and asserts the serialized datetime= request preserves microseconds and widens the inclusive upper bound to nine digits. Its covers 7-to-9 digit sub-microsecond timestamps case asserts that legal 7-, 8- and 9-digit timestamps in the bound's final microsecond fall within the pushed remote bound.
• StacUtilsTest gains a getFilterTemporal widens the inclusive upper bound to nanosecond precision case; the existing getFilterTemporal/addFiltersToUrl expectations were updated to the nine-digit, widened bounds (exact lower bound, +999 ns upper bound).

Did this PR include necessary documentation updates?

• No, this PR does not affect any public API so no need to change the documentation. (The Client.search datetime docstring was refreshed to show the .999999Z bounds.)

[Copilot]

Pull request overview

Fixes STAC datetime handling across the Python client and Spark-side STAC pushdown so that partial-date inputs (e.g. YYYY-mm) expand correctly and temporal bounds preserve microsecond precision end-to-end, avoiding dropped items at period boundaries.

Changes:

• Python: fix _expand_date for YYYY-mm (remove erroneous dt import, compute month end via calendar.monthrange, and use 23:59:59.999999Z inclusive upper bounds).
• Spark/Scala: preserve microseconds when converting Spark TimestampType literals for STAC temporal pushdown and widen inclusive upper bounds to the last nanosecond within the final microsecond for remote filtering.
• Tests: add/extend Scala and Python test coverage for month-end/leap-year expansion and boundary/fractional-second inclusion.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/optimization/SpatialTemporalFilterPushDownForStacScan.scala	Convert Spark timestamp literals from micros to LocalDateTime without truncating to milliseconds.
spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/io/stac/StacUtils.scala	Serialize temporal filters with 9 fractional digits and widen inclusive upper bounds by +999 ns.
spark/common/src/test/scala/org/apache/spark/sql/sedona_sql/optimization/SpatialTemporalFilterPushDownForStacScanTest.scala	New tests validating microsecond preservation and widened upper-bound pushdown behavior.
spark/common/src/test/scala/org/apache/spark/sql/sedona_sql/io/stac/StacUtilsTest.scala	Update/add expectations to match nanosecond-precision serialization and widened upper bounds.
python/sedona/spark/stac/collection_client.py	Fix _expand_date YYYY-mm expansion and make whole-period upper bounds microsecond-inclusive.
python/sedona/spark/stac/client.py	Refresh Client.search datetime docstring to reflect microsecond-inclusive bounds.
python/tests/stac/test_collection_client.py	Add tests for expanded date forms and for retaining items in the final fractional second of a period.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.