Skip to content

[GH-3273] GeoPandas: add distributed coverage validation - #3274

Merged
jiayuasu merged 3 commits into
apache:masterfrom
jiayuasu:feature/coverage-validation-native
Aug 25, 2026
Merged

[GH-3273] GeoPandas: add distributed coverage validation#3274
jiayuasu merged 3 commits into
apache:masterfrom
jiayuasu:feature/coverage-validation-native

Conversation

@jiayuasu

@jiayuasu jiayuasu commented Aug 20, 2026

Copy link
Copy Markdown
Member

Did you read the Contributor Guide?

Is this PR related to a ticket?

  • Yes, and the PR name follows the format [GH-XXX] my subject.

Closes #3273.

What changes were proposed in this PR?

This PR implements distributed polygonal coverage validation for the GeoPandas API.

It adds:

  • GeoSeries.is_valid_coverage() and GeoSeries.invalid_coverage_edges(), with delegation from GeoDataFrame;
  • one hidden Catalyst expression that validates a target geometry against its candidate coverage neighbors; and
  • GeoPandas API documentation, release notes, and English and Chinese tutorial coverage.

The implementation discovers polygonal candidates with a spatial self-join over gap-expanded envelopes, validates each distinct represented geometry on Spark executors, and maps the diagnostics back to one index-bearing evaluation of the input. It preserves physical duplicate rows, all index levels, CRS, null positions, and input order without relying on a cross-branch row identifier.

invalid_coverage_edges() remains lazy and does not collect geometry rows to the driver. is_valid_coverage() performs one distributed reduction to return a Python bool. The implementation uses one registered __sedona_internal_* expression and adds no public Spark SQL API, JVM adapter, or Python UDF.

gap_width must be a finite, non-negative numeric scalar. Missing geometries remain missing; geometries without polygonal components return empty invalid-edge lines. Coverage diagnostics preserve Z and, matching GeoPandas, omit M coordinates.

How was this patch tested?

  • CoverageValidationTest: 13 tests passed.
  • Spark 3.4 / Scala 2.12 focused expression and catalog suites: 7 tests passed.
  • Spark 4.1 / Scala 2.13 clean reactor: all 7 modules passed.
  • Spark 3.4 focused Python coverage matrix: 31 passed and 1 expected upstream-version parity skip.
  • Spark 4.1 focused Python coverage matrix: 32 tests passed.
  • A three-executor local-cluster regression repeatedly validated shuffled join/aggregate input without row/result mis-association.
  • Matching shaded-package builds, Spotless, Black, Markdown linting, Python compilation, and git diff --check passed.

The tests cover matching edges, overlaps, exact duplicates, holes, multipolygons, polygonal collections, mixed dimensional layouts, narrow gaps, null and nonpolygonal rows, invalid gap widths, duplicate and MultiIndex preservation, generated and MapType indexes, shuffled multi-executor execution, CRS preservation, hidden expression registration, native spatial-join planning, laziness, Spark Connect, and GeoPandas parity.

Performance

Coverage validation uses a distributed spatial self-join followed by per-geometry candidate aggregation and a geometry-keyed lookup onto the input rows. Geometry rows are not materialized on the driver. Executor memory scales with candidate density and gap_width; the materialized invalid-edge result restores pandas-on-Spark order after projecting away candidate arrays.

Did this PR include necessary documentation updates?

  • Yes, I am adding a new API. I am using the current SNAPSHOT version, v2.0.0.
  • Yes, I have updated the GeoPandas API documentation, release notes, and English and Chinese tutorials. There is no public Spark SQL function to document.

@jiayuasu
jiayuasu force-pushed the feature/coverage-validation-native branch from e9e3a11 to 67fdeef Compare August 21, 2026 21:29
@jiayuasu jiayuasu changed the title [GH-3273] Add internal coverage validation primitives [GH-3273] GeoPandas: add distributed coverage validation Aug 21, 2026
@jiayuasu
jiayuasu force-pushed the feature/coverage-validation-native branch from 67fdeef to c87438e Compare August 23, 2026 06:20
@jiayuasu
jiayuasu force-pushed the feature/coverage-validation-native branch from c87438e to d79530a Compare August 23, 2026 06:32
@jiayuasu jiayuasu added this to the sedona-2.0.0 milestone Aug 25, 2026
@jiayuasu
jiayuasu marked this pull request as ready for review August 25, 2026 05:32
@jiayuasu
jiayuasu merged commit 62c9fca into apache:master Aug 25, 2026
45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GeoPandas: implement distributed coverage validation

1 participant