[GH-3273] GeoPandas: add distributed coverage validation - #3274
Merged
jiayuasu merged 3 commits intoAug 25, 2026
Conversation
jiayuasu
force-pushed
the
feature/coverage-validation-native
branch
from
August 21, 2026 21:29
e9e3a11 to
67fdeef
Compare
jiayuasu
force-pushed
the
feature/coverage-validation-native
branch
from
August 23, 2026 06:20
67fdeef to
c87438e
Compare
jiayuasu
force-pushed
the
feature/coverage-validation-native
branch
from
August 23, 2026 06:32
c87438e to
d79530a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Did you read the Contributor Guide?
Is this PR related to a ticket?
[GH-XXX] my subject.Closes #3273.
What changes were proposed in this PR?
This PR implements distributed polygonal coverage validation for the GeoPandas API.
It adds:
GeoSeries.is_valid_coverage()andGeoSeries.invalid_coverage_edges(), with delegation fromGeoDataFrame;The implementation discovers polygonal candidates with a spatial self-join over gap-expanded envelopes, validates each distinct represented geometry on Spark executors, and maps the diagnostics back to one index-bearing evaluation of the input. It preserves physical duplicate rows, all index levels, CRS, null positions, and input order without relying on a cross-branch row identifier.
invalid_coverage_edges()remains lazy and does not collect geometry rows to the driver.is_valid_coverage()performs one distributed reduction to return a Pythonbool. The implementation uses one registered__sedona_internal_*expression and adds no public Spark SQL API, JVM adapter, or Python UDF.gap_widthmust be a finite, non-negative numeric scalar. Missing geometries remain missing; geometries without polygonal components return empty invalid-edge lines. Coverage diagnostics preserve Z and, matching GeoPandas, omit M coordinates.How was this patch tested?
CoverageValidationTest: 13 tests passed.git diff --checkpassed.The tests cover matching edges, overlaps, exact duplicates, holes, multipolygons, polygonal collections, mixed dimensional layouts, narrow gaps, null and nonpolygonal rows, invalid gap widths, duplicate and MultiIndex preservation, generated and MapType indexes, shuffled multi-executor execution, CRS preservation, hidden expression registration, native spatial-join planning, laziness, Spark Connect, and GeoPandas parity.
Performance
Coverage validation uses a distributed spatial self-join followed by per-geometry candidate aggregation and a geometry-keyed lookup onto the input rows. Geometry rows are not materialized on the driver. Executor memory scales with candidate density and
gap_width; the materialized invalid-edge result restores pandas-on-Spark order after projecting away candidate arrays.Did this PR include necessary documentation updates?
v2.0.0.