Skip to content

fix(rust/sedona-spatial-join): Fix geography scalar functions failing over spatial join output - #1153

Merged
jiayuasu merged 1 commit into
apache:mainfrom
jiayuasu:fix/spatial-join-empty-batches
Aug 11, 2026
Merged

fix(rust/sedona-spatial-join): Fix geography scalar functions failing over spatial join output#1153
jiayuasu merged 1 commit into
apache:mainfrom
jiayuasu:fix/spatial-join-empty-batches

Conversation

@jiayuasu

Copy link
Copy Markdown
Member

What changes were proposed in this PR?

Applying a geography scalar function (ST_Area, ST_Distance, ST_Intersection, ...) to the output of a spatial join failed with SedonaCScalarKernelImpl::execute failed: Array input must not be empty. Two fixes, following the two directions suggested in the issue:

  1. Stop the spatial join from emitting empty batches. produce_last_result_batch unconditionally returned the assembled "unmatched tail rows" batch, which is always zero-row for join types that don't produce unmatched probe rows (e.g. inner joins). So every probe batch whose trailing rows had no match pushed one empty batch downstream. It now suppresses empty batches (build_joined_batch is still called for its visited-bitmap bookkeeping). This also helps consumers that reject empty batches, such as lonboard.

  2. Handle zero-length array inputs in ImportedScalarKernel. Imported C scalar kernels (sedona-s2geography) reject zero-length arrays because they broadcast scalar arguments using modulo indexing. The FFI wrapper now short-circuits num_rows == 0 and returns an empty array of the return type without crossing the C boundary. Empty batches are legal in DataFusion streams, so scalar UDFs must tolerate them regardless of which operator produced them.

Note the bug was broader than reported in the issue: joins with matches also failed whenever any probe batch had unmatched trailing rows (which is nearly always the case for selective joins), so this broke most geography spatial joins with scalar functions applied on top.

How was this patch tested?

  • New unit test empty_array_input in c/sedona-extension/src/scalar_kernel.rs: an imported kernel that rejects empty input (mimicking the sedona-s2geography kernels) invoked with a zero-length array now returns a zero-length array. Fails without fix 2.
  • New integration test test_no_empty_batches_in_join_output in rust/sedona-spatial-join: executes SpatialJoinExec partitions directly for a join with matches and a join with zero matches, asserting no emitted batch is empty. Fails without fix 1.
  • Verified the reproduction from Geography scalar function over spatial-join output fails with "Array input must not be empty" #1084 end to end with a --features s2geography build: the selective geography self-join now returns 0, NaN (matching the empty equi-join behavior), and ST_Area/ST_Intersection/ST_Distance work over join output with and without matches.
  • Full sedona-extension and sedona-spatial-join test suites pass.

Did this PR include necessary documentation updates?

  • No, this PR does not affect documentation.

Fixes #1084.

… over spatial join output

Applying a geography scalar function (ST_Area, ST_Distance, ...) to the
output of a spatial join failed with "Array input must not be empty"
whenever the join produced few or no matches.

Two fixes:

- The spatial join emitted one zero-row batch for every probe batch whose
  trailing rows had no matches: produce_last_result_batch unconditionally
  returned the assembled batch, which is always empty for join types that
  don't produce unmatched probe rows (e.g. inner joins). Suppress empty
  batches; this also fixes consumers like lonboard that reject them.

- Imported C scalar kernels (sedona-s2geography) reject zero-length array
  inputs because they broadcast scalar arguments using modulo indexing.
  Short-circuit empty inputs in ImportedScalarKernel and return an empty
  array of the return type, so empty batches from any operator are handled.

Fixes apache#1084
@github-actions
github-actions Bot requested a review from james-willis August 11, 2026 21:35
@jiayuasu
jiayuasu requested a review from paleolimbot August 11, 2026 21:36
@jiayuasu
jiayuasu merged commit 2458239 into apache:main Aug 11, 2026
18 checks passed
@jiayuasu
jiayuasu deleted the fix/spatial-join-empty-batches branch August 11, 2026 23:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Geography scalar function over spatial-join output fails with "Array input must not be empty"

2 participants