fix(rust/sedona-spatial-join): Fix geography scalar functions failing over spatial join output - #1153
Merged
Conversation
… over spatial join output Applying a geography scalar function (ST_Area, ST_Distance, ...) to the output of a spatial join failed with "Array input must not be empty" whenever the join produced few or no matches. Two fixes: - The spatial join emitted one zero-row batch for every probe batch whose trailing rows had no matches: produce_last_result_batch unconditionally returned the assembled batch, which is always empty for join types that don't produce unmatched probe rows (e.g. inner joins). Suppress empty batches; this also fixes consumers like lonboard that reject them. - Imported C scalar kernels (sedona-s2geography) reject zero-length array inputs because they broadcast scalar arguments using modulo indexing. Short-circuit empty inputs in ImportedScalarKernel and return an empty array of the return type, so empty batches from any operator are handled. Fixes apache#1084
james-willis
approved these changes
Aug 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Applying a geography scalar function (
ST_Area,ST_Distance,ST_Intersection, ...) to the output of a spatial join failed withSedonaCScalarKernelImpl::execute failed: Array input must not be empty. Two fixes, following the two directions suggested in the issue:Stop the spatial join from emitting empty batches.
produce_last_result_batchunconditionally returned the assembled "unmatched tail rows" batch, which is always zero-row for join types that don't produce unmatched probe rows (e.g. inner joins). So every probe batch whose trailing rows had no match pushed one empty batch downstream. It now suppresses empty batches (build_joined_batchis still called for its visited-bitmap bookkeeping). This also helps consumers that reject empty batches, such as lonboard.Handle zero-length array inputs in
ImportedScalarKernel. Imported C scalar kernels (sedona-s2geography) reject zero-length arrays because they broadcast scalar arguments using modulo indexing. The FFI wrapper now short-circuitsnum_rows == 0and returns an empty array of the return type without crossing the C boundary. Empty batches are legal in DataFusion streams, so scalar UDFs must tolerate them regardless of which operator produced them.Note the bug was broader than reported in the issue: joins with matches also failed whenever any probe batch had unmatched trailing rows (which is nearly always the case for selective joins), so this broke most geography spatial joins with scalar functions applied on top.
How was this patch tested?
empty_array_inputinc/sedona-extension/src/scalar_kernel.rs: an imported kernel that rejects empty input (mimicking the sedona-s2geography kernels) invoked with a zero-length array now returns a zero-length array. Fails without fix 2.test_no_empty_batches_in_join_outputinrust/sedona-spatial-join: executesSpatialJoinExecpartitions directly for a join with matches and a join with zero matches, asserting no emitted batch is empty. Fails without fix 1.--features s2geographybuild: the selective geography self-join now returns0, NaN(matching the empty equi-join behavior), andST_Area/ST_Intersection/ST_Distancework over join output with and without matches.sedona-extensionandsedona-spatial-jointest suites pass.Did this PR include necessary documentation updates?
Fixes #1084.