Skip to content

Disable memory-limited NLJ fallback for left-emitting joins with multi-partition right inputs#22671

Draft
kosiew wants to merge 3 commits into
apache:mainfrom
kosiew:memory-limited-22641
Draft

Disable memory-limited NLJ fallback for left-emitting joins with multi-partition right inputs#22671
kosiew wants to merge 3 commits into
apache:mainfrom
kosiew:memory-limited-22641

Conversation

@kosiew
Copy link
Copy Markdown
Contributor

@kosiew kosiew commented May 31, 2026

Which issue does this PR close?

Rationale for this change

The memory-limited NestedLoopJoin fallback path can produce incorrect results for joins that require final left-side emission when the right side has multiple partitions.

The fallback implementation tracks match state locally for each probe partition. Without global coordination across all right partitions, a partition may incorrectly emit unmatched, semi, anti, or mark results for left rows that actually match in another partition. This can lead to incorrect query results under memory pressure.

To preserve correctness, this change disables the memory-limited fallback for join types that depend on final left-side emission when the right side is partitioned. In these cases, the operator will continue to fail with resource exhaustion rather than produce incorrect output.

What changes are included in this PR?

  • Generalize the existing FULL JOIN fallback guard to cover all joins that require final left-side emission when right_partition_count > 1.

  • Replace the previous full_join_multi_partition check with a left_final_multi_partition check based on need_produce_result_in_final(join_type).

  • Update comments to document why memory-limited fallback is unsafe for these join types without cross-partition left-match tracking.

  • Adjust memory-limited join tests so that:

    • Supported join types continue to verify successful spill-based fallback.
    • LEFT, LEFT SEMI, LEFT ANTI, LEFT MARK, and FULL joins verify that fallback remains disabled and returns a resource exhaustion error under tight memory limits.
  • Add a dedicated regression test:

    • test_nlj_memory_limited_left_join_multi_partition_fallback_disabled
    • Verifies that a multi-partition LEFT JOIN does not use the fallback path under memory pressure.
    • Verifies that a single-partition LEFT JOIN can still spill and complete successfully.

Are these changes tested?

Yes.

The following tests were added or updated:

  • Added:

    • test_nlj_memory_limited_left_join_multi_partition_fallback_disabled
    • assert_multi_partition_join_oom
  • Updated memory-limited fallback coverage to verify:

    • Successful spill-based fallback for supported join types (Inner, Right, RightSemi, RightAnti, RightMark).
    • Resource exhaustion behavior for left-emitting join types with multi-partition right inputs (Left, LeftSemi, LeftAnti, LeftMark, Full).

Are there any user-facing changes?

Yes.

Under memory pressure, NestedLoopJoin will no longer attempt the memory-limited fallback path for joins that require final left-side emission when the right side has multiple partitions. Instead of potentially returning incorrect results, these queries will fail with a resource exhaustion error until cross-partition coordination is implemented.

This change prioritizes correctness and ensures join results do not vary based on memory availability.

LLM-generated code disclosure

This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed and tested.

kosiew added 3 commits May 31, 2026 20:40
…regression tests

- Disabled NLJ spill fallback when right_partition_count > 1 and when the join requires final left-side emission (Left, LeftSemi, LeftAnti, LeftMark, Full).
- Added rationale comment to clarify the changes.
- Introduced regression tests for:
- Multi-partition LEFT join with tight memory resulting in OOM.
- Single-partition LEFT join with tight memory which still spills but succeeds.
- Updated overalloc test join scopes accordingly.
- Added private test helper `assert_multi_partition_join_oom`
- Replaced duplicate OOM assertion blocks
- Renamed `_columns` and `_batches` to `_`
… and improved caller parameters

- Enhanced assert_multi_partition_join_oom to create a fresh constrained TaskContext.
- Updated callers to pass memory_limit and batch_size parameters.
- Removed duplicate context setup in tests for better clarity and efficiency.
@github-actions github-actions Bot added the physical-plan Changes to the physical-plan crate label May 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Correctness guard: disable memory-limited NestedLoopJoin fallback for LEFT-family joins with multi-partition right side

1 participant