Skip to content

[AURON #2176] Implement native support for lead window function#2188

Open
weimingdiit wants to merge 2 commits intoapache:masterfrom
weimingdiit:feat/native-window-lead
Open

[AURON #2176] Implement native support for lead window function#2188
weimingdiit wants to merge 2 commits intoapache:masterfrom
weimingdiit:feat/native-window-lead

Conversation

@weimingdiit
Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #2176

Rationale for this change

Auron’s native window support previously covered rank-like functions and a subset of aggregate window functions, but did not support offset-based window functions such as lead(...).

This PR extends native window coverage with a conservative first step:

  • support lead(...)
  • preserve Spark-compatible behavior for input, offset, and default
  • keep unsupported semantics out of the native path rather than approximating them incorrectly

What changes are included in this PR?

This PR:

  • adds Lead handling in NativeWindowBase
  • extends the protobuf/planner window function enum with LEAD
  • adds native planner support to decode LEAD into the native window plan
  • introduces a native LeadProcessor in datafusion-ext-plans
  • evaluates lead using Spark-compatible offset/default/null behavior
  • adds a full-partition processing path for lead so that lookahead works correctly across input batches
  • adds Rust regression coverage for cross-batch lead
  • adds Scala regression tests for:
    • native lead(...) execution
    • Spark fallback for lead(... ) IGNORE NULLS

The native implementation supports Spark semantics for:

  • lead(input)

    • default offset is 1
    • default value is null
  • lead(input, offset, default)

    • returns the value of input at the offsetth row after the current row in the same window partition
    • if the target row exists and input there is null, returns null
    • if the target row does not exist, returns default

Supported scope in this PR:

  • standard RESPECT NULLS behavior

Not supported natively in this PR:

  • IGNORE NULLS

Unsupported IGNORE NULLS queries continue to fall back to Spark to preserve correctness.

Are there any user-facing changes?

Yes.
Queries using lead(...) can now remain on Auron’s native window execution path when they use supported semantics.
Queries using unsupported lead(... ) IGNORE NULLS behavior will continue to fall back to Spark.

How was this patch tested?

CI.

@weimingdiit weimingdiit changed the title [AURON #2176]Implement native support for lead window function [AURON #2176] Implement native support for lead window function Apr 9, 2026
@weimingdiit weimingdiit force-pushed the feat/native-window-lead branch from efe00fc to 5fd852f Compare April 9, 2026 12:09
Signed-off-by: weimingdiit <weimingdiit@gmail.com>
@weimingdiit weimingdiit force-pushed the feat/native-window-lead branch from 5fd852f to 7f1fef3 Compare April 9, 2026 13:20
Signed-off-by: weimingdiit <weimingdiit@gmail.com>
@weimingdiit weimingdiit marked this pull request as ready for review April 10, 2026 01:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement native support for lead window function

1 participant