Skip to content

Add IGNORE NULLS support for LEAD and LAG window functions#18202

Open
yashmayya wants to merge 1 commit intoapache:masterfrom
yashmayya:ignore-nulls-lead-lag
Open

Add IGNORE NULLS support for LEAD and LAG window functions#18202
yashmayya wants to merge 1 commit intoapache:masterfrom
yashmayya:ignore-nulls-lead-lag

Conversation

@yashmayya
Copy link
Copy Markdown
Contributor

  • Replace custom PinotLeadWindowFunction / PinotLagWindowFunction (which blocked IGNORE NULLS at the Calcite level) with standard SqlStdOperatorTable.LEAD / SqlStdOperatorTable.LAG
  • Add processRowsIgnoreNulls() to LeadValueWindowFunction and LagValueWindowFunction using a bounded ArrayDeque for O(N) time and O(offset) memory
  • LEAD scans right-to-left and LAG scans left-to-right, each maintaining a sliding window of the last offset non-null values
  • Support all three operand forms: LEAD/LAG(col), LEAD/LAG(col, offset), and LEAD/LAG(col, offset, default)
  • Add 7 unit tests covering default offset, custom offset, offset with default value, all-nulls edge case, and multiple partitions
  • Add 5 resource-based query tests including LEAD/LAG IGNORE NULLS with various configurations and an explicit RESPECT NULLS regression guard
  • Add @Nullable annotation to WindowFunction.extractValueFromRow()
  • Fix pre-existing Javadoc typo in LeadValueWindowFunction ("LAG" → "LEAD")

@yashmayya yashmayya force-pushed the ignore-nulls-lead-lag branch from e7fa9b1 to 50c6b87 Compare April 14, 2026 23:38
@yashmayya yashmayya added multi-stage Related to the multi-stage query engine window-functions Related to SQL window functions on the multi-stage query engine labels Apr 14, 2026
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Apr 15, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 63.34%. Comparing base (d299729) to head (50c6b87).
⚠️ Report is 10 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master   #18202      +/-   ##
============================================
+ Coverage     63.31%   63.34%   +0.03%     
  Complexity     1627     1627              
============================================
  Files          3229     3238       +9     
  Lines        196707   197020     +313     
  Branches      30408    30471      +63     
============================================
+ Hits         124538   124801     +263     
- Misses        62192    62219      +27     
- Partials       9977    10000      +23     
Flag Coverage Δ
custom-integration1 100.00% <ø> (ø)
integration 100.00% <ø> (ø)
integration1 100.00% <ø> (ø)
integration2 0.00% <ø> (ø)
java-11 63.32% <100.00%> (+8.06%) ⬆️
java-21 63.32% <100.00%> (+0.02%) ⬆️
temurin 63.34% <100.00%> (+0.03%) ⬆️
unittests 63.34% <100.00%> (+0.03%) ⬆️
unittests1 55.29% <100.00%> (+<0.01%) ⬆️
unittests2 34.95% <0.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

multi-stage Related to the multi-stage query engine window-functions Related to SQL window functions on the multi-stage query engine

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants