Skip to content

feat(cudf): Support multi-column RANGE Window aggregates - #20

Merged
winningsix merged 4 commits into
HighPerfDataAccelerator:devfrom
thirtiseven:feature/cudf-multi-column-range-window
Jul 21, 2026
Merged

feat(cudf): Support multi-column RANGE Window aggregates#20
winningsix merged 4 commits into
HighPerfDataAccelerator:devfrom
thirtiseven:feature/cudf-multi-column-range-window

Conversation

@thirtiseven

@thirtiseven thirtiseven commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • support aggregate RANGE Window frames with multiple ORDER BY columns, including mixed directions and null ordering
  • preserve streaming behavior across input batches and spill while treating complete multi-column peer keys consistently
  • bound exchange batch concatenation by bytes and materialize nested UCX partition windows before their sliced backing storage is released
  • add focused coverage for multi-column aggregate Window selection, cross-batch and spill execution, and sliced struct RANGE partitioning

Why

The cuDF Window adapter assumed that aggregate RANGE frames had a single ordering key. Spark workloads with multiple ordering columns were therefore rejected under strict cuDF replacement, even though the input was already correctly partitioned and sorted. Once that path was enabled, wide nested payloads also exposed two exchange lifetime and batching issues: concatenation could grow without a byte bound, and sliced nested columns could outlive their backing storage during UCX partitioning.

Impact

Multi-column aggregate RANGE windows can stay fully native and stream bounded batches through MPP. The exchange changes keep wide and nested payloads bounded and owned across asynchronous partition delivery.

Validation

  • native Gluten/Velox target compiled and linked successfully with this branch
  • Netflix Job 144 completed with strict MPP and cuDF fallback disabled
  • the fresh Iceberg target contains exactly 51,528,228 rows in four data files
  • no fallback, retry, failed task, or GPU OOM was observed
  • companion Spark Gluten PR carries the planner, exchange configuration, and multi-peer root placement changes

Companion PR

NVIDIA/spark-gluten#71 carries the Spark planner, exchange configuration, and multi-peer root placement changes.

@github-actions github-actions Bot added the cudf label Jul 21, 2026
@github-actions

Copy link
Copy Markdown

Selective Build Plan

Linux release with adapters is running a full build (changes touch velox/experimental/ or velox/external/). See the CI workflows README for what this means.


Selective build plan

@thirtiseven thirtiseven changed the title Support multi-column RANGE Window aggregates feat(cudf): Support multi-column RANGE Window aggregates Jul 21, 2026
@thirtiseven
thirtiseven marked this pull request as ready for review July 21, 2026 07:59
…umn-range-window

# Conflicts:
#	velox/experimental/cudf/exec/CudfWindow.cpp
@thirtiseven

Copy link
Copy Markdown
Collaborator Author

Updated onto the latest dev (b4f5aea9c). The CudfWindow.cpp conflict was resolved by preserving the upstream ordered first/first_value multi-key path alongside aggregate multi-column RANGE support, while retaining the unsupported multi-key rank/dense_rank guard.

Validation after the merge:

  • incremental build of velox_cudf_adapter_operator_test and ucx_exchange_test
  • velox_cudf_adapter_operator_test: 18/18 passed
  • focused RANGE/sliced-struct UCX tests: 3/3 passed

No benchmark query was rerun for this conflict refresh.

@winningsix
winningsix merged commit 954e7ba into HighPerfDataAccelerator:dev Jul 21, 2026
43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants