Skip to content

feat: arrow device array list view export#8219

Open
0ax1 wants to merge 10 commits into
developfrom
ad/cudf-list-export
Open

feat: arrow device array list view export#8219
0ax1 wants to merge 10 commits into
developfrom
ad/cudf-list-export

Conversation

@0ax1
Copy link
Copy Markdown
Contributor

@0ax1 0ax1 commented Jun 2, 2026

Summary

Adds CUDA Arrow Device export for Vortex ListViewArray by exporting it as standard Arrow List for cuDF.

  • GPU export path for contiguous list-views using device-built i32 Arrow offsets
  • GPU rebuild path for supported non-contiguous primitive list-views
  • Host fallback via CPU ListViewArrayListArray rebuild when GPU export cannot handle the shape
  • CUB exclusive-scan wrapper used by the rebuild path
  • New CUDA kernels, tests, e2e coverage, and benchmarks for list-view export

@0ax1 0ax1 added the changelog/feature A new feature label Jun 2, 2026
Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
@0ax1 0ax1 force-pushed the ad/cudf-list-export branch from 9912bc6 to 7ceec68 Compare June 2, 2026 15:07
@0ax1 0ax1 requested review from connortsui20 and robert3005 June 2, 2026 15:07
@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented Jun 2, 2026

Merging this PR will improve performance by 18.46%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 3 improved benchmarks
❌ 1 regressed benchmark
✅ 1271 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation bitwise_not_vortex_buffer_mut[128] 246.1 ns 275.3 ns -10.6%
Simulation chunked_bool_canonical_into[(1000, 10)] 45.4 µs 30.5 µs +49.13%
Simulation chunked_varbinview_canonical_into[(1000, 10)] 197.7 µs 161.5 µs +22.43%
Simulation chunked_varbinview_into_canonical[(1000, 10)] 212.3 µs 176 µs +20.64%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ad/cudf-list-export (d9d49ad) with develop (66335d4)

Open in CodSpeed

0ax1 and others added 4 commits June 2, 2026 15:11
Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
cuDF's from_arrow_device_column has no Arrow FixedSizeList -> cudf type
mapping, so including a fixed_lists column in the exported struct made the
harness fail with "Unsupported type_id conversion to cudf". Vortex still
exports FixedSizeList correctly (covered by the test_export_fixed_size_list
unit test in vortex-cuda); the column simply cannot round-trip through
cuDF's device-array import. Keep the lists column, which cuDF supports.

Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread vortex-test/e2e-cuda/src/lib.rs Outdated
Comment thread vortex-cuda/src/arrow/canonical.rs Outdated
Comment thread vortex-cuda/src/arrow/canonical.rs
0ax1 added 4 commits June 3, 2026 09:04
Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
@0ax1
Copy link
Copy Markdown
Contributor Author

0ax1 commented Jun 3, 2026

depends on a new cudf harness release: vortex-data/cudf-test-harness#6

@0ax1 0ax1 requested a review from robert3005 June 3, 2026 13:39
Signed-off-by: Alexander Droste <alexander.droste@protonmail.com>
@0ax1 0ax1 force-pushed the ad/cudf-list-export branch from 03c3e13 to d9d49ad Compare June 3, 2026 13:40
Comment thread vortex-cuda/benches/list_view_cuda.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants