Skip to content

[WIP] Bump Apache DF to 55 - #9334

Draft
AdamGS wants to merge 3 commits into
developfrom
adamg/df-55
Draft

[WIP] Bump Apache DF to 55#9334
AdamGS wants to merge 3 commits into
developfrom
adamg/df-55

Conversation

@AdamGS

@AdamGS AdamGS commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

Periodic update to DataFusion

What changes are included in this PR?

  1. Bump Arrow to 59.2
  2. Bump DF to 55.0.0 (Release DataFusion 55.0.0 (Jul / Aug 2026) apache/datafusion#22393)
  3. Bump geoarrow dependency (requires release to support Arrow 59)

Currently this PR also includes bumping our MSRV to 1.95 because geoarrow requires that (DataFusion only requires 1.94).

What APIs are changed? Are there any user-facing changes?

Just the dependency changes

AdamGS added 3 commits August 10, 2026 13:18
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
@AdamGS AdamGS added changelog/chore A trivial change ext/datafusion Relates to the DataFusion integration labels Aug 10, 2026
@AdamGS AdamGS added the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Aug 10, 2026
@github-actions github-actions Bot removed the action/bench-sql Run SQL benchmarks without Vortex Compact on this PR label Aug 10, 2026
@codspeed-hq

codspeed-hq Bot commented Aug 10, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 2.18%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 27 improved benchmarks
❌ 22 regressed benchmarks
✅ 1938 untouched benchmarks
⏩ 42 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation set_indices_vortex_buffer[65536] 121.2 µs 182.6 µs -33.64%
Simulation set_indices_vortex_buffer[16384] 31.6 µs 47.1 µs -32.93%
Simulation compact_sliced[(16384, 10)] 1.4 ms 2 ms -28.55%
Simulation set_indices_vortex_buffer[2048] 5.4 µs 7.5 µs -27.89%
Simulation compact_sliced[(4096, 10)] 385.2 µs 527.6 µs -27%
Simulation in_place_non_nullable[65536] 49.2 µs 66.5 µs -26.1%
Simulation copy_nullable[65536] 1.3 ms 1.7 ms -26.03%
Simulation set_indices_vortex_buffer[1024] 3.6 µs 4.7 µs -24.29%
Simulation decompress[u64, (10000, 4)] 318 µs 403.8 µs -21.24%
Simulation primitive_i64_vortex 3.3 ms 4 ms -18.23%
Simulation set_indices[(16384, 0.01)] 13.5 µs 16.2 µs -16.7%
Simulation div_decimal_i64_nonnull 942.2 µs 1,114.9 µs -15.49%
WallTime cuda/bitpacked_u8/unpack/3bw[100M] 299.2 µs 350.9 µs -14.74%
Simulation new_raw_prim_test_between[i32, 2048] 76.7 µs 88.7 µs -13.56%
Simulation set_indices[(16384, 0.1)] 36.1 µs 41.8 µs -13.52%
Simulation set_indices[(16384, 0.5)] 137.3 µs 156.3 µs -12.2%
Simulation set_indices[(16384, 0.9)] 237.9 µs 269.7 µs -11.77%
Simulation set_indices[(16384, 0.99)] 256 µs 289.4 µs -11.57%
Simulation compress_fsst[(1000, 64, 8)] 1 ms 1.2 ms -10.86%
Simulation compress_alp[f32, (10000, 0.01, 1.0)] 395.2 µs 442.5 µs -10.69%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing adamg/df-55 (ba2ac36) with develop (4b37237)

Open in CodSpeed

Footnotes

  1. 42 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change ext/datafusion Relates to the DataFusion integration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant