chore: dependency update Arrow to 58 and DataFusion to 53 by timsaucer · Pull Request #6496 · lance-format/lance

timsaucer · 2026-04-13T14:31:19Z

This PR is a dependency update only for Arrow and DataFusion. It includes the minimal changes necessary for the upgrade and is not designed to have any additional impact.

Closes #6137

github-actions · 2026-04-13T14:31:43Z

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

timsaucer

Ok, there's a few things we need to take care of but otherwise looks in decent shape.

timsaucer · 2026-04-13T14:35:00Z

                None,
                PartitionMode::CollectLeft,
                NullEquality::NullEqualsNull,
+                false,


Need to investigate if this is a null-aware join.

null-aware only matters for very specific anti-join patterns and we never configure an anti-join today.

timsaucer · 2026-04-13T14:35:30Z

 use datafusion::functions_aggregate;
 use datafusion::logical_expr::{Expr, ScalarUDF, col, lit};
 use datafusion::physical_expr::PhysicalSortExpr;
+#[allow(deprecated)]


Replacing CoalesceBatchesExec should probably be a new PR.

timsaucer · 2026-04-13T14:36:18Z

                    None,
                    PartitionMode::CollectLeft,
                    NullEquality::NullEqualsNull,
+                    false,


Also check here if this is a null aware join

Ditto, not a concern

timsaucer · 2026-04-13T14:36:36Z

    }

    /// Take row indices produced by input plan from the dataset (with projection)
+    #[allow(deprecated)]


Suggested change

#[allow(deprecated)]

#[expect(deprecated)]

Yeah, let's use expect.

timsaucer · 2026-04-13T14:38:45Z

-    fn statistics(&self) -> DataFusionResult<Statistics> {
-        Ok(Statistics {
-            num_rows: Precision::Inexact(
-                self.query.k * self.query.refine_factor.unwrap_or(1) as usize,
-            ),
-            ..Statistics::new_unknown(self.schema().as_ref())
-        })
-    }


We probably need to implement partition statistics here, not just drop it.

Did statistics get fully removed now? I know it had been deprecated for a bit. Losing this is probably fine for now if we make sure to add a follow-up.

timsaucer · 2026-04-13T14:38:59Z


 use super::TakeExec;
 use arrow_schema::Schema as ArrowSchema;
+#[allow(deprecated)]


Suggested change

#[allow(deprecated)]

#[expect(deprecated)]

timsaucer · 2026-04-13T14:39:08Z

 }

 impl PhysicalOptimizerRule for CoalesceTake {
+    #[allow(deprecated)]


Suggested change

#[allow(deprecated)]

#[expect(deprecated)]

timsaucer · 2026-04-13T14:41:30Z

 "futures",
 "itertools 0.14.0",
 "log",
- "object_store",


We cannot get past having 2 versions of object_store until opendal 0.56 is released

opendal has been released, let's go!

codecov · 2026-04-14T21:29:40Z

Codecov Report

❌ Patch coverage is 90.83333% with 11 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/io/exec/knn.rs	81.25%	3 Missing ⚠️
...rc/dataset/mem_wal/memtable/scanner/exec/vector.rs	0.00%	2 Missing ⚠️
rust/lance-index/src/scalar/btree.rs	0.00%	1 Missing ⚠️
...e/src/dataset/mem_wal/memtable/scanner/exec/fts.rs	50.00%	1 Missing ⚠️
...ce/src/dataset/mem_wal/scanner/exec/bloom_guard.rs	50.00%	1 Missing ⚠️
...src/dataset/mem_wal/scanner/exec/coalesce_first.rs	50.00%	1 Missing ⚠️
...e/src/dataset/mem_wal/scanner/exec/filter_stale.rs	50.00%	1 Missing ⚠️
rust/lance/src/io/exec/filtered_read.rs	50.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

Upgrade arrow 57 -> 58, datafusion 52.1 -> 53.0, parquet 57 -> 58, geoarrow 0.7 -> 0.8, geodatafusion 0.3 -> 0.4, and datafusion-python >=52,<53 -> >=53,<54. Adapt to DataFusion 53 breaking changes: - PlanProperties now wrapped in Arc (ExecutionPlan::properties returns &Arc<PlanProperties>) - ExecutionPlan::statistics() removed from trait - HashJoinExec::try_new gains null_aware: bool parameter - SimplifyContext::new replaced with SimplifyContext::default() builder - PGBitwiseNot renamed to BitwiseNot (sqlparser update) - ScalarValue::RunEndEncoded new variant handled - MemoryReservation methods no longer require &mut self Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…3 upgrade The DataFusion 53 upgrade guide specifies that statistics() should be replaced with partition_statistics(), not just removed. Three ExecutionPlan implementations had meaningful row-count statistics that were lost in the initial upgrade: - LanceScanExec: computes row counts from fragment metadata - ANNIvfPartitionExec: reports minimum_nprobes as exact row count - ScalarIndexExec: reports Precision::Exact(2) for index expression results Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Update Python bindings for pyo3 0.26→0.28 API changes: - FromPyObject now takes two lifetime params and uses `extract` instead of `extract_bound` - Replace deprecated `downcast()` with `cast()` across all files - Add `from_py_object` / `skip_from_py_object` to all Clone pyclass types - Replace deprecated `PyCapsule::reference` with `pointer_checked` Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

DataFusion 53 changed the EnsureCooperative optimizer to skip wrapping leaf nodes when a cooperative ancestor (CoalescePartitionsExec) already exists, so CooperativeExec no longer appears around LanceRead in these plans. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

timsaucer · 2026-04-15T11:31:43Z

It looks like opendal should be releasing soon so maybe we end up holding for a little: apache/opendal#7283

timsaucer · 2026-04-16T14:30:28Z

Ok, we're in the green! I've identified a couple of things to follow up on / check but otherwise looking good IMO. @wjones127 / @westonpace do you have thoughts about merging or waiting for the opendal update before we move forward?

gotocoding-DB · 2026-04-24T13:43:46Z

Hey @timsaucer (sorry for so many pings, Tim)! Do we have any unpublished news here?
Or do we still wait review from @wjones127 and @westonpace?

westonpace

Thanks for this work Tim. I'll see if I can get some opinion on OpenDAL. Is that the only reason this is in draft? I'd like to get this in as some of our users are encountering apache/datafusion@a6a4df9

westonpace · 2026-04-27T12:54:26Z

                None,
                PartitionMode::CollectLeft,
                NullEquality::NullEqualsNull,
+                false,


null-aware only matters for very specific anti-join patterns and we never configure an anti-join today.

westonpace · 2026-04-27T12:57:07Z

                    None,
                    PartitionMode::CollectLeft,
                    NullEquality::NullEqualsNull,
+                    false,


Ditto, not a concern

westonpace · 2026-04-27T12:57:55Z

    }

    /// Take row indices produced by input plan from the dataset (with projection)
+    #[allow(deprecated)]


Yeah, let's use expect.

westonpace · 2026-04-27T12:59:54Z

-    fn statistics(&self) -> DataFusionResult<Statistics> {
-        Ok(Statistics {
-            num_rows: Precision::Inexact(
-                self.query.k * self.query.refine_factor.unwrap_or(1) as usize,
-            ),
-            ..Statistics::new_unknown(self.schema().as_ref())
-        })
-    }


Did statistics get fully removed now? I know it had been deprecated for a bit. Losing this is probably fine for now if we make sure to add a follow-up.

westonpace · 2026-04-27T13:32:00Z

Ok, so after more review I'm fairly certain that we don't ever use object_store indirectly via Datafusion, so from a pure lance perspective I see no problem with having multiple versions of object_store.

I think the biggest headache would be for users that are using Datafusion themselves and using Lance via Rust (that might be you Tim 😆).

In this case if you were adding your object_store implementation to Datafusion's object store registry and then also trying to add it to Lance's object store registry you would run into some headaches.

However, if you're just using object_store indirectly (e.g. you use Parquet scanners via Datafusion and you give Lance object store URIs when you connect) you'd probably be ok.

westonpace · 2026-04-29T02:42:41Z

Wanting to get this in soon so I've rebased and submitted as #6638

github-actions Bot added A-python Python bindings A-java Java bindings + JNI labels Apr 13, 2026

timsaucer commented Apr 13, 2026

View reviewed changes

timsaucer changed the title ~~Update Arrow to 58 and DataFusion to 53~~ chore: dependency update Arrow to 58 and DataFusion to 53 Apr 14, 2026

github-actions Bot added the chore label Apr 14, 2026

timsaucer and others added 8 commits April 15, 2026 07:21

cargo fmt

21d26c4

update pyo3 dep

b4f00a4

update method for extracting ffi objects

b2dfd77

cargo fmt

5140c22

timsaucer force-pushed the tsaucer/arrow57-datafusion53 branch from 4c70c57 to 5140c22 Compare April 15, 2026 11:23

docstring update

2657b1f

westonpace approved these changes Apr 27, 2026

View reviewed changes

timsaucer closed this May 22, 2026

Uh oh!

Conversation

timsaucer commented Apr 13, 2026

Uh oh!

github-actions Bot commented Apr 13, 2026

Uh oh!

timsaucer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

timsaucer commented Apr 15, 2026

Uh oh!

timsaucer commented Apr 16, 2026

Uh oh!

gotocoding-DB commented Apr 24, 2026

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

westonpace commented Apr 27, 2026

Uh oh!

westonpace commented Apr 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov Bot commented Apr 14, 2026 •

edited

Loading