Skip to content

build(deps): upgrade datafusion, arrow, and other deps#504

Merged
xushiyan merged 3 commits intoapache:mainfrom
xushiyan:upgrade-deps
Jan 1, 2026
Merged

build(deps): upgrade datafusion, arrow, and other deps#504
xushiyan merged 3 commits intoapache:mainfrom
xushiyan:upgrade-deps

Conversation

@xushiyan
Copy link
Member

@xushiyan xushiyan commented Jan 1, 2026

Description

Upgrade:

  • Datafusion
  • Arrow
  • PyO3
  • Rust lang
  • and others

How are the changes test-covered

  • N/A
  • Automated tests (unit and/or integration tests)
  • Manual tests
    • Details are described below

Copilot AI review requested due to automatic review settings January 1, 2026 22:05
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR upgrades major dependencies including DataFusion (46.0.0 → 51.0.0), Arrow (54.2.0 → 57.1.0), PyO3 (0.23 → 0.26), and several other dependencies. It also updates the Rust toolchain version and makes necessary API adaptations for the upgraded libraries.

Key Changes

  • Upgraded DataFusion from 46.0.0 to 51.0.0 and Arrow from 54.2.0 to 57.1.0, with corresponding API changes for FileScanConfig builder pattern and Expr::Literal variants
  • Upgraded PyO3 from 0.23 to 0.26, migrating return types from PyObject to Py<PyAny> with .unbind() pattern
  • Updated Rust toolchain to version 1.88 and adjusted type signatures (e.g., usize to u64 for file sizes)

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated no comments.

Show a summary per file
File Description
rust-toolchain.toml Updates Rust toolchain channel from 1.87 to 1.88
Cargo.toml Updates rust-version and upgrades arrow, datafusion, object_store, prost, bytes, chrono, strum, and url dependencies
python/Cargo.toml Upgrades pyo3 dependency from ~0.23 to ~0.26
python/src/internal.rs Adapts Python bindings for PyO3 0.26: changes return types from PyObject to Py<PyAny>, adds .unbind() calls, and updates base_file_size from usize to u64
crates/datafusion/src/lib.rs Adapts to DataFusion 51.0.0 API: uses FileScanConfigBuilder instead of FileScanConfig::new(), updates Expr::Literal pattern matching, adds crypto field, and converts to FileGroup type
crates/datafusion/src/util/expr.rs Updates Expr::Literal pattern matching to include second parameter
crates/core/src/storage/mod.rs Updates ParquetObjectReader::new() API to accept obj_path instead of obj_meta and chain with .with_file_size()
crates/core/src/storage/file_metadata.rs Changes size field type from usize to u64 for compatibility with object_store API
crates/core/src/avro_to_arrow/schema.rs Adds support for new Arrow decimal types: Decimal32 and Decimal64

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@codecov
Copy link

codecov bot commented Jan 1, 2026

Codecov Report

❌ Patch coverage is 92.85714% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 87.63%. Comparing base (72b376e) to head (6724907).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
crates/datafusion/src/util/expr.rs 50.00% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #504   +/-   ##
=======================================
  Coverage   87.62%   87.63%           
=======================================
  Files          65       65           
  Lines        4106     4109    +3     
=======================================
+ Hits         3598     3601    +3     
  Misses        508      508           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@xushiyan xushiyan added this to the release-0.5.0 milestone Jan 1, 2026
@xushiyan xushiyan merged commit 7a275c6 into apache:main Jan 1, 2026
15 checks passed
xushiyan added a commit that referenced this pull request Jan 2, 2026
@xushiyan xushiyan linked an issue Jan 3, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Upgrade datafusion, arrow, and related deps and code

2 participants