build(deps): upgrade datafusion, arrow, and other deps#504
build(deps): upgrade datafusion, arrow, and other deps#504xushiyan merged 3 commits intoapache:mainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This PR upgrades major dependencies including DataFusion (46.0.0 → 51.0.0), Arrow (54.2.0 → 57.1.0), PyO3 (0.23 → 0.26), and several other dependencies. It also updates the Rust toolchain version and makes necessary API adaptations for the upgraded libraries.
Key Changes
- Upgraded DataFusion from 46.0.0 to 51.0.0 and Arrow from 54.2.0 to 57.1.0, with corresponding API changes for
FileScanConfigbuilder pattern andExpr::Literalvariants - Upgraded PyO3 from 0.23 to 0.26, migrating return types from
PyObjecttoPy<PyAny>with.unbind()pattern - Updated Rust toolchain to version 1.88 and adjusted type signatures (e.g.,
usizetou64for file sizes)
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| rust-toolchain.toml | Updates Rust toolchain channel from 1.87 to 1.88 |
| Cargo.toml | Updates rust-version and upgrades arrow, datafusion, object_store, prost, bytes, chrono, strum, and url dependencies |
| python/Cargo.toml | Upgrades pyo3 dependency from ~0.23 to ~0.26 |
| python/src/internal.rs | Adapts Python bindings for PyO3 0.26: changes return types from PyObject to Py<PyAny>, adds .unbind() calls, and updates base_file_size from usize to u64 |
| crates/datafusion/src/lib.rs | Adapts to DataFusion 51.0.0 API: uses FileScanConfigBuilder instead of FileScanConfig::new(), updates Expr::Literal pattern matching, adds crypto field, and converts to FileGroup type |
| crates/datafusion/src/util/expr.rs | Updates Expr::Literal pattern matching to include second parameter |
| crates/core/src/storage/mod.rs | Updates ParquetObjectReader::new() API to accept obj_path instead of obj_meta and chain with .with_file_size() |
| crates/core/src/storage/file_metadata.rs | Changes size field type from usize to u64 for compatibility with object_store API |
| crates/core/src/avro_to_arrow/schema.rs | Adds support for new Arrow decimal types: Decimal32 and Decimal64 |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #504 +/- ##
=======================================
Coverage 87.62% 87.63%
=======================================
Files 65 65
Lines 4106 4109 +3
=======================================
+ Hits 3598 3601 +3
Misses 508 508 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Description
Upgrade:
How are the changes test-covered