chore: Update to arrow/parquet 59.0.0 by alamb · Pull Request #22744 · apache/datafusion

alamb · 2026-06-03T21:14:48Z

Which issue does this PR close?

related to Release arrow-rs / parquet Major version 59.0.0 (May 2026) arrow-rs#9110

WIP: I am using this PR to test the arrow release

Rationale for this change

Update to latest version of arrow/parquet

What changes are included in this PR?

Update to arrow/parquet 59.0.0

Are these changes tested?

By CI

Are there any user-facing changes?

New dependency

alamb · 2026-06-03T21:35:29Z

                .bloom_filter_properties(&ColumnPath::from(""))
                .expect("expected bloom properties!")
-                .fpp,
+                .fpp(),


These fields are made private in

refactor: make BloomFilterProperties fpp/ndv private with accessors arrow-rs#9969

alamb · 2026-06-03T21:37:05Z

        };
        if let Some(bloom_filter_ndv) = bloom_filter_ndv {
-            builder = builder.set_bloom_filter_ndv(*bloom_filter_ndv);
+            builder = builder.set_bloom_filter_max_ndv(*bloom_filter_ndv);


due to feat(parquet): add BloomFilterPropertiesBuilder arrow-rs#9877 which deprecated set_bloom_filtr_ndv

alamb · 2026-06-03T21:38:27Z

-                ndv: DEFAULT_BLOOM_FILTER_NDV
-            }),
+            Some(
+                &BloomFilterProperties::builder()


Properties are now built with builder:

feat(parquet): add BloomFilterPropertiesBuilder arrow-rs#9877

alamb · 2026-06-03T21:38:50Z

-            &[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
-            &[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 5, 6],
-        ]))],
+        vec![Arc::new(


from impl was removed because it could panic

Replace From<Vec<_>> impls with TryFroms for FixedSizeBinaryArray arrow-rs#10019

alamb · 2026-06-03T21:39:39Z

-                scale: 2,
-                precision: 9,
-            })
+            .with_logical_type(LogicalType::decimal(2, 9))


have to use new helpers added in

Add helper functions to create LogicalType struct variants arrow-rs#9996
As the types were changed in

Use Thrift macro to generate Parquet LogicalType serialization code arrow-rs#9997

alamb · 2026-06-03T21:40:05Z

 ]

-[[package]]
-name = "integer-encoding"


yay for removing older deps

alamb · 2026-06-03T21:40:12Z

 "cfg-if",
 ]

-[[package]]


no more thrift! We now use the entirely new thrift encoder and not the thrift generator

github-actions · 2026-06-03T21:47:36Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
error: running 'cargo update' on crate 'datafusion' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^59.0.0"`
candidate versions found which didn't match: 58.3.0, 58.2.0, 58.1.0, ...
location searched: crates.io index
required by package `datafusion v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/core)`
    ... which satisfies path dependency `datafusion` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks/local-datafusion-53_1_0-default-44d05ba70c24be25)`

-----
error: failed to update dependencies for crate datafusion v53.1.0
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/core --features array_expressions,avro,backtrace,bzip2,compression,crypto_expressions,datafusion-datasource-avro,datafusion-datasource-parquet,datafusion-functions-nested,datafusion-sql,datetime_expressions,default,encoding_expressions,extended_tests,flate2,force_hash_collisions,liblzma,math_expressions,nested_expressions,parquet,parquet_encryption,recursive_protection,regex_expressions,serde,sql,sqlparser,string_expressions,unicode_expressions,zstd &&
          cargo update

error: running 'cargo update' on crate 'datafusion-common' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^59.0.0"`
candidate versions found which didn't match: 58.3.0, 58.2.0, 58.1.0, ...
location searched: crates.io index
required by package `datafusion-common v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/common)`
    ... which satisfies path dependency `datafusion-common` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks/local-datafusion_common-53_1_0-default-b6ddb7c2d8eda5c6)`

-----
error: failed to update dependencies for crate datafusion-common v53.1.0
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/common --features backtrace,force_hash_collisions,object_store,parquet,parquet_encryption,recursive_protection,sql,sqlparser &&
          cargo update

error: running 'cargo update' on crate 'datafusion-datasource-parquet' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^59.0.0"`
candidate versions found which didn't match: 58.3.0, 58.2.0, 58.1.0, ...
location searched: crates.io index
required by package `datafusion-datasource-parquet v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource-parquet)`
    ... which satisfies path dependency `datafusion-datasource-parquet` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks/local-datafusion_datasource_parquet-53_1_0-default-6b9780d27fce52ad)`

-----
error: failed to update dependencies for crate datafusion-datasource-parquet v53.1.0
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/datasource-parquet --features parquet_encryption &&
          cargo update

error: running 'cargo update' on crate 'datafusion-physical-expr' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^59.0.0"`
candidate versions found which didn't match: 58.3.0, 58.2.0, 58.1.0, ...
location searched: crates.io index
required by package `datafusion-physical-expr v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr)`
    ... which satisfies path dependency `datafusion-physical-expr` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks/local-datafusion_physical_expr-53_1_0-default-10f6c42332b9f7a0)`

-----
error: failed to update dependencies for crate datafusion-physical-expr v53.1.0
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/physical-expr --features proto,recursive_protection &&
          cargo update

error: running 'cargo update' on crate 'datafusion-physical-expr-common' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^59.0.0"`
candidate versions found which didn't match: 58.3.0, 58.2.0, 58.1.0, ...
location searched: crates.io index
required by package `datafusion-physical-expr-common v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr-common)`
    ... which satisfies path dependency `datafusion-physical-expr-common` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks/local-datafusion_physical_expr_common-53_1_0-default-f9b0c7d7466ac2ba)`

-----
error: failed to update dependencies for crate datafusion-physical-expr-common v53.1.0
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/physical-expr-common --features default,proto &&
          cargo update

error: running 'cargo update' on crate 'datafusion-physical-plan' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^59.0.0"`
candidate versions found which didn't match: 58.3.0, 58.2.0, 58.1.0, ...
location searched: crates.io index
required by package `datafusion-physical-plan v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-plan)`
    ... which satisfies path dependency `datafusion-physical-plan` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks/local-datafusion_physical_plan-53_1_0-default-b18d4d8250a7053b)`

-----
error: failed to update dependencies for crate datafusion-physical-plan v53.1.0
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/physical-plan --features force_hash_collisions,proto,test_utils,tokio_coop,tokio_coop_fallback &&
          cargo update

error: running 'cargo update' on crate 'datafusion-spark' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^59.0.0"`
candidate versions found which didn't match: 58.3.0, 58.2.0, 58.1.0, ...
location searched: crates.io index
required by package `datafusion-spark v53.1.0 (/home/runner/work/datafusion/datafusion/datafusion/spark)`
    ... which satisfies path dependency `datafusion-spark` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks/local-datafusion_spark-53_1_0-default-6f3319ca5d884369)`

-----
error: failed to update dependencies for crate datafusion-spark v53.1.0
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/spark --features core,datafusion,default &&
          cargo update

error: aborting due to failure to run 'cargo update' for crate datafusion v53.1.0

chore: Update to arrow/parquet 59.0.0

843c6ce

alamb mentioned this pull request Jun 3, 2026

Release arrow-rs / parquet Major version 59.0.0 (May 2026) apache/arrow-rs#9110

Open

12 tasks

Update code for API changes

55c7eb3

github-actions Bot added physical-expr Changes to the physical-expr crates core Core DataFusion crate common Related to common crate datasource Changes to the datasource crate physical-plan Changes to the physical-plan crate spark labels Jun 3, 2026

alamb commented Jun 3, 2026

View reviewed changes

github-actions Bot added the auto detected api change Auto detected API change label Jun 3, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Update to arrow/parquet 59.0.0#22744

chore: Update to arrow/parquet 59.0.0#22744
alamb wants to merge 2 commits into
apache:mainfrom
alamb:alamb/update_arrow_59

alamb commented Jun 3, 2026

Uh oh!

alamb Jun 3, 2026

Uh oh!

alamb Jun 3, 2026

Uh oh!

alamb Jun 3, 2026

Uh oh!

alamb Jun 3, 2026

Uh oh!

alamb Jun 3, 2026

Uh oh!

alamb Jun 3, 2026

Uh oh!

alamb Jun 3, 2026

Uh oh!

github-actions Bot commented Jun 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

alamb commented Jun 3, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant