Upgrade to pyo3 0.21.0

[abhiaagarwal]

Description

pyo3 recently released 0.21.0, which came with some big breaking changes (see: migration guide). This requires rewriting a good chunk of functions, but brings increased performance + memory safety. Also comes with an experimental async feature, to expose rust async functions to python async (with no interop required).

Related Issue(s)

[abhiaagarwal]

take

[abhiaagarwal]

this turned out to be way more complicated than I anticipated, as it depends on datafusion updating its pyo3 to 0.21 (tracking apache/datafusion#9808) and arrow updating (apache/arrow-rs#5566). I patched all the dependencies and successfully compiled, but a bunch of tests related to datafusion are now failing.

abhiaagarwal@785dada

[ion-elgreco]

@abhiaagarwal let me guess, are you seeing tests failing with?

"Wrong number of children"

[abhiaagarwal]

@ion-elgreco exactly right :) how did you know?

[ion-elgreco]

@abhiaagarwal magic 🪄 haha

No I also tried bumping it, and @universalmind303 but we all got that error.

I opened an issue in Datafusion repo this morning, to see if they know what the issue might be

[abhiaagarwal]

@ion-elgreco yeah, I have 32 tests failing.

failures:
    data_catalog::storage::tests::test_query_table
    delta_datafusion::expr::test::test_expr_sql
    delta_datafusion::tests::delta_scan_case_sensitive
    delta_datafusion::tests::delta_scan_mixed_partition_order
    delta_datafusion::tests::delta_table_provider_with_config
    operations::delete::tests::test_delete_nested
    operations::delete::tests::test_delete_null
    operations::delete::tests::test_delete_on_mixed_columns
    operations::delete::tests::test_delete_on_nonpartition_column
    operations::delete::tests::test_delete_on_partition_column
    operations::merge::tests::test_merge
    operations::merge::tests::test_merge_case_sensitive
    operations::merge::tests::test_merge_delete_matched
    operations::merge::tests::test_merge_delete_not_matched
    operations::merge::tests::test_merge_empty_table
    operations::merge::tests::test_merge_no_alias
    operations::merge::tests::test_merge_partition_filtered
    operations::merge::tests::test_merge_partitions
    operations::merge::tests::test_merge_partitions_skipping
    operations::merge::tests::test_merge_pushdowns
    operations::merge::tests::test_merge_pushdowns_partitioned
    operations::merge::tests::test_merge_str
    operations::merge::tests::test_merge_with_alias_mix
    operations::update::tests::test_update_case_sensitive
    operations::update::tests::test_update_no_predicate
    operations::update::tests::test_update_non_partition
    operations::update::tests::test_update_null
    operations::update::tests::test_update_partitions
    operations::write::tests::test_nested_struct
    operations::write::tests::test_replace_where
    operations::write::tests::test_replace_where_partitioned
    operations::write::tests::test_write_different_types

test result: FAILED. 214 passed; 32 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.76s

[universalmind303]

@abhiaagarwal @ion-elgreco those are valid errors. Datafusion added some new optimization rules that break a lot of older plans. I am actively working on getting all of the code upgraded to be compatible with this rule. #2249. Should have this reviewable within the next few days.

[ion-elgreco]

@universalmind303 I opened an issue in Datafusion since I was curious if they knew the issue, but I haven't gotten time to look at it yet.

But it might be useful to see: apache/datafusion#9895

[ion-elgreco]

@abhiaagarwal you should be able to bump arrow now, just merged @universalmind303's chore PR

[abhiaagarwal]

@ion-elgreco arrow is not updated yet, and the branch with the 0.21 pyo3 update is also broken. I'm gonna hold off until things stabilize