feat(python): Write dataframes as delta tables

[chitralverma]

Allows users to write a Polars DataFrame as a delta lake table.

[Currently, this is a work in progress as its blocked by https://github.com/delta-io/delta-rs/pull/1044]

Checklist

• resolve like in scans if URI is provided
• delta-polars type validations
• Update min supported delta version after deltalake supports large types.
• verify behaviour on cloud stores
• test cases
• examples
• build isolated whl for final tests
• update with the latest master
• Verify for non-large types

Closes #2858 and #7574

[stinodego]

Hey @chitralverma , just a quick check: I see you have a PR out at delta-rs for supporting LargeBinary/LargeString. Have you also thought about supporting LargeList? This is another 'blocker' I had identified.

[chitralverma]

let me add that to the existing PR as well

[stinodego]

Maybe casting unsigned -> signed integers should also be added on the delta-rs side, then?

[chitralverma]

Maybe casting unsigned -> signed integers should also be added on the delta-rs side, then?

This I think we will have to do on our side I think. but I can check with the delta team

[stinodego]

Maybe casting unsigned -> signed integers should also be added on the delta-rs side, then?

This I think we will have to do on our side I think. but I can check with the delta team

We can definitely do that on the Polars side, but if you're mapping Arrow types to Delta supported types in delta-rs anyway, it would make sense to include casting unsigned integers as well.

I'd suggest trying to cast to a signed integer of the same bit size, and raising an error when this is not possible.

[chitralverma]

Maybe casting unsigned -> signed integers should also be added on the delta-rs side, then?

This I think we will have to do on our side I think. but I can check with the delta team

We can definitely do that on the Polars side, but if you're mapping Arrow types to Delta supported types in delta-rs anyway, it would make sense to include casting unsigned integers as well.

I'd suggest trying to cast to a signed integer of the same bit size, and raising an error when this is not possible.

Done! added uint* and float16.

Now the following remain,

    DataType::Null => {}
    DataType::Time32(_) => {}
    DataType::Time64(_) => {}
    DataType::Duration(_) => {}
    DataType::Interval(_) => {}
    DataType::Union(_, _, _) => {}
    DataType::Dictionary(_, _) => {} # polars categorical gets converted to this
    DataType::RunEndEncoded(_, _) => {}

[chitralverma]

long time!

So we have a new rust release for delta-rs. A python release should be on the way soon, so we can actually have this feature in polars as well.

[chitralverma]

Finally, the delta-rs release is in place and I can start working on this again

[chitralverma]

@stinodego @ritchie46 finally marked this as ready for review!
please check it out at your convenience.

[stinodego]

Amazing! Reviewing this later today.

[ritchie46]

I read the code. Not that familiar with delta-tables' API, so I will leave this one to @stinodego.

[stinodego]

Looks great! Just a few remarks.

[chitralverma]

Looks great! Just a few remarks.

@stinodego thanks for the review. made the requested changes.

[stinodego]

Great! Looks like all is in order.