feat(python): expose rust writer as additional engine v2 #1891

ion-elgreco · 2023-11-20T21:44:49Z

Description

Adds rust writer as additional engine in python
Adds overwrite schema functionality to the rust writer. @roeap feel free to point out improvements 😄

A couple gaps will exist between current Rust writer and pyarrow writer. We will have to solve this in a later PR:

Replacewhere (partition filter / predicate) overwrite
(users however can solve this by doing DeltaTabel.delete and then append)

Related Issue(s)

closes Add a Rust-backed engine for write_deltalake #1861

@junjunjd

Exposes added `convert to delta` functionality by @junjunjd to Python API. - closes delta-io#1767 --------- Co-authored-by: Robert Pack <42610831+roeap@users.noreply.github.com>

# Description This refactors the merge operation to use DataFusion's DataFrame and LogicalPlan APIs The NLJ is eliminated and the query planner can pick the optimal join operator. This also enables the operation to use multiple threads and should result in significant speed up. Merge is still limited to using a single thread in some area. When collecting benchmarks, I encountered multiple OoM issues with Datafusion's hash join implementation. There are multiple tickets upstream open regarding this. For now, I've limited the number of partitions to just 1 to prevent this. Predicates passed as SQL are also easier to use now. Manual casting was required to ensure data types were aligned. Now the logical plan will perform type coercion when optimizing the plan. # Related Issues - enhances delta-io#850 - closes delta-io#1790 - closes delta-io#1753

# Description Implements benchmarks that are similar to Spark's Delta benchmarks. Enable us to have a standard benchmark to measure improvements to merge and some pieces can be factored out to build a framework for bench marking delta workflows.

This reverts commit 57565b5.

rtyler · 2023-11-21T05:53:50Z

@ion-elgreco should #1872 be closed with this being its replacement?

ion-elgreco · 2023-11-21T09:44:57Z

@rtyler yes will do that!

Signed-off-by: Nikolay Ulmasov <ulmasov@hotmail.com>

# Description Adds docs on how to append, overwrite, delete rows, and Z Order Delta tables. Will add much more detailed pages in the future. Just getting the high-level skeleton of the docs developed.

…#1836) # Description get_actions wrongly assumes that partition_columns from schema and partitionValues from log must be the same. This is not true since partition_columns are logical column names while partitionValues are physical column names. Tests pending # Related Issue(s) - closes delta-io#1835 # Documentation https://github.com/delta-io/delta/blob/master/PROTOCOL.md#writer-requirements-for-column-mapping "Track partition values and column level statistics with the physical name of the column in the transaction log." --------- Co-authored-by: Will Jones <willjones127@gmail.com>

@stinodego

…iter/merge (delta-io#1820) This ports some functionality that @stinodego and I had worked on in Polars. Where we converted a pyarrow schema to a compatible delta schema. It converts the following: - uint -> int - timestamp(any timeunit) -> timestamp(us) I adjusted the functionality to do schema conversion from large to normal when necessary, which is still needed in MERGE as workaround delta-io#1753. Additional things I've added: - Schema conversion for every input in write_deltalake/merge - Add Pandas dataframe conversion - Add Pandas dataframe as input in merge - closes delta-io#686 - closes delta-io#1467 --------- Co-authored-by: Will Jones <willjones127@gmail.com>

roeap

looking good, just some minor comments :)

roeap · 2023-11-25T12:42:21Z

crates/deltalake-core/src/operations/write.rs

+                    metadata.schema = schema.clone().try_into().unwrap();
+                    let metadata_action = Metadata::try_from(metadata).unwrap();
+                    actions.push(Action::Metadata(metadata_action));


let's handle these unwraps.

@roeap Can you check if the change is ok? I think you want that it returns the error, so I used the question mark operator

yes, exactly right :)

python/deltalake/writer.py

scarman-db · 2023-11-29T17:40:36Z

crates/deltalake-core/src/operations/write.rs

+                    .or_else(|_| this.snapshot.arrow_schema())
+                    .unwrap_or(schema.clone());
+
+                if schema != table_schema {


Do the schemas here have Eq or PartialEq on them? This might not be as straight forward as this.

It has PartialEq on them. This piece was also reused from src/lib.rs in the python module

@roeap

- Adds rust writer as additional engine in python - Adds overwrite schema functionality to the rust writer. @roeap feel free to point out improvements 😄 A couple gaps will exist between current Rust writer and pyarrow writer. We will have to solve this in a later PR: - Replacewhere (partition filter / predicate) overwrite (users however can solve this by doing DeltaTabel.delete and then append) - closes delta-io#1861 --------- Signed-off-by: Nikolay Ulmasov <ulmasov@hotmail.com> Co-authored-by: Robert Pack <robstar.pack@gmail.com> Co-authored-by: Robert Pack <42610831+roeap@users.noreply.github.com> Co-authored-by: David Blajda <db@davidblajda.com> Co-authored-by: Nikolay Ulmasov <ulmasov@hotmail.com> Co-authored-by: Matthew Powers <matthewkevinpowers@gmail.com> Co-authored-by: Thomas Frederik Hoeck <44194839+thomasfrederikhoeck@users.noreply.github.com> Co-authored-by: Adrian Ehrsam <5270024+aersam@users.noreply.github.com> Co-authored-by: Will Jones <willjones127@gmail.com> Co-authored-by: Marijn Valk <marijncv@hotmail.com>

ion-elgreco and others added 22 commits November 18, 2023 15:38

first version

ca2acb8

add try from uri with storage options

3f55470

Start to enable overwrite_schema

714cc56

add tests to check rust py03 writer

52f0d6a

remove comment

c067d4d

rename and clean up

5c5f247

add float type support in partition cols

717b7c7

check for pandas

dae361e

add support for name and desc

3052f12

fmt

01f0194

improve tests and add config support

9911574

parametrize write benchmark

a410dfa

add LargeUtf8 support in partition stringify

8c976b6

refactor: express log schema in delta types

57565b5

feat(python): expose convert_to_deltalake (delta-io#1842)

49a298b

Exposes added `convert to delta` functionality by @junjunjd to Python API. - closes delta-io#1767 --------- Co-authored-by: Robert Pack <42610831+roeap@users.noreply.github.com>

Revert "refactor: express log schema in delta types"

07113c6

This reverts commit 57565b5.

Merge branch 'main' into feat/expose_rust_writer_as_optional_engine

3a8c026

formatting

3e25561

use fromstr

d9a4ce0

add overwrite_schema support

e3c7189

ion-elgreco requested review from wjones127, fvaleye, roeap and rtyler as code owners November 20, 2023 21:44

github-actions bot added binding/python Issues for the Python package binding/rust Issues for the Rust crate crate/core labels Nov 20, 2023

fix clippy of unrelated code ?

5af1251

cargo fmt

3744384

ion-elgreco mentioned this pull request Nov 21, 2023

feat(python): expose rust writer as additional engine #1872

Closed

ion-elgreco modified the milestone: python v0.14 Nov 22, 2023

r3stl355 and others added 12 commits November 25, 2023 09:38

resolve delta-io#1860

2e1f0c9

Signed-off-by: Nikolay Ulmasov <ulmasov@hotmail.com>

docs: on append, overwrite, delete and z-ordering (delta-io#1897)

fee4d77

# Description Adds docs on how to append, overwrite, delete rows, and Z Order Delta tables. Will add much more detailed pages in the future. Just getting the high-level skeleton of the docs developed.

docs: update python docs link in readme.md (delta-io#1899)

3173ad7

ci: run doctest in CI for Python API examples (delta-io#1840)

c38b518

resolve merge

573e8fe

fix bug from conflict

bb5815a

Merge branch 'main' into fix/expose_writer_rust_to_python_v2

8c56194

remove commented code

381df0d

expose predicate, handle error better, and add overloads

3ab7687

fmt

e73eea3

roeap reviewed Nov 25, 2023

View reviewed changes

ion-elgreco added 2 commits November 25, 2023 16:14

use ? instead of unwrap

b4f9695

Merge branch 'main' into fix/expose_writer_rust_to_python_v2

50e7257

rtyler approved these changes Nov 29, 2023

View reviewed changes

scarman-db reviewed Nov 29, 2023

View reviewed changes

ion-elgreco merged commit e6ad2e0 into delta-io:main Nov 29, 2023
24 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(python): expose rust writer as additional engine v2 #1891

feat(python): expose rust writer as additional engine v2 #1891

ion-elgreco commented Nov 20, 2023 •

edited

Loading

rtyler commented Nov 21, 2023

ion-elgreco commented Nov 21, 2023

roeap left a comment

roeap Nov 25, 2023

ion-elgreco Nov 25, 2023

roeap Nov 25, 2023

scarman-db Nov 29, 2023

ion-elgreco Nov 29, 2023

feat(python): expose rust writer as additional engine v2 #1891

feat(python): expose rust writer as additional engine v2 #1891

Conversation

ion-elgreco commented Nov 20, 2023 • edited Loading

Description

Related Issue(s)

rtyler commented Nov 21, 2023

ion-elgreco commented Nov 21, 2023

roeap left a comment

Choose a reason for hiding this comment

roeap Nov 25, 2023

Choose a reason for hiding this comment

ion-elgreco Nov 25, 2023

Choose a reason for hiding this comment

roeap Nov 25, 2023

Choose a reason for hiding this comment

scarman-db Nov 29, 2023

Choose a reason for hiding this comment

ion-elgreco Nov 29, 2023

Choose a reason for hiding this comment

ion-elgreco commented Nov 20, 2023 •

edited

Loading