Add deltalake support in AWS S3 with Pandas #1834

fvaleye · 2022-12-02T17:22:32Z

Feature or Bugfix

Feature

Detail

Delta Lake is an open-source storage framework, a Python library delta-rs is available to access it using Pandas. Integrating deltalake in pandas makes the support work out of the box for pandas users in lambda.
The first version is only a read operation converting into a Pandas DataFrame from AWS S3 or AWS Glue, other methods could be added (the writer is still experimental atm).

Relates

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

malachi-constant · 2022-12-02T18:38:05Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: fea90fb
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant · 2022-12-02T22:20:06Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 59b6757
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant · 2022-12-02T22:25:33Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 2d409c6
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant · 2022-12-05T19:22:50Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 23b5e4b
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant · 2022-12-05T20:37:25Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: bf211fa
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jaidisido · 2022-12-06T11:41:34Z

awswrangler/s3/_read_deltalake.py

+_logger: logging.Logger = logging.getLogger(__name__)
+
+
+def read_deltalake(


Keeping with the spirit of awswrangler, perhaps we should combine read_deltalake and read_deltalake_from_glue into a single method. For instance, wr.s3.read_parquet currently handles both reading from S3 directly or via the Glue catalog.

We did face some overloading issues with this strategy in the past though, so happy to be convinced otherwise and keen to hear others thoughts on this

I kept the implementation separated for the moment since deltalake already has its own way of getting the S3 location of a Glue table in Rust.

As a side note: we could also remove the Glue integration to start only with S3.

jaidisido · 2022-12-06T11:47:14Z

awswrangler/s3/_read_deltalake.py

+    without_files: bool = False,
+    partitions: Optional[List[Tuple[str, str, Any]]] = None,
+    columns: Optional[List[str]] = None,
+    filesystem: Optional[Union[str, pa_fs.FileSystem]] = None,


awswrangler is entirely based on boto3 for credentials management. Potentially we could build the Pyarrow FileSystem from the boto3 session. However, it might not be necessary altogether (see comment on to_pandas below).

I changed the implementation by relying only on the boto3 session to build the storage_options for the storage backend of the DeltaTable while keeping the possibility to add more configuration via s3_additional_kwargs.

jaidisido · 2022-12-06T11:53:56Z

awswrangler/s3/_read_deltalake.py

+        storage_options=storage_options,
+        without_files=without_files,
+    )
+    return table.to_pandas(partitions=partitions, columns=columns, filesystem=filesystem)


Looking into the source for this to_pandas implementation, it seems that it goes S3 -> Arrow Table -> Pandas DF. The user-provided conversion arguments seem limited though (partitions, columns, filesystem). Within awswrangler we tend to provide some sane defaults for this conversion and then simply delegate to the user to provide whatever arguments they want to override via a pyarrow_additional_kwargs argument. That dict is then forwarded to the to_pandas method. For consistency we could follow something similar here.

Good idea! I created pyarrow_additional_kwargs and s3_additional_kwargs based on your suggestion!

…ers to provide more flexibility for the user when reading a DeltaTable

malachi-constant · 2022-12-09T08:42:42Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: e5b4694
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

jaidisido

I have compiled my suggested changes in this commit.

I would have created a PR against your branch but I get permission denied on your fork.

If you are happy with the changes, you can either grant me access to the fork or I can create a new PR with this branch.

The only outstanding item in my view is making a decision about whether to support the from_data_catalog method or drop it, and if so whether it should be within a single method or not.

I would vote for dropping it. One reason being that I was unable to create an integration test that would read from the Glue catalog. That is because it does not seem like the write_deltalake API support it for now. So if we cannot test it, I would rather not include it.

fvaleye · 2022-12-09T16:32:32Z

Thanks @jaidisido

I am totally aligned with your changes and your suggestions. All good for me, I included your commits in this PR 👍

malachi-constant · 2022-12-09T16:54:28Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 6dcde18
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant · 2022-12-09T16:57:49Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: a0516ea
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

malachi-constant · 2022-12-12T10:46:33Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 79a6088
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Add deltalake support in S3

e8db7f7

fvaleye force-pushed the feature/add-deltalake-support-in-s3 branch 2 times, most recently from fa8c7e6 to fea90fb Compare December 2, 2022 18:34

fvaleye force-pushed the feature/add-deltalake-support-in-s3 branch from fea90fb to 59b6757 Compare December 2, 2022 22:16

fvaleye force-pushed the feature/add-deltalake-support-in-s3 branch from 59b6757 to 2d409c6 Compare December 2, 2022 22:21

malachi-constant marked this pull request as draft December 2, 2022 23:31

Merge branch 'main' into feature/add-deltalake-support-in-s3

23b5e4b

fvaleye force-pushed the feature/add-deltalake-support-in-s3 branch from 2d409c6 to 23b5e4b Compare December 5, 2022 18:58

fvaleye force-pushed the feature/add-deltalake-support-in-s3 branch from bf211fa to 1dd60da Compare December 5, 2022 19:40

Merge branch 'main' into feature/add-deltalake-support-in-s3

cc03a66

fvaleye force-pushed the feature/add-deltalake-support-in-s3 branch from 1dd60da to cc03a66 Compare December 5, 2022 19:42

fvaleye marked this pull request as ready for review December 5, 2022 19:49

malachi-constant requested review from cnfait, jaidisido, malachi-constant, kukushking and LeonLuttenberger December 5, 2022 20:10

malachi-constant assigned malachi-constant and fvaleye and unassigned malachi-constant Dec 5, 2022

malachi-constant added the enhancement New feature or request label Dec 5, 2022

malachi-constant added this to the 2.19.0 milestone Dec 5, 2022

jaidisido reviewed Dec 6, 2022

View reviewed changes

fvaleye requested review from jaidisido and kukushking and removed request for cnfait, malachi-constant and LeonLuttenberger December 7, 2022 19:46

fvaleye force-pushed the feature/add-deltalake-support-in-s3 branch from 3eef47f to 62c88e1 Compare December 7, 2022 20:01

Add pyarrow_additional_kwargs and s3_additional_kwargs in the paramet…

d1267db

…ers to provide more flexibility for the user when reading a DeltaTable

fvaleye force-pushed the feature/add-deltalake-support-in-s3 branch from 62c88e1 to d1267db Compare December 7, 2022 20:01

fvaleye and others added 3 commits December 7, 2022 21:02

Merge branch 'main' into feature/add-deltalake-support-in-s3

2cee056

Enable SSE, enforce HTTPS, enable node-to-node encryption (aws#1851)

08ca60d

Merge branch 'main' into feature/add-deltalake-support-in-s3

e5b4694

jaidisido added 2 commits December 9, 2022 13:50

(feat): Refactor and add tests

9497a60

Minor - missing space

6dcde18

jaidisido requested changes Dec 9, 2022

View reviewed changes

Merge branch 'main' into feature/add-deltalake-support-in-s3

a0516ea

fvaleye requested review from jaidisido and kukushking and removed request for kukushking and jaidisido December 9, 2022 18:34

Remove Glue catalog read until write is available

79a6088

jaidisido approved these changes Dec 12, 2022

View reviewed changes

jaidisido merged commit fec1f7e into aws:main Dec 12, 2022

This was referenced Jan 28, 2023

Expand AWS Lambda layer pyarrow build? #1969

Closed

extending arrow to include python build modules #1977

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add deltalake support in AWS S3 with Pandas #1834

Add deltalake support in AWS S3 with Pandas #1834

fvaleye commented Dec 2, 2022

malachi-constant commented Dec 2, 2022

malachi-constant commented Dec 2, 2022

malachi-constant commented Dec 2, 2022

malachi-constant commented Dec 5, 2022

malachi-constant commented Dec 5, 2022

jaidisido Dec 6, 2022 •

edited

Loading

fvaleye Dec 6, 2022 •

edited

Loading

jaidisido Dec 6, 2022

fvaleye Dec 6, 2022 •

edited

Loading

jaidisido Dec 6, 2022

fvaleye Dec 6, 2022

malachi-constant commented Dec 9, 2022

jaidisido left a comment •

edited

Loading

fvaleye commented Dec 9, 2022 •

edited

Loading

malachi-constant commented Dec 9, 2022

malachi-constant commented Dec 9, 2022

malachi-constant commented Dec 12, 2022

		_logger: logging.Logger = logging.getLogger(__name__)


		def read_deltalake(

Add deltalake support in AWS S3 with Pandas #1834

Add deltalake support in AWS S3 with Pandas #1834

Conversation

fvaleye commented Dec 2, 2022

Feature or Bugfix

Detail

Relates

malachi-constant commented Dec 2, 2022

AWS CodeBuild CI Report

malachi-constant commented Dec 2, 2022

AWS CodeBuild CI Report

malachi-constant commented Dec 2, 2022

AWS CodeBuild CI Report

malachi-constant commented Dec 5, 2022

AWS CodeBuild CI Report

malachi-constant commented Dec 5, 2022

AWS CodeBuild CI Report

jaidisido Dec 6, 2022 • edited Loading

Choose a reason for hiding this comment

fvaleye Dec 6, 2022 • edited Loading

Choose a reason for hiding this comment

jaidisido Dec 6, 2022

Choose a reason for hiding this comment

fvaleye Dec 6, 2022 • edited Loading

Choose a reason for hiding this comment

jaidisido Dec 6, 2022

Choose a reason for hiding this comment

fvaleye Dec 6, 2022

Choose a reason for hiding this comment

malachi-constant commented Dec 9, 2022

AWS CodeBuild CI Report

jaidisido left a comment • edited Loading

Choose a reason for hiding this comment

fvaleye commented Dec 9, 2022 • edited Loading

malachi-constant commented Dec 9, 2022

AWS CodeBuild CI Report

malachi-constant commented Dec 9, 2022

AWS CodeBuild CI Report

malachi-constant commented Dec 12, 2022

AWS CodeBuild CI Report

jaidisido Dec 6, 2022 •

edited

Loading

fvaleye Dec 6, 2022 •

edited

Loading

fvaleye Dec 6, 2022 •

edited

Loading

jaidisido left a comment •

edited

Loading

fvaleye commented Dec 9, 2022 •

edited

Loading