Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Delta support for s3 #11

Closed
ligao101 opened this issue Apr 25, 2019 · 3 comments
Closed

Delta support for s3 #11

ligao101 opened this issue Apr 25, 2019 · 3 comments

Comments

@ligao101
Copy link

hi Delta community,

Could we add support for eventual consistent blob storage such as AWS s3?

Thanks
@amiorin
Copy link

amiorin commented Apr 27, 2019

@tdas

Are you guys thinking to use DynamoDB Lock Client in the rename block? In my company, we have many AWS Accounts but I think we could accept to create a DynamoDB table writable by all our AWS Accounts with sts:AssumeRole.

To avoid denial of service because of malicious internal users I would create a sidekick API to generate row-level DynamoDB credentials based on a challenge like Let's Encrypt. If you can write a "635d14bc-7c47-4a31-b01b-df3a8bd673b6" in this _delta_log/challenge-fd8b78b0-847c-4b94-9996-3a455582b947 then I'll give you the credentials to acquire the lock.

To authenticate and authorize the users of the sidekick API a would create a pluggable module. We use OAuth, but I can imagine that sts:GetCallerIdentity could also be a good candidate.

@binary132
Copy link

binary132 commented May 7, 2019 •
edited

Lacking native S3 support is a deal-breaker for me to propose Delta as a serious solution at my company.

@tdas
Copy link
Contributor

tdas commented May 10, 2019

Hey all,

We are working towards adding S3 support. I have created an official issue #39 to track S3 support work. I am closing this ticket for now, please continue the discussion on that issue.

@tdas tdas closed this as completed May 10, 2019
LantaoJin added a commit to LantaoJin/delta that referenced this issue Mar 24, 2020
@LantaoJin
[CARMEL-2218][FOLLOWUP] fix version (delta-io#11)
c3531bc
LantaoJin added a commit to LantaoJin/delta that referenced this issue Mar 12, 2021
@LantaoJin
[CARMEL-4055] Overwrite previous data anyway when create delta table (d…
d74298a 
…elta-io#11)
jbguerraz pushed a commit to jbguerraz/delta that referenced this issue Jul 6, 2022
@JassAbidi
Merge pull request delta-io#11 from delta-io/master
9e96cd7 
update fork
tdas pushed a commit to tdas/delta that referenced this issue May 31, 2023
@zsxwing
Consistent schema in Hive and Delta metadata (delta-io#11)
ea20363 
Right now we require the partition columns should be after the data columns.

This PR adds a new DeltaInputSplit to remove the above limitation and also adds validation to ensure Hive's schema is always consistent with Delta's metadata regarding column types and order.
andreaschat-db added a commit to andreaschat-db/delta that referenced this issue Apr 23, 2024
@andreaschat-db
# This is a combination of 12 commits.
ea7b635 
# This is the 1st commit message:

flush

# This is the commit message delta-io#2:

flush

# This is the commit message delta-io#3:

First sane version without isRowDeleted

# This is the commit message delta-io#4:

Hack RowIndexMarkingFilters

# This is the commit message delta-io#5:

Add support for non-vectorized readers

# This is the commit message delta-io#6:

Metadata column fix

# This is the commit message delta-io#7:

Avoid non-deterministic UDF to filter deleted rows

# This is the commit message delta-io#8:

metadata with Expression ID

# This is the commit message delta-io#9:

Fix complex views issue

# This is the commit message delta-io#10:

Tests

# This is the commit message delta-io#11:

cleaning

# This is the commit message delta-io#12:

More tests and fixes
andreaschat-db added a commit to andreaschat-db/delta that referenced this issue Apr 23, 2024
@andreaschat-db
# This is a combination of 16 commits.
ad044ef 
# This is the 1st commit message:

flush

# This is the commit message delta-io#2:

flush

# This is the commit message delta-io#3:

First sane version without isRowDeleted

# This is the commit message delta-io#4:

Hack RowIndexMarkingFilters

# This is the commit message delta-io#5:

Add support for non-vectorized readers

# This is the commit message delta-io#6:

Metadata column fix

# This is the commit message delta-io#7:

Avoid non-deterministic UDF to filter deleted rows

# This is the commit message delta-io#8:

metadata with Expression ID

# This is the commit message delta-io#9:

Fix complex views issue

# This is the commit message delta-io#10:

Tests

# This is the commit message delta-io#11:

cleaning

# This is the commit message delta-io#12:

More tests and fixes

# This is the commit message delta-io#13:

Partial cleaning

# This is the commit message delta-io#14:

cleaning and improvements

# This is the commit message delta-io#15:

cleaning and improvements

# This is the commit message delta-io#16:

Clean RowIndexFilter
andreaschat-db added a commit to andreaschat-db/delta that referenced this issue Apr 26, 2024
@andreaschat-db
# This is a combination of 12 commits.
40b0238 
# This is the 1st commit message:

flush

# This is the commit message delta-io#2:

flush

# This is the commit message delta-io#3:

First sane version without isRowDeleted

# This is the commit message delta-io#4:

Hack RowIndexMarkingFilters

# This is the commit message delta-io#5:

Add support for non-vectorized readers

# This is the commit message delta-io#6:

Metadata column fix

# This is the commit message delta-io#7:

Avoid non-deterministic UDF to filter deleted rows

# This is the commit message delta-io#8:

metadata with Expression ID

# This is the commit message delta-io#9:

Fix complex views issue

# This is the commit message delta-io#10:

Tests

# This is the commit message delta-io#11:

cleaning

# This is the commit message delta-io#12:

More tests and fixes
andreaschat-db added a commit to andreaschat-db/delta that referenced this issue Apr 26, 2024
@andreaschat-db
# This is a combination of 16 commits.
0826e53 
# This is the 1st commit message:

flush

# This is the commit message delta-io#2:

flush

# This is the commit message delta-io#3:

First sane version without isRowDeleted

# This is the commit message delta-io#4:

Hack RowIndexMarkingFilters

# This is the commit message delta-io#5:

Add support for non-vectorized readers

# This is the commit message delta-io#6:

Metadata column fix

# This is the commit message delta-io#7:

Avoid non-deterministic UDF to filter deleted rows

# This is the commit message delta-io#8:

metadata with Expression ID

# This is the commit message delta-io#9:

Fix complex views issue

# This is the commit message delta-io#10:

Tests

# This is the commit message delta-io#11:

cleaning

# This is the commit message delta-io#12:

More tests and fixes

# This is the commit message delta-io#13:

Partial cleaning

# This is the commit message delta-io#14:

cleaning and improvements

# This is the commit message delta-io#15:

cleaning and improvements

# This is the commit message delta-io#16:

Clean RowIndexFilter
@galadrielwithlaptop galadrielwithlaptop mentioned this issue May 14, 2024
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@amiorin @tdas @binary132 @ligao101