Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEAT] Allow for selection of append/overwrite/overwrite_partitions options when writing data #1768

Open
jaychia opened this issue Jan 9, 2024 · 1 comment · May be fixed by #1986
Open

[FEAT] Allow for selection of append/overwrite/overwrite_partitions options when writing data #1768

jaychia opened this issue Jan 9, 2024 · 1 comment · May be fixed by #1986
Assignees
@colin-ho
Labels
p2 Nice to have features

Comments

@jaychia
Copy link
Contributor

jaychia commented Jan 9, 2024

Is your feature request related to a problem? Please describe.

When writing data, Daft currently performs an append by default.

We should additionally provide options to:

  1. Overwrite the entire table (delete and then write)
  2. Overwrite only dirty partitions (delete and then overwrite any partitions contained by the DataFrame)
@jaychia
Copy link
Contributor Author

jaychia commented Feb 13, 2024

We can tackle append + overwrite first, and make a separate ticket for overwrite_partitions

@jaychia jaychia removed the priority label Feb 13, 2024
@jaychia jaychia added the p2 Nice to have features label Feb 20, 2024
@colin-ho colin-ho linked a pull request Mar 11, 2024 that will close this issue
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@colin-ho colin-ho

Labels
p2 Nice to have features
Projects
Daft-OSS
Status: In Review
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[FEAT] Allow overwrite / overwrite partitions for write operations Eventual-Inc/Daft
2 participants
@jaychia @colin-ho