You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have a Delta Lake on S3 with over 2TB of data, which we write to daily. using we use write_deltalake (writing new partitions every day with partition filters)
We noticed a significant decline in read performance after a few weeks. which led to further investigation I discovered that no checkpoint files were being written. Currently, I am at over 4000 transaction JSON files, and no checkpoint file is there.
As far as I know, Delta's default behavior includes checkpointing after the 10th version. Is there a way to enforce this or trigger it manually?
Delta-rs version: 0.10.0
Binding:
Environment:
Cloud provider: AWS
OS: macOs
Other:
We have a Delta Lake on S3 with over 2TB of data, which we write to daily. using we use
write_deltalake
(writing new partitions every day withpartition filters
)We noticed a significant decline in read performance after a few weeks. which led to further investigation I discovered that no checkpoint files were being written. Currently, I am at over 4000 transaction JSON files, and no checkpoint file is there.
As far as I know, Delta's default behavior includes checkpointing after the 10th version. Is there a way to enforce this or trigger it manually?
The text was updated successfully, but these errors were encountered: