Information about the removal of S3DynamoDbLogStore

[rtyler]

In #4464 the S3DynamoDbLogStore support was removed and instead S3-based conditional put semantics are now the default supported mode for writes on S3.

S3DynamoDbLogStore allowed coordinated commits with a similarly configured Delta/Spark writers on the same table. As of the 1.0.0 release of delta-rs this coordination will no longer work.

⚡ More details to follow later once the PR is merged.

[rtyler]

The original S3DynamoDbLogStore implementation was contributed by the Samba TV team in 2022 to Delta/Spark and was adopted by delta-rs for multi-process write coordination. While the use of S3DynamoDbLogStore was strongly recommended for running multiple concurrent delta-rs processes, many AWS users relied on it for architectures where delta-rs based writers would add incremental commits to a Delta table while running periodic Spark-based OPTIMIZE commands.

The S3DynamoDbLogStore functionality is not required for multi-process writes by delta-rs since the introduction of consistent conditional writes by AWS S3 in late 2024.

The Delta Lake project has never really invested in the S3DynamoDbLogStore support and its support is likely to be removed in Delta/Spark "any day now" as it relies on a now completely unsupported AWS SDK version.

There is related investment in the Catalog Managed tables RFC which supercedes the Managed Commits work. This work does not directly address the conditional put interoperability between delta-rs based writers and Delta/Spark writers, since it appears that Delta/Spark still does not support this functionality properly.

See:

• [BUG] Concurrent write to _delta_log on AWS S3 Delta Table results in data loss delta#1830
• [Feature Request] [S3DynamoDBLogStore] S3 Conditional Writes delta#3596
• [QUESTION] Multi-cluster write to S3 without dynamoDB delta#4813

Either way 🔥 delta-rs 1.0 will not support S3DynamoDbLogStore. 🔥

Users who require this interoperability should remain on the 0.32.x release
branch which will receive some backported bugfixes.

Configuration changes

The migration is mostly removing configuration rather than adding it. Drop AWS_S3_LOCKING_PROVIDER=dynamodb (and the DELTA_DYNAMO_TABLE_NAME / DELTA_DYNAMO_* overrides) from your storage_options and environment entirely. If those keys are still present after the upgrade, delta-rs will not silently lock with DynamoDB — the S3LogStoreFactory now logs an error pointing at
this discussion falls through to the default logstore, so leaving them set just produces noise.

from deltalake import write_deltalake
import pandas as pd

df = pd.DataFrame({"x": [1, 2, 3]})
write_deltalake("s3://path/to/table", df)  # conditional put, no lock table

A note on allow_unsafe_rename

If at any point you used AWS_S3_ALLOW_UNSAFE_RENAME=true as an escape hatch to avoid DynamoDB, you can and should remove it too. That flag disables commit safety entirely (it was only ever appropriate for single-writer scenarios), and the whole point of the conditional-put default is that you no longer need to trade safety for convenience. Setting it now actively opts you out of the safe path by routing back through the legacy S3StorageBackend.