Updating Table log retention configuration with write_deltalake silently changes nothing

[liamphmurphy]

Environment

Delta-rs version:
python-v0.15.1

Binding:
Same as above

Environment:

• Other: Local, tested on s3 with localstack and local filesystem

---

Bug

What happened: We have a use case of needing to update our log retention from the default 30 days to 7 days. We already have a set of delta tables made. When testing locally, I wanted to see if an existing delta table could have its log retention updated when the delta table is already in place.

From my testing, changing the configuration mapping passed into the write_deltalake between runs silently does nothing.

What you expected to happen:

If this is a supported use-case, then I'd expect to see the table's configuration updated when changing the configuration mapping.

If it is not a supported use-case, then I'd expect some kind of Exception.

How to reproduce it:

First, I write a Delta table like so, and then print it's metadata object:

write_deltalake(
          s3_path, 
          table,
          schema=pyarrow_schema,
          mode="append",
          partition_by=["date","hour"]
)
delta_table = deltalake.DeltaTable(s3_path) 
print(delta_table.metadata())

As expected, it prints out a Table metadata object with an empty configuration:

Metadata(id: cf08d466-4126-403b-ae22-12649565275d, name: None, description: None, partition_columns: ['date', 'hour'], created_time: 1705964444931, configuration: {})

Then on a second run, on the same table, where the configuration changes and printing the metadata object:

write_deltalake(
  s3_path, 
  table,
  schema=pyarrow_schema,
  mode="append",
  partition_by=["date","hour"],
  configuration={"logRetentionDuration": "7"}
)
delta_table = deltalake.DeltaTable(s3_path) 
print(delta_table.metadata())

The write gives no indication of a failure but after printing the table, the configuration object is empty. It displays the same object as above.

[ion-elgreco]

@liamphmurphy this is to be expected because only the first write operation can add a configuration, while consecutive changes to the config should be done with Alter table properties. I started working on unsetting some properties, will work on creating a set_table_properties soon

[liamphmurphy]

@liamphmurphy this is to be expected because only the first write operation can add a configuration, while consecutive changes to the config should be done with Alter table properties. I started working on unsetting some properties, will work on creating a set_table_properties soon

@ion-elgreco Thanks for the quick response! So if I understand correctly, the use case will be supported but the functionality is not in place yet?

[ion-elgreco]

Yes, but it will be a separate operation you need to trigger in the future. DeltaTable().alter.set_table_properties()

[ion-elgreco]

You can track that one here: #1663