delta-rs does not appear to support tables with liquid clustering

[nchammas]

Not sure if this should be filed as a feature request instead.

Environment

Delta-rs version: 0.15.0

Binding: Python

Environment:

• Cloud provider: Databricks / AWS

---

Bug

What happened: I am unable to open a Delta table that has liquid clustering active.

What you expected to happen: Open the table successfully.

How to reproduce it:

spark.range(100).createOrReplaceTempView("test")

CREATE OR REPLACE TABLE nchammas.deltalakers_test
CLUSTER BY (id)
AS
SELECT *
FROM test;

import deltalake

deltalake.DeltaTable("/dbfs/user/hive/warehouse/nchammas.db/deltalakers_test")

This yields the following error:

DeltaProtocolError: Invalid JSON in log record, version=0, line=`{"domainMetadata":{"domain":"delta.liquid",
"configuration": "{\"clusteringColumns\":[{\"physicalName\":[\"id\"]}],\"domainName\":\"delta.liquid\"}",
"removed":false}}`, err=`unknown variant `domainMetadata`, expected one of `metaData`, `cdc`, `add`,
`remove`, `txn`, `protocol`, `commitInfo` at line 1 column 17`

There are other variations on this error, depending on whether the table has been optimized or cached. But they all mention domainMetadata, which was presumably added as part of liquid clustering.

[rtyler]

Yep this looks like an error to me 😆 I know where the problem is, if you would be willing to share a zip of your test table or something like that, that would save me some time in creating a reproduction case

[nchammas]

I included a way to create my test table from scratch. Does that work for you?

[rtyler]

@nchammas I do not have a Databricks environment which I can use to replicate the exact table structure. I don't know if the DBR you may have used has any special pre-release or proprietary Delta Lake bits in it, which is why I ask for the table itself.

Otherwise, I will just consult with the Delta protocol and add our missing attributes which I believe liquid clustering should be using.

[nchammas]

Here you go.

deltalakers_test.zip

I'm on DBR 13.3.