Delta Lake 4.3.1

Delta Lake 4.3.1

We are pleased to announce the release of Delta Lake 4.3.1, a patch release on top of 4.3.0 with targeted bug fixes for OAuth authentication in the Delta REST Catalog, S3A fast listing through FilterFileSystem wrappers, and UC managed-table metadata handling.

Highlights

• [Spark] Fix OAuth case-sensitivity bug in Delta REST Catalog authentication: Fixes a bug where CaseInsensitiveStringMap.entrySet() lowercased camelCase OAuth config keys (e.g. oauth.clientId), breaking Delta REST Catalog authentication.

Delta Spark

Delta Spark 4.3.1 is built on Apache Spark 4.1.0 and Apache Spark 4.0.1. As with Apache Spark, we publish Maven artifacts for Scala 2.13.

• Maven artifacts for Spark 4.1.0:
  • delta-spark_4.1_2.13
  • delta-connect-client_4.1_2.13
  • delta-connect-common_4.1_2.13
  • delta-connect-server_4.1_2.13
  • delta-contribs_2.13
  • delta-storage
  • delta-storage-s3-dynamodb
• Maven artifacts for Spark 4.0.1:
  • delta-spark_4.0_2.13
  • delta-connect-client_4.0_2.13
  • delta-connect-common_4.0_2.13
  • delta-connect-server_4.0_2.13
  • delta-contribs_2.13
  • delta-storage
  • delta-storage-s3-dynamodb
• Backward-compatibility artifacts (no Spark version in the name; default to Spark 4.1.0):
  • delta-spark_2.13
  • delta-connect-client_2.13
  • delta-connect-common_2.13
  • delta-connect-server_2.13
• Python artifact:
  • delta-spark

Bug fixes in this release:

• Support delta.enabledFastS3AListFrom with OSS UnityCatalog (>=0.4.1). Because the unitycatalog spark connector introduced CredScopedFileSystem, which relies on S3AFileSystem under the hood, this fix unwraps the internal filesystem of CredScopedFileSystem and handles the casting to enable fast S3A listing.
• Don't persist is_managed_location into Delta table metadata: This reserved DSv2 catalog property was leaking into the committed Metadata.configuration at table creation. It is now filtered out at AbstractDeltaCatalog.createTable and synthesized at load for managed tables, matching the behaviour of Spark's V1Table.

Delta Kernel

The Delta Kernel project is a set of Java libraries for building Delta connectors that read and write Delta tables without needing to understand the Delta protocol directly.

• Maven artifacts:
  • delta-kernel-api
  • delta-kernel-defaults
  • delta-kernel-unitycatalog

No changes to Delta Kernel in this patch release.

Delta UniForm

Delta UniForm's delta-iceberg and delta-hudi modules automatically keep Apache Iceberg and Apache Hudi metadata in sync with Delta commits, so Iceberg and Hudi readers can query Delta tables without data duplication.

• Maven artifacts:
  • delta-iceberg_2.13
  • delta-hudi_2.13

No changes to Delta UniForm in this patch release.

Delta Sharing

Delta Sharing is a Spark DataSource that lets clients run batch, streaming, CDF, and time-travel reads on tables shared via the Delta Sharing protocol. The 2.13 suffix indicates Scala 2.13.

• Maven artifacts for Spark 4.1.0:
  • delta-sharing-spark_4.1_2.13
• Maven artifacts for Spark 4.0.1:
  • delta-sharing-spark_4.0_2.13
• Backward-compatibility artifacts (no Spark version in the name; default to Spark 4.1.0):
  • delta-sharing-spark_2.13

No changes to Delta Sharing in this patch release.

Delta Flink

The Kernel-based delta-flink connector (experimental) is released as part of this patch.

• Maven artifact:
  • delta-flink

No changes to Delta Flink in this patch release.

Credits

Daniel Wang, Murali Ramanujam, Zheng Hu, Rakesh Veeramacheneni, Tathagata Das, Timothy Wang, Vishnu Chandrashekhar, Xin Huang, Yi Li