Skip to content

Releases: delta-io/delta

Release list

Delta Lake 4.4.0

Delta Lake 4.4.0 Latest
Latest

Choose a tag to compare

View all tags
@timothyw553 timothyw553 released this 20 Aug 16:52
v4.4.0
88c008d
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

We are excited to announce the release of Delta Lake 4.4.0. This release adds Apache Spark 4.2 support, expands integration with the UC Delta Table API, and adds identity-column and generated-column support to Delta Spark.

Highlights

  • [Spark] Apache Spark 4.2 support: Delta Spark, Delta Connect, and Delta Sharing now support Apache Spark 4.2.0.
  • [Kernel] [Flink] UC Delta API integration: Delta Kernel and the experimental Flink connector can use the UC Delta API for catalog-managed tables.
  • [Spark] Identity columns in SQL DDL: CREATE TABLE now supports GENERATED ALWAYS AS IDENTITY and GENERATED BY DEFAULT AS IDENTITY.
  • [Spark] Generated columns with Unity Catalog: Delta tables created through Unity Catalog can now define generated columns.

Delta Spark

Delta Spark 4.4.0 is built for Apache Spark 4.2.0, Apache Spark 4.1.0, and Apache Spark 4.0.1. As with Apache Spark, Maven artifacts are published for Scala 2.13.

The key features of this release are:

  • Apache Spark 4.2 support (#7223): Delta Spark, Delta Connect, and Delta Sharing are now built and tested with Spark 4.2.0 in addition to Spark 4.0.1 and 4.1.0.
  • Identity columns in Spark SQL DDL (#7062): CREATE TABLE now accepts Spark's GENERATED ALWAYS AS IDENTITY and GENERATED BY DEFAULT AS IDENTITY syntax, including custom start and increment values.
  • Generated columns with Unity Catalog (#7290): Generated-column expressions are now preserved during table creation. Delta computes or validates generated values during writes using the declared expressions.
  • SHOW PARTITIONS support (#6102): Users can inspect the partitions of a partitioned Delta table with the standard Spark SQL command.
  • VOID column support (#6965, #6966): On Spark 4.1 and later, Delta reads now preserve VOID (NullType) columns instead of failing or silently dropping them. This includes time-travel and path-based reads.

Delta Kernel

Delta Kernel is a set of Java libraries for building Delta connectors that read and write Delta tables without requiring each connector to implement the Delta protocol directly.

The key changes in this release are:

  • UC Delta API integration (#7159): The Unity Catalog committer can use the UC Delta API for catalog-managed tables, including structured protocol, metadata, and domain-metadata updates.
  • Safer concurrent writes (#7163): Concurrent transactions that remove or update the same data file now conflict instead of allowing duplicate active files or lost deletion-vector updates.
  • Correct partition output for column-mapped tables (#7212): Kernel write contexts now translate logical partition names to physical in target paths and AddFile.partitionValues, allowing partitioned writes to name- and id-mapped tables to round-trip correctly.
  • Protocol-recommended timestamp partitions (#7287): Newly written TIMESTAMP partition values use UTC ISO-8601 strings. Pre-epoch TIMESTAMP_NTZ values are also serialized correctly.
  • Fewer object-store metadata requests (#6856, #7099): Known file lengths are passed when opening files, and redundant existence probes are avoided during log listing. This reduces HEAD requests and prevents failures with scoped credentials that permit listing but not directory-level HEAD operations.

Delta UniForm

Delta UniForm's delta-iceberg module keeps Apache Iceberg metadata synchronized with Delta commits, allowing Iceberg readers to query Delta tables without duplicating data.

Note: In Delta 4.4, delta-iceberg_2.13 supports Spark 4.1 and is not compatible with Spark 4.2.

The key changes in this release are:

  • Atomic table initialization (#6880): UniForm metadata can be initialized atomically with Delta table creation.
  • Improved Iceberg conversion correctness (#7047, #7187): Metadata-only statistics commits are handled correctly, and UniForm table properties are preserved during saveAsTable overwrite.

Delta Sharing

Delta Sharing is a Spark DataSource that supports batch, streaming, CDF, and time-travel reads on tables shared through the Delta Sharing protocol.

The key changes in this release are:

  • Delta Sharing client 1.4.1 (#7281): The Spark connector now uses Delta Sharing client 1.4.1.
  • Protocol changes within streaming and CDF ranges (#7305, #7319): Opt-in historical-protocol support allows a shared streaming or CDF query to cross a table protocol upgrade without reconstructing an inconsistent local log. The switches are disabled by default and require corresponding server support.
  • Independent CDF streaming format resolution (#7003): CDF streaming can control automatic Parquet-versus-Delta response-format resolution independently from ordinary streaming.

Delta Flink

The Kernel-based delta-flink connector remains experimental. Delta 4.4.0 supports Apache Flink 2.0.2, 2.1.3, 2.2.1, and 2.3.0.

Read more
Assets 2
Loading

Delta Lake 3.3.3

Delta Lake 3.3.3

Choose a tag to compare

View all tags
@seewishnew seewishnew released this 12 Aug 21:01
v3.3.3
0ce5dd9
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

We are pleased to announce the release of Delta Lake 3.3.3, a patch release on top of 3.3.2 with targeted fixes for Delta Sharing cache correctness, transaction log retention safety, and a Kernel Parquet-reader configuration bug, plus a Delta Sharing client upgrade.

Note: this patch release does not publish delta-iceberg (UniForm/Iceberg support) — see Delta UniForm below for details and follow-up plan.

Highlights

  • [Spark] Fix transaction log retention during metadata cleanup: Log cleanup could previously delete transaction log files still needed to reconstruct table versions within the retention window on tables with infrequent commits/checkpoints, breaking time travel and CDF reads.
  • [Delta Sharing] Fix deletion vector cache refresh bug: Cache refresh could drop deletion-vector fileId→URL mappings, causing long-running queries against shared DV tables to fail with "file id not found" errors.
  • [Delta Sharing] Upgrade delta-sharing-client 1.2.2 → 1.2.8: Picks up more resilient OAuth token parsing and improved retry/error-reporting behavior.
  • [Spark] Add RANDOMIZE_FILE_PREFIXES support: Wires up the previously-declared delta.randomizeFilePrefixes / delta.randomPrefixLength table properties on the write path, to spread S3 object keys for high-throughput workloads. Opt-in; default behavior unchanged.
  • [Spark] Fix Spark 3.5.6 compatibility for TRUNCATE TABLE: Spark 3.5.6 tightened V2Writes validation; Delta tables now correctly declare TRUNCATE/OVERWRITE_BY_FILTER capability.
  • [Kernel] Fix missing Hadoop configuration in Parquet reader: The Hadoop Configuration was not being passed when Kernel opened Parquet footers, so custom filesystem/credential settings could be silently ignored.

Delta Spark

Delta Spark 3.3.3 is built on Apache Spark 3.5.6. Maven artifacts are published for both Scala 2.12 and 2.13.

Fixes and changes in this release:

  • Fix transaction log retention during metadata cleanup: Delete eligible log files only if a newer checkpoint exists before the retention cutoff, so time travel and CDF reads to retained versions keep working. Tables with a low commit/checkpoint rate may retain log files slightly beyond the nominal cutoff as a result.
  • Add RANDOMIZE_FILE_PREFIXES support: delta.randomizeFilePrefixes / delta.randomPrefixLength table properties are now honored on writes, placing new data files under a random subdirectory prefix to spread S3 request load. Opt-in; default write path is unchanged.
  • Fix Spark 3.5.6 compatibility for TRUNCATE TABLE: StagedDeltaTableV2 now implements SupportsTruncate and declares TRUNCATE/OVERWRITE_BY_FILTER capabilities, fixing TRUNCATE TABLE under Spark 3.5.6's stricter V2Writes validation.

Delta Kernel

The Delta Kernel project is a set of Java libraries for building Delta connectors without needing to implement the Delta protocol directly.

Fixes in this release:

  • Fix missing Hadoop configuration parameter when creating the Parquet reader: A follow-up to an earlier IllegalAccessError fix had dropped the Hadoop Configuration when constructing the ParquetReader used to read footers, so custom Hadoop filesystem settings could be ignored during Kernel reads.

Delta UniForm

Delta UniForm's delta-iceberg and delta-hudi modules keep Apache Iceberg and Apache Hudi metadata in sync with Delta commits.

  • delta-iceberg is not published in 3.3.3. There will be no delta-iceberg artifact in this release. There are no changes to delta-iceberg from the previous version. Existing delta-iceberg_2.12/delta-iceberg_2.13 v3.3.2 artifacts remain available and are forward-compatible with delta-spark 3.3.3.
  • Maven artifacts:

No changes to Delta Hudi in this patch release (rebuilt against the updated build toolchain only).

Delta Sharing

Delta Sharing is a Spark DataSource for batch, streaming, CDF, and time-travel reads on tables shared via the Delta Sharing protocol.

Fixes and changes in this release:

  • Fix deletion vector cache refresh dropping fileId→URL entries: Cache refresh previously dropped deletion-vector file mappings registered during initial cache setup, causing long-running queries against DV-enabled shared tables to fail with "file id not found" errors after a refresh.
  • Upgrade delta-sharing-client to 1.2.8 (from 1.2.2): brings more resilient OAuth expires_in parsing, adaptive retry/backoff on retryable server errors, and clearer error reporting from streaming responses.

Delta Standalone

No changes to Delta Standalone in this patch release.

Delta Hive Connector

No changes to the Delta Hive connector in this patch release.

Delta Flink

The Kernel-based delta-flink connector (experimental).

No changes to Delta Flink in this patch release.

Credits

Daniel Mattos, Felipe Pessoto, Geeta Krishna Panda, littlegrasscao, Sun Cao, Venki Korukanti, Vishnu Chandrashekhar, Yingyi Bu

Loading

Delta Lake 4.3.1

Delta Lake 4.3.1

Choose a tag to compare

View all tags
@seewishnew seewishnew released this 08 Jul 04:59
v4.3.1
54ce026
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Delta Lake 4.3.1

We are pleased to announce the release of Delta Lake 4.3.1, a patch release on top of 4.3.0 with targeted bug fixes for OAuth authentication in the Delta REST Catalog, S3A fast listing through FilterFileSystem wrappers, and UC managed-table metadata handling.

Highlights

  • [Spark] Fix OAuth case-sensitivity bug in Delta REST Catalog authentication: Fixes a bug where CaseInsensitiveStringMap.entrySet() lowercased camelCase OAuth config keys (e.g. oauth.clientId), breaking Delta REST Catalog authentication.

Delta Spark

Delta Spark 4.3.1 is built on Apache Spark 4.1.0 and Apache Spark 4.0.1. As with Apache Spark, we publish Maven artifacts for Scala 2.13.

Bug fixes in this release:

  • Support delta.enabledFastS3AListFrom with OSS UnityCatalog (>=0.4.1). Because the unitycatalog spark connector introduced CredScopedFileSystem, which relies on S3AFileSystem under the hood, this fix unwraps the internal filesystem of CredScopedFileSystem and handles the casting to enable fast S3A listing.
  • Don't persist is_managed_location into Delta table metadata: This reserved DSv2 catalog property was leaking into the committed Metadata.configuration at table creation. It is now filtered out at AbstractDeltaCatalog.createTable and synthesized at load for managed tables, matching the behaviour of Spark's V1Table.

Delta Kernel

The Delta Kernel project is a set of Java libraries for building Delta connectors that read and write Delta tables without needing to understand the Delta protocol directly.

No changes to Delta Kernel in this patch release.

Delta UniForm

Delta UniForm's delta-iceberg and delta-hudi modules automatically keep Apache Iceberg and Apache Hudi metadata in sync with Delta commits, so Iceberg and Hudi readers can query Delta tables without data duplication.

No changes to Delta UniForm in this patch release.

Delta Sharing

Delta Sharing is a Spark DataSource that lets clients run batch, streaming, CDF, and time-travel reads on tables shared via the Delta Sharing protocol. The 2.13 suffix indicates Scala 2.13.

No changes to Delta Sharing in this patch release.

Delta Flink

The Kernel-based delta-flink connector (experimental) is released as part of this patch.

No changes to Delta Flink in this patch release.

Credits

Daniel Wang, Murali Ramanujam, Zheng Hu, Rakesh Veeramacheneni, Tathagata Das, Timothy Wang, Vishnu Chandrashekhar, Xin Huang, Yi Li

Loading

Delta Lake 4.3.0

Delta Lake 4.3.0

Choose a tag to compare

View all tags
@seewishnew seewishnew released this 18 Jun 04:26
v4.3.0
76c12a2
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

We are excited to announce the release of Delta Lake 4.3.0, which delivers new features, performance improvements, and protocol updates across Delta Spark, Kernel, UniForm, Sharing, and Flink. See the highlights below for the marquee changes.

Highlights

  • [Spark] Unity Catalog Delta REST API integration: Spark now supports the UC Delta API, using Unity Catalog as the source of truth for managed Delta tables. With server-side commit validation, server-advertised table features, and intent-based metadata updates, this integration provides consistent, safe access for Spark today and sets the foundation for future support across Flink, Trino, and other engines.
  • [Spark] Selective data replacement with replaceOn and replaceUsing DataFrame APIs: Spark now supports selectively replacing table data with the result of a DataFrame. Use replaceUsing to replace rows that match on specified columns, or replaceOn to replace rows that satisfy a user-defined condition.
  • [UniForm] Atomic + incremental Iceberg conversion and Spark 4.1 support: UniForm now writes Iceberg metadata atomically with the Delta commit, incrementally converts only the changed log range.
  • [Sharing] Streaming and CDF Support: Delta Sharing in 4.3.0 improves Spark Structured Streaming and Batch CDF support with automatic Delta response resolution, Parquet-to-Delta streaming conversion, shared DeltaFormattable streaming CDF support, and Trigger.AvailableNow support for shared tables.

Delta Spark

Delta Spark 4.3.0 is built on Apache Spark 4.1.0 and Apache Spark 4.0.1. As with Apache Spark, we publish Maven artifacts for Scala 2.13.

The key features of this release are:

  • Unity Catalog Delta REST API integration : Delta Spark now uses the new UC Delta REST API by default for UC-managed Delta tables. Managed Delta operations, including table loads, CREATE / CTAS, REPLACE, and all other metadata-changing writes such as DML, schema evolution, auto-merge, and supported ALTER TABLE updates, are routed through the new API. Non-Delta tables and external tables, including name-based and path-based access, continue to use the legacy delegate.
  • Delta DSv2 Connector with Delta Kernel (Experimental) : adds new features like support for batch writes to Catalog-Managed tables, Streaming source support including all read options and Catalog-driven batch CDC (SELECT … CHANGES FROM VERSION/TIMESTAMP, DV-aware) - gated behind spark.databricks.delta.changelogV2.enabled.
  • V2 Checkpoint performance hardening for large tables: V2 checkpoints now default to 50,000 actions per sidecar, so sidecar files are automatically split into multiple parts and checkpoint writes parallelize better out of the box.
  • Selectively replace data with 'replaceOn' and 'replaceUsing' DataFrame APIs: Use these options to replace part of the table with the result of a DataFrame. ‘replaceOn’ replaces rows that match a user-defined condition. ‘replaceUsing’ replaces rows where specified columns are equal. See Delta Lake API doc.
  • Implicit casting for DataFrame by-name writes: DataFrame writes that match by column name, except save() and saveAsTable().mode(“overwrite”), now apply Spark's implicit casts to align source values with the target schema, matching SQL INSERT BY NAME behavior.
  • Variant column statistics on write: Delta Spark now collects min/max statistics for Variant columns at write time, enabling data skipping on Variant-shredded tables.
  • REPLACE TABLE / RTAS / DPO production hardening: concurrency, source-materialization, and operational-metric coverage now spans the full set of REPLACE-style DataFrame writes introduced in Delta 4.2.

Other notable features and bug-fixes include:

Delta Kernel

The Delta Kernel project is a set of Java libraries for building Delta connectors that read and write Delta tables without needing to understand the Delta protocol directly.

The key features of this release are:

  • Incremental version-checksum construction: tableSizeBytes and numFiles are updated incrementally instead of via full log replay, making table-health diagnostics fast on large tables.
  • [Open tables with a missing _last_checkpo...
Read more
Loading

Delta Lake 4.2.0

Delta Lake 4.2.0

Choose a tag to compare

View all tags
@openinx openinx released this 16 Apr 05:00
v4.2.0
9f600c4

We are excited to announce the release of Delta Lake 4.2.0! This release includes significant new features, improved safety and compatibility, and important bug fixes.

Highlights

  • [Spark] Unity Catalog Managed Table enhancements (experimental): REPLACE TABLE / RTAS and Dynamic Partition Overwrite support, automatic table schema/properties sync to catalog on table creation.
  • [Spark] Delta Spark V2 connector - streaming read (experimental): enhance streaming read capabilities for catalog-managed table by supporting critical options like startingTimestamp and skipChangeCommits.
  • [Flink] New Kernel-based Flink connector (experimental): a brand-new Kernel-based delta-flink connector that enables Apache Flink to read, write, and interact with catalog-managed Delta tables.
  • [Kernel] Geospatial, Variant GA, and Collations table feature: Delta Kernel can now read and write tables using geometry/geography types with bounding-box data skipping, generally available Variant columns, and collated string types.
  • [Security] The Delta project has undergone a substantial hardening effort across multiple surface areas, including stronger validation and dependency security scanning to proactively reduce supply-chain risk.

Delta Spark

Delta Spark 4.2.0 is built on Apache Spark 4.1.0 and Apache Spark 4.0.1. Similar to Apache Spark, we have released Maven artifacts for Scala 2.13.

The key features of this release are:

  • Delta Spark V2 Streaming read: Enhance streaming read capabilities for catalog-managed tables. The Delta V2 Spark connector now supports key options including startingVersion, startingTimestamp, maxBytesPerTrigger, maxFilesPerTrigger, excludeRegex, skipChangeCommits, ignoreDeletes, ignoreChanges, and ignoreFileDeletion.
  • REPLACE TABLE/RTAS/DPO support: Previously, operations like REPLACE TABLE, RTAS, and Dynamic Partition Overwrite (DPO) were not supported for catalog-managed Delta tables. This change will enable these operations to be a single, atomic action, significantly improving the safety and reliability of users' catalog-managed tables.
  • Server-Side Planning: OAuth Support (preview): The server-side planning client now supports OAuth-based authentication when delegating scan planning to an external catalog server.
  • INSERT BY NAME with Schema Evolution: SQL INSERT ... BY NAME statements now support automatic schema evolution, adding missing columns to the target table when delta.schemaAutoMerge.enabled is set. This brings INSERT BY NAME behavior in line with INSERT SELECT with schema evolution.
  • Force Statistics Collection: A new table property delta.stats.skipping.forceOptimizeStatsCollection enables forcing file statistics collection during query optimization. This ensures accurate data skipping for tables where statistics may be absent or stale, without requiring a manual OPTIMIZE run.
  • Allow CDF Writes for Non-Data-Changing Operations: When Change Data Feed is enabled, write operations that produce no data changes — such as add-only or remove-only commits — are now permitted. This reduces unnecessary write failures in CDC pipelines that perform metadata-only or compaction operations.
  • Variant Type in Schema Conversion: Delta Spark now correctly handles Spark's VariantType during Delta schema conversion, enabling seamless schema operations on tables with Variant columns.

Other notable changes:

Delta Kernel

The Delta Kernel project is a set of Java and Rust libraries for building Delta connectors that can read and write to Delta tables without the need to understand the Delta protocol details.

The key features of this release are:

  • Geospatial table feature: Delta Kernel now supports reading and writing tables with geometry and geography columns, including bounding-box data skipping via the StGeometryBoxesIntersect predicate.
  • Variant GA table feature: The Variant data type is now generally available in Delta Kernel.
  • Lazy schema parsing: Tables with unsupported column types in kernel (e.g. VOID) can now be loaded without error. The schema is only parsed when explicitly accessed, so connectors that don't need the schema can still read metadata, configuration, and other table properties from these tables.
  • Improved CommitInfo compatibility: Tables whose commits were written by external engines that omit engineInfo, operation, or txnId fields can now be read without errors.
  • [Add vacuumProtocolCheck ...
Read more
Loading

Delta Lake 4.1.0

Delta Lake 4.1.0

Choose a tag to compare

View all tags
@openinx openinx released this 26 Feb 02:42
v4.1.0
7d2762e

We are excited to announce the release of Delta Lake 4.1.0! This release includes significant new features, performance improvements, and important platform upgrades.

Highlights

  • [Spark] Apache Spark 4.1.0 Support. The default build of Delta 4.1.0 leverages Apache Spark 4.1.0; however, it retains compatibility with Apache Spark 4.0.1.
  • [Spark] Catalog managed table enhancements (preview): Support UC managed table creation, batch read/write, streaming read/write.
  • [Spark] Spark V2 connector based on Delta Kernel API : A new Spark DataSource V2 connector backed by Delta Kernel, supporting streaming reads for catalog-managed tables.
  • [Spark] Server-Side Planning (preview): Delegate scan planning to catalog servers following the Apache Iceberg REST Catalog API. Supported filter, projection, and limits are pushed down to do the query planning.
  • [Spark] Conflict-free feature enablement: Enable Deletion Vectors and Column Mapping on existing tables without blocking or conflicting with concurrent writes.
  • [Kernel] Full support for catalog-managed tables, enabling Kernel-based connectors to interact with catalog-managed Delta tables (e.g., via Unity Catalog).

Delta Spark

Delta Spark 4.1.0 is built on Apache Spark 4.1.0 and Apache Spark 4.0.1. Similar to Apache Spark, we have released Maven artifacts for Scala 2.13.

Starting in Delta 4.1.0, Maven artifacts include a Spark version suffix (e.g., delta-spark_4.1_2.13 instead of delta-spark_2.13), with backward compatibility preserved in this release but dependency updates recommended. Separate artifacts are now published for Spark 4.1 and Spark 4.0 so users can choose the version matching their Spark runtime.

The key features of this release are:

  • Catalog-managed table enhancements: Delta Spark has added more support for Unity Catalog managed Delta tables via catalogManaged feature, enabling table creation, batch and streaming reads/writes (including time travel, and DML operations), history inspection, and OAuth-based authentication. This is still in preview and production usage is not recommended.
  • Delta v2 Spark Connector: A new Spark DataSource V2 connector backed by Delta Kernel supporting streaming read for catalog-managed table.
  • Server-Side Planning (preview): Delegate scan planning to an external catalog server, with filter, projection, and limit pushdown and multi-cloud credential support.
  • Atomic CTAS: CREATE TABLE AS SELECT for UC managed delta tables (MANAGED and EXTERNAL) is now fully atomic, working with UC 0.4.0. Other operations (including REPLACE TABLE, REPLACE TABLE AS SELECT, CREATE OR REPLACE TABLE, Dynamic Partition Overwrite) now fail fast instead of running in best-effort mode.
  • Conflict-free Deletion Vector enablement: Enable Deletion Vectors on existing tables without conflicting with concurrent transactions or requiring a maintenance window.
  • Conflict-free Column Mapping enablement: Enable Column Mapping on existing tables without conflicting with concurrent transactions or requiring a maintenance window.
  • Type widening enhancements: New decimal coercion modes during schema evolution (e.g., integer to decimal). The default automatic widening mode is now always.

Other notable changes:

  • Support writing AddFile stats as struct in checkpoints.
  • Fix multiple MERGE and INSERT struct null expansion bugs (#5612, #5658, #5759, #4759).
  • Fix timezone conversion for non-UTC timestamp partition values.
  • Fix metadata cleanup to retain files required for log reconstruction.
  • Fix DomainMetadata handling for RESTORE/CLONE and concurrent REPLACE.

Delta Kernel

The Delta Kernel project is a set of Java and Rust libraries for building Delta connectors that can read and write to Delta tables without the need to understand the Delta protocol details.

The key features of this release are:

  • Full catalog-managed tables support: enabling Kernel-based connectors to read from and write to Delta tables managed by a catalog such as Unity Catalog. This includes snapshot loading, time travel, change data feeds, commit publishing, and a reference Unity Catalog integration. Catalog-managed tables shift commit coordination from the filesystem to a managing catalog, unlocking capabilities like multi-table transactions.
  • Type widening support: Read tables with type widening enabled, including Iceberg compatibility checks for type changes.
  • Variant read/write support including variant stats.
  • Collation support: Writing collated tables, partition pruning, and data skipping with collations.
  • Checkpoint protection support to preserve table history during feature drops.

Delta Universal Format (UniForm)

Delta UniForm is enabled for Spark 4.0 in this release, restoring full Iceberg interoperability that was listed as a limitation in Delta 4.0.0. Both hudi and iceberg are currently not compatible with Spark 4.1, as support depends on upcoming releases providing Spark 4.1 compatible integratio...

Read more
Loading

Delta Lake 4.0.1

Delta Lake 4.0.1

Choose a tag to compare

View all tags
@timothyw553 timothyw553 released this 16 Jan 00:40
v4.0.1
14460c1

We are excited to announce the release of Delta Lake 4.0.1! This release contains important bug fixes to 4.0.0 and it is recommended that users update to 4.0.1.

Highlights

  • [Spark] Breaking change: rename managed table feature from catalogOwned-preview to catalogManaged; legacy ucTableId has also transitioned to the new managed-table io.unitycatalog.tableId
  • [Spark] Unity Catalog OAuth: add OAuth authentication via catalog-scoped auth.* configs; tokens are acquired and refreshed automatically; static tokens remain supported.
  • [Spark] Unity Catalog managed table table creation: enable creating UC-managed Delta table and always send table properties to the UC server as source of truth.
  • [Spark] Spark compatibility fix: addressed NoSuchMethodError in REORG TABLE … APPLY (PURGE) when running with Spark 4.0.1.

Component-specific bug fixes are detailed below.

Delta Spark

Delta Spark 4.0.1 is built on Apache Spark™ 4.0.1. Similar to Apache Spark, we have released Maven artifacts for Scala 2.13.

The key features of this release are:

  • Catalog Managed Tables: managed table feature rename. The feature formerly named catalogOwned‑preview is standardized as catalogManaged. The associated Unity Catalog table ID property is updated accordingly (ucTableId → io.unitycatalog.tableId) ; calls that still send the legacy key are handled for compatibility during creation.
  • Fix Spark binary compatibility issue that caused REORG TABLE … APPLY (PURGE) to fail with NoSuchMethodError on Spark 4.0.1 by switching to the stable constructor and retrieving SQL configs via SparkSession.active.sessionState.conf in executors
  • Introduced end‑to‑end tests for Unity Catalog DML on the 4.0.x line to validate UC‑backed execution paths; scope is test‑only with no public API changes.

Unity Catalog Support

  • Official compatibility with UC 0.3.1. Delta 4.0.1 is officially tested with UC integration tests validating end‑to‑end behavior with UC 0.3.1. See the UC 0.3.1 release for the corresponding connector capabilities and APIs.

  • Support Unity Catalog OAuth authentication. Use catalog‑scoped auth.* configuration; tokens are automatically acquired and refreshed, avoiding embedded static tokens. Legacy static‑token configs continue to work for backward compatibility.

    Enable OAuth on a Spark catalog alias that points to UC:

    # Point a Spark catalog alias at Unity Catalog
spark.sql.catalog.mycatalog = "io.unitycatalog.connectors.spark.UCSingleCatalog"
spark.sql.catalog.mycatalog.uri = "https://<your-workspace-host>"

# OAuth (dynamic tokens) — supported keys
spark.sql.catalog.mycatalog.auth.type               = "oauth"
spark.sql.catalog.mycatalog.auth.oauth.uri          = "https://<auth-server-endpoint>"
spark.sql.catalog.mycatalog.auth.oauth.clientId     = "<client-id>"
spark.sql.catalog.mycatalog.auth.oauth.clientSecret = "<client-secret>"

# Static token (legacy-compatible)
spark.sql.catalog.mycatalog.auth.type  = "static"
spark.sql.catalog.mycatalog.auth.token = "<personal-access-token>"
# Legacy key also supported: spark.sql.catalog.mycatalog.token

    And run with Delta’s Spark extensions as usual:

    --conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" \
--conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"

  • Support Unity Catalog Managed Delta table creation. You can now create UC‑managed Delta tables via standard CREATE TABLE on a UC‑backed Spark catalog; at creation time, Delta sends table properties to the UC server so the server is the source of truth.

    Example:

    CREATE TABLE mycatalog.my_schema.events (
  id BIGINT,
  ts TIMESTAMP,
  data STRING
) 
USING delta 
TBLPROPERTIES (
  'delta.feature.catalogManaged' = 'supported'
);

Compatibility

  • Hadoop version: When using the Delta 4.0.1 release with Spark 4.0.1 please use Hadoop 3.4.x; Using Hadoop 3.3.x may lead to dependency conflicts and unsupported/runtime issues. Please update any 3.3.x references in commands and dependency coordinates to 3.4.x.

Credits

Allison Portis, Anudeep Konaboina, Dhruv Arya, Felipe Pessoto, Lukas Rupprecht, Oussama Saoudi, Tathagata Das, Timothy Wang, Yi Li, Hao Jiang, Zheng Hu

Loading

Delta Lake 4.0.0

Delta Lake 4.0.0

Choose a tag to compare

View all tags
@allisonport-db allisonport-db released this 09 Jun 16:22
v4.0.0
6d055c5

We are excited to announce the final release of Delta Lake 4.0.0! This release includes several exciting new features.

Highlights

  • [Spark] Preview support for catalog-managed tables, a new table feature that transforms Delta Lake into a catalog-oriented lakehouse table format. This feature is still in the RFC stage, and as such, the protocol is still under development and is subject to change.
  • [Spark] Delta Connect is an extension for Spark Connect which enables the usage of Delta over Spark Connect, allowing Delta to be used with the decoupled client-server architecture of Spark Connect.
  • [Spark] Support for the Variant data type to enable semi-structured storage and data processing, for flexibility and performance.
  • [Spark] Support a new DROP FEATURE implementation that allows dropping table features instantly without truncating history.
  • [Kernel] Support for reading and writing version checksum.
  • [Kernel] Support reading log compaction files for better performance during snapshot construction, and support writing log compaction files as a post commit hook.
  • [Kernel] Support for the Clustered Table feature which enables defining and updating the clustering columns on a table.
  • [Kernel] Support for writing to row tracking enabled tables.
  • [Kernel] Support for writing file statistics to the Delta log when they are provided by the engine. This enables data skipping using query filters at read time.

Details by each component.

Sunset of Delta Standalone and dependent connectors

Currently, Delta Standalone and its dependent connectors, including Delta Flink and Delta Hive, are no longer under active development. Starting in Delta 4.0 we will not be releasing these projects as part of the 4.x Delta releases. These connectors are in maintenance mode and, going forward, will only receive critical security fixes and high-severity bug patches in the 3.x series. We are committed to a full transition from Delta Standalone to Delta Kernel and a future Kernel-based Flink connector.

Delta Spark

Delta Spark 4.0 is built on Apache Spark™ 4.0 . Similar to Apache Spark, we have released Maven artifacts for Scala 2.13.

The key features of this release are:

  • Delta Connect adds Spark Connect support to Scala and Python APIs of Delta Lake for Apache Spark. Spark Connect is a new project released in Apache Spark 4.0 that adds a decoupled client-server infrastructure which allows remote connectivity from Spark from everywhere. Delta Connect makes the DeltaTable interfaces compatible with the new Spark Connect protocol. For more information on how to use Delta Connect, see the Delta Connect documentation. Delta Connect is currently in preview.
  • Preview support for catalog-managed tables: Delta Spark now supports reading from and writing to tables that have the catalogOwned-preview feature enabled. This feature allows a catalog to broker all commits to the table it manages, giving the catalog the control and visibility it needs to prevent invalid operations (e.g. commits that violate foreign key constraints), enforce security and access controls, and opens the door for future performance optimizations. Currently write support includes INSERT, MERGE INTO, UPDATE, and DELETE operations.
    • Note: this feature is still in the RFC stage, and as such, the protocol is still under development and is subject to change. The catalogOwned-preview feature should not be enabled for production tables and tables created with this preview feature enabled may not be compatible with future Delta Spark releases.
  • Support for Variant data type: The Variant data type is a new Apache Spark data type. The Variant data type enables flexible, and efficient processing of semi-structured data, without a user-specified schema. Variant data does not require a fixed schema on write. Instead, Variant data is queried using the schema-on-read approach. The Variant data type allows flexible ingestion by not requiring a write schema, and enables faster processing with the Spark Variant binary encoding format. This feature was originally released in preview as part of Delta 4.0.0 Preview, as of 4.0.0 this feature is no longer in preview. Please see the documentation and the example for more details.
  • Preview support for shredded variants: Shredded variants are a storage optimization which allow for efficient sub-field extraction at the cost of higher write overhead, showing up to 20x read performance improvement. Shredded Variant data is stored according to the Parquet Variant Shredding specification. See the variantShredding RFC for more details.
    • Note that this feature is in preview and that tables created with this preview feature enabled may not be compatible with future Delta Spark releases.
  • Type Widening now supports a broader set of type changes and is no longer in preview. This feature allows you to change the data type of a column in your Delta table without rewriting the underlying data files. See the type widening documentation for a list of all supported type changes and additional information. Delta 3.3 or above is required to read tables with type widening enabled.
  • Support dropping table features without truncating history: The current drop feature implementation requires the execution of the command twice with a 24 hour waiting time in between. In addition, it also results in the truncation of the history of the Delta table to the last 24 hours. The new DROP FEATURE implementation allows dropping features instantly without truncating history. Dropping a feature introduces a new writer feature to the table, the checkpointProtection feature.
    • Dropping a feature with the new behaviour can be achieved as follows:
    ALTER TABLE table_name DROP FEATURE feature_name
    • We can still drop a feature with the old behavior as follows:
    ALTER TABLE table_name DROP FEATURE feature_name TRUNCATE HISTORY
    • The checkpointProtection feature can be dropped with history truncation.

Other notable changes include:

  • Support dropping table features using the DeltaTable Scala/Python APIs with deltaTable.dropFeatureSupport.
  • Support dropping the deletionVector table feature.
  • Support DataFrameReader options to unblock non-additive schema changes when streaming.
  • Invariant checks for DML commands to detect potential bugs in Delta or Spark earlier during execution and prevent committing the transaction in these cases.
  • Support the timestampdiff and timestampadd expressions for generated columns.
  • Support sorting within partitions when Z-ordering. This can be e...
Read more
Loading

Delta Lake 3.3.2

Delta Lake 3.3.2

Choose a tag to compare

View all tags
@raveeram-db raveeram-db released this 31 May 00:20
v3.3.2
051aac5

We are excited to announce the release of Delta Lake 3.3.2! This release contains several important bug fixes and improvements to the 3.3.1 release and it is recommended that users upgrade to 3.3.2.

Component specific bug fixes are detailed below.

Delta Spark

Delta Spark 3.3.2 is built on Apache Spark™ 3.5.3. Similarly to Apache Spark, we have released Maven artifacts for both Scala 2.12 and Scala 2.13.

The key fixes in this release are:

  • Fix to clean up stale checksum files during Metadata cleanup to improve table maintenance

Delta Kernel

The key fixes in this release are:

  • Kernel improvement to eliminate dependencies on package-private Parquet classes for better compatibility with JVM environments with multiple class loaders.

Other projects

Delta Flink (Delta-Standalone based)

The key fixes in this release are:

  • Flink fix to correct mapping between Delta's BinaryType and Flink's data types for improved type compatibility.

Credits

Dhruv Arya, Prakhar Jain, Venkateshwar Korukanti, Scott Sandre

Loading

Delta Lake 3.3.1

Delta Lake 3.3.1

Choose a tag to compare

View all tags
@raveeram-db raveeram-db released this 05 May 03:21
v3.3.1
bee74a2

We are excited to announce the release of Delta Lake 3.3.1! This release contains a few bug fixes to the 3.3.0 release and it is recommended that users upgrade to 3.3.1.

Component specific bug fixes are detailed below.

Delta Spark

Delta Spark 3.3.1 is built on Apache Spark™ 3.5.3. Similarly to Apache Spark, we have released Maven artifacts for both Scala 2.12 and Scala 2.13.

The key fixes in this release are:

  • Fix to allow user-specified on read if consistent with the table schema
  • Documentation update for Row Tracking to include Row Tracking Backfill introduced in Delta 3.3

Delta Kernel

The key fixes in this release are:

  • Kernel fix to handle non-uniform value types in map[string, string] in delta commit files

Other projects

No fixes or changes were made in the components below in this release but the corresponding artifacts are listed.

Delta Universal Format (UniForm)

Delta Sharing Spark

Credits

Wenchen Fan, Thang Long Vu

Loading