[Feature Request] [Connectors] [Kernel] Request for a Flink connector using the Delta Kernel

[raveeram-db]

Feature request

Which Delta project/connector is this regarding?

• Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Overview

This feature request is to build a Connector for Flink (specifically Flink 2.0) using the Java Delta Kernel (which is purpose-built to enable Connector development while abstracting away Delta protocol details) instead of the existing, deprecated Delta Standalone based Flink Connector.

Motivation

The current Flink connector is built on Delta Standalone, which is no longer under active development as noted in the Delta 3.1.0 release notes. Delta Kernel is the forward-looking foundation for connectors, with support for features like Deletion Vectors, Column Mapping, IcebergCompatV1/V2 etc. A Flink connector built on the Kernel would unlock access to these features and better align with the long-term direction of the Delta project.

Further details

[Public] [External] Design Doc: Delta-Flink Sink built on Delta Kernel

Willingness to contribute

The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?

• Yes. I can contribute this feature independently.
• Yes. I would be willing to contribute this feature with guidance from the Delta Lake community.
• No. I cannot contribute this feature at this time.

[mohsenrezaeithe]

I'm interested in contributing to this with guidance from the community.

[newfront]

The flink 2 kernel based connector is here (it is currently experimental and will stabalize in future releases). You can use https://repo1.maven.org/maven2/io/delta/delta-flink/4.2.0/ to get started with the new Kernel powered Flink connector.

https://github.com/delta-io/delta/blob/master/flink/docker/2.0/docker-compose.yml