Moving the DataFusion integration into a separate repository

[gabotechs]

I’d like to explore decoupling the development lifecycle of iceberg-datafusion from iceberg-rust by moving the existing DataFusion integration into a crate maintained in a separate repository.

As discussed in datafusion-distributed #599, we are exploring incorporating the integration into datafusion-distributed, making native distributed Iceberg support a core offering of that project. Another option worth exploring would be to create a dedicated repository, such as datafusion-contrib/datafusion-iceberg.

Regardless of its eventual home, the goal would be to allow the community to advance the Iceberg+DataFusion integration without adding review pressure for iceberg-rust maintainers. The integration could evolve at its own pace and be maintained by a different or broader group of contributors.

Some reasons why this may be worth considering:

• There appears to be a vision of making iceberg-rust engine-agnostic (related discussion, cc @blackmwk).
• Several integration-related PRs have had difficulty receiving review attention in this repository, such as #2671 and #2521.
• Other DataFusion-related projects have expressed interest in hosting an Iceberg integration, including Ballista #2174 and datafusion-distributed #599.

I’d like to hear from the community (particularly the iceberg-rust maintainers) about the possibility of moving iceberg-datafusion out of the iceberg-rust repository. This could allow projects such as Ballista and datafusion-distributed to share common functionality and contribute to the integration independently of the iceberg-rust release and review lifecycle.

[alamb]

Here is a past conversation as well

• [Discussion] Reduce the need for iceberg-rust forks / how can we grow the community? #1797

[alamb]

I’d like to explore decoupling the development lifecycle of iceberg-datafusion from iceberg-rust by moving the existing DataFusion integration into a crate maintained in a separate repository.

FWIW another option would be to leave the iceberg- datafusion crate in this repo for testing, and then create a new integration (maybe fork, maybe something entirely new) in datafusion-contrib that focused on staying up to date with the latest datafusion

That might be less disruptive to the iceberg-rust community, at least in the short term. As we see what grows in datafusion-contrib we could then evaluate if it made sense to move / combine the crates

FWIW I am happy to make a datafusion-contrib repo for this work

[gabotechs]

Yeah, I'd probably price this in in any solution. Whether an extra repo pops up or not with a new Iceberg integration, maintainers of this project are always free to keep the current integration here.

[alamb]

cc @Xuanwo and @CTTY (a recent active reviewer / merger)