Kafka Provider using confluent_kafka

[knxacgcg]

There is an open PR #7407 which tries to address the lack of a Kafka implementation, but has gone stale and does not seem feature complete.

Is a confluent_kafka based implementation (for Sensor and Producer) acceptable in place of the kafka-python implementation? We are currently working on one and would like to contribute back to the main project.

It supports:

• KafkaProducer
• KafkaConsumer
• Serializer/Deserializers (BYO or use confluents)
• Schema Registry Support
• Authentication Support

[aviflax]

(Looks like #7407 has been superseded by #12388, which unfortunately also seems to be stalled.)

FWIW, I’m working on a new DAG and the provider you’re describing would be perfect for my needs. I’d love to see it bundled into Airflow, but if there’s any way for me to use it in the meantime, even if it’s a WIP, that’d be super excellent. I’d be happy to contribute bug reports, bug fixes, docs, etc.

[knxacgcg]

hey @aviflax , @potiuk, thanks for the lively discussion!
the python client used by #12388 doesnt seem to be as fully featured as the confluent client, so it would pretty much be an entire re-write of that issue.

I'm happy to deliver this as a plugin as well. I was actually a little unsure of the difference between a Plugin and a custom provider package. Is the difference only the Plugin class definition?

Ideally, I'd like to push this as part of airflow's main project for supportability / ease of use, but plugin / or custom provider would also work.

[aviflax]

I'm happy to deliver this as a plugin as well.

Great!

I was actually a little unsure of the difference between a Plugin and a custom provider package. Is the difference only the Plugin class definition?

Yeah, I don’t know; I’m still fairly n00bish with Airflow. @potiuk maybe you could clarify this for us?

@knxacgcg in the meantime I’d love to get a peek at your code. Is it accessible anywhere?

Thanks!

[potiuk]

Provider is the best and yeah - it should be part of community - located in 'airflow/providers/kafka' - alongside other Apache providers. Plugins are mainly if you want to modify airflow UI components (custom views or custom 'airflow' behaviour as a whole, or add custom macros. Providers are when you introduce integration with another 'service' or 'tool' through operators/sensors/transfers and accompanying connections.

[knxacgcg]

@aviflax , @potiuk

thanks for the clarification!
We need to clean up some things and should have an initial fork / PR out in the next couple weeks

[aviflax]

Excellent! Thank you @potiuk and @knxacgcg!