[Question] [Feature] Add sensor capability to listen to Kafka Pipeline events

[sharmalalit]

Dear Airflow Maintainers,

I was looking at AirFlow documentation [0] and trying to figure out it it is possible to wait for an event in Kafka Pipeline to occur and then generate a task in DAG.

Is it possible to extend existing sensors to achieve this.

[0] - http://pythonhosted.org/airflow/concepts.html#operators

[yuanfeiz]

I am working on the similar feature, would you like to provide more information about your requirements?

• Would you configure anything besides the topic, for example, the offset, partition, consumer-groups?
• Would you use a cluster of broker servers or just use one single server?
• Do you need more fine-grained control over the message fetching, for example, the timeout?