Restructure the Kafka related operators

[kamir]

Proposal:

We create a wayang-kafka module inside of wayang-platforms.
This leads to a cleaner approach to add these operators.

Right now we edit both wayang-java and wayang-spark, where both of them contain versions of operators related to reading and writing to Kafka.

Conceptually, having all implementations for the specific APIs in wayang-kafka would be neat.

Then you could have wayang-kafka/src/main/java/JavaKafkaOperator wayang-kafka/src/main/java/SparkKafkaOperator etc. and the respective platforms themselves don't have to concern about Kafka.

It is planned to implement this as soon as the initial demos for Kafka integration are ready.

Target: Release 1.1 - Q1-2025.

[zkaoudi]

To take this further, we create another module called wayang-io (or some other name) and inside this parent module we place the wayang-kafka module that @kamir mentioned above. The reason for that being: Kafka is not a processing platform, only reads or writes data. So inside wayang-io we can have all the data sources and the data sinks.

[novatechflow]

+1 - that's a great idea! There we can also put our REST API or pure storage modules.