Kafka Support?

[gwthm-in]

Can Skale take the data from Kakfa topics ?

[mvertes]

A kafka connector is not yet integrated on the worker side.

However, on the master side one could consume and produce data using a nodeJS kafka connector such as kafka-node (other choices exist as well), and stream it to skale datasets for workers consumption using sc.objectStream. Keep in mind that it would be a non scalable workaround.

The real solution is of course to integrate the kafka module in the worker, and map one worker per kafka partition, which would allow parallel reading or writing to kafka. Something we plan to do, after adding a cassandra connector and avro support.

[tommedema]

@mvertes what are future plans of skale? is this being actively maintained, e.g. are feature requests such as worker kafka support being considered?