Break Kafka operators into their own streamsx.kafka repository?

[Alex-Cook4]

I would to start a discussion on whether we should move Kafka into it's own streamsx.kafka repository.

Kafka still has not done a 1.0 release, therefore has been much more volatile than the other messaging clients. The move from 0.8 to 0.9 clients broke compatibility with 0.8 Kafka brokers. I have just proposed #244 to upgrade to Kafka 0.10, which will break compatibility with 0.9 Kafka brokers.

For developers who want the latest fixes for other messaging clients, but are still on older (now not supported) Kafka brokers, you can end up in situations where you want different versions of the same toolkit for the same application, which can't be done.

A concrete example of this is users who want to use RabbitMQ and Kafka 0.8. Since RabbitMQ was added after upgrading to Kafka 0.9, there is no way to have the same application work with RabbitMQ and Kafka 0.8 without breaking up the toolkit.

Thoughts?

[mikespicer]

I would support Kafka being in its own toolkit. In addition to the versioning issues I also think it will help with visibility as Kafka is a very popular system and it would be beneficial to have our support for it be more visible. I expect that it could also help with application bundle sizes.

[chanskw]

+1. I am in favor of breaking up the messaging toolkit into smaller toolkits in general.

I think the messaging toolkit should be splitted into these going forward:
com.ibm.streamsx.messaging.jms
com.ibm.streamsx.messaging.kafka
com.ibm.streamsx.messaging.rabbitmq
com.ibm.streamsx.messaging.mqtt

For XMS, I propose to not move it forward. If people need this, they can still use the "old" messaging toolkits.

[Alex-Cook4]

I like the idea of splitting them all out...would we still want the "messaging" in the namespace for all of them? Maybe just:

com.ibm.streamsx.jms
com.ibm.streamsx.kafka etc

[chanskw]

I kept the namespace for compatibility reasons. So, when users move to the new toolkits, the operators remain compatible and do not need to modify code.

[mikespicer]

• Splitting out the kafka and rabbitMQ toolkits makes sense.
• I'm not sure that splitting out JMS, MQTT and XMS is helpful or necessary. They have a number of common elements and are fairly stable.
• Having XMS only being supported in the old version of the toolkit with all the others in it seems like we would be abandoning it and I don't think thats a good idea.

[chanskw]

I think we should consider deprecating xms.

As discussed before, there is really no difference in support between JMS and XMS. In our performance testing, JMS is at least 2x faster than XMS. XMS is much harder to install and does not support PPC and RH7. The recommended approach is to use JMS instead of XMS. So we are just keeping it around for compatibility purposes.

[chanskw]

As for MQTT and JMS, MQTT has been going through some changes to support various security schemes for the IOT platform. I also proposed to split them up for consistency, and to make the app bundle size smaller.

[Alex-Cook4]

Do we want to revisit this? Still seems like a good idea...

[chanskw]

streamsx.kafka is created. @cancilla is going to work on moving the operators over.

[schubon]

Customers are using the (from-scratch) new toolkit https://github.com/IBMStreams/streamsx.kafka that was started from scratch. If you still encounter the issue there, open an issue over there, please.