[#1190] Apache TubeMQ (InLong) Adapter & Sink

[SteveYurongSu]

Background

Apache TubeMQ (now is a submodule of Apache Inlong) is a trillion-records-scale distributed messaging queue (MQ) system, focuses on data transmission and storage under massive data. Compared to many open source MQ projects, TubeMQ has unique advantages in terms of stability, performance, and low cost.

It offers a variety of features:

• Pluggable transport protocols, such as TCP, SSL
• Support big-data and streaming ecosystem integration
• Message retroactivity by time or offset
• Efficient pull and push consumption model
• Flexible distributed scale-out deployment architecture
• Feature-rich administrative dashboard for configuration
• Authentication and authorization

For more information:

• TubeMQ VS Kafka

Approach

Connect Adapter:
Introduce the TubeMQProtocol for reading data from TubeMQ brokers.

Sink:
Introduce the TubeMQPublisherSink for writing data to TubeMQ brokers.

Remarks

PR introduces (a) breaking change(s): <yes/no>
no

PR introduces (a) deprecation(s): <yes/no>
no

[tenthe]

Thanks a lot for the PR.
I added a minor comment, but this is a rather small change.

One more hint. For sinks we have a newer API StreamPipesDataSink, this requires only one class instead of three. Here are two example [1], [2].

This can also be addressed later, the requested changes for the PR are only related to the wrong service name.

[1] https://github.com/apache/streampipes/blob/dev/streampipes-extensions/streampipes-sinks-internal-jvm/src/main/java/org/apache/streampipes/sinks/internal/jvm/datalake/DataLakeSink.java
[2] https://github.com/apache/streampipes/blob/dev/streampipes-extensions/streampipes-sinks-brokers-jvm/src/main/java/org/apache/streampipes/sinks/brokers/jvm/pulsar/PulsarPublisherSink.java

[SteveYurongSu]

@tenthe Thanks a lot for the reviewing!

Sorry for the wrong service name. I have fixed them.

One more hint. For sinks we have a newer API StreamPipesDataSink, this requires only one class instead of three. Here are two example [1], [2].

Yes, this time I already implemented TubeMQPublisherSink based on the latest API StreamPipesDataSink. The implementation experience was much better :)

[bossenti]

thanks a lot for your contribution @SteveYurongSu 🙏🏼
Impressive how far you got on your own!
How was your experience with the extensibility of StreamPipes for adapters and sinks? Did you get the required information easily?

[bossenti]

Not directly required, but it would be awesome if you could add a short descriptionof the input parameters here

[SteveYurongSu]

Good point! Description is added :D

[bossenti]

Thanks a lot 🙏🏼

[dominikriemer]

Awesome, thank you!

[SteveYurongSu]

How was your experience with the extensibility of StreamPipes for adapters and sinks? Did you get the required information easily?

Overall, the experience was very good.

I mainly got the necessary information from the following two links:

• https://streampipes.apache.org/docs/docs/extend-setup.html
• https://cwiki.apache.org/confluence/display/STREAMPIPES/Extensions

In addition, I have to say that streampipes are too suitable for iot developers! I'm going to contribute some more code related to the IoTDB eco :)

[bossenti]

Glad to hear!
That would be awesome 🤩
We really appreciate your contributions ❤️

[vesense]

Nick work! Thanks @SteveYurongSu