Remove Kafka hard dependency in Uber mode

[lbroudoux]

Reason/Context

When enabling Asynchronous features, Kafka is used today as the communication backend between the main app container and the async-minion container. In fact we're just using a single microcks-services-updates topic so that changes to a service definition can be propagated to the async-minion. See https://microcks.io/documentation/installing/deployment-options/#architecture-for-async-api

Using Kafka makes sense at scale because it allows true decoupling between the number of app containers and the number and specialization of async-minion containers (some may be focused on different protocols, different publication frequencies, etc...). Also, Kafka is the number 1 broker used in big organizations.

With the introduction of the uber distribution, we see a growing demand for using Microcks within the development inner-loop using Testcontainers. In this use-case Kafka represents an extra container (that can consume lots of resources) that may be used only by a few developers as we see a rise in usages of pure WebSocket communication or other stuff like AWS SNS and SQS (probably using LocalStack as well). Moreover, in this situation, we need 1 app container and 1 async-minion container as we don't have to support huge loads in a single developer workflow.

That's the reason why we'd like to remove the hard dependency to Kafka if enabling Asynchronous features on uber mode (and on Testcontainers as a consequence). The fact is that Kafka communication can be easily replaced by WebSocket connection between the 2 containers.

Description

When using uber image and uber Spring profile:

• the service change events will be published through a WebSocket on app container only. The path of this WS endpoint could be /api/services-updates to align with Kafka topic names and existing APIs,
• the async-minion component will connect and listen to this WS endpoint, not Kafka broker.

That way, enabling Asynchronous features will only require an addition container.

Implementation ideas

WebSocket publication and dependencies will only be integrated into the microcks-uber container image. The publication channel will replace the existing Kafka one in core Microcks.

On the other hand, we also plan to have a microcks-async-minion-uber container image where the Kafka @Incoming connector will be disabled and replaced by a WebSocket client connected to the main app container.

The 2 uber images will be also configured to allow reasonable defaults and ease the pain regarding configuration through Testcontainers interface.

[lbroudoux]

This dependency has now been dropped and you can use quay.io/microcks/microcks-async-minion-uber:nightly in combination with quay.io/microcks/microcks-uber:nightly.

For a quick test, you can run:

docker run -p 8585:8080 -p 8686:9090 -it quay.io/microcks/microcks-uber:nightly

to start a simple standalone instance of Microcks main container. Then, start the new minion with the information on how to connect to Microcks main container.

docker run -p 8586:8081 -e MICROCKS_HOST_PORT=host.docker.internal:8585 -it quay.io/microcks/microcks-async-minion-uber:nightly

As soon as started, this one takes care of connecting through WebSocket to get updates from the main container 😉