This is the confluent stack with MSSQL, MySQL, MongoDB, Postgres and ElasticSearch, all tweaked to support the Materialized Views and Streaming ETL ksqlDB tutorials, seperated into linked docker compose stacks.
Neither tutorial worked exactly as documented at the time as this writing, there were errors in the confluent-hub commands to install the connectors (--worker-configs was not set to /dev/null), and the docker-compose.yml (missing docker images for latest, seems they autogenerate the docs but don't always validate the image tag), so unfortunately nothing just worked - initially.
Also, neither tutorial works on Windows, oddly, since the entire example uses docker... for some odd reason the tutorials use the confluent-hub command on the host, and confluent-hub does not work on Windows directly (though it seems to work in WSL2 if not blocked by a corporate security team).
Yet the entire example uses docker!
So rather then fix their tutorials one off it seemed more practical to roll everything into one development stack that could be used post-tutorial, on any platform.
It seems someone at Confluent does not understand the fundamental concept behind docker.
With all this, you can jump to "start the stack" in each tutorial, and follow the Kafka related commands, which are all OS agnostic. Everything works on Windows.
NOTE: The full stack will require a machine with at least 16GB of RAM, 32GB to be useful for development. 64GB is best. Make sure to increase your default Docker Desktop resource settings to at least 16GB of RAM and 4 vCPU, or the stack is likely to experience instability and resource starvation crashes (the ksqldb-server tends to die suddenly). The stacks were broken into seperate docker compose segments to allow for easier management of machine resources. They all connect to the same default docker network created by the confluent stack initially.
On the test Windows machine Docker Docker Swap resources were also increased. The confluent stack alone consumed 12.36GB of RAM idle at the 5 min mark. With the postgres-mongo-elastic stack added 14.04 GB RAM.
Side note for VM users... If you are running Docker Desktop for Windows in a VM using nested virtualization WSL2 will not be your friend, and any performance gains will be lost as the VMMem process eats up CPU. The test machine uses Hyper-V as the Docker Desktop engine to mitigate this issue, with much better results in CPU usage and usability.
All stacks are dependent on the confluent stack network, which must be started first.
the confluent stack, from the downloaded original-cp-stack, modified to pre-load required connectors.
joins an mssql db to the confluent stack for CDC testing
joins a mysql db to the confuent for the mat view tutorial
joins the postgres, mongo and elastic servers for the streaming etl tutorial
cd stacks/confluent
docker-compose up -d # starts the confluent stack
cd ../postgres-mongo-elastic
docker-compose up -d # starts the servers required for the streaming ETL tutorial
On Windows, the command to start the stack is slightly different (no hyphen):
docker compose up -d
Instead of doubling up on the ksqlDB server and turning it into a connect server, this approach simply connects the ksqlDB server to the connect server, the proper way, and loads the connectors at startup. You can add additional connectors as required in the confluent stack connect server startup.