docs: add basic docker-compose file and instructions

[big-andy-coates]

Description

Added a sample docker-compose file with some steps for how to use. Brings up multi-node KSQL with dependants.

Only outstanding issue is it doesn't look like community members can build docker images locally, as they won't have access to a AWS based docker repo, right? Does anyone know a way around this, or plans to make this publicly available? cc @apurvam @rodesai @agavra

Testing done

manually checked it all works.

Reviewer checklist

• Ensure docs are updated if necessary. (eg. if a user visible feature is being added or changed).
• Ensure relevant issues are linked (description should include text like "Fixes #")

[vinothchandar]

Public dockerHub? We have stuff here already https://hub.docker.com/r/confluentinc/ksqldb-server

[vinothchandar]

As you mentioned, it would be good to have this be done without the need for aws ecr, if possible.

I will let you make the call and merge when ready.

[vinothchandar]

can we just say ksqldb-server-1 and ksqldb-server-2? We can't really control active/standby assignment and ensure this container will be active always right?

[big-andy-coates]

It's not about active / standby. It's about scaling the cluster.

The primary is at a known address and can't be scaled. The other is at an unknown address and can be scaled.

I originally started with server-1 and server-2, but the command line is better this way around e.g. docker-compose up -scale ksqldb-server 3 = 4 node cluster.

Might switch ksqldb-server to additional-ksqldb-server e.g. docker-compose up -scale addtional-ksqldb-server 3, which is probably more intuitive that its a 4 node cluster.

[vinothchandar]

Just want to check if this should be 29092? since thats the port that gets exposed for the kafka container?

[big-andy-coates]

This is correct.

[agavra]

FYI - if you want to build the image without ECR you can change the dockerfile not to depend on cp-base-new and just use something open source like the openjdk:8-jre-alpine image

[agavra]

Also cc @MichaelDrogalis @derekjn - do we want to have this docker compose in the repo differ from whats on https://ksqldb.io/quickstart.html?

[derekjn]

Also cc @MichaelDrogalis @derekjn - do we want to have this docker compose in the repo differ from whats on https://ksqldb.io/quickstart.html?

I think it's fine if they're different. The quickstart is for getting a minimal development environment up and running. As long as the same images are being used everywhere, I don't think there is a need to maintain uniformity across all of the different docker-compose files that may be using these images.

[big-andy-coates]

Public dockerHub? We have stuff here already https://hub.docker.com/r/confluentinc/ksqldb-server

I'm aware, but its the build that depends on our ECR.

FYI - if you want to build the image without ECR you can change the dockerfile not to depend on cp-base-new and just use something open source like the openjdk:8-jre-alpine image

Good to know - I'll take a look. However if this something we want to do? I'm not sure of the repercussions...

It seems our only unavailable dependency is cp-base-new. However there is a cp-base image published. I'll ping the build team to see what the difference is, which we should be using, and if we can publish the new one to docker hub...

I'll raise follow up if we're switching base image.

Also cc @MichaelDrogalis @derekjn - do we want to have this docker compose in the repo differ from whats on https://ksqldb.io/quickstart.html?

Ah, didn't know about that. We should maybe keep them inline, the difference between the one in this PR and the one in the example is:

• This one uses latest version, docs on uses hardcoded version. Using latest is probably less overhead when versions change.
• This one defaults to two KSQL nodes and can be scaled: docker-compose up -scale additional-ksqldb-server 2. Not required, but useful.
• This one sets rf and min isr on the transaction log, which I believe is required with latest Kafka, so the example on will need updating with this. e.g.
• This one depends on SR, which enables Avro format.

cc @JimGalasyn

[JimGalasyn]

@big-andy-coates This is awesome! @ybyzek may find it useful, too. I'll copy it over to the docs-md directory.