This is a Docker image appropriate for running Flink in Kuberenetes. You can also run it locally with docker-compose in which case you get two containers by default:
flink-jobmanager- runs a Flink Job Manager in cluster mode and exposes a port for Flink and a port for the WebUI.flink-taskmanager- runs a Flink Task Manager and connects to the Flink Job Manager via static DNS nameflink-jobmanager.
The Docker setup is heavily influenced by docker-flink
You only need to build the Docker image if you have changed Dockerfile or the startup shell script, otherwise skip to the next step and start using directly.
To build, get the code from Github, change as desired and build an image by running make
Get the docker-compose.yml from Github and then use the following snippets
docker-compose up -d will start a Job Manager with a single Task Manager in background.
docker-compose scale flink-taskmanager=5 will scale to 5 Task Managers.
- Copy the Flink job JAR to the Job Manager
docker cp /path/to/job.jar $(docker ps --filter name=flink-jobmanager --format={{.ID}}):/job.jar to
- Copy the data to each Flink node if necessary
for i in $(docker ps --filter name=flink --format={{.ID}}); do
docker cp /path/to/data.csv $i:/data.csv
done- Run the job
docker exec -it $(docker ps --filter name=flink-jobmanager --format={{.ID}}) flink run -c <your_job_class> /job.jar [optional params]
where optional params could for example point to the dataset copied at the previous step.
Navigate to http://localhost:8081
docker-compose down shuts down the cluster.
You are very welcome to open an issue or a PR on Github.