Start building Docker images and publishing them to Docker Hub (or other storage)

[ivanyu]

Users will benefit from having Karapace in the form of Docker image. For example, this might be useful for test environments.

[krisek]

I started to build a Dockerfile, current challenges:

• current HEAD depends on aiven/kafka-python origin/aiven-20212201 (luckily it can be installed with pip install https://github.com/aiven/kafka-python/archive/aiven-20212201.zip)
• karapace is not distributed as pip package, and it cannot be installed from git; ie. pip install doesn't work from git archive due to various issues (starting with versions.py - unfortunately version: don't read Makefile short_ver #155 doesn't fix it, karapace/version.py missing when building with pip #157 opened), therefore a prebuilt pip package needs to be copied into the container, which means the container build is more complicated
• it's a minor issue (most probably I miss something here) that for whatever reason I couldn't manage multistage build, the resulting container works only if you leave all build dependencies in the container image

[hackaugusto]

Hey @krisek, thanks for your comment!

I'm also working on updating our docker recipe, the draft PR is here #149. Maybe that could interesting for you to look at. One important note, that Dockerfile is only optimized for reuse of the caching layers, it seems you're also interested in optimizing the image size, let me know how important is that for you. (Edit: I have updated the Dockerfile to generate a small image).

Here are some additional details for each of your points:

1. Unfortunately the packaging tooling does not support installations from git since 2019-01-22 [setuptools]. Here we have a few options:
   • Always have the code in master branch working with unpatched dependencies. This is far from ideal, since the package available in public PyPI repository would be behind in bugfixes/functionality, but at least would provide a way to install karapace with pip.
   • Release to PyPI a -aiven version of the dependencies, with the patches applied. This would allow karapace to be installed with a plain pip install, but this not really desirable for other reasons. Namely we would prefer to contribute back to our dependencies instead of forking them, this would also lead to package conflicts, and probably general confusion on which package to install.
   • Use pip with a requirements.txt/contraints.txt file. This is the preferred approach ref, but it can be kinda of painful as you have experienced. In this case what I can do is improve our installation documentation (sorry about that, you got me in the process of updating our packaging).
2. There are many things happening here. (1) A PyPI package would not really work for the reasons mentioned above. Our idea is to rely on the docker images to fix the issue. (2) Installation requires the VCS repo, it won't work out-of-the-box with source only. This is again a tooling problem since there are multiple ways of sourcing the version. Using the source control allows us to tag exactly which commit is being used, this is significantly better for installs that are not a release, the alternative would be to use a postfix like -snapshot or -$(date) which only tells us that it is not a release installation. (3) There are two approaches one can take here for a image generation:
   • Generate a .whl outside of the container and install it in the container
   • Expose the whole repo to the container to the container and do the source install from it. Both approaches are valid, I went with this second option just because all the commands are contained in a Dockerfile instead of having multiple files that have to be executed in an specific order.
3. I can't say much about this, it depends on which dependencies you're talking about. The list install_requires are the runtime dependencies, all of those are required for the code to run.

Hope that helps.

[krisek]

it seems you're also interested in optimizing the image size, let me know how important is that for you

Let's say, not the size of the image, but the unncessary dependencies: it can save me from legthy discussions with our security departement if we can avoid gcc or any sort of unnecessary dependencies on a docker image going to production environments.

2. alternative would be to use a postfix like -snapshot or -$(date) which only tells us that it is not a release installation.

Would this be accepted as PR? I can do it quickly.

[hackaugusto]

Would this be accepted as PR? I can do it quickly.

Let's keep the current approach for the moment, I was just trying to clarify the options and show the rationale for using the VCS for the version. Thanks for the help though :)

[hackaugusto]

Images are being published to https://hub.docker.com/r/aivenoy/karapace/