Publish container images for kcp

[lburgazzoli]

One area where kcp could also be useful is for testing controllers and operators and having a container image could simplify the test set-up as example, I'm currently working on an operator written using the java-operator-sdk where I don't need all the features and controller provided by a standard kubernetes distro but only a control plane so having kcp available as a container image would be very useful as I can leverage testcontainers to spin up a kcp instance as part of the test lifecycle.

[imjasonh]

Images are already published in CI to GHCR: https://github.com/kcp-dev/kcp/pkgs/container/kcp%2Fsyncer-c2e3073d5026a8f7f2c47a50c16bdbec -- with tags for short commit SHAs:

docker pull ghcr.io/kcp-dev/kcp/syncer-c2e3073d5026a8f7f2c47a50c16bdbec:7471fb9

We can improve this over time (in particular the image name, which is terrible). If you try it out and give feedback about what worked/didn't that would help us improve it.

[lburgazzoli]

As per #111, I tough the image is just the syncer, not kcp.
Am I wrong ?

[imjasonh]

You are totally correct, sorry for misreading.

It should be easy enough to build the image, but if the intention is to run kcp inside a Kubernetes cluster, I'm not sure that's something we've really explored or prioritized yet, so things might not work very well. In particular, kcp presents a K8s control plane backed by etcd, which should run fine inside K8s with its own etcd storage, but it might end up being confusing to have both running.

[lburgazzoli]

if the intention is to run kcp inside a Kubernetes cluster

Not at all, my only goal is to be able to spin up kcp as part of my integration tests running locally on my PC or as part of GitHub Actions. As today I use kind or minikube but they are overkill for my use case where in essence I don't need any of the controller that they ship.

[metacosm]

Hehe, good to see you're on top of things, @lburgazzoli! I was thinking that kcp could indeed be interesting in a testing context for operators.

[lburgazzoli]

It should be easy enough to build the image,

so yes it is quite simple to generate a container image for kcp with ko however, what should be the best and easier way to access the kcp ?

I initially tough to use volumes thus having the .kcp persisted outside of the container but it may cause troubles:

➜ docker run --rm -ti -v $PWD/conf:/.kcp/data:Z -p 6443:6443 ko.local/kcp.go start

{
  "level":"warn",
  "ts":1634314144.2230017,
  "caller":"fileutil/fileutil.go:57",
  "msg":"check file permission","error":"directory \".kcp/data\" exist, but the permission is \"dtrwxr-xr-x\". The recommended permission is \"-rwx------\" to prevent possible unprivileged access to the data"
}

Even if kcp starts, then content of the volume is owned by root and it is not accessible by standard users:

➜ ll conf/
total 16
-rw-------. 1 root root 6896 Oct 15 18:27 admin.kubeconfig
-rw-r--r--. 1 root root 2275 Oct 15 18:27 apiserver.crt
-rw-------. 1 root root 1675 Oct 15 18:27 apiserver.key
drwx------. 4 root root   29 Oct 15 18:27 member
drwx------. 5 root root   42 Oct 15 18:27 secrets

This can be fixed by using podman or rootles docker but it may require some additional set-up so, is there a better way to access a kcp instance running in a container ?

[ncdc]

We are now publishing a kcp image (see https://github.com/kcp-dev/kcp/pkgs/container/kcp).

@lburgazzoli can your test automation use a docker cp/podman cp command to copy the admin.kubeconfig out of the container for use by your tests?

[metacosm]

@lburgazzoli let me know how it goes, would be interested to see if this works nicely in GitHub Actions…

[lburgazzoli]

@ncdc @metacosm added an initial testcontainer impl that spins up a kcp server https://github.com/lburgazzoli/testcontainers-kcp

[ncdc]

I believe we can close this. If there are still any issues, please let us know.