MicroK8s' built-in registry declared for the Camel K installation, but not used - integration installations fail

[bthj]

Installing Camel K to a local MicroK8s cluster with

kamel install --registry-insecure true --registry localhost:32000 --kaniko-build-cache=false --force --wait --example

successfully installs the camel-k-operator but the example integration installed with the --example flag fails to build, with those messages:

$ kubectl logs -f camel-k-kit-bpu91bq2032o5jv6daig-builder
STEP 1: FROM adoptopenjdk/openjdk8:slim
Getting image source signatures
Copying blob sha256:24e9ea251047e2f8f6e81eee2c7468307906b95d988f92039c6e13a6d764ae3e
Copying blob sha256:f11b29a9c7306674a9479158c1b4259938af11b97359d9ac02030cc1095e9ed1
Copying blob sha256:5bed26d33875e6da1d9ff9a1054c5fef3bbeb22ee979e14b72acf72528de007b
Copying blob sha256:930bda195c84cf132344bf38edcad255317382f910503fef234a9ce3bff0f4dd
Copying blob sha256:022f48766a856d5901451ec6d39fd887807f9c3fa64dd45dabaa935ba70561e7
Copying blob sha256:78bf9a5ad49e4ae42a83f4995ade4efc096f78fd38299cf05bc041e8cdda2a36
Copying blob sha256:0a5ab47c17c313991b8420037ab81f08ca958498b4331e2a352e530019f42b05
Copying config sha256:2d20634b71d274c9aa0c13463b0c92f177cc8a5b1ee5e5947f8b14e2045ba014
Writing manifest to image destination
Storing signatures
level=error msg="Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: permission denied"
error creating build container: The following failures happened while trying to pull image specified by "adoptopenjdk/openjdk8:slim" based on search registries in /etc/containers/registries.conf:
* "localhost/adoptopenjdk/openjdk8:slim": Error initializing source docker://localhost/adoptopenjdk/openjdk8:slim: error pinging docker registry localhost: Get http://localhost/v2/: dial tcp [::1]:80: connect: connection refused
* "docker.io/adoptopenjdk/openjdk8:slim": Error committing the finished image: error adding layer with blob "sha256:5bed26d33875e6da1d9ff9a1054c5fef3bbeb22ee979e14b72acf72528de007b": ApplyLayer exit status 1 stdout:  stderr: permission denied
* "registry.fedoraproject.org/adoptopenjdk/openjdk8:slim": Error initializing source docker://registry.fedoraproject.org/adoptopenjdk/openjdk8:slim: Error reading manifest slim in registry.fedoraproject.org/adoptopenjdk/openjdk8: manifest unknown: manifest unknown
* "registry.access.redhat.com/adoptopenjdk/openjdk8:slim": Error initializing source docker://registry.access.redhat.com/adoptopenjdk/openjdk8:slim: Error reading manifest slim in registry.access.redhat.com/adoptopenjdk/openjdk8: name unknown: Repo not found
* "registry.centos.org/adoptopenjdk/openjdk8:slim": Error initializing source docker://registry.centos.org/adoptopenjdk/openjdk8:slim: Error reading manifest slim in registry.centos.org/adoptopenjdk/openjdk8: manifest unknown: manifest unknown
* "quay.io/adoptopenjdk/openjdk8:slim": Error initializing source docker://quay.io/adoptopenjdk/openjdk8:slim: Error reading manifest slim in quay.io/adoptopenjdk/openjdk8: unauthorized: access to the requested resource is not authorized
level=error msg="exit status 1"

MicroK8s' built-in registry is referenced in the kamel install command with the flags --registry-insecure true --registry localhost:32000 and that registry has proven to be up and running properly for other deployments.

According to the log output, the registry declared with the installation flags isn't being called, but rather http://localhost/v2/(so not localhost on port 32000 as desired).

Is there something I need to add to the Camel K installation command?

[PhilAndrew]

Hi @bthj have you been able to get camel-k to run with MicroK8S? I had the same problem, but I would like the two to work together.

[lsalotto-ingenia]

Hi all, any updates on this? Did any of you got this working?

[bthj]

Hi - haven't had an opportunity to look further into this.

[ReggieCarey]

I just tried to install on MicroK8s and it complained initially.

However I attempted the following and it appears to have worked:

$ kamel install --registry-insecure true --registry localhost:32000/v1
OLM is not available in the cluster. Fallback to regular installation.
Camel K installed in namespace default 

$ kubectl get all
NAME                                    READY   STATUS    RESTARTS   AGE
pod/camel-k-operator-7bb969fb8b-t4jqd   1/1     Running   0          108s

NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.152.183.1   <none>        443/TCP   17h

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/camel-k-operator   1/1     1            1           108s

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/camel-k-operator-7bb969fb8b   1         1         1       108s

[tadayosi]

@ReggieCarey What happens if you run some example app with the operator? The original issue is when you run some app on MicroK8s. Thanks.

[ReggieCarey]

I could not get integrations to work.

…

Sent from my iPhone

On Oct 7, 2021, at 21:18, Tadayoshi Sato ***@***.***> wrote:  @ReggieCarey What happens if you run some example app with the operator? The original issue is when you run some app on MicroK8s. Thanks. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[johnpoth]

Ah yes I think this happens because both camel-k and the container runtime (so containerd in this case) need to access the image registry in a unified way. While using localhost:32000 will work locally and for containerd, it will not work in the pod camel-k runs in.

One easy way to to reach the image registry from the camel-k pod is to use the image registry's ClusterIP set in it's Service, so something like:

$kubectl get service registry -n container-registry -o yaml
apiVersion: v1
kind: Service
...
  clusterIP: 10.152.183.32
  clusterIPs:
  - 10.152.183.32

Now using 10.152.183.32:5000 will work in the pod camel-k is running on so pushing images will work. However containerd won't be able to pull images and create integration pods. It will try to use https and complain.

To setup containerd to use 10.152.183.32:5000 as an insecure registry, one can edit it's configuration file. When MicroK8s is installed with snap it is located under /var/snap/microk8s/current/args/containerd-template.toml more info here. You can then simply add:

[plugins."io.containerd.grpc.v1.cri".registry.mirrors."10.152.183.32:5000"]
        endpoint = ["http://10.152.183.32:5000"]

Then restart MicroK8s as stated in the docs. Then camel-k can be installed properly with (replace with your registry's ClusterIP)

kamel install --registry="10.152.183.32:5000" --registry-insecure=true

I just tested locally and I'm able to run integrations successfully.

HTH !

ps: it might cool to write a doc about installing camel-k on MicroK8s if people are interested

[github-actions]

This issue has been automatically marked as stale due to 90 days of inactivity.
It will be closed if no further activity occurs within 15 days.
If you think that’s incorrect or the issue should never stale, please simply write any comment.
Thanks for your contributions!