Unable to run an integration on Minikube

[michaeljmcd]

I created a Minikube cluster with the registry and olm addons and then follow the steps outlined below.

podman machine init --cpus 2 -m 4096
podman machine start
minikube start --driver=podman --container-runtime=containerd --addons=registry,dashboard,olm
kubectl config set-context --current --namespace=$NS 
kamel install --global
kamel init Example.java # Just the default, no additions
kubectl describe integration example

No events get added to the integration and no Camel pods are added to the namespace. Versions of the clients are listed below and this is all on MacOS 12.4.

$ kamel version
Camel K Client 1.9.2

$ minikube version                                                                                                                                              
minikube version: v1.26.0
commit: f4b412861bb746be73053c9f6d2895f12cf78565

$ kubectl version                                                                                                                                                    
Client Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.17-dispatcher", GitCommit:"a39a896b5018d0c800124a36757433c660fd0880", GitTreeState:"clean", BuildDate:"2021-01-28T22:06:27Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:18:48Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}

The registry created by Minikube works, as I can push custom images to it as described https://minikube.sigs.k8s.io/docs/handbook/registry/#docker-on-macos and create pods from the images I create.

Is there a step I'm missing? Any other steps I can take to debug what is going on?

[tadayosi]

Hi, there is one more important step left to run your integration. It is:

kamel run Example.java

[michaeljmcd]

Hi, there is one more important step left to run your integration. It is:

kamel run Example.java

My bad, that's a bit of a documentation fail. Yes, I ran kamel run Example.java before describe integrations. The describe command did return the object (which wouldn't have been the case if run had been omitted entirely) but never one that progressed to actually running.

[tadayosi]

@michaeljmcd Ah ok, then if you chose kamel install --global method, run the following command in your namespace before creating an integration:

kamel install --skip-operator-setup -n $NS

[michaeljmcd]

So, starting fresh, I get CRD installation errors.

$ export NS=example
$ minikube delete
$ minikube start --driver=podman --container-runtime=containerd --addons=registry,dashboard,olm
$ kubectl create namespace $NS 
$ kubectl config set-context --current --namespace=$NS 
$ kamel install --global                                                                                                                                         
OLM is available in the cluster
Error: cannot check CRD installation after 90 seconds
# Re-run due to the message above.
$ kamel install --global                                                                                                                                          
OLM is available in the cluster
OLM resources are already available: skipping installation
Error: cannot check CRD installation after 90 seconds
$ kamel install --skip-operator-setup -n $NS
Error: cannot check CRD installation after 90 seconds
$ kubectl get pods -n $NS                                                                                                                                                   
No resources found in example namespace.

I'll try it again here in a bit.

[michaeljmcd]

OK, I think my previous error was an error installing OLM which I worked around by manually pulling the image first.

With the following sequence:

$ export NS=example
$ minikube delete
$ minikube start --driver=podman --container-runtime=containerd --addons=registry,dashboard
$ podman pull quay.io/operator-framework/olm # I got image pull errors without manually pulilng the image.
$ operator-sdk olm install
$ kubectl create namespace $NS 
$ kamel install --global 
$ kubectl config set-context --current --namespace=$NS 
$ kamel install --skip-operator-setup -n $NS
$ kamel init Example.java
$ kamel run -n $NS Example.java

I get the following events:

$ k describe integration example 
[...]
Events:
  Type     Reason                       Age                    From                                Message
  ----     ------                       ----                   ----                                -------
  Normal   IntegrationConditionChanged  2m59s                  camel-k-integration-controller      Condition "IntegrationPlatformAvailable" is "False" for Integration example: example/camel-k
  Normal   IntegrationPhaseUpdated      2m59s                  camel-k-integration-controller      Integration "example" in phase "Waiting For Platform"
  Warning  IntegrationError             2m59s (x3 over 2m59s)  camel-k-integration-controller      Cannot reconcile Integration example: error during trait customization: unable to find catalog matching version requirement: runtime=, provider=quarkus
  Normal   IntegrationConditionChanged  2m                     camel-k-integration-controller      Condition "IntegrationPlatformAvailable" is "True" for Integration example: example/camel-k
  Normal   IntegrationPhaseUpdated      2m                     camel-k-integration-controller      Integration "example" in phase "Initialization"
  Normal   IntegrationPhaseUpdated      2m                     camel-k-integration-controller      Integration "example" in phase "Building Kit"
  Normal   IntegrationConditionChanged  2m                     camel-k-integration-controller      Condition "IntegrationKitAvailable" is "False" for Integration example: creating a new integration kit
  Normal   ReasonRelatedObjectChanged   119s                   camel-k-integration-kit-controller  Integration Kit "kit-cb7ek9ssarkc73860meg", created by Integration "example", changed phase to "Build Submitted"
  Normal   ReasonRelatedObjectChanged   119s                   camel-k-build-controller            Build "kit-cb7ek9ssarkc73860meg", created by Integration "example", changed phase to "Scheduling"
  Normal   ReasonRelatedObjectChanged   119s                   camel-k-build-controller            Build "kit-cb7ek9ssarkc73860meg", created by Integration "example", changed phase to "Pending"
  Normal   ReasonRelatedObjectChanged   119s                   camel-k-build-controller            Build "kit-cb7ek9ssarkc73860meg", created by Integration "example", changed phase to "Running"
  Normal   ReasonRelatedObjectChanged   119s                   camel-k-integration-kit-controller  Integration Kit "kit-cb7ek9ssarkc73860meg", created by Integration "example", changed phase to "Build Running"
  Normal   IntegrationConditionChanged  57s                    camel-k-integration-controller      Condition "Ready" is "False" for Integration example: 0/1 updated replicas
  Normal   ReasonRelatedObjectChanged   57s                    camel-k-build-controller            Build "kit-cb7ek9ssarkc73860meg", created by Integration "example", changed phase to "Succeeded"
  Normal   IntegrationConditionChanged  57s                    camel-k-integration-controller      Condition "IntegrationKitAvailable" is "True" for Integration example: kit-cb7ek9ssarkc73860meg
  Normal   IntegrationPhaseUpdated      57s                    camel-k-integration-controller      Integration "example" in phase "Deploying"
  Normal   IntegrationConditionChanged  57s                    camel-k-integration-controller      Condition "CronJobAvailable" is "False" for Integration example: different controller strategy used (deployment)
  Normal   IntegrationConditionChanged  57s                    camel-k-integration-controller      Condition "DeploymentAvailable" is "True" for Integration example: deployment name is example
  Normal   IntegrationConditionChanged  57s                    camel-k-integration-controller      Condition "ServiceAvailable" is "False" for Integration example: no http service required
  Normal   IntegrationConditionChanged  57s                    camel-k-integration-controller      Condition "ExposureAvailable" is "False" for Integration example: no host or service defined
  Normal   ReasonRelatedObjectChanged   57s                    camel-k-integration-kit-controller  Integration Kit "kit-cb7ek9ssarkc73860meg", created by Integration "example", changed phase to "Ready"
  Normal   IntegrationConditionChanged  42s (x2 over 57s)      camel-k-integration-controller      (combined from similar events): Condition "Ready" is "False" for Integration example: Back-off pulling image "10.100.48.193/example/camel-k-kit-cb7ek9ssarkc73860meg@sha256:5761f6d045fe3549834462b619ef73396e84756ce03f2f25cf9cc9dfe4951e11"
  Normal   IntegrationConditionChanged  42s (x2 over 57s)      camel-k-integration-controller      (combined from similar events): Condition "Ready" is "False" for Integration example: Back-off pulling image "10.100.48.193/example/camel-k-kit-cb7ek9ssarkc73860meg@sha256:5761f6d045fe3549834462b619ef73396e84756ce03f2f25cf9cc9dfe4951e11"
  Normal   IntegrationPhaseUpdated      29s (x2 over 57s)      camel-k-integration-controller      Integration "example" in phase "Running"
  Normal   IntegrationConditionChanged  29s                    camel-k-integration-controller      Condition "Ready" is "False" for Integration example: 0/1 ready replicas
  Normal   IntegrationPhaseUpdated      14s (x2 over 42s)      camel-k-integration-controller      Integration "example" in phase "Error"
  Normal   IntegrationConditionChanged  14s                    camel-k-integration-controller      Condition "Ready" is "False" for Integration example: Back-off pulling image "10.100.48.193/example/camel-k-kit-cb7ek9ssarkc73860meg@sha256:5761f6d045fe3549834462b619ef73396e84756ce03f2f25cf9cc9dfe4951e11"

[tadayosi]

What are the events when you see the related k describe ik and k describe ikb?

[michaeljmcd]

Events from k describe ik:

Events:
  Type    Reason                      Age    From                                Message
  ----    ------                      ----   ----                                -------
  Normal  IntegrationKitPhaseUpdated  2m44s  camel-k-integration-kit-controller  Integration Kit "kit-cb82mq4tgdqc73a5603g" in phase "Build Submitted"
  Normal  IntegrationKitPhaseUpdated  2m44s  camel-k-integration-kit-controller  Integration Kit "kit-cb82mq4tgdqc73a5603g" in phase "Build Running"
  Normal  IntegrationKitPhaseUpdated  63s    camel-k-integration-kit-controller  Integration Kit "kit-cb82mq4tgdqc73a5603g" in phase "Ready"

Events from k describe ikb:

Events:
  Type    Reason             Age    From                      Message
  ----    ------             ----   ----                      -------
  Normal  BuildPhaseUpdated  3m54s  camel-k-build-controller  Build "kit-cb82mq4tgdqc73a5603g" in phase "Scheduling"
  Normal  BuildPhaseUpdated  3m54s  camel-k-build-controller  Build "kit-cb82mq4tgdqc73a5603g" in phase "Pending"
  Normal  BuildPhaseUpdated  3m54s  camel-k-build-controller  Build "kit-cb82mq4tgdqc73a5603g" in phase "Running"
  Normal  BuildPhaseUpdated  2m14s  camel-k-build-controller  Build "kit-cb82mq4tgdqc73a5603g" in phase "Succeeded"

[michaeljmcd]

After experimenting a little more, I think this is tied to the detection of the registry. If I use Docker Hub, everything works on both Minikube and Kind.

[michaeljmcd]

Continuing the experiment, if I run the registry:

CONTAINER ID  IMAGE                                                                                           COMMAND               CREATED            STATUS                PORTS                      NAMES
2c726839ea18  docker.io/library/registry:2                                                                    /etc/docker/regis...  About an hour ago  Up About an hour ago  127.0.0.1:5001->5000/tcp   kind-registry
ae0d4e841510  docker.io/kindest/node@sha256:0866296e693efe1fed79d5e6c7af8df71fc73ae45e3679af05342239cdc5bc8e                        16 minutes ago     Up 16 minutes ago     127.0.0.1:53017->6443/tcp  kind-control-plane
```

With the registry attached to the Kind network and pull the IP:

```
$ podman inspect kind-registry -f='{{json .NetworkSettings.Networks.kind}}'

{"EndpointID":"","Gateway":"10.89.0.1","IPAddress":"10.89.0.4","IPPrefixLen":24,"IPv6Gateway":"fc00:f853:ccd:e793::1","GlobalIPv6Address":"fc00:f853:ccd:e793::4","GlobalIPv6PrefixLen":64,"MacAddress":"c2:52:5a:54:81:d2","NetworkID":"kind","DriverOpts":null,"IPAMConfig":null,"Links":null,"Aliases":["2c726839ea18"]}
```

Then install Camel-K with that address:

```
$ kamel install --olm=false -n $NS --registry 10.89.0.4:5000 --organization myorg --registry-insecure true --wait
```

And run the default integration, I get a successful build:

```
k get builds                                                                                                                                                                                    
NAME                       PHASE       AGE   STARTED   DURATION          ATTEMPTS
kit-cbcq60phte5c73bqjoq0   Succeeded   12m   12m       1m30.171809865s   

A successful image push:

$ curl -X GET http://localhost:5001/v2/_catalog                                                                                                                                                   
{"repositories":["myorg/camel-k-kit-cbcq60phte5c73bqjoq0"]}

And a failed pod initialization:

$ k describe pod example-59b4bc7758-d8d2d
....
Events:
  Type     Reason     Age                   From                         Message
  ----     ------     ----                  ----                         -------
  Normal   Scheduled  12m                   default-scheduler            Successfully assigned example/example-59b4bc7758-d8d2d to kind-control-plane
  Normal   Pulling    10m (x4 over 12m)     kubelet, kind-control-plane  Pulling image "10.89.0.4:5000/myorg/camel-k-kit-cbcq60phte5c73bqjoq0@sha256:1bd03e9065dfd97939773f15ad83e87976f05ca3cd99d577bdc236fc36600ae0"
  Warning  Failed     10m (x4 over 12m)     kubelet, kind-control-plane  Failed to pull image "10.89.0.4:5000/myorg/camel-k-kit-cbcq60phte5c73bqjoq0@sha256:1bd03e9065dfd97939773f15ad83e87976f05ca3cd99d577bdc236fc36600ae0": rpc error: code = Unknown desc = failed to pull and unpack image "10.89.0.4:5000/myorg/camel-k-kit-cbcq60phte5c73bqjoq0@sha256:1bd03e9065dfd97939773f15ad83e87976f05ca3cd99d577bdc236fc36600ae0": failed to resolve reference "10.89.0.4:5000/myorg/camel-k-kit-cbcq60phte5c73bqjoq0@sha256:1bd03e9065dfd97939773f15ad83e87976f05ca3cd99d577bdc236fc36600ae0": failed to do request: Head "https://10.89.0.4:5000/v2/myorg/camel-k-kit-cbcq60phte5c73bqjoq0/manifests/sha256:1bd03e9065dfd97939773f15ad83e87976f05ca3cd99d577bdc236fc36600ae0": http: server gave HTTP response to HTTPS client
  Warning  Failed     10m (x4 over 12m)     kubelet, kind-control-plane  Error: ErrImagePull
  Warning  Failed     10m (x6 over 12m)     kubelet, kind-control-plane  Error: ImagePullBackOff
  Normal   BackOff    2m19s (x42 over 12m)  kubelet, kind-control-plane  Back-off pulling image "10.89.0.4:5000/myorg/camel-k-kit-cbcq60phte5c73bqjoq0@sha256:1bd03e9065dfd97939773f15ad83e87976f05ca3cd99d577bdc236fc36600ae0"

The error appears to be that the insecure registry setting is being respected by the push but not the pull.

[tadayosi]

Hi,

Sorry, my bad. When you installed operator with kamel install --global you shouldn't need to run the following. My previous instruction was wrong.

kamel install --skip-operator-setup -n $NS

What happens if you do the same steps above but without kamel install --skip-operator-setup? Does it go back to the original issue?

[github-actions]

This issue has been automatically marked as stale due to 90 days of inactivity.
It will be closed if no further activity occurs within 15 days.
If you think that’s incorrect or the issue should never stale, please simply write any comment.
Thanks for your contributions!