helm install chart failed: failed pre-install: timed out waiting for the condition / job failed: BackoffLimitExceeded

[chaosi-zju]

What happened:

➜  karmada git:(master) helm install karmada -n karmada-system --create-namespace --dependency-update ./charts/karmada --debug 
...
Error: INSTALLATION FAILED: failed pre-install: 1 error occurred:
        * timed out waiting for the condition


helm.go:84: [debug] failed pre-install: 1 error occurred:
        * timed out waiting for the condition


INSTALLATION FAILED
main.newInstallCmd.func2
        helm.sh/helm/v3/cmd/helm/install.go:147
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/cobra@v1.6.1/command.go:916
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/cobra@v1.6.1/command.go:1044
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/cobra@v1.6.1/command.go:968
main.main
        helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
        runtime/proc.go:250
runtime.goexit
        runtime/asm_amd64.s:1598

What you expected to happen:

install success

How to reproduce it (as minimally and precisely as possible):

git clone https://github.com/karmada-io/karmada
cd karmada
helm install karmada -n karmada-system --create-namespace --dependency-update ./charts/karmada --debug 

Anything else we need to know?:

Environment:

• Karmada version: master
• kubectl-karmada or karmadactl version (the result of kubectl-karmada version or karmadactl version):
• Others: k8s v1.26.0

[chaosi-zju]

/assign chaosi-zju

[chaosi-zju]

I got the bug report today in two scenes (using a kind cluster)：

---

1、DNS broken

Phenomenon：

➜  karmada git:(dev) kubectl get pods -o wide -A
NAMESPACE            NAME                                         READY   STATUS                  RESTARTS   AGE   IP           NODE                 NOMINATED NODE   READINESS GATES
karmada-system       karmada-pre-install-6ppns                    0/1     Init:ImagePullBackOff   0          12m   10.244.0.5   host-control-plane   <none>           <none>
...
➜  karmada git:(dev) kubectl describe pods karmada-pre-install-6ppns -n karmada-system 
...
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  12m                   default-scheduler  Successfully assigned karmada-system/karmada-pre-install-6ppns to host-control-plane
  Warning  Failed     9m38s                 kubelet            Failed to pull image "docker.io/cfssl/cfssl:latest": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/cfssl/cfssl:latest": failed to copy: httpReadSeeker: failed open: failed to do request: Get "https://registry-1.docker.io/v2/cfssl/cfssl/blobs/sha256:5997e4205ef719aca0e749aaed2000f49a2485a8f3f6861ad9582f90395754c5": dial tcp: lookup registry-1.docker.io on 172.18.0.1:53: no such host

Solution:

append below text to /etc/resolv.conf：

nameserver 8.8.8.8

---

2、Too slow to pull images

Phenomenon：

if it is too slow to pull docker.io/cfssl/cfssl:latest, you will see：

➜  karmada git:(dev) kubectl describe pods karmada-pre-install-6ppns -n karmada-system 
...
  Warning  Failed            46s                  kubelet            Failed to pull image "docker.io/cfssl/cfssl:latest": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/cfssl/cfssl:latest": failed to read expected number of bytes: unexpected EOF
  Warning  Failed            46s                  kubelet            Error: ErrImagePull
  Normal   BackOff           46s                  kubelet            Back-off pulling image "docker.io/cfssl/cfssl:latest"

if it is too slow to pull other images, you will see:

➜  karmada git:(dev) helm install karmada -n karmada-system --create-namespace --dependency-update ./charts/karmada --debug
...
client.go:773: [debug] karmada-post-install: Jobs active: 0, jobs failed: 6, jobs succeeded: 0
client.go:734: [debug] Add/Modify event for karmada-post-install: MODIFIED
Error: INSTALLATION FAILED: failed post-install: 1 error occurred:
        * job failed: BackoffLimitExceeded


helm.go:84: [debug] failed post-install: 1 error occurred:
        * job failed: BackoffLimitExceeded

Solution:

pull image by kind load docker-image in advance before executing helm install ..., full steps like these:

kind create cluster --name karmada-host --kubeconfig ~/.kube/config --image kindest/node:v1.26.0

cat >/tmp/pull-image.sh<<EOF
#!/bin/bash

docker pull docker.io/cfssl/cfssl:latest
docker pull docker.io/bitnami/kubectl:latest
docker pull docker.io/karmada/karmada-scheduler:latest
docker pull docker.io/karmada/karmada-webhook:latest
docker pull docker.io/karmada/karmada-controller-manager:latest
docker pull docker.io/karmada/karmada-aggregated-apiserver:latest
docker pull docker.io/karmada/karmada-agent:latest
docker pull docker.io/karmada/karmada-scheduler-estimator:latest
docker pull docker.io/karmada/karmada-descheduler:latest
docker pull docker.io/karmada/karmada-search:latest
docker pull registry.k8s.io/kube-controller-manager:v1.25.4
docker pull registry.k8s.io/kube-apiserver:v1.25.4
docker pull registry.k8s.io/etcd:3.5.9-0

kind load docker-image docker.io/cfssl/cfssl:latest --name karmada-host
kind load docker-image docker.io/bitnami/kubectl:latest --name karmada-host
kind load docker-image docker.io/karmada/karmada-scheduler:latest --name karmada-host
kind load docker-image docker.io/karmada/karmada-webhook:latest --name karmada-host
kind load docker-image docker.io/karmada/karmada-controller-manager:latest --name karmada-host
kind load docker-image docker.io/karmada/karmada-aggregated-apiserver:latest --name karmada-host
kind load docker-image docker.io/karmada/karmada-agent:latest --name karmada-host
kind load docker-image docker.io/karmada/karmada-scheduler-estimator:latest --name karmada-host
kind load docker-image docker.io/karmada/karmada-descheduler:latest --name karmada-host
kind load docker-image docker.io/karmada/karmada-search:latest --name karmada-host
kind load docker-image registry.k8s.io/kube-apiserver:v1.25.4 --name karmada-host
kind load docker-image registry.k8s.io/kube-controller-manager:v1.25.4 --name karmada-host
kind load docker-image registry.k8s.io/etcd:3.5.9-0 --name karmada-host
EOF

bash /tmp/pull-image.sh

git clone https://github.com/karmada-io/karmada
cd karmada
helm install karmada -n karmada-system --create-namespace --dependency-update ./charts/karmada --debug

pay attention to the image version of kube-apiserver/kube-controller-manager/etcd shall be modified according to your chart values.

then, the helm install process will be fast and smooth.

[chaosi-zju]

/remove-kind bug
/kind question

[chaosi-zju]

/close