panic: dial tcp - i/o timeout

[Pyrrha]

Environment

Installation method: [official](https://kubernetes.io/fr/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#check-required-ports) [documentation](https://kubernetes.io/fr/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/) + [dashboard](https://kubernetes.io/fr/docs/tasks/access-application-cluster/web-ui-dashboard/)
Kubernetes version: 1.20.2
Dashboard version: v2.1.0
Operating system: Debian GNU/Linux 10 (buster)

Steps to reproduce

1. Nothing to report: kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/recommended.yaml
2. NTR: kubectl proxy
3. Opening http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/

Observed result

After waiting 5 to 10 seconds:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "error trying to reach service: dial tcp 192.168.63.69:8443: i/o timeout",
  "code": 500
}

Or, sometimes happens immediately:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "no endpoints available for service \"https:kubernetes-dashboard:\"",
  "reason": "ServiceUnavailable",
  "code": 503
}

The cluster is fine, only a pod with Kubernetes dashboard is marked as Error

NAMESPACE              NAME                                             READY   STATUS    RESTARTS   AGE
kube-system            pod/calico-kube-controllers-bcc6f659f-7dvhn      1/1     Running   0          4h13m
kube-system            pod/calico-node-b75j6                            1/1     Running   0          4h13m
kube-system            pod/calico-node-g8nrr                            1/1     Running   0          4h12m
kube-system            pod/coredns-74ff55c5b-2vscm                      1/1     Running   0          4h14m
kube-system            pod/coredns-74ff55c5b-6vn49                      1/1     Running   0          4h14m
kube-system            pod/etcd-clu010001                               1/1     Running   0          4h14m
kube-system            pod/kube-apiserver-clu010001                     1/1     Running   0          4h14m
kube-system            pod/kube-controller-manager-clu010001            1/1     Running   0          4h14m
kube-system            pod/kube-proxy-b94g4                             1/1     Running   0          4h14m
kube-system            pod/kube-proxy-jljwb                             1/1     Running   0          4h12m
kube-system            pod/kube-scheduler-clu010001                     1/1     Running   0          4h14m
kubernetes-dashboard   pod/dashboard-metrics-scraper-79c5968bdc-qw2jr   1/1     Running   0          77m
kubernetes-dashboard   pod/kubernetes-dashboard-7448ffc97b-qv45k        0/1     Error     19         77m

NAMESPACE              NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
default                service/kubernetes                  ClusterIP   10.96.0.1       <none>        443/TCP                  4h14m
kube-system            service/kube-dns                    ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP   4h14m
kubernetes-dashboard   service/dashboard-metrics-scraper   ClusterIP   10.103.228.80   <none>        8000/TCP                 77m
kubernetes-dashboard   service/kubernetes-dashboard        ClusterIP   10.108.54.69    <none>        443/TCP                  77m

NAMESPACE     NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
kube-system   daemonset.apps/calico-node   2         2         2       2            2           beta.kubernetes.io/os=linux   4h13m
kube-system   daemonset.apps/kube-proxy    2         2         2       2            2           kubernetes.io/os=linux        4h14m

NAMESPACE              NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
kube-system            deployment.apps/calico-kube-controllers     1/1     1            1           4h13m
kube-system            deployment.apps/coredns                     2/2     2            2           4h14m
kubernetes-dashboard   deployment.apps/dashboard-metrics-scraper   1/1     1            1           77m
kubernetes-dashboard   deployment.apps/kubernetes-dashboard        0/1     1            0           77m

NAMESPACE              NAME                                                   DESIRED   CURRENT   READY   AGE
kube-system            replicaset.apps/calico-kube-controllers-bcc6f659f      1         1         1       4h13m
kube-system            replicaset.apps/coredns-74ff55c5b                      2         2         2       4h14m
kubernetes-dashboard   replicaset.apps/dashboard-metrics-scraper-79c5968bdc   1         1         1       77m
kubernetes-dashboard   replicaset.apps/kubernetes-dashboard-7448ffc97b        1         1         0       77m

And when investigating:

2021/01/17 22:53:34 Starting overwatch
2021/01/17 22:53:34 Using namespace: kubernetes-dashboard
2021/01/17 22:53:34 Using in-cluster config to connect to apiserver
2021/01/17 22:53:34 Using secret token for csrf signing
2021/01/17 22:53:34 Initializing csrf token from kubernetes-dashboard-csrf secret
panic: Get "https://10.96.0.1:443/api/v1/namespaces/kubernetes-dashboard/secrets/kubernetes-dashboard-csrf": dial tcp 10.96.0.1:443: i/o timeout

goroutine 1 [running]:
github.com/kubernetes/dashboard/src/app/backend/client/csrf.(*csrfTokenManager).init(0xc0003a9160)
	/home/runner/work/dashboard/dashboard/src/app/backend/client/csrf/manager.go:41 +0x413
github.com/kubernetes/dashboard/src/app/backend/client/csrf.NewCsrfTokenManager(...)
	/home/runner/work/dashboard/dashboard/src/app/backend/client/csrf/manager.go:66
github.com/kubernetes/dashboard/src/app/backend/client.(*clientManager).initCSRFKey(0xc00013c080)
	/home/runner/work/dashboard/dashboard/src/app/backend/client/manager.go:502 +0xc6
github.com/kubernetes/dashboard/src/app/backend/client.(*clientManager).init(0xc00013c080)
	/home/runner/work/dashboard/dashboard/src/app/backend/client/manager.go:470 +0x47
github.com/kubernetes/dashboard/src/app/backend/client.NewClientManager(...)
	/home/runner/work/dashboard/dashboard/src/app/backend/client/manager.go:551
main.main()
	/home/runner/work/dashboard/dashboard/src/app/backend/dashboard.go:105 +0x21c

Expected result

Dashboard accessible

Comments

I'm pretty new in Kubernetes, have seen #5322 and #4938, and a few other.

Do not hesitate to ask precisely if more information are needed.

Thanks for your time.

[floreks]

This means that there is some kind of networking issue in your cluster and Dashboard can't properly connect to the API server to initialize resources. Not a bug.

/close

[k8s-ci-robot]

@floreks: Closing this issue.

In response to this:

This means that there is some kind of networking issue in your cluster and Dashboard can't properly connect to the API to initialize resources. Not a bug.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Pyrrha]

After looking into this, it was an installation mistake.
I encourage people having this same problem to check a recap of install commands. Especially installing network driver before joining cluster with another nodes.