Kubernetes-dashboard unreachable (sometimes)

[mwaag]

Environment-Info:
Cloudstack 4.20.3.0
Hypervisor: VMware 7.0.3
Kubernetes-Iso: v1.36.0 from https://download.cloudstack.org/cks/setup-v1.36.0-x86_64.iso

Hi,

we're experimenting with cloudstack kubernetes service right now and in some cases the kubernetes-dashboard started locally with
kubectl --kubeconfig /path/to/kube.conf proxy

gives us a ServiceUnavailability-Message when called in browser

So we started investigating and found out that the cloud-controller-manager often is in state CrashLoopBackOff and collect many restarts after a while:

kubectl --kubeconfig /mnt/d/Users/mwa/Downloads/vpc3-kube_kube.conf get pods --all-namespaces
NAMESPACE              NAME                                                    READY   STATUS             RESTARTS         AGE
kube-system            cloud-controller-manager-8648c8d5f-fmcwt                0/1     CrashLoopBackOff   899 (5m6s ago)   3d20h
kube-system            coredns-589f44dc88-pwgz8                                1/1     Running            0                3d20h
kube-system            coredns-589f44dc88-x7kdm                                1/1     Running            0                3d20h
kube-system            etcd-vpc3-kube-control-19edf733c33                      1/1     Running            0                3d20h
kube-system            kube-apiserver-vpc3-kube-control-19edf733c33            1/1     Running            0                3d20h
kube-system            kube-controller-manager-vpc3-kube-control-19edf733c33   1/1     Running            2 (26h ago)      3d20h
kube-system            kube-proxy-6sv67                                        1/1     Running            0                3d20h
kube-system            kube-proxy-srm5w                                        1/1     Running            0                3d20h
kube-system            kube-scheduler-vpc3-kube-control-19edf733c33            1/1     Running            2 (26h ago)      3d20h
kube-system            weave-net-d8zc8                                         2/2     Running            0                3d20h
kube-system            weave-net-ttdnl                                         2/2     Running            1 (3d20h ago)    3d20h
kubernetes-dashboard   dashboard-metrics-scraper-6c78786c6f-vvbkx              1/1     Running            0                3d20h
kubernetes-dashboard   kubernetes-dashboard-57db5448b5-smgln                   1/1     Running            0                3d20h

The log shows us a timeout trying to reach the management-server (192.168.0.87)

I0623 07:03:27.461933       1 serving.go:348] Generated self-signed cert in-memory
W0623 07:03:40.762340       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
F0623 07:04:10.764367       1 main.go:74] Cloud provider could not be initialized: could not init cloud provider "external-cloudstack": Get "https://cloudportal0.test.c.fraunhofer.de/clien
t/api/?apiKey=0s-l6KLik5QE7lTSd6NzlWEI5k0PSlOuFRvHREMILPg2Pm9R-_Nu7KPcUd2UriWb5lEqAA7SVQbJnJfrmjo6aA&command=listManagementServersMetrics&expires=2026-06-23T07%3A18%3A40Z&response=json&sig
natureversion=3&signature=R7yOS7KhVxOowH2g8rW4U2J753Y%3D": dial tcp 192.168.0.87:443: i/o timeout

This brought us to among other things to #12037 and #10258 and #9267

Leading us to the questions:

• Is the impossible callback to our management-server the root cause for this? (we had some dashboards running in some tests before although we used a completely wrong endpoint.url in our globals because we forgot about this setting in some earlier stadium).
• Will deployed applications be reachable at all? (we will test this parallel to this posting)

Since we're consider the VR-Networks as insecure in general and the cloudstack-ui is an internal-only resource would the reverse-proxy + firewall solution mentioned in the other discussions be advised for our scenario as well?

[chunkyen]

I would say that the crash is definitely caused by the inability for the CCM to reach the management server endpoint. I have a working kube 1.35 cluster with a fucntional CCM, but as soon as i stop the management server, I observe this in the CCM pod

Note: It is a bit concerning that the api key is logged.

As soon as the management server is back up, the CCM pod started sucessfully