Cluster failing to start: Failed to list with EOF and net/http: TLS handshake timeout

[glennodickson]

Please any help would be really appreciated!!

• Openshift: 4.1.18
• Platform: UPI
• Loadbalancer: haproxy
• DNS: Dnsmasq (as attached)
• PXEBoot: Default (as attached)

Attached:

• Logbundle zip file attached containing logs from Bootstrap and master
  log-bundle.tar.zip
  Loadbalancer config file:
  haproxy.txt
  Dnsmasq config file:
  dnsmasq.txt

What happened?

0. Generated all ignition files and ensured that hidden files have been cleared.
1. Boostrap start and waits for etcd
2. Masters 0,1,2 starts and downloads config from bootstrap
3. Bootstrap signals as above "committed proposal" for each etcd
4. Then errors "Failed to fetch discovery .... :6443 ... connection refused" (see below)
5. Now bootstrap process Kube-apiserver (6443) seems to restarts constantly

all firewalls are disabled on DNS and load balancer servers.

Could it be:

• Certificate related?
• Race condition where PC not faster enough (shouldn't be i9 8 phy. cores)

Output Hightlights:

Bootstrap starts:
Masters 0/1/2 dowload config and etcd are noticed

Nov 19 12:48:07 bootstrap bootkube.sh[1379]: Error: unhealthy cluster
Nov 19 12:48:08 bootstrap bootkube.sh[1379]: etcdctl failed. Retrying in 5 seconds...
Nov 19 12:49:42 bootstrap bootkube.sh[1379]: https://etcd-0.test.fritz.box:2379 is healthy: successfully committed proposal: took = 502.844138ms
Nov 19 12:50:20 bootstrap bootkube.sh[1379]: https://etcd-2.test.fritz.box:2379 is healthy: successfully committed proposal: took = 109.785824ms
Nov 19 12:50:25 bootstrap bootkube.sh[1379]: https://etcd-1.test.fritz.box:2379 is healthy: successfully committed proposal: took = 43.563555ms
Nov 19 12:50:26 bootstrap bootkube.sh[1379]: etcd cluster up. Killing etcd certificate signer...
Nov 19 12:50:32 bootstrap bootkube.sh[1379]: e0d21c36b68a594bc7b3260bfe9521df3c3dde39beaa352c5a690a4d33744eb2
Nov 19 12:50:33 bootstrap bootkube.sh[1379]: Starting cluster-bootstrap...
Nov 19 12:51:08 bootstrap bootkube.sh[1379]: Starting temporary bootstrap control plane...
Nov 19 12:51:08 bootstrap bootkube.sh[1379]: [#1] failed to fetch discovery: Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Nov 19 12:51:09 bootstrap bootkube.sh[1379]: E1119 12:51:09.175787       1 reflector.go:134] github.com/openshift/cluster-bootstrap/pkg/start/status.go:66: Failed to list *v1.Pod: Get https://api.test.fritz.box:6443/api/v1/pods: EOF

[core@bootstrap ~]$ watch netstat -plnt
tcp        0      0 0.0.0.0:57839           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -
tcp6       0      0 :::53471                :::*                    LISTEN      -
tcp6       0      0 :::9537                 :::*                    LISTEN      -
tcp6       0      0 :::10250                :::*                    LISTEN      -
tcp6       0      0 :::6443                 :::*                    LISTEN      -
tcp6       0      0 :::9099                 :::*                    LISTEN      -
tcp6       0      0 :::19531                :::*                    LISTEN      -
tcp6       0      0 :::10255                :::*                    LISTEN      -
tcp6       0      0 :::111                  :::*                    LISTEN      -
tcp6       0      0 :::10259                :::*                    LISTEN      -

Masters restart and shortly after bootstrap service (6443) restarts :

[core@bootstrap ~]$ journalctl -b -f -u bootkube.service
...
Nov 19 13:24:04 bootstrap bootkube.sh[31694]: E1119 13:24:04.841357       1 reflector.go:134] github.com/openshift/cluster-bootstrap/pkg/start/status.go:66: Failed to list *v1.Pod: Get https://api.test.fritz.box:6443/api/v1/pods: EOF
Nov 19 13:24:06 bootstrap bootkube.sh[31694]: E1119 13:24:06.332132       1 reflector.go:134] github.com/openshift/cluster-bootstrap/pkg/start/status.go:66: Failed to list *v1.Pod: Get https://api.test.fritz.box:6443/api/v1/pods: EOF
Nov 19 13:24:07 bootstrap bootkube.sh[31694]: E1119 13:24:07.440597       1 reflector.go:134] github.com/openshift/cluster-bootstrap/pkg/start/status.go:66: Failed to list *v1.Pod: Get https://api.test.fritz.box:6443/api/v1/pods: EOF
Nov 19 13:24:08 bootstrap bootkube.sh[31694]: E1119 13:24:08.497494       1 reflector.go:134] github.com/openshift/cluster-bootstrap/pkg/start/status.go:66: Failed to list *v1.Pod: Get https://api.test.fritz.box:6443/api/v1/pods: EOF
Nov 19 13:24:09 bootstrap bootkube.sh[31694]: E1119 13:24:09.522265       1 reflector.go:134] github.com/openshift/cluster-bootstrap/pkg/start/status.go:66: Failed to list *v1.Pod: Get https://api.test.fritz.box:6443/api/v1/pods: EOF
Nov 19 13:24:10 bootstrap bootkube.sh[31694]: E1119 13:24:10.597813       1 reflector.go:134] github.com/openshift/cluster-bootstrap/pkg/start/status.go:66: Failed to list *v1.Pod: Get https://api.test.fritz.box:6443/api/v1/pods: EOF
Nov 19 13:24:21 bootstrap bootkube.sh[31694]: E1119 13:24:21.630283       1 reflector.go:134] github.com/openshift/cluster-bootstrap/pkg/start/status.go:66: Failed to list *v1.Pod: Get https://api.test.fritz.box:6443/api/v1/pods: net/http: TLS handshake timeout
Nov 19 13:24:33 bootstrap bootkube.sh[31694]: E1119 13:24:33.132800       1 reflector.go:134] github.com/openshift/cluster-bootstrap/pkg/start/status.go:66: Failed to list *v1.Pod: Get https://api.test.fritz.box:6443/api/v1/pods: net/http: TLS handshake timeout
Nov 19 13:24:44 bootstrap bootkube.sh[31694]: E1119 13:24:44.168483       1 reflector.go:134] github.com/openshift/cluster-bootstrap/pkg/start/status.go:66: Failed to list *v1.Pod: Get https://api.test.fritz.box:6443/api/v1/pods: net/http: TLS handshake timeout
Nov 19 13:24:55 bootstrap bootkube.sh[31694]: E1119 13:24:55.276698       1 reflector.go:134] github.com/openshift/cluster-bootstrap/pkg/start/status.go:66: Failed to list *v1.Pod: Get https://api.test.fritz.box:6443/api/v1/pods: net/http: TLS handshake timeout
Nov 19 13:25:06 bootstrap bootkube.sh[31694]: E1119 13:25:06.414705       1 reflector.go:134] github.com/openshift/cluster-bootstrap/pkg/start/status.go:66: Failed to list *v1.Pod: Get https://api.test.fritz.box:6443/api/v1/pods: net/http: TLS handshake timeout
Nov 19 13:25:17 bootstrap bootkube.sh[31694]: E1119 13:25:17.438228       1 reflector.go:134] github.com/openshift/cluster-bootstrap/pkg/start/status.go:66: Failed to list *v1.Pod: Get https://api.test.fritz.box:6443/api/v1/pods: EOF\

masters 0/1/2

Nov 19 13:33:51 master0 hyperkube[2876]: E1119 13:33:51.904677    2876 kubelet.go:2276] node "master0" not found
Nov 19 13:33:52 master0 hyperkube[2876]: E1119 13:33:52.008815    2876 kubelet.go:2276] node "master0" not found
Nov 19 13:33:52 master0 hyperkube[2876]: I1119 13:33:52.031523    2876 eviction_manager.go:229] eviction manager: synchronize housekeeping
Nov 19 13:33:52 master0 hyperkube[2876]: E1119 13:33:52.038710    2876 eviction_manager.go:246] eviction manager: failed to get summary stats: failed to get node info: node "master0" not found
Nov 19 13:33:52 master0 hyperkube[2876]: E1119 13:33:52.109972    2876 kubelet.go:2276] node "master0" not found
Nov 19 13:33:52 master0 hyperkube[2876]: E1119 13:33:52.215617    2876 kubelet.go:2276] node "master0" not found
Nov 19 13:33:52 master0 hyperkube[2876]: E1119 13:33:52.304643    2876 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://api-int.test.fritz.box:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster0&limit=500&resourceVersion=0: EOF
Nov 19 13:33:52 master0 hyperkube[2876]: E1119 13:33:52.319586    2876 kubelet.go:2276] node "master0" not found
Nov 19 13:33:52 master0 hyperkube[2876]: E1119 13:33:52.421361    2876 kubelet.go:2276] node "master0" not found
Nov 19 13:33:52 master0 hyperkube[2876]: E1119 13:33:52.523345    2876 kubelet.go:2276] node "master0" not found
Nov 19 13:33:52 master0 hyperkube[2876]: E1119 13:33:52.625008    2876 kubelet.go:2276] node "master0" not found
Nov 19 13:33:52 master0 hyperkube[2876]: E1119 13:33:52.727907    2876 kubelet.go:2276] node "master0" not found
Nov 19 13:33:52 master0 hyperkube[2876]: E1119 13:33:52.799153    2876 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Get https://api-int.test.fritz.box:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster0&limit=500&resourceVersion=0: EOF
Nov 19 13:33:52 master0 hyperkube[2876]: E1119 13:33:52.809021    2876 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Get https://api-int.test.fritz.box:6443/api/v1/services?limit=500&resourceVersion=0: EOF
Nov 19 13:33:52 master0 hyperkube[2876]: E1119 13:33:52.849490    2876 kubelet.go:2276] node "master0" not found
Nov 19 13:33:52 master0 hyperkube[2876]: E1119 13:33:52.955039    2876 kubelet.go:2276] node "master0" not found
Nov 19 13:33:53 master0 hyperkube[2876]: E1119 13:33:53.056800    2876 kubelet.go:2276] node "master0" not found
Nov 19 13:33:53 master0 hyperkube[2876]: E1119 13:33:53.159608    2876 kubelet.go:2276] node "master0" not found
Nov 19 13:33:53 master0 hyperkube[2876]: E1119 13:33:53.262447    2876 kubelet.go:2276] node "master0" not found
Nov 19 13:33:53 master0 hyperkube[2876]: I1119 13:33:53.313235    2876 reflector.go:160] Listing and watching *v1.Pod from k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47
Nov 19 13:33:53 master0 hyperkube[2876]: E1119 13:33:53.374617    2876 kubelet.go:2276] node "master0" not found
Nov 19 13:33:53 master0 hyperkube[2876]: E1119 13:33:53.485333    2876 kubelet.go:2276] node "master0" not found
Nov 19 13:33:53 master0 hyperkube[2876]: E1119 13:33:53.600113    2876 kubelet.go:2276] node "master0" not found
Nov 19 13:33:53 master0 hyperkube[2876]: E1119 13:33:53.713770    2876 kubelet.go:2276] node "master0" not found

What you expected to happen?

Start cluster successfully
Services are estalshed on masters

How to reproduce it

PXE Boot:

• DNS/DHCP boots bootstrap and master
• Loadbalancer
  log-bundle.tar.zip

[abhinavdahiya]

bootstrap/journals/kubelet.log

Nov 19 13:08:20 bootstrap hyperkube[1188]: I1119 13:08:20.694996    1188 prober.go:111] Readiness probe for "bootstrap-kube-apiserver-bootstrap_kube-system(b9e9105943a7883a082cb084abd53815):kube-apiserver" failed (failure): HTTP probe failed with statuscode: 500

the bootstrap-kube-apiserver is failing.. and the gather failed to capture logs...

can you capture the logs for the bootstrap-kube-apiserver on the bootstrap-host using sudo crictl logs <>

[glennodickson]

Thanks @abhinavdahiya
I've started the cluster and looked at a few crashloopbacks of the kube-apiserver and they all seem to display the same messages in the logs as below:

[core@bootstrap ~]$ sudo crictl logs -f c0e4f62fe51d7

W1121 11:46:29.865632       1 kubecontrolplane_default.go:24] open /etc/kubernetes/secrets/service-account.pub: no such file or directory
Flag --insecure-port has been deprecated, This flag will be removed in a future version.
I1121 11:46:29.956536       1 server.go:69] `kube-apiserver [--admission-control-config-file=/tmp/kubeapiserver-admission-config.yaml130896078 --allow-privileged=true --anonymous-auth=false --audit-log-format=json --audit-log-maxage=0 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/log/kube-apiserver/audit.log --audit-policy-file=openshift.local.audit/policy.yaml --audit-webhook-config-file= --authorization-mode=RBAC --authorization-mode=Node --bind-address=0.0.0.0 --client-ca-file=/etc/kubernetes/secrets/kube-apiserver-complete-client-ca-bundle.crt --cors-allowed-origins=//127\.0\.0\.1(:|$) --cors-allowed-origins=//localhost(:|$) --enable-aggregator-routing=true --enable-logs-handler=false --enable-swagger-ui=true --endpoint-reconciler-type=lease --etcd-cafile=/etc/kubernetes/secrets/etcd-ca-bundle.crt --etcd-certfile=/etc/kubernetes/secrets/etcd-client.crt --etcd-keyfile=/etc/kubernetes/secrets/etcd-client.key --etcd-prefix=kubernetes.io --etcd-servers=https://etcd-0.test.fritz.box:2379,https://etcd-1.test.fritz.box:2379,https://etcd-2.test.fritz.box:2379 --event-ttl=3h --http2-max-streams-per-connection=2000 --insecure-port=0 --kubelet-certificate-authority=/etc/kubernetes/secrets/kubelet-client-ca-bundle.crt --kubelet-client-certificate=/etc/kubernetes/secrets/kube-apiserver-to-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/secrets/kube-apiserver-to-kubelet-client.key --kubelet-https=true --kubelet-preferred-address-types=Hostname --kubelet-preferred-address-types=InternalIP --kubelet-preferred-address-types=ExternalIP --kubelet-read-only-port=0 --kubernetes-service-node-port=0 --max-mutating-requests-inflight=600 --max-requests-inflight=1200 --min-request-timeout=3600 --minimal-shutdown-duration=70s --proxy-client-cert-file=/etc/kubernetes/secrets/apiserver-proxy.crt --proxy-client-key-file=/etc/kubernetes/secrets/apiserver-proxy.key --requestheader-allowed-names=kube-apiserver-proxy --requestheader-allowed-names=system:kube-apiserver-proxy --requestheader-allowed-names=system:openshift-aggregator --requestheader-client-ca-file=/etc/kubernetes/secrets/aggregator-signer.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-cluster-ip-range=172.30.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --storage-media-type=application/vnd.kubernetes.protobuf --tls-cert-file=/etc/kubernetes/secrets/kube-apiserver-service-network-server.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 --tls-cipher-suites=TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA --tls-cipher-suites=TLS_RSA_WITH_AES_128_GCM_SHA256 --tls-cipher-suites=TLS_RSA_WITH_AES_256_GCM_SHA384 --tls-cipher-suites=TLS_RSA_WITH_AES_128_CBC_SHA --tls-cipher-suites=TLS_RSA_WITH_AES_256_CBC_SHA --tls-min-version=VersionTLS12 --tls-private-key-file=/etc/kubernetes/secrets/kube-apiserver-service-network-server.key --tls-sni-cert-key=/etc/kubernetes/secrets/kube-apiserver-service-network-server.crt,/etc/kubernetes/secrets/kube-apiserver-service-network-server.key:kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster.local --tls-sni-cert-key=/etc/kubernetes/secrets/kube-apiserver-localhost-server.crt,/etc/kubernetes/secrets/kube-apiserver-localhost-server.key:localhost,127.0.0.1,::1 --tls-sni-cert-key=/etc/kubernetes/secrets/kube-apiserver-lb-server.crt,/etc/kubernetes/secrets/kube-apiserver-lb-server.key --tls-sni-cert-key=/etc/kubernetes/secrets/kube-apiserver-internal-lb-server.crt,/etc/kubernetes/secrets/kube-apiserver-internal-lb-server.key]`
I1121 11:46:29.979571       1 server.go:573] external host was not specified, using 192.168.179.72
W1121 11:46:29.982686       1 server.go:589] No TLS key provided, service account token authentication disabled
I1121 11:46:29.993209       1 server.go:146] Version: v1.13.4+c2a5caf
F1121 11:46:30.009399       1 cmd.go:71] failed to load SNI cert and key: open /etc/kubernetes/secrets/kube-apiserver-service-network-server.crt: no such file or directory

a number of bootstrap pods restart ...

 sudo crictl ps -a  

CONTAINER ID        IMAGE                                                                                                                    CREATED
  STATE               NAME                       ATTEMPT             POD ID
7256ca89d8f6a       f9aab9ed42c63e7c7053d97d6792b12e70f0963fc2132c3933b19cd719533bfd                                                         25 seconds ago
  Running             kube-apiserver             4                   22156e15b80ba
1b48800d472c1       quay.io/openshift-release-dev/ocp-release@sha256:420633acf3fc7572372fe2df758152f6ab1f53a21c79a6c4b741fa0394c7df3a        2 minutes ago
  Running             cluster-version-operator   2                   33834de6ae5d2
b2256673d4c21       6b8e6cd707aa76d451b2dabf66d4169ec3c549ca6f3a12ad3d4eb78afec093fe                                                         3 minutes ago
  Exited              kube-controller-manager    6                   947fc318d8720
bae050f30e91c       f9aab9ed42c63e7c7053d97d6792b12e70f0963fc2132c3933b19cd719533bfd                                                         3 minutes ago
  Exited              kube-apiserver             3                   22156e15b80ba
7c84ad4c31217       quay.io/openshift-release-dev/ocp-release@sha256:420633acf3fc7572372fe2df758152f6ab1f53a21c79a6c4b741fa0394c7df3a        9 minutes ago
  Exited              cluster-version-operator   1                   33834de6ae5d2
b11012d0ecc7c       quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d6ba37d0613ea9b0ee4a1b5b8e083f070fc8651eab09e266a50f6c66148dd72   13 minutes ago
  Running             kube-scheduler             0                   f2065f8ff3910

master constantly display:

sudo journalctl -xef
Nov 21 13:20:07 master0 hyperkube[1781]: E1121 13:20:07.579872    1781 kubelet.go:2276] node "master0" not found
Nov 21 13:20:07 master0 hyperkube[1781]: E1121 13:20:07.690048    1781 kubelet.go:2276] node "master0" not found
Nov 21 13:20:07 master0 hyperkube[1781]: E1121 13:20:07.799093    1781 kubelet.go:2276] node "master0" not found
Nov 21 13:20:07 master0 hyperkube[1781]: E1121 13:20:07.901795    1781 kubelet.go:2276] node "master0" not found
Nov 21 13:20:08 master0 hyperkube[1781]: E1121 13:20:08.003027    1781 kubelet.go:2276] node "master0" not found
Nov 21 13:20:08 master0 hyperkube[1781]: I1121 13:20:08.060214    1781 reflector.go:160] Listing and watching *v1.Pod from k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47

Any help would very appreciated, thanks,

[tonymalynn9]

I've run into this issue multiple times now. Is there any update on this issue?

[glennodickson]

@tonymalynn9 what version are you using?

[tonymalynn9]

@glennodickson I've attempted 4.1 and 4.2
Unfortunately I'll have to find a gap hopefully on the weekend to retry and produce logs or provide anymore info.

All DNS is correct according to dig.
HAproxy config - https://github.com/openshift-tigerteam/guides/blob/master/ocp4/ocp4-haproxy.cfg
DHCP config - https://github.com/openshift-tigerteam/guides/blob/master/ocp4/ocp4-dhcpd.conf

Guides that I've followed:

• https://blog.openshift.com/deploying-a-user-provisioned-infrastructure-environment-for-openshift-4-1-on-vsphere/

• https://blog.openshift.com/openshift-4-2-vsphere-install-with-static-ips/

[glennodickson]

So are you using dnsmasq and haproxy?

Yes, I found that Dig verifies the DNS. The installation mostly works except, when the masters boot for the second time (automatically) they have issues. I believe Boostrap attempts to transfer control over to the masters and at this time the errors I've previously mentioned start.

I'm suspicious that this is where the issue is and need someone to have a once-over my config files to verify them.

[glennodickson]

@tonymalynn9 I see you are using vSphere. Did you get it to work?

I got it working with vSphere but now I'm using Proxmox however the config is slightly different using dnsmasq instead of Bind on vSphere. Not sure what cause the errors in initial post where "Master 0" (and 1,2) cannot get resolved.

[glennodickson]

/label platform/baremetal

[glennodickson]

I've managed to get a log-bundle. Any help would be really appreciated. I'm really stuck.

Not sure why but also noticed that approve-csr.log contains:

bootstrap approve-csr.sh[1245]: The connection to the server localhost:6443 was refused - did you specify the right host or port

Also unsure why but Openshift.log contains:

Dec 27 16:10:17 bootstrap openshift.sh[1243]: error: unable to recognize "./99_kubeadmin-password-secret.yaml": Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused
Dec 27 16:10:17 bootstrap openshift.sh[1243]: kubectl create --filename ./99_kubeadmin-password-secret.yaml failed. Retrying in 5 seconds...

log-bundle-20191227165539.tar.gz

[volkermauel]

running into the same issue
dns is setup correctly, trying to bootstrap a single master.

logs contain the same errors
bootstrap approve-csr.sh[1245]: The connection to the server localhost:6443 was refused - did you specify the right host or port

Dec 27 16:10:17 bootstrap openshift.sh[1243]: error: unable to recognize "./99_kubeadmin-password-secret.yaml": Get https://localhost:6443/api?timeout=32s: dial tcp [::1]:6443: connect: connection refused Dec 27 16:10:17 bootstrap openshift.sh[1243]: kubectl create --filename ./99_kubeadmin-password-secret.yaml failed. Retrying in 5 seconds...

[glennodickson]

@volkermauel Did you resolve this?
Did you use VMWare, Proxmox etc?
Did you use DNSMASQ and/orHAProxy? If. so do you have associated files?
Unsure why there are issues with with secrets ?

[volkermauel]

Hey,

i figured i still had an entry in my LB-Config wrong (using HAProxy).
I tested the setup completely on Hyper-V on my local machine, since that was easily available.

I'm not entirely sure what did fix the issue, but i guess killing the container that had trouble on the bootstrap-node resolved it.

do so, by ssh-ing into the bootstrap machine, then "su" to get root-rights.
with "podman ps" you can list the containers running. 2 of them should have a status of "exited n minutes ago"
simply remove both of these with "podman rm CONTAINERID"

Hope this helps, let me know :)

[glennodickson]

@volkermauel thanks for the quick reply, very appreciated.
I'm using Proxmox btw.
You said you have a wrong entry in your HA Proxy

1. what was that out of curiosity pls?

I'll try killing container.
2. I assume you reinstalled cluster from scratch and then as mentioned in your prev. post. you killed container on Bootstrap ?

Thanks for your help.