Lots of "the server could not find the requested resource" errors

[johanneskastl]

I found that cert-manager did not issue a certificate using webhook-hetzner today. The certificate stays in status "False". This did work last week, with the same version of the webhook.

Not sure if it is related, but I noticed that the webhook-hetzner pod spits out lots of warnings:

W0324 05:54:37.343426       1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1beta3.FlowSchema: the server could not find the requested resource
E0324 05:54:37.343518       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1beta3.FlowSchema: failed to list *v1beta3.FlowSchema: the server could not find the requested resource
W0324 05:54:39.185135       1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1beta3.PriorityLevelConfiguration: the server could not find the requested resource
E0324 05:54:39.185242       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1beta3.PriorityLevelConfiguration: failed to list *v1beta3.PriorityLevelConfiguration: the server could not find the requested resource
W0324 05:55:16.260741       1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1beta3.FlowSchema: the server could not find the requested resource
E0324 05:55:16.260840       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1beta3.FlowSchema: failed to list *v1beta3.FlowSchema: the server could not find the requested resource
W0324 05:55:31.768006       1 reflector.go:424] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: failed to list *v1beta3.PriorityLevelConfiguration: the server could not find the requested resource
E0324 05:55:31.768095       1 reflector.go:140] pkg/mod/k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169: Failed to watch *v1beta3.PriorityLevelConfiguration: failed to list *v1beta3.PriorityLevelConfiguration: the server could not find the requested resource

[vadimkim]

Hi,
Error shows that it can't find resource connected to FlowSchema. PriorityLevelConfigurations is a part of this commit:
8aee159
Entire thread is connected to RBAC: #35
Check, if you have ClusterRole and ClusterRoleBinding from RBAC template. I am not 100% sure, but this might be an issue

[johanneskastl]

Thanks for the reply. I only set the groupName during helm installation, so anything else is default.

$ k get clusterroles|grep hetzner
cert-manager-webhook-hetzner:domain-solver                             2023-03-24T05:49:52Z
cert-manager-webhook-hetzner:flowcontrol-solver                        2023-03-24T05:49:52Z
$ k get clusterrolebindings|grep hetzner
cert-manager-webhook-hetzner:auth-delegator            ClusterRole/system:auth-delegator                                  24h
cert-manager-webhook-hetzner:domain-solver             ClusterRole/cert-manager-webhook-hetzner:domain-solver             24h
cert-manager-webhook-hetzner:flowcontrol-solver        ClusterRole/cert-manager-webhook-hetzner:flowcontrol-solver        24h
$

[vadimkim]

What version of webhook are you using?

[johanneskastl]

cert-manager-webhook-hetzner-1.2.2, installed via the helm chart.

values.yaml only contains:

groupName: <redacted>

Multiple k3s clusters, all running v1.25.7+k3s1

[diegombeltran]

Hi,

kubectl api-resources

Check which versions does your k8s API support. In my case, I'm running a kubeadm local 1.24 cluster and my highest supported version is: flowcontrol.apiserver.k8s.io/v1beta2

I'm still looking for the best way to fix this.

EDIT: As stated here, flowcontrol.apiserver.k8s.io/v1beta3 should be supported from version 1.26 and later.

[diegombeltran]

Hi again,

Upgrading to 1.26 fixes the issue. I deployed the cluster on Debian so I had to manually upgrade containerd to avoid kubelet to break.

[bernatvadell]

Is it not possible to use some old version compatible kubernetes 1.25.9?