Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cert-manager-webhook-dnsimple throws errors on GKE running Kubernetes 1.17.15-gke.800 #8

Closed
parmus opened this issue Jan 19, 2021 · 9 comments

Comments

@parmus
Copy link

parmus commented Jan 19, 2021

Deploying cert-manager-webhook-dnsimple on GKE cluster running Kubernetes 1.17.15-gke.800 result in the follow errors:

I0119 23:04:51.030632       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0119 23:04:51.030689       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0119 23:04:51.030723       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0119 23:04:51.030721       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0119 23:04:51.030725       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0119 23:04:51.030818       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0119 23:04:51.031057       1 dynamic_serving_content.go:130] Starting serving-cert::/tls/tls.crt::/tls/tls.key
I0119 23:04:51.032297       1 secure_serving.go:197] Serving securely on [::]:443
I0119 23:04:51.032772       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0119 23:04:51.033118       1 apf_controller.go:249] Starting API Priority and Fairness config controller
E0119 23:04:51.036000       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.PriorityLevelConfiguration: failed to list *v1beta1.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-webhook-dnsimple" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0119 23:04:51.036045       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.FlowSchema: failed to list *v1beta1.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-webhook-dnsimple" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
I0119 23:04:51.130931       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file 
I0119 23:04:51.130993       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController 
I0119 23:04:51.131026       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file 
E0119 23:04:52.179462       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.FlowSchema: failed to list *v1beta1.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-webhook-dnsimple" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0119 23:04:52.188664       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.PriorityLevelConfiguration: failed to list *v1beta1.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-webhook-dnsimple" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0119 23:04:53.896237       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.PriorityLevelConfiguration: failed to list *v1beta1.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-webhook-dnsimple" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0119 23:04:54.880564       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.FlowSchema: failed to list *v1beta1.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-webhook-dnsimple" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0119 23:04:57.600281       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.PriorityLevelConfiguration: failed to list *v1beta1.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-webhook-dnsimple" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0119 23:04:58.393101       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.FlowSchema: failed to list *v1beta1.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-webhook-dnsimple" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0119 23:05:05.928899       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.PriorityLevelConfiguration: failed to list *v1beta1.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-webhook-dnsimple" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0119 23:05:08.093017       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0/tools/cache/reflector.go:167: Failed to watch *v1beta1.FlowSchema: failed to list *v1beta1.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-webhook-dnsimple" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope

The problem seems to be a mismatch between the v0.20.0 client libraries and Kubernetes 1.17. Downgrading to the v0.19.0 client libraries solves the problem. Kubernetes 1.17.15-gke.800 is the latest stable release on GKE, so upgrading the cluster is not an option for users running production environments on stable.

cert-manager-webhook-dnsimple seems to work despite these errors, but on the other hand, there is nothing in cert-manager-webhook-dnsimple that needs the newer client libraries.

@arnediekmann
Copy link

Hey, I just saw this error in our clusters an google'd it and then ended here in the ticket of my own project 😅 - this totally got lost in my inbox. Sorry! I think a downgrade is the most viable solution, but I will check what the other webhooks are doing and will report back.

@ebrianne
Copy link

Looking around I noticed that one can add in the rbac yaml file to the ClusterRole :domain-solver (from https://github.com/gattytto/cert-manager-acme-he-webhook/blob/master/deploy/acme-he-webhook/templates/rbac.yaml)

  - apiGroups:
      - "flowcontrol.apiserver.k8s.io"
    resources:
      - 'prioritylevelconfigurations'
      - 'flowschemas'
    verbs:
      - 'list'
      - 'watch'

But it seems to still not solve the problem. I get the same issue ebrianne/cert-manager-webhook-duckdns#2

@parmus
Copy link
Author

parmus commented Mar 25, 2021

@ebrianne Which Kubernetes version are you seeing this with?

@parmus
Copy link
Author

parmus commented Mar 25, 2021

@arnediekmann Any news?

@parmus
Copy link
Author

parmus commented Mar 25, 2021

Just to be clear: FlowSchema and PriorityLevelConfiguration were in Alpha in Kubernetes v1.19 (https://v1-19.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#flowschema-v1alpha1-flowcontrol-apiserver-k8s-io and https://v1-18.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#prioritylevelconfiguration-v1alpha1-flowcontrol-apiserver-k8s-io). They don't enter Beta until v1.20. So any client library that tries to list those resources in the Beta namespace will fail on a pre-1.20 cluster.

@ebrianne
Copy link

@parmus I am using a k3s cluster v1.20.4 and thought I could solve the problem finally. Last time I downgraded to 1.19 for the client which solved the problem as indeed the feature was in alpha at that time. It seems at the moment a viable solution.

@arnediekmann
Copy link

Whoops, this got auto-closed by my commit. I just release version 0.1.0. In our clusters (Version 1.19.8) 11fb703 and the release seem to do the trick. But please do check in your environments and report back. Sorry for taking so long with this and thanks for your patience 😇

@arnediekmann
Copy link

@parmus somewhat off-topic but the release also encompasses your PRs. Thanks again for those contributions!

@deyaeddin
Copy link

I found this issue when I was searching solution for the same problem, I solved it by adding new clusterRole/Binding to the webhook service account (not the cert-manager service account ) ...like this:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: {{ include "cert-manager-webhook-hetzner.fullname" . }}:flowcontrol-solver
  labels:
    app: {{ include "cert-manager-webhook-hetzner.name" . }}
    chart: {{ include "cert-manager-webhook-hetzner.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
rules:
  - apiGroups:
      - "flowcontrol.apiserver.k8s.io"
    resources:
      - 'prioritylevelconfigurations'
      - 'flowschemas'
    verbs:
      - 'list'
      - 'watch'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: {{ include "cert-manager-webhook-hetzner.fullname" . }}:flowcontrol-solver
  labels:
    app: {{ include "cert-manager-webhook-hetzner.name" . }}
    chart: {{ include "cert-manager-webhook-hetzner.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: {{ include "cert-manager-webhook-hetzner.fullname" . }}:flowcontrol-solver
subjects:
  - apiGroup: ""
    kind: ServiceAccount
    name: {{ include "cert-manager-webhook-hetzner.fullname" . }}
    namespace: {{ .Release.Namespace | quote }}
---

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants