[stable/cert-manager] v0.6.0 - Internal error occurred: failed calling admission webhook "certificates.admission.certmanager.k8s.io": the server is currently unable to handle the request #10869
Comments
|
I think the issue #10856 might be indeed somehow related. |
|
Good to know! I have a helm-let's-encrypt to upgrade going forward as well, so I'm not going to do that until this is resolved. /cc @kragniz |
|
today I had to do a complete fallback to 0.5.2 as cert-manager 0.6.0 caused very strange side effect on my k8s gke environment: |
|
did anyone already try with cert-manager 0.6.5? |
|
@rmuehlbauer Trying with v0.6.5, same error... Any workarounds? |
|
at least not to my knowledge... |
|
today I had some time to dig somewhat deeper and finally could resolve the issue with new cert-manager versions - maybe you guys can use this to sort things out on your side. TL;DR: After working my way thorough cert-manager's "getting started guide" and "troubleshooting guide", I found a Note (on the very bottom on the troubleshooting guide) saying: "If the job continues to fail, please read the Webhook docs for additional information." I hope this piece of information helps a bit to sort out situations on your side. |
|
issue was solved by allowing my k8s master to access its nodes on port 6443 - which is used on the cert manager webhook pod |
Since the apiserver containers run with host networking and Pod networking is not routable outside the cluster, we can simply open up traffic for the relevant port. Reference to the issue: helm/charts#10869 (comment)
What is the gcloud command line to create this rule? |
|
i would like to see that gke command as well please |
|
For the record, I did not have this problem when setting up cert-manager. From my notes, here is literally everything that was needed to set up the cert-manager on a new cluster. # Install the CustomResourceDefinition resources separately
kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/00-crds.yaml
# Create the namespace for cert-manager
kubectl create namespace cert-manager
# Label the cert-manager namespace to disable resource validation
kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true
# Add the Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io
# Update your local Helm chart repository cache
helm repo update
# Install the cert-manager Helm chart
helm install \
--name cert-manager \
--namespace cert-manager \
--version v0.7.0 \
jetstack/cert-manager
kubectl get pods --namespace cert-manager
# Setup cluster issuer (using letsencrypt)
cat <<'EOF' | kubectl create -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-production
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: 'gajus@gajus.com'
privateKeySecretRef:
name: letsencrypt-production
http01: {}
EOF
# Set up certficate (replace with your details)
cat <<'EOF' | kubectl replace -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: queryalert-com
namespace: default
spec:
secretName: queryalert-com-tls
issuerRef:
name: letsencrypt-production
kind: ClusterIssuer
commonName: queryalert.com
dnsNames:
- queryalert.com
acme:
config:
- http01:
ingressClass: nginx
domains:
- queryalert.com
EOF
Then just update Ingress: apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: {{ .Release.Name | quote }}
labels:
{{- include "release_labels" . | indent 4 }}
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
+ certmanager.k8s.io/cluster-issuer: 'letsencrypt-production'
+ certmanager.k8s.io/acme-challenge-type: http01
+spec:
+ tls:
+ - hosts:
+ - queryalert.com
+ secretName: queryalert-com-tls
rules:
- host: queryalert.com
http:
paths:
- path: /api
backend:
serviceName: {{ .Release.Name | quote }}
servicePort: 8080
|
|
with 0.7.0 it works indeed thx |
please have a look at #10869 (comment) |
|
So, we are running a private cluster on GKE: I created firewall Then, I tried repeating steps from here #10869 (comment) and getting the same error when trying to create ClusterIssuer: Could you point me what else am I missing? |
hmm...your SOURCE and TAGS variables seem to be populated with the correct values - at least when I tried your commands in my environment. |
|
@rmuehlbauer thanks, good observation, I haven't really checked if rules were applied and they weren't, I am not sure why, maybe it has something to do with custon node pools or preemptible nodes. UPDATE: It was network, damn it, our cluster doesn't run on default network. I've updated my commands One more update, I got a step further with connections, but now getting this: I was updating from 0.5.2, so maybe something got messed up along those lines, I may try clean install again a bit later |
|
@mlushpenko I think you are hitting some RBAC issues - hava a look at https://docs.cert-manager.io/en/latest/getting-started/install.html - especially about the note regarding RBAC and GKE...hopefully that fixes your problem |
|
@rmuehlbauer thanks for suggestion, although looks fine: I was testing by running helm with my permissions, but it does look related to RBAC as it states in the error log. It feels to me like whoever is calling the API (probably webhook pod or cert-manager) is not running with specific SA because it tries to use anonymous user:
Do you have idea about validation process? I read |
This hint helps me solve the problem. Some notes here: for cluster created by kops, cross subnet mode may need to be enabled explicitly on AWS when using calico. apiVersion: kops/v1alpha2
kind: Cluster
metadata:
name: k8s.local
spec:
networking:
# Ref: https://github.com/kubernetes/kops/blob/master/docs/networking.md#enable-cross-subnet-mode-in-calico-aws-only
calico:
crossSubnet: true |
I dont actually have
https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/projects.zones.clusters Anyone have a workaround here that doesn't involve all the madness with private clusters? Trying to set this up on a vpc-native gke cluster. |
|
@oshalygin CIDR is 0.0.0.0/0 on a public cluster |
|
Credit to @skuro for the gcloud commands to add the firewall: |
|
In my case, I was deploying via terraform cert-manager and cert-manager-issuer charts in the same script. |
Is this a request for help?:
Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT (maybe)
Version of Helm and Kubernetes:
→ helm version
Client: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}
→ kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.7", GitCommit:"0c38c362511b20a098d7cd855f1314dad92c2780", GitTreeState:"clean", BuildDate:"2018-08-20T10:09:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.6-gke.3", GitCommit:"04ad69a117f331df6272a343b5d8f9e2aee5ab0c", GitTreeState:"clean", BuildDate:"2019-01-10T00:39:15Z", GoVersion:"go1.10.3b4", Compiler:"gc", Platform:"linux/amd64"}
Which chart:
cert-manager Version 0.6
What happened:
after upgrading the cert-manager pod's log is full of messages:
controller.go:147] certificates controller: Re-queuing item "some-certificate" due to error processing: Internal error occurred: failed calling admission webhook "certificates.admission.certmanager.k8s.io": the server is currently unable to handle the requestWhat you expected to happen:
How to reproduce it (as minimally and precisely as possible):
I've also done "helm delete --purge" and reinstalled the chart again - same behaviour.
Therefore, I can reproduce the issue by just installing the cert-manager chart
Anything else we need to know:
I've followed the install/upgrade instructions on https://cert-manager.readthedocs.io/en/latest/admin/upgrading/index.html and the upgrade went smooth without any problems. Also after the upgrade, all the pods are in "running" state.
The text was updated successfully, but these errors were encountered: