kubernetes 1.16: secret "cert-manager-webhook-webhook-tls" not found #2484
Comments
|
Can confirm, currently no way of getting this into an operational state. |
|
You are also running k8s v.1.16? |
|
No, actually running on 1.17.0. However, I did spin up a 1.16 cluster and had the same issue, not sure. Maybe issue started with changes made from k8s 1.16, will start digging into it in a few days. |
|
same problem here on gke with kubernetes 1.14 |
|
@g0blin79 with which version of cert-manager? Did you try other versions? |
|
@papanito 0.12.0 but same problem with 0.11.x |
|
So I tried some more stuff today, no success. Rolling everything back inside the actual cluster and doing everything as the docs say didn't help at all. Has anyone else tried something that helped by any chance? |
|
Neither do I. I had it working before as well |
|
Same problem with cert-manager 0.8.1 on kube 1.15.3 in an AWS kops cluster. I'm currently debugging this so it's a fresh cluster where the only thing running on it is cert-manager. Installed with (basically): I'm getting things like When I inspect the webhook pod I see: Here's some logs: logs-from-webhook-in-cert-manager-webhook-dfcbcc64b-6tg7k.txt Some standouts: Obviously looks like some kinda permissions error because I see the "cert-manager-webhook-webhook-tl" secret in the cert-manager namespace Still lookin |
|
@austinpray interesting! At least your secrets are being created. Many people here have issues with the secrets itself, so it seems that you're one step ahead of us. Have you tried setting the permissions of the service account manually and retry it? Maybe that gets it working?! |
|
Interesting indeed, in my case I never saw the secrets created. Mine is also a fresh cluster, setup from scratch - using rke - and does not run anything at the moment. Also the installtion with e.g. helm does not give any indication that something went wrong helm install \
cert-manager \
--namespace cert-manager \
--version v0.12.0 \
jetstack/cert-manager
NAME: cert-manager
LAST DEPLOYED: Mon Dec 30 15:48:19 2019
NAMESPACE: cert-manager
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
cert-manager has been deployed successfully!
In order to begin issuing certificates, you will need to set up a ClusterIssuer
or Issuer resource (for example, by creating a 'letsencrypt-staging' issuer).
More information on the different types of issuers and how to configure them
can be found in our documentation:
https://docs.cert-manager.io/en/latest/reference/issuers.html
For information on how to configure cert-manager to automatically provision
Certificates for Ingress resources, take a look at the `ingress-shim`
documentation: |
|
Does anyone know where to find the source for the webhook? Is it open sourced somewhere or is it just the built image? |
|
Apparently the controller is responsible
So looking at the code it seems it's created here: Don't know if that helps you @filipweidemann |
|
Thank you @papanito. Yes that helps, until someone with an actual solution comes up I just want to try and work it out as well. Fingers crossed. :) |
|
By the way, are you guys running high availability clusters by any chance? Because I am and I want to rule out that my API servers are somehow messing with it :P |
|
Nope simple cluster with 2 nodes at the moment |
|
@filipweidemann yep I've got 3 masters and 3 nodes |
|
Okay so that's also a dead end... |
|
Far fetched but can it be related to the underlying os? What are you guys using? My nodes run on Debian 10. |
|
Debian 10 on my nodes as well @papanito |
|
For me the issue was fixed after a proper label on a namespace cert-manager was added. I had there label But in my case (helm chart version 0.8.1) I had to put there
When it was done, finally missing certificates for the cluster were generated - at first cert-manager-webhook-ca (took around a minute) and then cert-manager-webhook-webhook-tls (also not immediately). To be sure that you have similar problem, check list of your issuers and certificates: Precious source to clarify it was in https://cert-manager-munnerz.readthedocs.io/en/stable/admin/resource-validation-webhook.html It says that webhook is enabled if cert-manager has such new name of label - can be added by
I struggled with this cert-manager for 2 days deleting, cleaning, reinstalling, adding already generated in another cluster certificates - but nothing helped until this label was assigned... Most important - do not hurry and track status of certificates via
|
|
Damn, I see. They changed their namespace! I am not sure if I used the cert-manager.io one, but rather the old deprecated one for version 0.12 Gonna give this a shot again @dladlk |
|
Doesn't help with the 0.12 version of cert-manager. Same error, no webhook-tls secret is being created. |
|
I just removed one worker node, ran |
|
@dakale thanks for the hint. I've added the label, however still the same problem. Also deleting the complete namespace and re-create |
|
Just an idea: could it all be related to some networking issues? |
|
Are you guys running Calico? |
|
Yes running calico with nft enabled for calico dameonset according to projectcalico/calico#2322 |
|
Alright. I had calico running as well, but also tried Flannel in the meantime, same result. I was getting sick and tired of wasting my time with trying to understand what the whole chain of errors leading to this error actually looks like, and I don't want to waste more time, so here's something that I tried a few minutes ago out of straight up anger and it worked (hacky, but it works): First of all, taint one of your master nodes (doesn't matter if you're running HA clusters or single node control planes), then assign a label to the tainted master, something like After you've done this, I hope no one finding this issue with deeper understanding of this whole chain is throwing up when he/she sees it, but hey. It's working for now. However, I'd gladly appreciate any attention from maintainers or alike, so we should keep this issue open. Something is strange if the secrets are only being created on master nodes... |
|
thanks @filipweidemann for your input this saved my day ;-) However I figured that tainting may not been necessary, I've did the following
Result kubectl -n cert-manager get pods
NAME READY STATUS RESTARTS AGE
cert-manager-55798cbfdf-mtbz6 1/1 Running 0 3m38s
cert-manager-cainjector-5b5d88b76b-drgbm 1/1 Running 0 3m38s
cert-manager-webhook-656f59b5d5-zn6sb 1/1 Running 0 3m38s |
|
@papanito Thanks, now it works on raspberry as well |
|
Thanks goes all to @filipweidemann he figured it out |
|
Good catch @papanito, didn't know tainting was optional. We migrated to another cloud host and surprise, everything is working now, even without the fix. |
|
Also experiencing this on a GKE 1.15 cluster using v0.12. I tried copying the secret from another cluster, which solves this issue, but I'm now facing problems further down the pipeline, and I'm not sure if they are related to this or not. |
|
Is there anybody from |
|
I'm not sure if this is the issue anyone else in this thread is running into, but I was able to solve this error by deploying everything into the |
|
Hi, First of all, thanks to the maintainers for the time and effort put into this OSS project. I have been dealing with this issue for the past few days, banging my head against a wall as to why things didn't work as they should. Some context: I have 2 clusters, both on GCP, one being production, and another one being a scaled-down version, for staging/testing. I had successfully deployed v0.12 to staging with no issues, but were facing this particular issue on the production cluster. I had tried copying the secret from the staging to production, which seemed to solve this issue, but where facing other problems further down the pipeline, where Stuff I tried:
In the end, here's what I learned, and how it fixed the problem for me: So, the solution for me was:
Hope this helps someone else |
|
@tiagojsag mhh you used Just guessing and putting my thought here |
|
Hi, I had successfully installed v0.12 using helm 3 on my staging cluster - where there was no "hidden" v0.8 cert-manager, like on my prod cluster. So while I can't be sure this is the cause, IMO the fault is that old v0.8 installed with helm 2 I had on my production cluster, and not necessarily helm 3. But then again, I did run multiple uninstalls on my prod cluster, so I have no idea what may have been left behind that would cause this issue... :/ |
|
BTW, stupid detail that may help (or may be totally useless) when debugging: On my staging cluster, where I am using helm 3, this happens: However, on my production cluster, where I am using helm 2, both commands return the expected list of resources |
|
Thanks @ioben , your solution works good to me with helmv2. I have no idea why setting global.leaderElection.namespace="cert-manager" resolves the issue of no secret of cert-manager-webhook-tls previously. |
|
Taking a look through this, it seems like a lot of the issues here are caused by multiple different versions of cert-manager installed, often due to upgrading from Helm 2 to 3. When uninstalling cert-manager, please follow the instructions here fully: https://cert-manager.io/docs/installation/uninstall/ This should fully remove all old resources. After that, it should be safe to install the latest version of cert-manager using any of our supported installed methods (Helm 2, 3, or static manifests). |
|
Just in case somebody hits this issue and needs to work with web proxys… Check your settings for I had something like this: - name: NO_PROXY
value: int.company.com\,localhost\,127.0.0.1\,10.0.0.0/8\,172.16.0.0/12\,192.168.0.0/16\,100.64.0.0/10instead of - name: NO_PROXY
value: int.company.com,localhost,127.0.0.1,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,100.64.0.0/10 |
* Initial commit * update perms * set validation rule for namespace helm/charts#10856 * because lint * set namespace for leader election cert-manager/cert-manager#2484 * fix namespace indents * update ignore Co-authored-by: Jen <jhasenau@greenpeace.org>
Describe the bug:
Installing
cert-managerends withExpected behaviour:
No errors, pods start without errors
Steps to reproduce the bug:
Simply install
cert-managerfromhelmorstatic manifestsAnything else we need to know?:
Installation result with
helmand the pods
and there is definitively no such secret
cert-manager-webhook-webhook-tlsPod details
cert-manager-cainjectorkubectl describe pod cert-manager-cainjector-6659d6844d-mpxc7 -n cert-manager Name: cert-manager-cainjector-6659d6844d-mpxc7 Namespace: cert-manager Priority: 0 Node: x.x.x.x/192.168.100.2 Start Time: Tue, 17 Dec 2019 17:55:34 +0100 Labels: app=cainjector app.kubernetes.io/instance=cert-manager app.kubernetes.io/managed-by=Tiller app.kubernetes.io/name=cainjector helm.sh/chart=cert-manager-v0.12.0 pod-template-hash=6659d6844d Annotations: cni.projectcalico.org/podIP: 10.42.111.203/32 Status: Running IP: 10.42.111.203 IPs: IP: 10.42.111.203 Controlled By: ReplicaSet/cert-manager-cainjector-6659d6844d Containers: cert-manager: Container ID: docker://674aeca3b8baed3c230c349e9bfea0f50b3cc287adddb6733e282e306712ed49 Image: quay.io/jetstack/cert-manager-cainjector:v0.12.0 Image ID: docker-pullable://quay.io/jetstack/cert-manager-cainjector@sha256:9ff6923f6c567573103816796df283d03256bc7a9edb7450542e106b349cf34a Port: <none> Host Port: <none> Args: --v=2 --leader-election-namespace=kube-system State: Terminated Reason: Error Exit Code: 255 Started: Tue, 17 Dec 2019 17:56:11 +0100 Finished: Tue, 17 Dec 2019 17:56:41 +0100 Last State: Terminated Reason: Error Exit Code: 255 Started: Tue, 17 Dec 2019 17:55:38 +0100 Finished: Tue, 17 Dec 2019 17:56:08 +0100 Ready: False Restart Count: 1 Environment: POD_NAMESPACE: cert-manager (v1:metadata.namespace) Mounts: /var/run/secrets/kubernetes.io/serviceaccount from cert-manager-cainjector-token-lhz85 (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: cert-manager-cainjector-token-lhz85: Type: Secret (a volume populated by a Secret) SecretName: cert-manager-cainjector-token-lhz85 Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> default-scheduler Successfully assigned cert-manager/cert-manager-cainjector-6659d6844d-mpxc7 to x.x.x.x Normal Pulled 9s (x2 over 42s) kubelet, x.x.x.x Container image "quay.io/jetstack/cert-manager-cainjector:v0.12.0" already present on machine Normal Created 8s (x2 over 41s) kubelet, x.x.x.x Created container cert-manager Normal Started 8s (x2 over 41s) kubelet, x.x.x.x Started container cert-manager Warning BackOff <invalid> kubelet, x.x.x.x Back-off restarting failed containerPod details
cert-manager-webhookkubectl describe pod cert-manager-webhook-547567b88f-b7fzk -n cert-manager Name: cert-manager-webhook-547567b88f-b7fzk Namespace: cert-manager Priority: 0 Node: x.x.x.x/192.168.100.1 Start Time: Tue, 17 Dec 2019 17:55:36 +0100 Labels: app=webhook app.kubernetes.io/instance=cert-manager app.kubernetes.io/managed-by=Tiller app.kubernetes.io/name=webhook helm.sh/chart=cert-manager-v0.12.0 pod-template-hash=547567b88f Annotations: <none> Status: Pending IP: IPs: <none> Controlled By: ReplicaSet/cert-manager-webhook-547567b88f Containers: cert-manager: Container ID: Image: quay.io/jetstack/cert-manager-webhook:v0.12.0 Image ID: Port: <none> Host Port: <none> Args: --v=2 --secure-port=10250 --tls-cert-file=/certs/tls.crt --tls-private-key-file=/certs/tls.key State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Liveness: http-get http://:6080/livez delay=0s timeout=1s period=10s #success=1 #failure=3 Readiness: http-get http://:6080/healthz delay=0s timeout=1s period=10s #success=1 #failure=3 Environment: POD_NAMESPACE: cert-manager (v1:metadata.namespace) Mounts: /certs from certs (rw) /var/run/secrets/kubernetes.io/serviceaccount from cert-manager-webhook-token-lf56p (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: certs: Type: Secret (a volume populated by a Secret) SecretName: cert-manager-webhook-tls Optional: false cert-manager-webhook-token-lf56p: Type: Secret (a volume populated by a Secret) SecretName: cert-manager-webhook-token-lf56p Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> default-scheduler Successfully assigned cert-manager/cert-manager-webhook-547567b88f-b7fzk to y.y.y.y Warning FailedMount <invalid> kubelet, y.y.y.y Unable to attach or mount volumes: unmounted volumes=[certs], unattached volumes=[cert-manager-webhook-token-lf56p certs]: timed out waiting for the condition Warning FailedMount <invalid> (x9 over 118s) kubelet, y.y.y.y MountVolume.SetUp failed for volume "certs" : secret "cert-manager-webhook-tls" not foundpossible related issues (mostly closed)
Environment details::
v1.16.2baremetal0.10.0,0.11.0and0.12.0helmandstatic manifests/kind bug
The text was updated successfully, but these errors were encountered: