Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Internal error occurred: failed calling webhook #2918

Closed
9p4 opened this issue May 15, 2020 · 20 comments
Closed

Internal error occurred: failed calling webhook #2918

9p4 opened this issue May 15, 2020 · 20 comments
Labels
triage/support Indicates an issue that is a support question.

Comments

@9p4
Copy link

9p4 commented May 15, 2020 •
edited

When testing the installation, I get "Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=30s: context deadline exceeded"

Versions:
Kubernetes: Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

OS: CentOS 7

Update: I have removed this environment and will not be able to test any fixes.

Cert Manager: v0.15.0

Installed from Helm using official documentation on a two-node cluster.

To reproduce:

  1. Using OpenNebula, create Kubernetes service as outlined (here)[https://docs.opennebula.io/appliances/service/kubernetes.html]
  2. Install Helm on machine that has access to cluster
  3. Follow (this)[https://cert-manager.io/docs/installation/kubernetes/] to install cert-manager.
  4. Test cert-manager as outlined in section "Verifying the installation" in installation guide.

Expected results:

  • Test passes

Additional Notes:

  • I already have an ingress controller running (HAProxy).
  • I think it may be an issue with the ca-injector (https is not being served because no certificate)
  • All pods are in running state

Logs (kubectl logs -l app=cert-manager -n cert-manager):

I0515 02:41:15.606858       1 reflector.go:175] Starting reflector *v1alpha2.CertificateRequest (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.606974       1 reflector.go:175] Starting reflector *v1alpha2.Order (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.607980       1 reflector.go:175] Starting reflector *v1alpha2.ClusterIssuer (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.608285       1 reflector.go:175] Starting reflector *v1alpha2.Certificate (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.798737       1 controller.go:141] cert-manager/controller/ingress-shim "msg"="syncing item" "key"="default/[redacted]" 
I0515 02:41:15.801614       1 sync.go:50] cert-manager/controller/ingress-shim "msg"="not syncing ingress resource as it does not contain a \"cert-manager.io/issuer\" or \"cert-manager.io/cluster-issuer\" annotation" "resource_kind"="Ingress" "resource_name"="[redacted]" "resource_namespace"="default" 
I0515 02:41:15.801753       1 controller.go:147] cert-manager/controller/ingress-shim "msg"="finished processing work item" "key"="default/[redacted]" 
I0515 02:41:15.801852       1 controller.go:141] cert-manager/controller/ingress-shim "msg"="syncing item" "key"="ingress-controller/[redacted]" 
I0515 02:41:15.802295       1 sync.go:50] cert-manager/controller/ingress-shim "msg"="not syncing ingress resource as it does not contain a \"cert-manager.io/issuer\" or \"cert-manager.io/cluster-issuer\" annotation" "resource_kind"="Ingress" "resource_name"="[redacted]" "resource_namespace"="ingress-controller" 
I0515 02:41:15.802427       1 controller.go:147] cert-manager/controller/ingress-shim "msg"="finished processing work item" "key"="ingress-controller/[redacted]

kubectl logs -n cert-manager cert-manager-7cb75cf6b4-wjndg:

I0515 02:40:08.138598       1 start.go:76] cert-manager "msg"="starting controller"  "git-commit"="1d6ecc9cf8d841782acb5f3d3c28467c24c5fd18" "version"="v0.15.0"
W0515 02:40:08.139801       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0515 02:40:08.151968       1 controller.go:193] cert-manager/controller/build-context "msg"="configured acme dns01 nameservers" "nameservers"=["10.96.0.10:53"] 
I0515 02:40:08.154500       1 controller.go:156] cert-manager/controller "msg"="starting leader election"  
I0515 02:40:08.155604       1 metrics.go:202] cert-manager/metrics "msg"="listening for connections on" "address"="0.0.0.0:9402" 
I0515 02:40:08.163373       1 leaderelection.go:242] attempting to acquire leader lease  kube-system/cert-manager-controller...
I0515 02:41:15.266989       1 leaderelection.go:252] successfully acquired lease kube-system/cert-manager-controller
I0515 02:41:15.282469       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="clusterissuers" 
I0515 02:41:15.282627       1 controller.go:89] cert-manager/controller/clusterissuers "msg"="starting control loop"  
I0515 02:41:15.282805       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="ingress-shim" 
I0515 02:41:15.282879       1 controller.go:89] cert-manager/controller/ingress-shim "msg"="starting control loop"  
I0515 02:41:15.283158       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="issuers" 
I0515 02:41:15.283233       1 controller.go:89] cert-manager/controller/issuers "msg"="starting control loop"  
I0515 02:41:15.295088       1 reflector.go:175] Starting reflector *v1.Secret (5m0s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.598703       1 controller.go:172] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="vault"
I0515 02:41:15.599100       1 controller.go:113] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="CertificateIssuing" 
I0515 02:41:15.599165       1 controller.go:113] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="CertificateKeyManager" 
I0515 02:41:15.599212       1 controller.go:113] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="CertificateRequestManager" 
I0515 02:41:15.599265       1 controller.go:113] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="CertificateTrigger" 
I0515 02:41:15.599495       1 controller.go:172] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="ca"
I0515 02:41:15.599758       1 controller.go:172] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="selfsigned"
I0515 02:41:15.600138       1 controller.go:172] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="venafi"
I0515 02:41:15.600471       1 controller.go:113] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="CertificateReadiness" 
I0515 02:41:15.600748       1 controller.go:172] cert-manager/controller/certificaterequests "msg"="new certificate request controller registered"  "type"="acme"
I0515 02:41:15.601283       1 reflector.go:175] Starting reflector *v1.Service (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.602575       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="challenges" 
I0515 02:41:15.602640       1 controller.go:89] cert-manager/controller/challenges "msg"="starting control loop"  
I0515 02:41:15.602754       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-vault" 
I0515 02:41:15.602800       1 controller.go:89] cert-manager/controller/certificaterequests-issuer-vault "msg"="starting control loop"  
I0515 02:41:15.603111       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="orders" 
I0515 02:41:15.603201       1 controller.go:89] cert-manager/controller/orders "msg"="starting control loop"  
I0515 02:41:15.603371       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-ca" 
I0515 02:41:15.603452       1 controller.go:89] cert-manager/controller/certificaterequests-issuer-ca "msg"="starting control loop"  
I0515 02:41:15.603737       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-selfsigned" 
I0515 02:41:15.603832       1 controller.go:89] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="starting control loop"  
I0515 02:41:15.603990       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-venafi" 
I0515 02:41:15.613745       1 controller.go:89] cert-manager/controller/certificaterequests-issuer-venafi "msg"="starting control loop"  
I0515 02:41:15.614747       1 reflector.go:175] Starting reflector *v1.Secret (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.620672       1 reflector.go:175] Starting reflector *v1beta1.Ingress (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.623644       1 reflector.go:175] Starting reflector *v1.Pod (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.606219       1 reflector.go:175] Starting reflector *v1alpha2.Issuer (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.606446       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="certificates" 
I0515 02:41:15.630209       1 controller.go:89] cert-manager/controller/certificates "msg"="starting control loop"  
I0515 02:41:15.606552       1 controller.go:131] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-acme" 
I0515 02:41:15.630508       1 controller.go:89] cert-manager/controller/certificaterequests-issuer-acme "msg"="starting control loop"  
I0515 02:41:15.606730       1 reflector.go:175] Starting reflector *v1alpha2.Challenge (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.606858       1 reflector.go:175] Starting reflector *v1alpha2.CertificateRequest (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.606974       1 reflector.go:175] Starting reflector *v1alpha2.Order (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.607980       1 reflector.go:175] Starting reflector *v1alpha2.ClusterIssuer (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.608285       1 reflector.go:175] Starting reflector *v1alpha2.Certificate (30s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:41:15.798737       1 controller.go:141] cert-manager/controller/ingress-shim "msg"="syncing item" "key"="default/[redacted]" 
I0515 02:41:15.801614       1 sync.go:50] cert-manager/controller/ingress-shim "msg"="not syncing ingress resource as it does not contain a \"cert-manager.io/issuer\" or \"cert-manager.io/cluster-issuer\" annotation" "resource_kind"="Ingress" "resource_name"="[redacted]" "resource_namespace"="default" 
I0515 02:41:15.801753       1 controller.go:147] cert-manager/controller/ingress-shim "msg"="finished processing work item" "key"="default/[redacted]" 
I0515 02:41:15.801852       1 controller.go:141] cert-manager/controller/ingress-shim "msg"="syncing item" "key"="ingress-controller/[redacted]" 
I0515 02:41:15.802295       1 sync.go:50] cert-manager/controller/ingress-shim "msg"="not syncing ingress resource as it does not contain a \"cert-manager.io/issuer\" or \"cert-manager.io/cluster-issuer\" annotation" "resource_kind"="Ingress" "resource_name"="[redacted]" "resource_namespace"="ingress-controller" 
I0515 02:41:15.802427       1 controller.go:147] cert-manager/controller/ingress-shim "msg"="finished processing work item" "key"="ingress-controller/[redacted]"

kubectl logs -n cert-manager cert-manager-cainjector-759496659c-6sgkj:

I0515 02:40:11.989165       1 start.go:82] starting ca-injector v0.15.0 (revision 1d6ecc9cf8d841782acb5f3d3c28467c24c5fd18)
I0515 02:40:13.862263       1 request.go:621] Throttling request took 1.040883608s, request: GET:https://10.96.0.1:443/apis/authentication.k8s.io/v1beta1?timeout=32s
I0515 02:40:14.862445       1 request.go:621] Throttling request took 2.040553795s, request: GET:https://10.96.0.1:443/apis/policy/v1beta1?timeout=32s
I0515 02:40:14.927206       1 setup.go:81] cert-manager "msg"="unable to register injector which is still in an alpha phase. Enable the feature on the API server in order to use this injector"  "injector"="auditsink"
I0515 02:40:14.927500       1 leaderelection.go:242] attempting to acquire leader lease  kube-system/cert-manager-cainjector-leader-election-core...
I0515 02:40:14.928256       1 reflector.go:175] Starting reflector *v1beta1.MutatingWebhookConfiguration (9h46m26.488119031s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:14.929974       1 reflector.go:175] Starting reflector *v1beta1.ValidatingWebhookConfiguration (10h4m16.134207791s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:14.931292       1 reflector.go:175] Starting reflector *v1beta1.APIService (10h5m42.401194473s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:14.931532       1 reflector.go:175] Starting reflector *v1beta1.CustomResourceDefinition (9h35m47.994364568s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:15.013269       1 setup.go:81] cert-manager "msg"="unable to register injector which is still in an alpha phase. Enable the feature on the API server in order to use this injector"  "injector"="auditsink"
I0515 02:40:15.013463       1 leaderelection.go:242] attempting to acquire leader lease  kube-system/cert-manager-cainjector-leader-election...
I0515 02:40:15.014502       1 reflector.go:175] Starting reflector *v1beta1.MutatingWebhookConfiguration (9h13m40.588931011s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:15.016105       1 reflector.go:175] Starting reflector *v1beta1.ValidatingWebhookConfiguration (9h30m32.361430045s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:15.017189       1 reflector.go:175] Starting reflector *v1beta1.APIService (9h51m38.977119063s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:15.017855       1 reflector.go:175] Starting reflector *v1beta1.CustomResourceDefinition (9h54m34.145270079s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:30.600083       1 leaderelection.go:252] successfully acquired lease kube-system/cert-manager-cainjector-leader-election-core
I0515 02:40:30.603665       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="customresourcedefinition" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"group":"","names":{"plural":"","kind":""},"scope":""},"status":{"conditions":null,"acceptedNames":{"plural":"","kind":""},"storedVersions":null}}}
I0515 02:40:30.606195       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="validatingwebhookconfiguration" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:30.607488       1 recorder.go:52] cert-manager/controller-runtime/manager/events "msg"="Normal"  "message"="cert-manager-cainjector-759496659c-6sgkj_acd9b7ea-feba-4e4b-bccf-260937b09f4c became leader" "object"={"kind":"ConfigMap","namespace":"kube-system","name":"cert-manager-cainjector-leader-election-core","uid":"4a7550a8-39e2-4576-bccb-dd302185a1ae","apiVersion":"v1","resourceVersion":"702208"} "reason"="LeaderElection"
I0515 02:40:30.607694       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="customresourcedefinition" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:30.608466       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="mutatingwebhookconfiguration" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:30.609639       1 reflector.go:175] Starting reflector *v1.Secret (9h52m51.343450322s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:30.609874       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="apiservice" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"groupPriorityMinimum":0,"versionPriority":0},"status":{}}}
I0515 02:40:30.610176       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="apiservice" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:30.610339       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="mutatingwebhookconfiguration" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:30.614417       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="validatingwebhookconfiguration" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:30.909916       1 controller.go:171] cert-manager/controller-runtime/controller "msg"="Starting Controller"  "controller"="customresourcedefinition"
I0515 02:40:30.910146       1 controller.go:190] cert-manager/controller-runtime/controller "msg"="Starting workers"  "controller"="customresourcedefinition" "worker count"=1
I0515 02:40:30.910811       1 controller.go:171] cert-manager/controller-runtime/controller "msg"="Starting Controller"  "controller"="mutatingwebhookconfiguration"
I0515 02:40:30.910850       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"blockaffinities.crd.projectcalico.org"}
I0515 02:40:30.910952       1 controller.go:190] cert-manager/controller-runtime/controller "msg"="Starting workers"  "controller"="mutatingwebhookconfiguration" "worker count"=1
I0515 02:40:30.911200       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"clusterinformations.crd.projectcalico.org"}
I0515 02:40:30.911394       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ipamconfigs.crd.projectcalico.org"}
I0515 02:40:30.912233       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="certificaterequests.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:30.918379       1 controller.go:171] cert-manager/controller-runtime/controller "msg"="Starting Controller"  "controller"="validatingwebhookconfiguration"
I0515 02:40:30.918553       1 controller.go:190] cert-manager/controller-runtime/controller "msg"="Starting workers"  "controller"="validatingwebhookconfiguration" "worker count"=1
I0515 02:40:30.918953       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:30.921947       1 controller.go:171] cert-manager/controller-runtime/controller "msg"="Starting Controller"  "controller"="apiservice"
I0515 02:40:30.911416       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:30.922633       1 controller.go:190] cert-manager/controller-runtime/controller "msg"="Starting workers"  "controller"="apiservice" "worker count"=1
I0515 02:40:30.923953       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.getambassador.io"}
I0515 02:40:30.924414       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.scheduling.k8s.io"}
I0515 02:40:30.924611       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.crd.projectcalico.org"}
I0515 02:40:30.925479       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1alpha3.cert-manager.io"}
I0515 02:40:30.925691       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.networking.k8s.io"}
I0515 02:40:30.925860       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.events.k8s.io"}
I0515 02:40:30.926269       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1alpha2.acme.cert-manager.io"}
I0515 02:40:30.926444       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.admissionregistration.k8s.io"}
I0515 02:40:30.926594       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v2beta1.autoscaling"}
I0515 02:40:30.926761       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.autoscaling"}
I0515 02:40:30.926950       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.apiextensions.k8s.io"}
I0515 02:40:30.927179       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.authorization.k8s.io"}
I0515 02:40:30.927338       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v2beta2.autoscaling"}
I0515 02:40:30.928849       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1alpha3.acme.cert-manager.io"}
I0515 02:40:30.929405       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.policy"}
I0515 02:40:30.929635       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.rbac.authorization.k8s.io"}
I0515 02:40:30.929807       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.getambassador.io"}
I0515 02:40:30.929974       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta2.getambassador.io"}
I0515 02:40:30.941776       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1alpha2.cert-manager.io"}
I0515 02:40:30.942130       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.node.k8s.io"}
I0515 02:40:30.942318       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.authorization.k8s.io"}
I0515 02:40:30.942482       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.batch"}
I0515 02:40:30.942637       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.batch"}
I0515 02:40:30.942796       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v2.getambassador.io"}
I0515 02:40:30.942965       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.authentication.k8s.io"}
I0515 02:40:30.943207       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.networking.k8s.io"}
I0515 02:40:30.943366       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.certificates.k8s.io"}
I0515 02:40:30.943528       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.coordination.k8s.io"}
I0515 02:40:30.943687       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1."}
I0515 02:40:30.943854       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.apiextensions.k8s.io"}
I0515 02:40:30.944353       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.admissionregistration.k8s.io"}
I0515 02:40:30.944531       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.authentication.k8s.io"}
I0515 02:40:30.944941       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.extensions"}
I0515 02:40:30.945235       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.storage.k8s.io"}
I0515 02:40:30.945485       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.apps"}
I0515 02:40:30.945653       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.rbac.authorization.k8s.io"}
I0515 02:40:30.945815       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.discovery.k8s.io"}
I0515 02:40:30.945990       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.coordination.k8s.io"}
I0515 02:40:30.946365       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.scheduling.k8s.io"}
I0515 02:40:30.946554       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.storage.k8s.io"}
I0515 02:40:30.994487       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" 
I0515 02:40:30.994656       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="mutatingwebhookconfiguration" "request"={"Namespace":"","Name":"cert-manager-webhook"}
I0515 02:40:30.995001       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:31.049958       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" 
I0515 02:40:31.050230       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="validatingwebhookconfiguration" "request"={"Namespace":"","Name":"cert-manager-webhook"}
I0515 02:40:31.050477       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:31.133179       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" 
I0515 02:40:31.133342       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="validatingwebhookconfiguration" "request"={"Namespace":"","Name":"cert-manager-webhook"}
I0515 02:40:31.134607       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" 
I0515 02:40:31.134798       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="mutatingwebhookconfiguration" "request"={"Namespace":"","Name":"cert-manager-webhook"}
I0515 02:40:31.160086       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="certificaterequests.cert-manager.io" "resource_namespace"="" 
I0515 02:40:31.160243       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"certificaterequests.cert-manager.io"}
I0515 02:40:31.160538       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"felixconfigurations.crd.projectcalico.org"}
I0515 02:40:31.162777       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="clusterissuers.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:31.700089       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="clusterissuers.cert-manager.io" "resource_namespace"="" 
I0515 02:40:31.700285       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"clusterissuers.cert-manager.io"}
I0515 02:40:31.700561       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"bgpconfigurations.crd.projectcalico.org"}
I0515 02:40:31.700858       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"mappings.getambassador.io"}
I0515 02:40:31.701168       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"filterpolicies.getambassador.io"}
I0515 02:40:31.701363       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"tcpmappings.getambassador.io"}
I0515 02:40:31.701531       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ipamblocks.crd.projectcalico.org"}
I0515 02:40:31.703460       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="issuers.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:32.410093       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="issuers.cert-manager.io" "resource_namespace"="" 
I0515 02:40:32.410261       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"issuers.cert-manager.io"}
I0515 02:40:32.410505       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"authservices.getambassador.io"}
I0515 02:40:32.410674       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ratelimitservices.getambassador.io"}
I0515 02:40:32.410836       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"tlscontexts.getambassador.io"}
I0515 02:40:32.411735       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="certificates.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:32.991902       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="certificates.cert-manager.io" "resource_namespace"="" 
I0515 02:40:32.992454       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"certificates.cert-manager.io"}
I0515 02:40:32.992768       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"globalnetworksets.crd.projectcalico.org"}
I0515 02:40:32.993596       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"consulresolvers.getambassador.io"}
I0515 02:40:32.993849       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ipamhandles.crd.projectcalico.org"}
I0515 02:40:32.994179       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ippools.crd.projectcalico.org"}
I0515 02:40:32.994353       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ratelimits.getambassador.io"}
I0515 02:40:32.994665       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="orders.acme.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:33.241307       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="orders.acme.cert-manager.io" "resource_namespace"="" 
I0515 02:40:33.241490       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"orders.acme.cert-manager.io"}
I0515 02:40:33.241736       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"networkpolicies.crd.projectcalico.org"}
I0515 02:40:33.241949       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"kubernetesendpointresolvers.getambassador.io"}
I0515 02:40:33.243082       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"hosts.getambassador.io"}
I0515 02:40:33.243277       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"tracingservices.getambassador.io"}
I0515 02:40:33.243445       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"filters.getambassador.io"}
I0515 02:40:33.243615       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"hostendpoints.crd.projectcalico.org"}
I0515 02:40:33.243806       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"networksets.crd.projectcalico.org"}
I0515 02:40:33.244797       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"modules.getambassador.io"}
I0515 02:40:33.245120       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"globalnetworkpolicies.crd.projectcalico.org"}
I0515 02:40:33.245369       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"logservices.getambassador.io"}
I0515 02:40:33.245544       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"kubernetesserviceresolvers.getambassador.io"}
I0515 02:40:33.247844       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="challenges.acme.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:33.711715       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="challenges.acme.cert-manager.io" "resource_namespace"="" 
I0515 02:40:33.711885       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"challenges.acme.cert-manager.io"}
I0515 02:40:33.712397       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"bgppeers.crd.projectcalico.org"}
I0515 02:40:33.712926       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="certificaterequests.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:33.820813       1 leaderelection.go:252] successfully acquired lease kube-system/cert-manager-cainjector-leader-election
I0515 02:40:33.823328       1 recorder.go:52] cert-manager/controller-runtime/manager/events "msg"="Normal"  "message"="cert-manager-cainjector-759496659c-6sgkj_6a3611b3-ae60-462d-88b6-3d71f9fba0d2 became leader" "object"={"kind":"ConfigMap","namespace":"kube-system","name":"cert-manager-cainjector-leader-election","uid":"8fad1895-8a0e-419d-a6e7-0f4603199533","apiVersion":"v1","resourceVersion":"702225"} "reason"="LeaderElection"
I0515 02:40:33.829504       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="certificaterequests.cert-manager.io" "resource_namespace"="" 
I0515 02:40:33.829656       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"certificaterequests.cert-manager.io"}
I0515 02:40:33.832279       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="customresourcedefinition" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"group":"","names":{"plural":"","kind":""},"scope":""},"status":{"conditions":null,"acceptedNames":{"plural":"","kind":""},"storedVersions":null}}}
I0515 02:40:33.835291       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="customresourcedefinition" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"secretName":"","issuerRef":{"name":""}},"status":{}}}
I0515 02:40:33.837113       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="mutatingwebhookconfiguration" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:33.837416       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="mutatingwebhookconfiguration" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"secretName":"","issuerRef":{"name":""}},"status":{}}}
I0515 02:40:33.837805       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="validatingwebhookconfiguration" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:33.838741       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="validatingwebhookconfiguration" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"secretName":"","issuerRef":{"name":""}},"status":{}}}
I0515 02:40:33.839460       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="apiservice" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"groupPriorityMinimum":0,"versionPriority":0},"status":{}}}
I0515 02:40:33.840463       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="clusterissuers.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:33.842624       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="apiservice" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"secretName":"","issuerRef":{"name":""}},"status":{}}}
I0515 02:40:33.843557       1 reflector.go:175] Starting reflector *v1alpha2.Certificate (9h24m37.730610426s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:34.334433       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="clusterissuers.cert-manager.io" "resource_namespace"="" 
I0515 02:40:34.334603       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"clusterissuers.cert-manager.io"}
I0515 02:40:34.336972       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="issuers.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:34.978360       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="issuers.cert-manager.io" "resource_namespace"="" 
I0515 02:40:34.978487       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"issuers.cert-manager.io"}
I0515 02:40:34.978959       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="certificates.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:35.236502       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="customresourcedefinition" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:35.237622       1 reflector.go:175] Starting reflector *v1.Secret (10h39m29.948285184s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:35.239889       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="validatingwebhookconfiguration" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:35.240226       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="mutatingwebhookconfiguration" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:35.243315       1 controller.go:164] cert-manager/controller-runtime/controller "msg"="Starting EventSource"  "controller"="apiservice" "source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0515 02:40:36.038184       1 controller.go:171] cert-manager/controller-runtime/controller "msg"="Starting Controller"  "controller"="customresourcedefinition"
I0515 02:40:36.038386       1 controller.go:190] cert-manager/controller-runtime/controller "msg"="Starting workers"  "controller"="customresourcedefinition" "worker count"=1
I0515 02:40:36.041377       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"clusterissuers.cert-manager.io"}
I0515 02:40:36.041629       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"tlscontexts.getambassador.io"}
I0515 02:40:36.041820       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"filters.getambassador.io"}
I0515 02:40:36.042079       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ipamhandles.crd.projectcalico.org"}
I0515 02:40:36.042384       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"certificaterequests.cert-manager.io"}
I0515 02:40:36.042566       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"clusterinformations.crd.projectcalico.org"}
I0515 02:40:36.042766       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ipamconfigs.crd.projectcalico.org"}
I0515 02:40:36.042972       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"kubernetesendpointresolvers.getambassador.io"}
I0515 02:40:36.043195       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"networksets.crd.projectcalico.org"}
I0515 02:40:36.043382       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ratelimitservices.getambassador.io"}
I0515 02:40:36.043555       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"hostendpoints.crd.projectcalico.org"}
I0515 02:40:36.043720       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ratelimits.getambassador.io"}
I0515 02:40:36.043890       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"globalnetworkpolicies.crd.projectcalico.org"}
I0515 02:40:36.044181       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"hosts.getambassador.io"}
I0515 02:40:36.044508       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"orders.acme.cert-manager.io"}
I0515 02:40:36.044715       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"mappings.getambassador.io"}
I0515 02:40:36.044892       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"globalnetworksets.crd.projectcalico.org"}
I0515 02:40:36.045912       1 controller.go:171] cert-manager/controller-runtime/controller "msg"="Starting Controller"  "controller"="apiservice"
I0515 02:40:36.046134       1 controller.go:190] cert-manager/controller-runtime/controller "msg"="Starting workers"  "controller"="apiservice" "worker count"=1
I0515 02:40:36.046411       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.authentication.k8s.io"}
I0515 02:40:36.046496       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"issuers.cert-manager.io"}
I0515 02:40:36.046575       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.events.k8s.io"}
I0515 02:40:36.046687       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"bgpconfigurations.crd.projectcalico.org"}
I0515 02:40:36.046727       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.batch"}
I0515 02:40:36.046857       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"tcpmappings.getambassador.io"}
I0515 02:40:36.046901       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1."}
I0515 02:40:36.047114       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1alpha3.cert-manager.io"}
I0515 02:40:36.047148       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"blockaffinities.crd.projectcalico.org"}
I0515 02:40:36.047258       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.apps"}
I0515 02:40:36.047328       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"kubernetesserviceresolvers.getambassador.io"}
I0515 02:40:36.047391       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta2.getambassador.io"}
I0515 02:40:36.047511       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"consulresolvers.getambassador.io"}
I0515 02:40:36.047690       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"networkpolicies.crd.projectcalico.org"}
I0515 02:40:36.047907       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"modules.getambassador.io"}
I0515 02:40:36.048163       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ippools.crd.projectcalico.org"}
I0515 02:40:36.048337       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"tracingservices.getambassador.io"}
I0515 02:40:36.049375       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"certificates.cert-manager.io"}
I0515 02:40:36.049577       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"felixconfigurations.crd.projectcalico.org"}
I0515 02:40:36.049788       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"authservices.getambassador.io"}
I0515 02:40:36.047528       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.storage.k8s.io"}
I0515 02:40:36.050933       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.networking.k8s.io"}
I0515 02:40:36.051120       1 controller.go:171] cert-manager/controller-runtime/controller "msg"="Starting Controller"  "controller"="mutatingwebhookconfiguration"
I0515 02:40:36.051201       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.apiextensions.k8s.io"}
I0515 02:40:36.051252       1 controller.go:190] cert-manager/controller-runtime/controller "msg"="Starting workers"  "controller"="mutatingwebhookconfiguration" "worker count"=1
I0515 02:40:36.051388       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.admissionregistration.k8s.io"}
I0515 02:40:36.051557       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.rbac.authorization.k8s.io"}
I0515 02:40:36.051723       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1alpha2.cert-manager.io"}
I0515 02:40:36.051906       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.discovery.k8s.io"}
I0515 02:40:36.052818       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="mutatingwebhookconfiguration" "request"={"Namespace":"","Name":"cert-manager-webhook"}
I0515 02:40:36.053334       1 controller.go:171] cert-manager/controller-runtime/controller "msg"="Starting Controller"  "controller"="validatingwebhookconfiguration"
I0515 02:40:36.053464       1 controller.go:190] cert-manager/controller-runtime/controller "msg"="Starting workers"  "controller"="validatingwebhookconfiguration" "worker count"=1
I0515 02:40:36.053757       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="validatingwebhookconfiguration" "request"={"Namespace":"","Name":"cert-manager-webhook"}
I0515 02:40:36.054862       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"challenges.acme.cert-manager.io"}
I0515 02:40:36.054991       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.scheduling.k8s.io"}
I0515 02:40:36.055193       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"ipamblocks.crd.projectcalico.org"}
I0515 02:40:36.055239       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.rbac.authorization.k8s.io"}
I0515 02:40:36.055403       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.node.k8s.io"}
I0515 02:40:36.055451       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"filterpolicies.getambassador.io"}
I0515 02:40:36.055566       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.authorization.k8s.io"}
I0515 02:40:36.055706       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.networking.k8s.io"}
I0515 02:40:36.055784       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"logservices.getambassador.io"}
I0515 02:40:36.055873       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1alpha3.acme.cert-manager.io"}
I0515 02:40:36.055967       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"bgppeers.crd.projectcalico.org"}
I0515 02:40:36.056099       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.getambassador.io"}
I0515 02:40:36.056244       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.scheduling.k8s.io"}
I0515 02:40:36.056383       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.policy"}
I0515 02:40:36.056527       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v2beta1.autoscaling"}
I0515 02:40:36.056719       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.authorization.k8s.io"}
I0515 02:40:36.056937       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.extensions"}
I0515 02:40:36.057431       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.storage.k8s.io"}
I0515 02:40:36.057608       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.admissionregistration.k8s.io"}
I0515 02:40:36.057843       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v2beta2.autoscaling"}
I0515 02:40:36.058927       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.autoscaling"}
I0515 02:40:36.059177       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.coordination.k8s.io"}
I0515 02:40:36.059364       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.batch"}
I0515 02:40:36.059505       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.authentication.k8s.io"}
I0515 02:40:36.059655       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.apiextensions.k8s.io"}
I0515 02:40:36.059847       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1alpha2.acme.cert-manager.io"}
I0515 02:40:36.060074       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1beta1.certificates.k8s.io"}
I0515 02:40:36.060224       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.coordination.k8s.io"}
I0515 02:40:36.060357       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v2.getambassador.io"}
I0515 02:40:36.060501       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.crd.projectcalico.org"}
I0515 02:40:36.060642       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="apiservice" "request"={"Namespace":"","Name":"v1.getambassador.io"}
I0515 02:40:36.303478       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="certificates.cert-manager.io" "resource_namespace"="" 
I0515 02:40:36.303698       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"certificates.cert-manager.io"}
I0515 02:40:36.311511       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="orders.acme.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:36.471880       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="orders.acme.cert-manager.io" "resource_namespace"="" 
I0515 02:40:36.472601       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"orders.acme.cert-manager.io"}
I0515 02:40:36.474510       1 sources.go:176] cert-manager/inject-controller "msg"="Extracting CA from Secret resource" "resource_kind"="CustomResourceDefinition" "resource_name"="challenges.acme.cert-manager.io" "resource_namespace"="" "secret"="cert-manager/cert-manager-webhook-ca"
I0515 02:40:36.866916       1 controller.go:172] cert-manager/inject-controller "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="challenges.acme.cert-manager.io" "resource_namespace"="" 
I0515 02:40:36.867931       1 controller.go:282] cert-manager/controller-runtime/controller "msg"="Successfully Reconciled"  "controller"="customresourcedefinition" "request"={"Namespace":"","Name":"challenges.acme.cert-manager.io"}

kubectl logs -n cert-manager cert-manager-webhook-7c75b89bf6-c4fr6:

W0515 02:40:11.031377       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0515 02:40:11.033489       1 webhook.go:63]  "msg"="using dynamic certificate generating using CA stored in Secret resource"  "secret_name"="cert-manager-webhook-ca" "secret_namespace"="cert-manager"
I0515 02:40:11.035773       1 server.go:139]  "msg"="listening for insecure healthz connections"  "address"=":6080"
I0515 02:40:11.036697       1 server.go:152]  "msg"="listening for secure connections"  "address"=":10250"
I0515 02:40:11.036890       1 server.go:178]  "msg"="registered pprof handlers"  
I0515 02:40:11.053553       1 reflector.go:175] Starting reflector *v1.Secret (1m0s) from external/io_k8s_client_go/tools/cache/reflector.go:125
I0515 02:40:11.252535       1 authority.go:313]  "msg"="Generating new root CA"  
I0515 02:40:11.838994       1 authority.go:248]  "msg"="Detected change in CA secret data, notifying watchers..."  
I0515 02:40:12.043864       1 dynamic_source.go:171]  "msg"="Generating new ECDSA private key"  
I0515 02:40:12.100600       1 dynamic_source.go:186]  "msg"="Signing new serving certificate"  
I0515 02:40:12.448962       1 dynamic_source.go:192]  "msg"="Signed new serving certificate"  
I0515 02:40:12.516189       1 dynamic_source.go:198]  "msg"="Updated serving TLS certificate"
@meyskens
Copy link
Contributor

Can you please get us the following info:

  • version of cert-manager installed
  • state and logs of the webhook pod in namespace cert-manager

Thanks

/triage support

@jetstack-bot jetstack-bot added the triage/support Indicates an issue that is a support question. label May 15, 2020
@meyskens meyskens changed the title Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=30s: context deadline exceeded Internal error occurred: failed calling webhook May 15, 2020
@9p4
Copy link
Author

9p4 commented May 15, 2020

@meyskens I have updated the question.

@thommay
Copy link

thommay commented May 31, 2020

Also seeing this on a freshly installed k8s 1.18.3 cluster with cert manager 0.15.1. Logs from the webhook pod are identical to the OP's.

@cachak
Copy link

cachak commented Jun 8, 2020

same issue with k8s 1.18.3 cluster with cert manager 0.15.1

@zedtux
Copy link

zedtux commented Jun 12, 2020 •
edited

same issue with k8s 1.15.12 and cert manager 0.15.1. Is there any workaround to this issue?

@milkeryildirim
Copy link

same issue with k8s 1.17.2 and cert-manager 0.15.1...

@estheban
Copy link

same issue with k8s 1.17.5 and cert-manager 0.14.3...

@zedtux
Copy link

zedtux commented Jun 17, 2020

What OS are you using because it was working fine with cert-manager 0.13 on Debian 9, but since I moved to Ubuntu 18.04, and I can't make it working anymore (I wrote a guide at that time).

@9p4
Copy link
Author

9p4 commented Jun 17, 2020

I’m running all my nodes on CentOS 7. I’ve updated my question.

@ltetrel
Copy link

ltetrel commented Jun 18, 2020 •
edited

I am also running on OpenNebula k8s cluster.
Reolved it by running the cert-manager pods on the master node.

First make sure to label your master node:
kubectl label nodes <master-name> kubernetes.io/role=master

Now install cert-manager with the following helm cmd:

helm install --name cert-manager --namespace cert-manager --version v0.15.1 \
jetstack/cert-manager \
--set nodeSelector."kubernetes\.io/role"=master  \
--set cainjector.nodeSelector."kubernetes\.io/role"=master \
--set webhook.nodeSelector."kubernetes\.io/role"=master

@ltetrel ltetrel mentioned this issue Jun 18, 2020
Closed
@0521ak47
Copy link

same issue with k8s 1.18.2 cluster calico with cert manager 0.15.1 and 0.13.1

@zxdyumiao
Copy link

same issue with k8s 1.17.2 and cert-manager 0.15.1

@ltetrel ltetrel mentioned this issue Jul 1, 2020
Closed
@a-dawg
Copy link

a-dawg commented Jul 14, 2020 •
edited

same issue with openshift 4.3 and cert manager 0.15.1
the port 10250 does not seem to be listening in the cert-manager-webhook pod container, however the liveness probe is up :(

@sboschman
Copy link

same issue after upgrading k8s 1.17.x to 1.18.6 (Rancher / flannel 0.12.0) with cert-manager 0.15.0, 0.15.2 and 0.16.0

couple of observations:

  • unable to list cert-manager resources (certs, orders, etc) using kubectl, same error as it triggers the webhook
  • port-forward to webhook service and pod respond to requests
  • using curl inside the kube-apiserver pod:
    • webhook service ip takes more than a minute to respond
    • webhook pod ip responds immediately

which took me to this issue with flannel: flannel-io/flannel#1243

after adding the route as proposed here things seem to start working again

@zedtux
Copy link

zedtux commented Aug 5, 2020 •
edited

I would like to post that now I no more have any issues with cert-manager in my Kubernetes 1.15.12 cluster.

I did ran the sonobuoy tool which highlighted that master nodes were failing to communicate with worker node after the provisioning tool Chef ran due to a restart of the VPN service (that allows masters and workers to talk to each other securely).
Reboot the nodes (was a testing cluster) made the e2e tests passing, then cert-manager worked again.

I'm now checking why restarting the VPN service prevents the nodes to communicate together, but that's another topic.

To be more precise on my case, I'm running sonobuoy run -p systemd-logs && watch -n 1 sonobuoy status and in 15 seconds max all tests should be completed and marked as passed.
After Chef ran on a node, running the same makes that node test stuck.

When that happen, after grabbing the sonobuoy-systemd-logs-daemon-set-XXXX pod name of that node, looking at the sonobuoy-worker container's logs, I can see:

kubectl logs -f sonobuoy-systemd-logs-daemon-set-464c7dbf3bf84b2d-kttf4 -n sonobuoy -c sonobuoy-worker
time="2020-08-05T09:37:01Z" level=info msg="Waiting for waitfile" waitfile=/tmp/results/done
time="2020-08-05T09:37:01Z" level=info msg="Starting to listen on port 8099 for progress updates and will relay them to https://[10.244.3.129]:8080/api/v1/progress/by-node/west-stg-mst-3/systemd-logs"
time="2020-08-05T09:37:02Z" level=info msg="Detected done file, transmitting result file" resultFile=/tmp/results/systemd_logs
time="2020-08-05T09:39:13Z" level=error msg="error entry for attempt: 1, verb: PUT, time: 2020-08-05 09:39:13.777742167 +0000 UTC m=+131.832465314, URL: https://[10.244.3.129]:8080/api/v1/results/by-node/west-stg-mst-3/systemd-logs: Put https://[10.244.3.129]:8080/api/v1/results/by-node/west-stg-mst-3/systemd-logs: dial tcp 10.244.3.129:8080: connect: connection timed out"

All in all the sonobuoy-systemd-logs-daemon-set-XXXX pod, from the defective node, tries to reach the sonobuoy pod from the sonobuoy namespace but can't (while other sonobuoy-systemd-logs-daemon-set-XXXX pods did well) and the test is "stuck".

Update: Actually I just discovered that when restarting the VPN service, the flannel network interface disappears and is never recreated! That explains why Kubernetes pods on that node can't communicate with other pods from the cluster.
Like I said in that issue, restarting Docker allows Flannel to create its network interface and the sonobuoy tests are passing again.

@dale88 dale88 mentioned this issue Oct 29, 2020
Closed
@meyskens
Copy link
Contributor

Going to close this one as it doesn't seem cert-manager related. Feel free to /reopen if needed.
If you think it could be a useful thing to document PRs to the documentations are always open!

@meyskens
Copy link
Contributor

/close

@jetstack-bot
Copy link
Contributor

@meyskens: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@alikarimii
Copy link

same issue when i want to create ingress service (kind: Ingress)
and i solved that by simple ping from ingress-nginx-controller to ingress-nginx-controller-admission.ingress-nginx.svc
shell to ingress-nginx-controller-ldkssk-sdjfn (this must be different in your env) and ping ingress-nginx-controller-admission.ingress-nginx.svc
i don't know why, but work

@alexpapworth
Copy link

@alikarimii Got any more commands for this one? Do you need to exec into the ingress-nginx-controller first?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
triage/support Indicates an issue that is a support question.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests