[BUG] Getting "webhook configurations error" #2179
Comments
om3171991
changed the title
|
What version of KinD on what type of OS? I use KinD myself nearly every day and don't see this. |
|
@chipzoller - I have created 3 clusters and observing the same issue - no webhook for kyverno. |
|
Hi @om3171991 - can you share the command that was used to install Kyverno? Besides, can you please attach the output of the command |
|
Hi @realshuting For installing Kyverno, We have uploaded the helm chart in our internal chartmuseum from below URL: and cmd used to install: helm install --kube-context kind-test kyverno kyverno --namespace kyverno ➜ kyverno git: ✗ test -n kyverno get ep |
|
Thanks @om3171991! It looks like there's no endpoint registered for Kyverno Pod, can you please describe the endpoint and inspect the issue? Here's what I have, you can see that the Pod's IP is listed: |
|
@realshuting - Can is it because the pod is not in a ready state (It's in CrashLoopBackOff because of the error mention in issue) kubectl --context kind-test -n kyverno get all NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE NAME READY UP-TO-DATE AVAILABLE AGE NAME DESIRED CURRENT READY AGE |
|
No, the Pod crashed because the Pod IP was not registered as the endpoint successfully. Can you provide the following output? |
|
➜ kyverno git : kubectl -n kyverno describe endpoint --context kind-test |
|
Sorry it should be |
|
➜ kyverno git : kubectl -n kyverno describe endpoints --context kind-test Events: Warning FailedToUpdateEndpoint 23m endpoint-controller Failed to update endpoint kyverno/kyverno-svc: Operation cannot be fulfilled on endpoints "kyverno-svc": the object has been modified; please apply your changes to the latest version and try again Name: kyverno-svc-metrics Events: Warning FailedToUpdateEndpoint 23m endpoint-controller Failed to update endpoint kyverno/kyverno-svc-metrics: Operation cannot be fulfilled on endpoints "kyverno-svc-metrics": the object has been modified; please apply your changes to the latest version and try again |
|
Interesting that there are two endpoints but your previous output only shows one Pod. Do you currently have 2 Kyverno pods running? If possible, can you re-install Kyverno and increase |
|
Reason for 2 IPs - I have run the kyverno helm upgrade command and the old pod is still there with CrashLoopBackOff ➜ kyverno git : kubectl --context kind-test -n kyverno get pod -o wide Increasing initialDelaySeconds - already tried that with 60 seconds for both ReadinessProbe and LivenessProbe but still, it is failing. Below is config which we are using : livenessProbe: readinessProbe: |
|
Were these Probes configured with a clean installation? I'll have to dig into the code and check if there's any dependency between the endpoints' status and the pod's. |
|
Yes. So to clear out if any confusion exist - I have reinstalled everything again. Please find the output for all commands below : ➜ kyverno git: kubectl --context kind-test -n kyverno get all NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE NAME READY UP-TO-DATE AVAILABLE AGE NAME DESIRED CURRENT READY AGE ➜ kubectl --context kind-test -n kyverno get pod -o wide ➜ kyverno git: kubectl -n kyverno describe endpoints --context kind-test Name: kyverno-svc-metrics ➜ kyverno git: kubectl -n kyverno --context kind-test get ep -n kyverno ➜ kyverno git: kubectl -n kyverno --context kind-test describe pod/kyverno-66548b7b4f-rlvhz ➜ kyverno git: kubectl -n kyverno --context kind-test logs pod/kyverno-66548b7b4f-rlvhz |
|
@realshuting - did you able to look and find something on this ? |
|
No @om3171991 , I'm currently focusing on 1.4.2 milestone completion, will revisit this issue after. |
|
@om3171991 I just install KinD 0.11.1 and Kyverno 1.4.1 from the upstream Helm repo on MacOS 10.14.6 running Docker Desktop for Mac v3.5.2 and I do not see an issue running Kyverno. |
|
@chipzoller - really appreciate that you tried on local setup but now I am totally confused why is it not working on my system. Any pointers what else I can check to fix this. |
|
I would clean up your setup by removing the KinD node image(s) you have and updating or reinstalling whatever you're using as an underlying engine (like Docker Desktop for Mac). Because Kyverno definitely does work on this combination, it's something specific to your setup. |
realshuting
added
the
end user
|
I've got the same error on my RPI cluster. I change the initialDelaySeconds of ReadinessProbe to 30, like @realshuting said. |
realshuting
modified the milestones:
Kyverno Release 1.5.1,
Kyverno Release 1.5.2,
Kyverno Release 1.6.0
|
We have run into this problem ("Timeout registering admission control webhooks") on startup. After becoming the leader, the pod seems to execute on the policies and then fails with this message, shutting down. This means that Kyverno never gets to a stable state. We aren't 100% sure (we aren't go experts), but evidence points to our issue being related to having Failed pods for reason "Shutdown". K8s doesn't clean these pods up automatically so you can examine them. When we have pods in this state for Kyverno, the new pods that are trying to start seem to get this error. Once we remove these Shutdown pods in the Kyverno namespace then Kyverno comes up and stabalizes. My guess is that Kyverno is doing something with the list of pods and is failing to account for shutdown pods. For instance, I see evidence in the code that it is trying to get the IP for the pods, but shutdown pods will not have an IP since they are not scheduled on a node. In our case, we are using Kyverno 1.5.1. We are in GKE using preemptible nodes, which means that our nodes shutdown and recycle at least 1x a day and evict any pods that were scheduled on them. This causes those pods to enter this "Failed, Shutdown" state. Pods in this state have the following information in them: Status: Failed Reason: Shutdown Message: Node is shutting, evicting pods Again, only circumstantial evidence so far - but it seems to be getting stronger every time we need to restart Kyverno (which is basically daily at this point). |
@awoodsprim - can you share logs for all Kyverno pods? I need to understand why the pods shut down. Was Kyverno functioning before? |
@nlamirault - this issue was fixed in 1.5.2 via #2757. |
|
Hi @awoodsprim - I did a few tests and was able to reproduce the same issue using the Kind cluster. The issue is related to how the node shuts down. Here are the tests I did: Test 1: running 1 Kyverno instance, remove a k8s node by "kubectl delete node "With this test, the old pod was terminated and the new pod was rescheduled to Test 2: running 1 Kyverno instance, remove a k8s node by "docker kill " (a Kind node is a container)There was no log after the container was killed since After a few seconds, the node became not ready and the pod remained running. This is because the kubelet on that node also died after docker kill, it can no longer update the pod status. Any operations on the matching resources (resources matched by configured policies) were rejected, see the below error message "connection refused". The only way to bring up a new kyverno instance is to delete After I removed all webhook configurations, the pod got rescheduled but went into The reason we added endpoint check is to solve #1740. The workaround is to run Kyverno with multiple replicas. After I scaled Kyverno up to 2 replicas, the issue was gone for test 2 (killed the node where kyverno leader/non-leader was scheduled on) and the cluster stayed healthy. Can you try the same and verify if it solves your issue? |
|
Unfortunately in my case, multiple replicas doesn't help. All nodes are shutdown by GKE simultaneously and it results in the same behavior. (No replicas are left standing)
Get Outlook for Android<https://aka.ms/AAb9ysg>
…________________________________
From: shuting ***@***.***>
Sent: Thursday, December 30, 2021 7:16:33 AM
To: kyverno/kyverno ***@***.***>
Cc: Alan Wood ***@***.***>; Mention ***@***.***>
Subject: Re: [kyverno/kyverno] [BUG] Getting "webhook configurations error" (#2179)
Hi @awoodsprim<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fawoodsprim&data=04%7C01%7Calan.wood%40obviohealth.com%7C95aa164d459545dbbaac08d9cb8e38c7%7C9b64610429da4dde80f9c008f147f05a%7C0%7C0%7C637764633967514251%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=CmOrKobhFwLAGz65ObhmcmdkzazN1C95WIW2gwnZr0o%3D&reserved=0> - I did a few tests and was able to reproduce the same issue using the Kind cluster. The issue is related to how the node shuts down. Here are the tests I did:
Test 1: running 1 Kyverno instance, remove a k8s node by "kubectl delete node "
With this test, the old pod was terminated and the new old was rescheduled to kind-worker5 by k8s. Kyverno was functioning after it came up.
I1230 11:07:30.160148 1 client.go:243] dclient/Poll "msg"="stopping registered resources sync"
I1230 11:07:30.160402 1 certmanager.go:175] CertManager "msg"="stopping cert renewer"
I1230 11:07:30.160415 1 informer.go:118] PolicyCacheController "msg"="shutting down"
I1230 11:07:30.160426 1 reportrequest.go:192] ReportChangeRequestGenerator "msg"="shutting down"
I1230 11:07:30.160440 1 controller.go:129] EventGenerator "msg"="shutting down"
I1230 11:07:30.160643 1 generate_controller.go:150] GenerateController "msg"="shutting down"
I1230 11:07:30.160664 1 reportcontroller.go:267] PolicyReportGenerator "msg"="shutting down"
I1230 11:07:30.160756 1 policy_controller.go:460] PolicyController "msg"="shutting down"
I1230 11:07:30.160799 1 controller.go:274] GenerateCleanUpController "msg"="shutting down"
I1230 11:07:30.202363 1 leaderelection.go:103] kyverno/LeaderElection "msg"="leadership lost, stopped leading" "id"="kyverno-7bc64dc96b-vvczf_9c951dae-5fd4-475f-a4a1-9250b8794506"
I1230 11:07:30.206415 1 leaderelection.go:103] webhookRegister/LeaderElection "msg"="leadership lost, stopped leading" "id"="kyverno-7bc64dc96b-vvczf_5283dc55-549a-4057-872e-2a05f9003f3b"
I1230 11:07:30.212881 1 registration.go:269] Register/cleanupKyvernoResource "msg"="updating Kyverno Pod, won't clean up Kyverno resources"
I1230 11:07:30.212993 1 main.go:536] setup "msg"="Kyverno shutdown successful"
rpc error: code = NotFound desc = an error occurred when try to find container "283924b9bf5e2eaeeaf273ba900f88f8cb7d6e05e5ee55b1a5d0a17b993f2100": not found%
✗ kg -n kyverno pod -o wide -w
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kyverno-7bc64dc96b-vvczf 1/1 Running 0 2m56s 10.244.2.2 kind-worker4 <none> <none>
kyverno-7bc64dc96b-vvczf 1/1 Terminating 0 4m1s 10.244.2.2 kind-worker4 <none> <none>
kyverno-7bc64dc96b-vvczf 1/1 Terminating 0 4m1s 10.244.2.2 kind-worker4 <none> <none>
kyverno-7bc64dc96b-btxmg 0/1 Pending 0 0s <none> <none> <none> <none>
kyverno-7bc64dc96b-btxmg 0/1 Pending 0 0s <none> kind-worker5 <none> <none>
kyverno-7bc64dc96b-btxmg 0/1 Init:0/1 0 0s <none> kind-worker5 <none> <none>
kyverno-7bc64dc96b-btxmg 0/1 Init:0/1 0 15s 10.244.5.2 kind-worker5 <none> <none>
kyverno-7bc64dc96b-btxmg 0/1 PodInitializing 0 33s 10.244.5.2 kind-worker5 <none> <none>
kyverno-7bc64dc96b-btxmg 0/1 Running 0 49s 10.244.5.2 kind-worker5 <none> <none>
kyverno-7bc64dc96b-btxmg 1/1 Running 0 60s 10.244.5.2 kind-worker5 <none> <none>
Test 2: running 1 Kyverno instance, remove a k8s node by "docker kill " (a Kind node is a container)
There was no log after the container was killed since docker kill kills the container immediately.
After a few seconds, the node became not ready and the pod remained running. This is because the kubelet on that node also died after docker kill, it can no longer update the pod status. Any operations on the matching resources (resources matched by configured policies) were rejected, see the below error message "connection refused". The only way to bring up a new kyverno instance is to delete mutatingwebhookconfigurations and validatingwebhookconfigurations to let the request pass through.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 4m5s default-scheduler Successfully assigned kyverno/kyverno-7bc64dc96b-btxmg to kind-worker5
Normal Pulling 4m5s kubelet Pulling image "ghcr.io/kyverno/kyvernopre:v1.5.2"
Normal Pulled 3m51s kubelet Successfully pulled image "ghcr.io/kyverno/kyvernopre:v1.5.2" in 13.650407899s
Normal Created 3m51s kubelet Created container kyverno-pre
Normal Started 3m51s kubelet Started container kyverno-pre
Normal Pulling 3m32s kubelet Pulling image "ghcr.io/kyverno/kyverno:v1.5.2"
Normal Pulled 3m17s kubelet Successfully pulled image "ghcr.io/kyverno/kyverno:v1.5.2" in 15.096770042s
Normal Created 3m17s kubelet Created container kyverno
Normal Started 3m17s kubelet Started container kyverno
Warning NodeNotReady 10s node-controller Node is not ready
✗ k run nginx --image=nginx:latest --image-pull-policy=IfNotPresent
Error from server (InternalError): Internal error occurred: failed calling webhook "mutate.kyverno.svc-fail": failed to call webhook: Post "https://kyverno-svc.kyverno.svc:443/mutate?timeout=10s": dial tcp 10.96.242.150:443: connect: connection refused
After I removed all webhook configurations, the pod got rescheduled but went into CrashLoopBackOff due to "Timeout registering admission control webhooks". And in this case, the Pod's IP was never registered in the endpoint (not sure why).
The reason we added endpoint check is to solve #1740<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkyverno%2Fkyverno%2Fissues%2F1740&data=04%7C01%7Calan.wood%40obviohealth.com%7C95aa164d459545dbbaac08d9cb8e38c7%7C9b64610429da4dde80f9c008f147f05a%7C0%7C0%7C637764633967514251%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=0FJ3815wFhoJyzDR%2FwcK7CZcQ%2Bsjh4jwVZofnKYEJlY%3D&reserved=0>.
✗ kg -n kyverno pod -o wide -w
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kyverno-7bc64dc96b-btxmg 1/1 Running 0 3m18s 10.244.5.2 kind-worker5 <none> <none>
kyverno-7bc64dc96b-btxmg 1/1 Running 0 3m55s 10.244.5.2 kind-worker5 <none> <none>
kyverno-7bc64dc96b-btxmg 1/1 Terminating 0 6m8s 10.244.5.2 kind-worker5 <none> <none>
kyverno-7bc64dc96b-zdh6h 0/1 Pending 0 0s <none> <none> <none> <none>
kyverno-7bc64dc96b-zdh6h 0/1 Pending 0 0s <none> kind-worker3 <none> <none>
kyverno-7bc64dc96b-zdh6h 0/1 Init:0/1 0 0s <none> kind-worker3 <none> <none>
kyverno-7bc64dc96b-zdh6h 0/1 Init:0/1 0 12s 10.244.1.2 kind-worker3 <none> <none>
kyverno-7bc64dc96b-zdh6h 0/1 PodInitializing 0 30s 10.244.1.2 kind-worker3 <none> <none>
kyverno-7bc64dc96b-zdh6h 0/1 Running 0 48s 10.244.1.2 kind-worker3 <none> <none>
kyverno-7bc64dc96b-zdh6h 1/1 Running 0 60s 10.244.1.2 kind-worker3 <none> <none>
kyverno-7bc64dc96b-zdh6h 0/1 Error 0 94s 10.244.1.2 kind-worker3 <none> <none>
kyverno-7bc64dc96b-zdh6h 0/1 Running 1 (2s ago) 95s 10.244.1.2 kind-worker3 <none> <none>
kyverno-7bc64dc96b-zdh6h 1/1 Running 1 (7s ago) 100s 10.244.1.2 kind-worker3 <none> <none>
kyverno-7bc64dc96b-zdh6h 0/1 Running 2 (0s ago) 2m1s 10.244.1.2 kind-worker3 <none> <none>
kyverno-7bc64dc96b-zdh6h 1/1 Running 2 (9s ago) 2m10s 10.244.1.2 kind-worker3 <none> <none>
kyverno-7bc64dc96b-zdh6h 0/1 Error 2 (31s ago) 2m32s 10.244.1.2 kind-worker3 <none> <none>
kyverno-7bc64dc96b-zdh6h 0/1 CrashLoopBackOff 2 (9s ago) 2m40s 10.244.1.2 kind-worker3 <none> <none>
The workaround is to run Kyverno with multiple replicas. After I scaled Kyverno up to 2 replicas, the issue was gone with test 2 and the cluster stayed healthy. Can you try the same and verify if it solves your issue?
—
Reply to this email directly, view it on GitHub<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkyverno%2Fkyverno%2Fissues%2F2179%23issuecomment-1003004320&data=04%7C01%7Calan.wood%40obviohealth.com%7C95aa164d459545dbbaac08d9cb8e38c7%7C9b64610429da4dde80f9c008f147f05a%7C0%7C0%7C637764633967514251%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=8HLxosSwoGi21NS%2Ba0VUn2JYGAst3bGQ%2BBOinCT26%2BM%3D&reserved=0>, or unsubscribe<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAPI66A435AW4HLLPHF7VQGLUTRESDANCNFSM5AYLCEUA&data=04%7C01%7Calan.wood%40obviohealth.com%7C95aa164d459545dbbaac08d9cb8e38c7%7C9b64610429da4dde80f9c008f147f05a%7C0%7C0%7C637764633967514251%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=xE670pWMKnpsLI4lWw1N11i0nOXzxKUK469NisjYPuc%3D&reserved=0>.
Triage notifications on the go with GitHub Mobile for iOS<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Calan.wood%40obviohealth.com%7C95aa164d459545dbbaac08d9cb8e38c7%7C9b64610429da4dde80f9c008f147f05a%7C0%7C0%7C637764633967514251%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=kyUr1W%2FlwCtlR1uMtlH%2BEkqs07cn7WUJPGwGbrJ8mBE%3D&reserved=0> or Android<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Calan.wood%40obviohealth.com%7C95aa164d459545dbbaac08d9cb8e38c7%7C9b64610429da4dde80f9c008f147f05a%7C0%7C0%7C637764633967514251%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=tx%2F7AmV45K9y9t%2BG%2BVuWfi37rKch4ZxaDhikjY7XpiE%3D&reserved=0>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
** This e-mail is from an external sender.**
|
Again the issue relates to the shut down process. Was Kyverno scaled down to zero before shutdown? If the Kyverno shuts down properly, it automatically cleans up all managed webhook configurations and won't impact the cluster. |
|
No, unfortunately GKE is rather harsh when it comes to pre-emptive nodes. It terminates the nodes and evicts the pods.
Scaling down to 0 and then back to any number of replicas does not fix this situation after this happens - the pods come back up and immediately start to fail again.
The only thing that seems to fix it is removing the pods that are still present in the evicted status (Failed, Shutdown). Once that happens, Kyverno recovers.
|
|
In my test 2, the container got deleted immediately so the webhook configurations were not garbage collected by the Kyverno instances thus blocking all the rest of the admission requests.
I'm asking if it's possible to scale Kyverno down to 0 before nodes shut down.
If there's no way to terminate Kyverno gracefully, the alternative is to delete existing Kyverno pods when the cluster restarts as you mentioned above. |
|
We are not in control of the node shutdown, so no we have no method that I'm aware of to intercept GKE shutting down the nodes.
As for the cleanup routine - we are planning to do this, but my question is why does Kyverno have this issue? We have numerous other validating and mutating webhook configurations that do not experience this problem even though they are also pre-empted like Kyverno is which points to it being a Kyverno issue.
Is there a situation where it scans / finds pods and is therefore picking up these pods which have no IP address? My guess is that the broken code may be in "webhookconfig/registration.go". There is a line there about getting the pod ip that looks suspect to me:
```
kyverno := pods.Items[0]
podIP, _, err := unstructured.NestedString(kyverno.UnstructuredContent(), "status", "podIP")
if err != nil {
return fmt.Errorf("failed to extract pod IP: %v", err)
}
if podIP == "" {
return fmt.Errorf("pod is not assigned to any node yet")
}
```
From what I can see you are getting a list of all the pods, and assuming that the first pod in the array is interesting. In my case, that is very likely not the case since the pod list will have shutdown pods in the array.
I've updated the verbose setting to level 6 and this error message was output to the logs:
```
│ kyverno I0103 15:55:01.013863 1 common.go:131] setup "msg"="Failed to register admission control webhooks" "reason"="pod is not assigned to any node yet" │
│ kyverno E0103 15:55:01.986167 1 main.go:405] setup "msg"="Timeout registering admission control webhooks" "error"=null
```
This seems to verify my suspicion.
Also not 100% sure what the future lines are doing:
```
for _, subset := range endpoint.Subsets {
if len(subset.Addresses) == 0 {
continue
}
for _, addr := range subset.Addresses {
if addr.IP == podIP {
wrc.log.Info("Endpoint ready", "ns", config.KyvernoNamespace, "name", config.KyvernoServiceName)
return nil
}
}
}
```
That seems again like you are comparing the endpoints to an arbitrary pod IP - does this work when there are multiple replicas across multiple nodes or are you just verifying that at least one of the endpoints is pointing to the first pod from the listed pods?
|
Yes, the endpoint check was added to solve #1740. I see there's an improvement that can be made when getting the pod's IP, I'll send a PR to check the pod's status when getting the IPs. Thanks for pointing this out. |
|
@realshuting Sounds great, happy we were able to track it down. Hopefully that additional check (along with us cleaning up shutdown pods) will fix our issue. |
|
Hi @awoodsprim - I sent the improvement via #2902, here's the test image that you can verify Can you test the fix and let me know if it works? If not, can you attach the kubelet's log? |
|
Hi @realshuting we are going to put the docker image live in our dev cluster. It can take up to 24 hours to hit the race condition, but we will let you know if by tomorrow this happens again. |
|
@realshuting Preliminary analysis looks like it solved the problem. We were able to see the Kyverno pod startup with a shutdown pod in the API return array earlier and the pod was able to recover (once the leadership lease was expired). We want it to go through one more cycle to be sure, but right now it looks good. |
|
Great! The fix will be out in 1.5.3 in the following days. Let me know if you see any other issues. |
|
Closing the issue, feel free to open if needed. |
|
Hi Team, I was using kyverno 1.7.3 and upgraded to 1.10.3, I removed absolutely everything and installed it from scratch. And I get the same error as the subject of this thread. I don't know how to repair it. I dont know why..but some policies was installed. |
|
Please open a new issue or discussion. |
|
@chipzoller observing the same issue on upgrading from 1.7.5 to 1.10.5 @roldyxoriginal can you please share the issue link if you got a chance to open a new issue? |
Hi Team,
We are facing an issue with the Kind Kubernetes cluster where we are seeing the below error message :
On checking pod logs, we find out that helm installation is not able to create webhook configurations :
E0721 16:03:38.718669 1 monitor.go:160] WebhookMonitor/registerWebhookIfNotPresent "msg"="missing webhooks" "error"="mutatingwebhookconfigurations.admissionregistration.k8s.io "kyverno-verify-mutating-webhook-cfg" not found"
E0721 16:03:38.808018 1 monitor.go:91] WebhookMonitor "msg"="" "error"="failed to register webhooks: Endpoint not ready"
I0721 16:03:39.078243 1 status.go:75] WebhookMonitor/WebhookStatusControl "msg"="updating deployment annotation" "name"="kyverno" "namespace"="kyverno" "key"="kyverno.io/webhookActive" "val"="true"
E0721 16:03:50.758993 1 main.go:363] setup "msg"="Timeout registering admission control webhooks" "error"=null
➜ kyverno git: kubectl --context kind-test get validatingwebhookconfigurations,mutatingwebhookconfigurations
No resources found
whereas we have installed kyverno on the local docker desktop and AKS cluster and it is running fine as expected - Kyverno pod is started successfully without failing any probes.
kind cluster details -
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
prod-control-plane Ready control-plane,master 7d4h v1.21.1 172.18.0.3 Ubuntu 21.04 5.10.25-linuxkit containerd://1.5.2
kyverno installation details -
➜ kyverno git: helm history --kube-context kind-test kyverno --namespace kyverno
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Wed Jul 21 21:27:39 2021 superseded kyverno-1.4.1 v1.4.1 Install complete
2 Wed Jul 21 21:30:03 2021 deployed kyverno-1.4.1 v1.4.1 Upgrade complete
I am not sure if I missed any step as the same helm chart repo is used to install in all clusters - docker desktop, AKS & Kind.
Please help us here as local development of policy is becoming difficult to do and also let me know if any other details is required.
cc: @realshuting
The text was updated successfully, but these errors were encountered: