Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MutatingWebhookConfiguration conflict when upgrading from 1.6 to 1.7 #88

Closed
NohaIhab opened this issue Jun 15, 2023 · 2 comments · Fixed by #90
Closed

MutatingWebhookConfiguration conflict when upgrading from 1.6 to 1.7 #88

NohaIhab opened this issue Jun 15, 2023 · 2 comments · Fixed by #90
Labels
bug Something isn't working

Comments

@NohaIhab
Copy link
Contributor

NohaIhab commented Jun 15, 2023

the name of mutatingWebhookConfiguration resource that the admission-webhook charm creates has been changed from admission-webhook in 1.6 to admission-webhook-mutating-webhook-configuration in 1.7.
In the case of upgrading CKF 1.6 to 1.7, this leads to 2 mutatingWebhookConfiguration objects existing in the cluster for the same webhook. As a result, when creating a notebook, the webhook gets triggered twice from both mutatingWebhookConfiguration objects and the notebook gets stuck in FailedCreate with a conflict on the PodDefaults being applied to the notebook pod.

@NohaIhab NohaIhab added the bug Something isn't working label Jun 15, 2023
@NohaIhab
Copy link
Contributor Author

NohaIhab commented Jun 15, 2023

Logs from the admission-webhook workload:

I0614 17:08:21.895892       1 main.go:660] 1 matching pod defaults, for pod testnb-0
I0614 17:08:21.895904       1 main.go:666] Matching PD detected of count 1, patching spec
I0614 17:08:21.895970       1 main.go:479] mutating pod: testnb-0
I0614 17:08:21.896013       1 main.go:681] applied poddefaults: access-ml-pipeline successfully on Pod: testnb-0 
I0614 17:08:21.904028       1 main.go:598] Entering mutatePods in mutating webhook
I0614 17:08:21.904575       1 main.go:624] Looking at pod annotations, found: map[kubectl.kubernetes.io/default-container:testnb kubectl.kubernetes.io/default-logs-container:testnb poddefault.admission.kubeflow.org/poddefault-access-ml-pipeline:34168 prometheus.io/path:/stats/prometheus prometheus.io/port:15020 prometheus.io/scrape:true sidecar.istio.io/status:{"initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["workload-socket","credential-socket","workload-certs","istio-envoy","istio-data","istio-podinfo","istio-token","istiod-ca-cert"],"imagePullSecrets":null,"revision":"default"}]
I0614 17:08:22.765280       1 main.go:644] fetched 1 poddefault(s) in namespace test
I0614 17:08:22.765324       1 main.go:660] 1 matching pod defaults, for pod testnb-0
I0614 17:08:22.765332       1 main.go:666] Matching PD detected of count 1, patching spec
E0614 17:08:22.765533       1 main.go:349] merging volumes for access-ml-pipeline has a conflict on volume-kf-pipeline-token: 

@NohaIhab
Copy link
Contributor Author

NohaIhab commented Jun 15, 2023

MutatingWebhookConfiguration objects in the cluster:

kubectl get mutatingwebhookconfigurations | grep admission-webhook
NAME                                                        WEBHOOKS   
kubeflow-admission-webhook                                  1          
kubeflow-admission-webhook-mutating-webhook-configuration   1          

NohaIhab added a commit that referenced this issue Jun 15, 2023
…figuration

fix: revert name change of webhook config fixes #88
NohaIhab added a commit that referenced this issue Jun 15, 2023
@kimwnasptd kimwnasptd added this to Needs Triage in MLOps Solution Issues via automation Jun 20, 2023
@kimwnasptd kimwnasptd moved this from Needs Triage to Labeled in MLOps Solution Issues Jun 20, 2023
NohaIhab added a commit that referenced this issue Jul 4, 2023
MLOps Solution Issues automation moved this from Labeled to Done Jul 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Development

Successfully merging a pull request may close this issue.

1 participant