Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: secret "milvus-operator-webhook-cert" not found #21961

Closed
1 task
Kellen-X opened this issue Feb 3, 2023 · 33 comments
Closed
1 task

[Bug]: secret "milvus-operator-webhook-cert" not found #21961

Kellen-X opened this issue Feb 3, 2023 · 33 comments
Assignees
Labels
kind/bug Issues or changes related a bug stale indicates no udpates for 30 days triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone

Comments

@Kellen-X
Copy link

Kellen-X commented Feb 3, 2023

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:  2.2.X
- Deployment mode(standalone or cluster):   cluster
- OS(Ubuntu or CentOS):   CentOS
- Others:   Alibaba Cloud Container Service for Kubernetes  1.24.6-aliyun.1  

Current Behavior

image
image

image

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

@Kellen-X Kellen-X added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 3, 2023
@Kellen-X Kellen-X changed the title [Bug]: [Bug]: secret "milvus-operator-webhook-cert" not found Feb 3, 2023
@LoveEachDay
Copy link
Contributor

/assign @haorenfsa

@haorenfsa
Copy link
Contributor

haorenfsa commented Feb 3, 2023

@Kellen-X Hi, thank you for this feedback. Could you tell me how did you install the milvus-operator? or which document did you follow?

Or you can try follow the instructions in https://github.com/milvus-io/milvus-operator/blob/main/docs/installation/installation.md to install / update the milvus-operator again

@Kellen-X
Copy link
Author

Kellen-X commented Feb 3, 2023

@haorenfsa
I installed using kubectl according to the official documentation. https://milvus.io/docs/install_cluster-milvusoperator.md

image

image

image

image

@haorenfsa
Copy link
Contributor

haorenfsa commented Feb 3, 2023

@Kellen-X Cloud you also provide the log of pod milvus-operator-check-2m4fr?

@yanliang567
Copy link
Contributor

/unassign

@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 3, 2023
@yanliang567 yanliang567 added this to the 2.2.3 milestone Feb 3, 2023
@Kellen-X
Copy link
Author

Kellen-X commented Feb 3, 2023

@Kellen-XCloud 你也提供pod的日志milvus-operator-check-2m4fr
@haorenfsa

kubectl logs -f milvus-operator-checker-fx4qq -n milvus-operator

I0203 09:24:24.193250 1 request.go:668] Waited for 1.017843536s due to client-side throttling, not priority and fairness, request: GET:https://192.168.0.1:443/apis/milvus.io/v1beta1?timeout=32s
2023/02/03 09:24:34 unable to install certificationfailed to create certificate: failed to create resource Certificate/milvus-operator-serving-cert: Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": context deadline exceeded

@haorenfsa
Copy link
Contributor

haorenfsa commented Feb 3, 2023

@Kellen-X It seems that milvus-operator failed to install the cert-manager in your kubernetes due to the time out.

You can try fix it by apply below yaml file with kubectl:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: milvus-operator-selfsigned-issuer
  namespace: milvus-operator
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: milvus-operator-serving-cert
  namespace: milvus-operator
spec:
  dnsNames:
  - milvus-operator-webhook-service.milvus-operator.svc
  - milvus-operator-webhook-service.milvus-operator.svc.cluster.local
  issuerRef:
    kind: Issuer
    name: milvus-operator-selfsigned-issuer
  secretName: milvus-operator-webhook-cert

@Kellen-X
Copy link
Author

Kellen-X commented Feb 3, 2023

@Kellen-X由于超时,milvus-operator 似乎不能在您的 kubernetes 中安装 cert-manager。

您可以尝试通过以下yaml文件应用来修改它kubectl

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: milvus-operator-selfsigned-issuer
  namespace: milvus-operator
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: milvus-operator-serving-cert
  namespace: milvus-operator
spec:
  dnsNames:
  - milvus-operator-webhook-service.milvus-operator.svc
  - milvus-operator-webhook-service.milvus-operator.svc.cluster.local
  issuerRef:
    kind: Issuer
    name: milvus-operator-selfsigned-issuer
  secretName: milvus-operator-webhook-cert

@haorenfsa
error
Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": context deadline exceeded

I use Alibaba Cloud products: "Alibaba Cloud Container Service for Kubernetes" (ack pro)

image

@haorenfsa
Copy link
Contributor

@Kellen-X Try deleting the old Certificate with kubectl delete Certificate milvus-operator-serving-cert , and kubectl apply again with additional option --timeout=60s?
If still not works, could please provide the log of the cert-manager pod?

@Kellen-X
Copy link
Author

Kellen-X commented Feb 5, 2023

@Kellen-X 尝试删除旧证书kubectl delete Certificate milvus-operator-serving-cert ,然后kubectl apply再使用其他选项--timeout=60s? 如果还是不行,能否提供一下cert-manager pod的日志?

@haorenfsa

kellen@kellendeMacBook-Pro milvus % kubectl get pods -n cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-86b6c9bf4b-ccpfz 1/1 Running 0 14m
cert-manager-cainjector-7d4868cf69-kqhz6 1/1 Running 0 14m
cert-manager-webhook-6cb6fbfc58-glzfq 1/1 Running 0 14m
kellen@kellendeMacBook-Pro milvus %
kellen@kellendeMacBook-Pro milvus %
kellen@kellendeMacBook-Pro milvus % kubectl logs cert-manager-86b6c9bf4b-ccpfz -n cert-manager
I0205 07:50:07.615283 1 start.go:75] cert-manager "msg"="starting controller" "git-commit"="b26bd256f124d480fcb198e1464854f41b0d0d2c" "version"="v1.5.3"
W0205 07:50:07.615338 1 client_config.go:615] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0205 07:50:07.618073 1 controller.go:268] cert-manager/controller/build-context "msg"="configured acme dns01 nameservers" "nameservers"=["192.168.0.10:53"]
I0205 07:50:07.620370 1 controller.go:85] cert-manager/controller "msg"="enabled controllers: [certificaterequests-approver certificaterequests-issuer-acme certificaterequests-issuer-ca certificaterequests-issuer-selfsigned certificaterequests-issuer-vault certificaterequests-issuer-venafi certificates-issuing certificates-key-manager certificates-metrics certificates-readiness certificates-request-manager certificates-revision-manager certificates-trigger challenges clusterissuers ingress-shim issuers orders]"
I0205 07:50:07.620523 1 controller.go:115] cert-manager/controller "msg"="starting leader election"
I0205 07:50:07.621548 1 leaderelection.go:243] attempting to acquire leader lease kube-system/cert-manager-controller...
I0205 07:50:07.621909 1 controller.go:105] cert-manager/controller "msg"="starting metrics server" "address"={"IP":"::","Port":9402,"Zone":""}
I0205 07:51:20.322941 1 leaderelection.go:253] successfully acquired lease kube-system/cert-manager-controller
I0205 07:51:20.427400 1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="challenges"
I0205 07:51:20.427401 1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-approver"
I0205 07:51:20.427437 1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-vault"
I0205 07:51:20.427509 1 controller.go:163] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-vault"
I0205 07:51:20.427588 1 controller.go:163] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-ca"
I0205 07:51:20.427599 1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="certificates-key-manager"
I0205 07:51:20.427606 1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-venafi"
I0205 07:51:20.427627 1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="certificates-readiness"
I0205 07:51:20.427648 1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="ingress-shim"
I0205 07:51:20.427660 1 controller.go:163] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-venafi"
I0205 07:51:20.427711 1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="issuers"
I0205 07:51:20.427718 1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="certificates-issuing"
I0205 07:51:20.427743 1 controller.go:163] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="gateway-shim"
I0205 07:51:20.427855 1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="orders"
I0205 07:51:20.427958 1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-acme"
I0205 07:51:20.427964 1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-ca"
I0205 07:51:20.427982 1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-selfsigned"
I0205 07:51:20.427992 1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="certificates-metrics"
I0205 07:51:20.428015 1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="certificates-request-manager"
I0205 07:51:20.428089 1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="certificates-revision-manager"
I0205 07:51:20.428381 1 controller.go:163] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-acme"
I0205 07:51:20.428418 1 controller.go:163] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-selfsigned"
I0205 07:51:20.428440 1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="certificates-trigger"
I0205 07:51:20.429016 1 controller.go:186] cert-manager/controller "msg"="starting controller" "controller"="clusterissuers"
kellen@kellendeMacBook-Pro milvus %
kellen@kellendeMacBook-Pro milvus %
kellen@kellendeMacBook-Pro milvus % kubectl logs cert-manager-cainjector-7d4868cf69-kqhz6 -n cert-manager
I0205 07:50:07.620420 1 start.go:107] "starting" version="v1.5.3" revision="b26bd256f124d480fcb198e1464854f41b0d0d2c"
I0205 07:50:08.671504 1 request.go:668] Waited for 1.019768485s due to client-side throttling, not priority and fairness, request: GET:https://192.168.0.1:443/apis/cert-manager.io/v1alpha2?timeout=32s
I0205 07:50:08.727088 1 leaderelection.go:243] attempting to acquire leader lease kube-system/cert-manager-cainjector-leader-election...
I0205 07:50:24.917381 1 leaderelection.go:253] successfully acquired lease kube-system/cert-manager-cainjector-leader-election
I0205 07:50:24.917560 1 recorder.go:104] cert-manager/controller-runtime/manager/events "msg"="Normal" "message"="cert-manager-cainjector-7d4868cf69-kqhz6_4183719a-9ab1-471a-ba3c-a7766a887fdd became leader" "object"={"kind":"ConfigMap","namespace":"kube-system","name":"cert-manager-cainjector-leader-election","uid":"6f511cd6-2fda-4398-9866-ae17d144274d","apiVersion":"v1","resourceVersion":"1447767"} "reason"="LeaderElection"
I0205 07:50:24.917589 1 recorder.go:104] cert-manager/controller-runtime/manager/events "msg"="Normal" "message"="cert-manager-cainjector-7d4868cf69-kqhz6_4183719a-9ab1-471a-ba3c-a7766a887fdd became leader" "object"={"kind":"Lease","namespace":"kube-system","name":"cert-manager-cainjector-leader-election","uid":"e7218a92-5ed5-4216-821b-32ddcb54bff9","apiVersion":"coordination.k8s.io/v1","resourceVersion":"1447768"} "reason"="LeaderElection"
I0205 07:50:25.118668 1 controller.go:165] cert-manager/certificate/customresourcedefinition/controller/controller-for-certificate-customresourcedefinition "msg"="Starting EventSource" "source"={}
I0205 07:50:25.118692 1 controller.go:165] cert-manager/certificate/customresourcedefinition/controller/controller-for-certificate-customresourcedefinition "msg"="Starting EventSource" "source"={}
I0205 07:50:25.118702 1 controller.go:165] cert-manager/certificate/customresourcedefinition/controller/controller-for-certificate-customresourcedefinition "msg"="Starting EventSource" "source"={}
I0205 07:50:25.118714 1 controller.go:173] cert-manager/certificate/customresourcedefinition/controller/controller-for-certificate-customresourcedefinition "msg"="Starting Controller"
I0205 07:50:25.118880 1 controller.go:165] cert-manager/secret/customresourcedefinition/controller/controller-for-secret-customresourcedefinition "msg"="Starting EventSource" "source"={}
I0205 07:50:25.118891 1 controller.go:165] cert-manager/secret/customresourcedefinition/controller/controller-for-secret-customresourcedefinition "msg"="Starting EventSource" "source"={}
I0205 07:50:25.118898 1 controller.go:173] cert-manager/secret/customresourcedefinition/controller/controller-for-secret-customresourcedefinition "msg"="Starting Controller"
I0205 07:50:25.119156 1 controller.go:165] cert-manager/certificate/mutatingwebhookconfiguration/controller/controller-for-certificate-mutatingwebhookconfiguration "msg"="Starting EventSource" "source"={}
I0205 07:50:25.119176 1 controller.go:165] cert-manager/certificate/mutatingwebhookconfiguration/controller/controller-for-certificate-mutatingwebhookconfiguration "msg"="Starting EventSource" "source"={}
I0205 07:50:25.119184 1 controller.go:165] cert-manager/certificate/mutatingwebhookconfiguration/controller/controller-for-certificate-mutatingwebhookconfiguration "msg"="Starting EventSource" "source"={}
I0205 07:50:25.119193 1 controller.go:173] cert-manager/certificate/mutatingwebhookconfiguration/controller/controller-for-certificate-mutatingwebhookconfiguration "msg"="Starting Controller"
I0205 07:50:25.119347 1 controller.go:165] cert-manager/secret/mutatingwebhookconfiguration/controller/controller-for-secret-mutatingwebhookconfiguration "msg"="Starting EventSource" "source"={}
I0205 07:50:25.119364 1 controller.go:165] cert-manager/secret/mutatingwebhookconfiguration/controller/controller-for-secret-mutatingwebhookconfiguration "msg"="Starting EventSource" "source"={}
I0205 07:50:25.119372 1 controller.go:173] cert-manager/secret/mutatingwebhookconfiguration/controller/controller-for-secret-mutatingwebhookconfiguration "msg"="Starting Controller"
I0205 07:50:25.119396 1 controller.go:165] cert-manager/secret/validatingwebhookconfiguration/controller/controller-for-secret-validatingwebhookconfiguration "msg"="Starting EventSource" "source"={}
I0205 07:50:25.119411 1 controller.go:165] cert-manager/secret/validatingwebhookconfiguration/controller/controller-for-secret-validatingwebhookconfiguration "msg"="Starting EventSource" "source"={}
I0205 07:50:25.119420 1 controller.go:173] cert-manager/secret/validatingwebhookconfiguration/controller/controller-for-secret-validatingwebhookconfiguration "msg"="Starting Controller"
I0205 07:50:25.119433 1 controller.go:165] cert-manager/secret/apiservice/controller/controller-for-secret-apiservice "msg"="Starting EventSource" "source"={}
I0205 07:50:25.119453 1 controller.go:165] cert-manager/secret/apiservice/controller/controller-for-secret-apiservice "msg"="Starting EventSource" "source"={}
I0205 07:50:25.119461 1 controller.go:173] cert-manager/secret/apiservice/controller/controller-for-secret-apiservice "msg"="Starting Controller"
I0205 07:50:25.119481 1 controller.go:165] cert-manager/certificate/validatingwebhookconfiguration/controller/controller-for-certificate-validatingwebhookconfiguration "msg"="Starting EventSource" "source"={}
I0205 07:50:25.119514 1 controller.go:165] cert-manager/certificate/validatingwebhookconfiguration/controller/controller-for-certificate-validatingwebhookconfiguration "msg"="Starting EventSource" "source"={}
I0205 07:50:25.119524 1 controller.go:165] cert-manager/certificate/validatingwebhookconfiguration/controller/controller-for-certificate-validatingwebhookconfiguration "msg"="Starting EventSource" "source"={}
I0205 07:50:25.119532 1 controller.go:173] cert-manager/certificate/validatingwebhookconfiguration/controller/controller-for-certificate-validatingwebhookconfiguration "msg"="Starting Controller"
I0205 07:50:25.119533 1 controller.go:165] cert-manager/certificate/apiservice/controller/controller-for-certificate-apiservice "msg"="Starting EventSource" "source"={}
I0205 07:50:25.119546 1 controller.go:165] cert-manager/certificate/apiservice/controller/controller-for-certificate-apiservice "msg"="Starting EventSource" "source"={}
I0205 07:50:25.119560 1 controller.go:165] cert-manager/certificate/apiservice/controller/controller-for-certificate-apiservice "msg"="Starting EventSource" "source"={}
I0205 07:50:25.119569 1 controller.go:173] cert-manager/certificate/apiservice/controller/controller-for-certificate-apiservice "msg"="Starting Controller"
I0205 07:50:25.220055 1 controller.go:207] cert-manager/secret/mutatingwebhookconfiguration/controller/controller-for-secret-mutatingwebhookconfiguration "msg"="Starting workers" "worker count"=1
I0205 07:50:25.220080 1 controller.go:207] cert-manager/secret/validatingwebhookconfiguration/controller/controller-for-secret-validatingwebhookconfiguration "msg"="Starting workers" "worker count"=1
I0205 07:50:25.220121 1 controller.go:207] cert-manager/secret/apiservice/controller/controller-for-secret-apiservice "msg"="Starting workers" "worker count"=1
I0205 07:50:25.220253 1 controller.go:207] cert-manager/secret/customresourcedefinition/controller/controller-for-secret-customresourcedefinition "msg"="Starting workers" "worker count"=1
I0205 07:50:25.230117 1 controller.go:178] cert-manager/secret/validatingwebhookconfiguration/generic-inject-reconciler "msg"="updated object" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "resource_version"="v1"
I0205 07:50:25.232520 1 controller.go:178] cert-manager/secret/mutatingwebhookconfiguration/generic-inject-reconciler "msg"="updated object" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "resource_version"="v1"
E0205 07:50:25.236467 1 controller.go:175] cert-manager/secret/validatingwebhookconfiguration/generic-inject-reconciler "msg"="unable to update target object with new CA data" "error"="Operation cannot be fulfilled on validatingwebhookconfigurations.admissionregistration.k8s.io "cert-manager-webhook": the object has been modified; please apply your changes to the latest version and try again" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "resource_version"="v1"
E0205 07:50:25.236515 1 controller.go:304] cert-manager/secret/validatingwebhookconfiguration/controller/controller-for-secret-validatingwebhookconfiguration "msg"="Reconciler error" "error"="Operation cannot be fulfilled on validatingwebhookconfigurations.admissionregistration.k8s.io "cert-manager-webhook": the object has been modified; please apply your changes to the latest version and try again" "name"="cert-manager-webhook" "namespace"=""
I0205 07:50:25.237264 1 controller.go:178] cert-manager/secret/mutatingwebhookconfiguration/generic-inject-reconciler "msg"="updated object" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "resource_version"="v1"
I0205 07:50:25.241063 1 controller.go:178] cert-manager/secret/validatingwebhookconfiguration/generic-inject-reconciler "msg"="updated object" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "resource_version"="v1"
I0205 07:50:25.248616 1 controller.go:178] cert-manager/secret/validatingwebhookconfiguration/generic-inject-reconciler "msg"="updated object" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="cert-manager-webhook" "resource_namespace"="" "resource_version"="v1"
I0205 07:50:25.252401 1 controller.go:178] cert-manager/secret/customresourcedefinition/generic-inject-reconciler "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="orders.acme.cert-manager.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:50:25.455560 1 controller.go:178] cert-manager/secret/customresourcedefinition/generic-inject-reconciler "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="clusterissuers.cert-manager.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:50:25.486755 1 controller.go:178] cert-manager/secret/customresourcedefinition/generic-inject-reconciler "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="certificaterequests.cert-manager.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:50:25.675567 1 controller.go:178] cert-manager/secret/customresourcedefinition/generic-inject-reconciler "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="issuers.cert-manager.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:50:25.820391 1 controller.go:178] cert-manager/secret/customresourcedefinition/generic-inject-reconciler "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="challenges.acme.cert-manager.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:50:25.864675 1 controller.go:178] cert-manager/secret/customresourcedefinition/generic-inject-reconciler "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="certificates.cert-manager.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:50:25.888551 1 controller.go:178] cert-manager/secret/customresourcedefinition/generic-inject-reconciler "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="orders.acme.cert-manager.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:50:26.080895 1 controller.go:178] cert-manager/secret/customresourcedefinition/generic-inject-reconciler "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="clusterissuers.cert-manager.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:50:26.102405 1 controller.go:178] cert-manager/secret/customresourcedefinition/generic-inject-reconciler "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="certificaterequests.cert-manager.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:50:26.283299 1 controller.go:178] cert-manager/secret/customresourcedefinition/generic-inject-reconciler "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="issuers.cert-manager.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:50:26.408150 1 controller.go:178] cert-manager/secret/customresourcedefinition/generic-inject-reconciler "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="challenges.acme.cert-manager.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:50:26.450527 1 controller.go:178] cert-manager/secret/customresourcedefinition/generic-inject-reconciler "msg"="updated object" "resource_kind"="CustomResourceDefinition" "resource_name"="certificates.cert-manager.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:50:27.320097 1 controller.go:207] cert-manager/certificate/customresourcedefinition/controller/controller-for-certificate-customresourcedefinition "msg"="Starting workers" "worker count"=1
I0205 07:50:27.320116 1 controller.go:207] cert-manager/certificate/mutatingwebhookconfiguration/controller/controller-for-certificate-mutatingwebhookconfiguration "msg"="Starting workers" "worker count"=1
I0205 07:50:27.320204 1 controller.go:207] cert-manager/certificate/validatingwebhookconfiguration/controller/controller-for-certificate-validatingwebhookconfiguration "msg"="Starting workers" "worker count"=1
E0205 07:50:27.320287 1 sources.go:114] cert-manager/certificate/mutatingwebhookconfiguration/generic-inject-reconciler "msg"="unable to fetch associated certificate" "error"="Certificate.cert-manager.io "milvus-operator-serving-cert" not found" "certificate"={"Namespace":"milvus-operator","Name":"milvus-operator-serving-cert"} "resource_kind"="MutatingWebhookConfiguration" "resource_name"="milvus-operator-mutating-webhook-configuration" "resource_namespace"="" "resource_version"="v1"
I0205 07:50:27.320319 1 controller.go:166] cert-manager/certificate/mutatingwebhookconfiguration/generic-inject-reconciler "msg"="could not find any ca data in data source for target" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="milvus-operator-mutating-webhook-configuration" "resource_namespace"="" "resource_version"="v1"
I0205 07:50:27.320287 1 controller.go:207] cert-manager/certificate/apiservice/controller/controller-for-certificate-apiservice "msg"="Starting workers" "worker count"=1
E0205 07:50:27.320353 1 sources.go:114] cert-manager/certificate/validatingwebhookconfiguration/generic-inject-reconciler "msg"="unable to fetch associated certificate" "error"="Certificate.cert-manager.io "milvus-operator-serving-cert" not found" "certificate"={"Namespace":"milvus-operator","Name":"milvus-operator-serving-cert"} "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="milvus-operator-validating-webhook-configuration" "resource_namespace"="" "resource_version"="v1"
E0205 07:50:27.320375 1 sources.go:114] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="unable to fetch associated certificate" "error"="Certificate.cert-manager.io "milvus-operator-serving-cert" not found" "certificate"={"Namespace":"milvus-operator","Name":"milvus-operator-serving-cert"} "resource_kind"="CustomResourceDefinition" "resource_name"="milvusupgrades.milvus.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:50:27.320389 1 controller.go:166] cert-manager/certificate/validatingwebhookconfiguration/generic-inject-reconciler "msg"="could not find any ca data in data source for target" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="milvus-operator-validating-webhook-configuration" "resource_namespace"="" "resource_version"="v1"
I0205 07:50:27.320419 1 controller.go:166] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="could not find any ca data in data source for target" "resource_kind"="CustomResourceDefinition" "resource_name"="milvusupgrades.milvus.io" "resource_namespace"="" "resource_version"="v1"
E0205 07:50:27.323455 1 sources.go:114] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="unable to fetch associated certificate" "error"="Certificate.cert-manager.io "milvus-operator-serving-cert" not found" "certificate"={"Namespace":"milvus-operator","Name":"milvus-operator-serving-cert"} "resource_kind"="CustomResourceDefinition" "resource_name"="milvuses.milvus.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:50:27.323512 1 controller.go:166] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="could not find any ca data in data source for target" "resource_kind"="CustomResourceDefinition" "resource_name"="milvuses.milvus.io" "resource_namespace"="" "resource_version"="v1"
E0205 07:50:27.327862 1 sources.go:114] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="unable to fetch associated certificate" "error"="Certificate.cert-manager.io "milvus-operator-serving-cert" not found" "certificate"={"Namespace":"milvus-operator","Name":"milvus-operator-serving-cert"} "resource_kind"="CustomResourceDefinition" "resource_name"="milvusclusters.milvus.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:50:27.327891 1 controller.go:166] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="could not find any ca data in data source for target" "resource_kind"="CustomResourceDefinition" "resource_name"="milvusclusters.milvus.io" "resource_namespace"="" "resource_version"="v1"
E0205 07:57:10.110212 1 sources.go:114] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="unable to fetch associated certificate" "error"="Certificate.cert-manager.io "milvus-operator-serving-cert" not found" "certificate"={"Namespace":"milvus-operator","Name":"milvus-operator-serving-cert"} "resource_kind"="CustomResourceDefinition" "resource_name"="milvusclusters.milvus.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:57:10.110244 1 controller.go:166] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="could not find any ca data in data source for target" "resource_kind"="CustomResourceDefinition" "resource_name"="milvusclusters.milvus.io" "resource_namespace"="" "resource_version"="v1"
E0205 07:57:10.166642 1 sources.go:114] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="unable to fetch associated certificate" "error"="Certificate.cert-manager.io "milvus-operator-serving-cert" not found" "certificate"={"Namespace":"milvus-operator","Name":"milvus-operator-serving-cert"} "resource_kind"="CustomResourceDefinition" "resource_name"="milvusclusters.milvus.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:57:10.166672 1 controller.go:166] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="could not find any ca data in data source for target" "resource_kind"="CustomResourceDefinition" "resource_name"="milvusclusters.milvus.io" "resource_namespace"="" "resource_version"="v1"
E0205 07:57:10.360715 1 sources.go:114] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="unable to fetch associated certificate" "error"="Certificate.cert-manager.io "milvus-operator-serving-cert" not found" "certificate"={"Namespace":"milvus-operator","Name":"milvus-operator-serving-cert"} "resource_kind"="CustomResourceDefinition" "resource_name"="milvusclusters.milvus.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:57:10.360747 1 controller.go:166] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="could not find any ca data in data source for target" "resource_kind"="CustomResourceDefinition" "resource_name"="milvusclusters.milvus.io" "resource_namespace"="" "resource_version"="v1"
E0205 07:57:10.754860 1 sources.go:114] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="unable to fetch associated certificate" "error"="Certificate.cert-manager.io "milvus-operator-serving-cert" not found" "certificate"={"Namespace":"milvus-operator","Name":"milvus-operator-serving-cert"} "resource_kind"="CustomResourceDefinition" "resource_name"="milvuses.milvus.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:57:10.754894 1 controller.go:166] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="could not find any ca data in data source for target" "resource_kind"="CustomResourceDefinition" "resource_name"="milvuses.milvus.io" "resource_namespace"="" "resource_version"="v1"
E0205 07:57:10.808981 1 sources.go:114] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="unable to fetch associated certificate" "error"="Certificate.cert-manager.io "milvus-operator-serving-cert" not found" "certificate"={"Namespace":"milvus-operator","Name":"milvus-operator-serving-cert"} "resource_kind"="CustomResourceDefinition" "resource_name"="milvusupgrades.milvus.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:57:10.809007 1 controller.go:166] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="could not find any ca data in data source for target" "resource_kind"="CustomResourceDefinition" "resource_name"="milvusupgrades.milvus.io" "resource_namespace"="" "resource_version"="v1"
E0205 07:57:10.878829 1 sources.go:114] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="unable to fetch associated certificate" "error"="Certificate.cert-manager.io "milvus-operator-serving-cert" not found" "certificate"={"Namespace":"milvus-operator","Name":"milvus-operator-serving-cert"} "resource_kind"="CustomResourceDefinition" "resource_name"="milvuses.milvus.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:57:10.878866 1 controller.go:166] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="could not find any ca data in data source for target" "resource_kind"="CustomResourceDefinition" "resource_name"="milvuses.milvus.io" "resource_namespace"="" "resource_version"="v1"
E0205 07:57:10.881963 1 sources.go:114] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="unable to fetch associated certificate" "error"="Certificate.cert-manager.io "milvus-operator-serving-cert" not found" "certificate"={"Namespace":"milvus-operator","Name":"milvus-operator-serving-cert"} "resource_kind"="CustomResourceDefinition" "resource_name"="milvusupgrades.milvus.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:57:10.881987 1 controller.go:166] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="could not find any ca data in data source for target" "resource_kind"="CustomResourceDefinition" "resource_name"="milvusupgrades.milvus.io" "resource_namespace"="" "resource_version"="v1"
E0205 07:57:11.021042 1 sources.go:114] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="unable to fetch associated certificate" "error"="Certificate.cert-manager.io "milvus-operator-serving-cert" not found" "certificate"={"Namespace":"milvus-operator","Name":"milvus-operator-serving-cert"} "resource_kind"="CustomResourceDefinition" "resource_name"="milvuses.milvus.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:57:11.021080 1 controller.go:166] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="could not find any ca data in data source for target" "resource_kind"="CustomResourceDefinition" "resource_name"="milvuses.milvus.io" "resource_namespace"="" "resource_version"="v1"
E0205 07:57:11.021162 1 sources.go:114] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="unable to fetch associated certificate" "error"="Certificate.cert-manager.io "milvus-operator-serving-cert" not found" "certificate"={"Namespace":"milvus-operator","Name":"milvus-operator-serving-cert"} "resource_kind"="CustomResourceDefinition" "resource_name"="milvusupgrades.milvus.io" "resource_namespace"="" "resource_version"="v1"
I0205 07:57:11.021184 1 controller.go:166] cert-manager/certificate/customresourcedefinition/generic-inject-reconciler "msg"="could not find any ca data in data source for target" "resource_kind"="CustomResourceDefinition" "resource_name"="milvusupgrades.milvus.io" "resource_namespace"="" "resource_version"="v1"
E0205 07:57:11.340445 1 sources.go:114] cert-manager/certificate/mutatingwebhookconfiguration/generic-inject-reconciler "msg"="unable to fetch associated certificate" "error"="Certificate.cert-manager.io "milvus-operator-serving-cert" not found" "certificate"={"Namespace":"milvus-operator","Name":"milvus-operator-serving-cert"} "resource_kind"="MutatingWebhookConfiguration" "resource_name"="milvus-operator-mutating-webhook-configuration" "resource_namespace"="" "resource_version"="v1"
I0205 07:57:11.340476 1 controller.go:166] cert-manager/certificate/mutatingwebhookconfiguration/generic-inject-reconciler "msg"="could not find any ca data in data source for target" "resource_kind"="MutatingWebhookConfiguration" "resource_name"="milvus-operator-mutating-webhook-configuration" "resource_namespace"="" "resource_version"="v1"
E0205 07:57:11.373631 1 sources.go:114] cert-manager/certificate/validatingwebhookconfiguration/generic-inject-reconciler "msg"="unable to fetch associated certificate" "error"="Certificate.cert-manager.io "milvus-operator-serving-cert" not found" "certificate"={"Namespace":"milvus-operator","Name":"milvus-operator-serving-cert"} "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="milvus-operator-validating-webhook-configuration" "resource_namespace"="" "resource_version"="v1"
I0205 07:57:11.373661 1 controller.go:166] cert-manager/certificate/validatingwebhookconfiguration/generic-inject-reconciler "msg"="could not find any ca data in data source for target" "resource_kind"="ValidatingWebhookConfiguration" "resource_name"="milvus-operator-validating-webhook-configuration" "resource_namespace"="" "resource_version"="v1"
kellen@kellendeMacBook-Pro milvus %
kellen@kellendeMacBook-Pro milvus %
kellen@kellendeMacBook-Pro milvus %
kellen@kellendeMacBook-Pro milvus % kubectl logs cert-manager-webhook-6cb6fbfc58-glzfq -n cert-manager
W0205 07:50:07.931074 1 client_config.go:615] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
W0205 07:50:07.937283 1 client_config.go:615] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0205 07:50:07.937415 1 webhook.go:70] cert-manager/webhook "msg"="using dynamic certificate generating using CA stored in Secret resource" "secret_name"="cert-manager-webhook-ca" "secret_namespace"="cert-manager"
I0205 07:50:07.937578 1 server.go:138] cert-manager/webhook "msg"="listening for insecure healthz connections" "address"=":6080"
I0205 07:50:07.937619 1 server.go:169] cert-manager/webhook "msg"="listening for secure connections" "address"=":10250"
I0205 07:50:07.937637 1 server.go:201] cert-manager/webhook "msg"="registered pprof handlers"
I0205 07:50:09.013033 1 dynamic_source.go:272] cert-manager/webhook "msg"="Updated serving TLS certificate"
kellen@kellendeMacBook-Pro milvus %
kellen@kellendeMacBook-Pro milvus %
kellen@kellendeMacBook-Pro milvus %
kellen@kellendeMacBook-Pro milvus %
kellen@kellendeMacBook-Pro milvus %
kellen@kellendeMacBook-Pro milvus % kubectl get pods -n milvus-operator
NAME READY STATUS RESTARTS AGE
milvus-operator-5c896c4c6d-r8v2j 0/1 ContainerCreating 0 8m42s
kellen@kellendeMacBook-Pro milvus %
kellen@kellendeMacBook-Pro milvus %
kellen@kellendeMacBook-Pro milvus % kubectl logs milvus-operator-5c896c4c6d-r8v2j -n milvus-operator
Error from server (BadRequest): container "manager" in pod "milvus-operator-5c896c4c6d-r8v2j" is waiting to start: ContainerCreating
kellen@kellendeMacBook-Pro milvus %

kellen@kellendeMacBook-Pro milvus % kubectl apply -f https://raw.githubusercontent.com/milvus-io/milvus-operator/main/deploy/manifests/deployment.yaml --timeout=60s
namespace/milvus-operator created
serviceaccount/milvus-operator created
serviceaccount/milvus-operator-checker created
customresourcedefinition.apiextensions.k8s.io/milvusclusters.milvus.io created
customresourcedefinition.apiextensions.k8s.io/milvuses.milvus.io created
customresourcedefinition.apiextensions.k8s.io/milvusupgrades.milvus.io created
clusterrole.rbac.authorization.k8s.io/milvus-operator-checker-role created
clusterrole.rbac.authorization.k8s.io/milvus-operator-manager-role created
clusterrolebinding.rbac.authorization.k8s.io/milvus-operator-checker-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/milvus-operator-manager-rolebinding created
role.rbac.authorization.k8s.io/milvus-operator-leader-election-role created
rolebinding.rbac.authorization.k8s.io/milvus-operator-leader-election-rolebinding created
service/milvus-operator-metrics-service created
service/milvus-operator-webhook-service created
deployment.apps/milvus-operator created
job.batch/milvus-operator-checker created
mutatingwebhookconfiguration.admissionregistration.k8s.io/milvus-operator-mutating-webhook-configuration created
validatingwebhookconfiguration.admissionregistration.k8s.io/milvus-operator-validating-webhook-configuration created
kellen@kellendeMacBook-Pro milvus % kubectl get pods -n milvus-operator
NAME READY STATUS RESTARTS AGE
milvus-operator-5c896c4c6d-r8v2j 0/1 ContainerCreating 0 61s
milvus-operator-checker-mz9rl 1/1 Running 2 (27s ago) 61s
kellen@kellendeMacBook-Pro milvus %
kellen@kellendeMacBook-Pro milvus %
kellen@kellendeMacBook-Pro milvus %
kellen@kellendeMacBook-Pro milvus % kubectl logs -f milvus-operator-checker-mz9rl -n milvus-operator
I0205 07:58:06.986006 1 request.go:668] Waited for 1.019273306s due to client-side throttling, not priority and fairness, request: GET:https://192.168.0.1:443/apis/policy/v1beta1?timeout=32s
2023/02/05 07:58:17 unable to install certificationfailed to create certificate: failed to create resource Certificate/milvus-operator-serving-cert: Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": context deadline exceeded

@haorenfsa
Copy link
Contributor

haorenfsa commented Feb 6, 2023

@Kellen-X I'm trying to reproduce it. I'll post the result here later.

@haorenfsa
Copy link
Contributor

haorenfsa commented Feb 6, 2023

@Kellen-X Plz try

kubectl apply -f cert.yaml --timeout=60s

The cert.yaml content:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: milvus-operator-selfsigned-issuer
  namespace: milvus-operator
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: milvus-operator-serving-cert
  namespace: milvus-operator
spec:
  dnsNames:
  - milvus-operator-webhook-service.milvus-operator.svc
  - milvus-operator-webhook-service.milvus-operator.svc.cluster.local
  issuerRef:
    kind: Issuer
    name: milvus-operator-selfsigned-issuer
  secretName: milvus-operator-webhook-cert

@Kellen-X
Copy link
Author

Kellen-X commented Feb 6, 2023

@Kellen-X Plz try

kubectl apply -f cert.yaml --timeout=60s

The cert.yaml content:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: milvus-operator-selfsigned-issuer
  namespace: milvus-operator
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: milvus-operator-serving-cert
  namespace: milvus-operator
spec:
  dnsNames:
  - milvus-operator-webhook-service.milvus-operator.svc
  - milvus-operator-webhook-service.milvus-operator.svc.cluster.local
  issuerRef:
    kind: Issuer
    name: milvus-operator-selfsigned-issuer
  secretName: milvus-operator-webhook-cert

@haorenfsa
image
image
image

@haorenfsa
Copy link
Contributor

Oh, cert-manager webhook has its own timeout config. Plz try this:

kubectl patch mutatingwebhookconfigurations cert-manager-webhook --type='json' -p '[{"op": "replace", "path": "/webhooks/0/timeoutSeconds", "value": 30}]'

# then apply again
kubectl apply -f cert.yaml --timeout=30s

@haorenfsa
Copy link
Contributor

If above still don't work, just delete it with kubectl delete mutatingwebhookconfigurations cert-manager-webhook, and try apply again

It's fine in a test environment. We don't create Certs manually, so we don't need the validation.

@yanliang567 yanliang567 modified the milestones: 2.2.3, 2.2.4 Feb 14, 2023
@Kellen-X
Copy link
Author

Kellen-X commented Feb 15, 2023

@haorenfsa

shell@Alicloud:~$ kubectl patch mutatingwebhookconfigurations cert-manager-webhook --type='json' -p '[{"op": "replace", "path": "/webhooks/0/timeoutSeconds", "value": 30}]'
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook patched
shell@Alicloud:~$ cat cert.yaml 
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: milvus-operator-selfsigned-issuer
  namespace: milvus-operator
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: milvus-operator-serving-cert
  namespace: milvus-operator
spec:
  dnsNames:
  - milvus-operator-webhook-service.milvus-operator.svc
  - milvus-operator-webhook-service.milvus-operator.svc.cluster.local
  issuerRef:
    kind: Issuer
    name: milvus-operator-selfsigned-issuer
  secretName: milvus-operator-webhook-cert
shell@Alicloud:~$ 
shell@Alicloud:~$ kubectl apply -f cert.yaml --timeout=30s
Error from server (InternalError): error when creating "cert.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=30s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Error from server (InternalError): error when creating "cert.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=30s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
shell@Alicloud:~$ 
shell@Alicloud:~$ kubectl delete mutatingwebhookconfigurations cert-manager-webhook
mutatingwebhookconfiguration.admissionregistration.k8s.io "cert-manager-webhook" deleted
shell@Alicloud:~$ kubectl apply -f cert.yaml --timeout=30s
Error from server (InternalError): error when creating "cert.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=10s": context deadline exceeded
Error from server (InternalError): error when creating "cert.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=10s": context deadline exceeded
shell@Alicloud:~$ 

@haorenfsa
Copy link
Contributor

plz kubectl delete validatingwebhookconfigurations cert-manager-webhook as well.

@Kellen-X
Copy link
Author

kubectl delete validatingwebhookconfigurations cert-manager-webhook也一样。

shell@Alicloud:~$ kubectl delete validatingwebhookconfigurations cert-manager-webhook
validatingwebhookconfiguration.admissionregistration.k8s.io "cert-manager-webhook" deleted
shell@Alicloud:~$ 
shell@Alicloud:~$ kubectl apply -f cert.yaml --timeout=60s
issuer.cert-manager.io/milvus-operator-selfsigned-issuer created
certificate.cert-manager.io/milvus-operator-serving-cert created
shell@Alicloud:~$ 
shell@Alicloud:~$ kubectl get pods -n cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-86b6c9bf4b-ccpfz              1/1     Running   0          9d
cert-manager-cainjector-7d4868cf69-kqhz6   1/1     Running   0          9d
cert-manager-webhook-6cb6fbfc58-glzfq      1/1     Running   0          9d
shell@Alicloud:~$ 
shell@Alicloud:~$ kubectl get pods -n milvus-operator
NAME                               READY   STATUS    RESTARTS   AGE
milvus-operator-5c896c4c6d-vl52p   1/1     Running   0          8d
shell@Alicloud:~$ 
shell@Alicloud:~$ kubectl apply -f https://raw.githubusercontent.com/milvus-io/milvus-operator/main/config/samples/milvus_cluster_default.yaml

Error from server (InternalError): error when creating "https://raw.githubusercontent.com/milvus-io/milvus-operator/main/config/samples/milvus_cluster_default.yaml": Internal error occurred: failed calling webhook "mmilvus.kb.io": failed to call webhook: Post "https://milvus-operator-webhook-service.milvus-operator.svc:443/mutate-milvus-io-v1beta1-milvus?timeout=10s": context deadline exceeded
shell@Alicloud:~$ 
shell@Alicloud:~$ kubectl get pods -n milvus-operator
NAME                               READY   STATUS    RESTARTS   AGE
milvus-operator-5c896c4c6d-vl52p   1/1     Running   0          8d
shell@Alicloud:~$ 
shell@Alicloud:~$ 
shell@Alicloud:~$ kubectl  logs milvus-operator-5c896c4c6d-vl52p -n milvus-operator
I0215 07:27:26.165885       1 request.go:668] Waited for 1.037849003s due to client-side throttling, not priority and fairness, request: GET:https://192.168.0.1:443/apis/scheduling.alibabacloud.com/v1alpha1?timeout=32s
2023-02-15T07:27:26.222Z        INFO    controller-runtime.metrics      metrics server is starting to listen    {"addr": ":8080"}
2023-02-15T07:27:26.222Z        INFO    controller-runtime.builder      Registering a mutating webhook  {"GVK": "milvus.io/v1beta1, Kind=Milvus", "path": "/mutate-milvus-io-v1beta1-milvus"}
2023-02-15T07:27:26.222Z        INFO    controller-runtime.webhook      registering webhook     {"path": "/mutate-milvus-io-v1beta1-milvus"}
2023-02-15T07:27:26.222Z        INFO    controller-runtime.builder      Registering a validating webhook        {"GVK": "milvus.io/v1beta1, Kind=Milvus", "path": "/validate-milvus-io-v1beta1-milvus"}
2023-02-15T07:27:26.222Z        INFO    controller-runtime.webhook      registering webhook     {"path": "/validate-milvus-io-v1beta1-milvus"}
2023-02-15T07:27:26.222Z        INFO    controller-runtime.webhook      registering webhook     {"path": "/convert"}
2023-02-15T07:27:26.222Z        INFO    controller-runtime.builder      conversion webhook enabled      {"object": {"metadata":{"creationTimestamp":null},"spec":{"components":{"paused":false,"disableMetric":false,"metricInterval":""},"dependencies":{"etcd":{"endpoints":null},"pulsar":{"endpoint":""},"kafka":{},"rocksmq":{"persistence":{"persistentVolumeClaim":{"spec":{"resources":{}}}}},"storage":{"type":"","secretRef":"","endpoint":""}},"config":null},"status":{"status":"","ingress":{"loadBalancer":{}},"replicas":{}}}}
2023-02-15T07:27:26.223Z        INFO    controller-runtime.builder      skip registering a mutating webhook, admission.Defaulter interface is not implemented   {"GVK": "milvus.io/v1alpha1, Kind=Milvus"}
2023-02-15T07:27:26.223Z        INFO    controller-runtime.builder      skip registering a validating webhook, admission.Validator interface is not implemented {"GVK": "milvus.io/v1alpha1, Kind=Milvus"}
2023-02-15T07:27:26.224Z        INFO    controller-runtime.builder      conversion webhook enabled      {"object": {"metadata":{"creationTimestamp":null},"spec":{"paused":false,"persistence":{"persistentVolumeClaim":{"spec":{"resources":{}}}},"dependencies":{"etcd":{"endpoints":null},"pulsar":{"endpoint":""},"kafka":{},"rocksmq":{"persistence":{"persistentVolumeClaim":{"spec":{"resources":{}}}}},"storage":{"type":"","secretRef":"","endpoint":""}},"config":null},"status":{"status":"","ingress":{"loadBalancer":{}},"replicas":{}}}}
2023-02-15T07:27:26.224Z        INFO    controller-runtime.builder      Registering a mutating webhook  {"GVK": "milvus.io/v1beta1, Kind=MilvusUpgrade", "path": "/mutate-milvus-io-v1beta1-milvusupgrade"}
2023-02-15T07:27:26.224Z        INFO    controller-runtime.webhook      registering webhook     {"path": "/mutate-milvus-io-v1beta1-milvusupgrade"}
2023-02-15T07:27:26.224Z        INFO    controller-runtime.builder      Registering a validating webhook        {"GVK": "milvus.io/v1beta1, Kind=MilvusUpgrade", "path": "/validate-milvus-io-v1beta1-milvusupgrade"}
2023-02-15T07:27:26.224Z        INFO    controller-runtime.webhook      registering webhook     {"path": "/validate-milvus-io-v1beta1-milvusupgrade"}
2023-02-15T07:27:26.224Z        INFO    setup   starting manager        {"version": "0.7.4", "milvus-helm version": "milvus-3.3.6"}
I0215 07:27:26.224321       1 leaderelection.go:243] attempting to acquire leader lease milvus-operator/71808ec5.milvus.io...
2023-02-15T07:27:26.224Z        INFO    controller-runtime.webhook.webhooks     starting webhook server
2023-02-15T07:27:26.224Z        INFO    controller-runtime.manager      starting metrics server {"path": "/metrics"}
2023-02-15T07:27:26.224Z        INFO    controller-runtime.certwatcher  Updated current TLS certificate
2023-02-15T07:27:26.224Z        INFO    controller-runtime.webhook      serving webhook server  {"host": "", "port": 9443}
2023-02-15T07:27:26.225Z        INFO    controller-runtime.certwatcher  Starting certificate watcher
I0215 07:27:26.244009       1 leaderelection.go:253] successfully acquired lease milvus-operator/71808ec5.milvus.io
2023-02-15T07:27:26.244Z        DEBUG   controller-runtime.manager.events       Normal  {"object": {"kind":"ConfigMap","namespace":"milvus-operator","name":"71808ec5.milvus.io","uid":"1d937d70-a377-48cb-81f3-761529c578d0","apiVersion":"v1","resourceVersion":"6549134"}, "reason": "LeaderElection", "message": "milvus-operator-5c896c4c6d-vl52p_fe054dbf-3792-4084-befb-b3373d878400 became leader"}
2023-02-15T07:27:26.244Z        DEBUG   controller-runtime.manager.events       Normal  {"object": {"kind":"Lease","namespace":"milvus-operator","name":"71808ec5.milvus.io","uid":"9cbad5ef-721f-4614-9cd3-f26b9f39b2f1","apiVersion":"coordination.k8s.io/v1","resourceVersion":"6549135"}, "reason": "LeaderElection", "message": "milvus-operator-5c896c4c6d-vl52p_fe054dbf-3792-4084-befb-b3373d878400 became leader"}
2023-02-15T07:27:26.244Z        INFO    controller-runtime.manager.controller.milvuscluster     Starting EventSource    {"reconciler group": "milvus.io", "reconciler kind": "MilvusCluster", "source": "kind source: /, Kind="}
2023-02-15T07:27:26.244Z        INFO    controller-runtime.manager.controller.milvuscluster     Starting Controller     {"reconciler group": "milvus.io", "reconciler kind": "MilvusCluster"}
2023-02-15T07:27:26.244Z        INFO    controller-runtime.manager.controller.milvus    Starting EventSource    {"reconciler group": "milvus.io", "reconciler kind": "Milvus", "source": "kind source: /, Kind="}
2023-02-15T07:27:26.244Z        INFO    controller-runtime.manager.controller.milvus    Starting Controller     {"reconciler group": "milvus.io", "reconciler kind": "Milvus"}
2023-02-15T07:27:26.244Z        INFO    controller-runtime.manager.controller.milvusupgrade     Starting EventSource    {"reconciler group": "milvus.io", "reconciler kind": "MilvusUpgrade", "source": "kind source: /, Kind="}
2023-02-15T07:27:26.244Z        INFO    controller-runtime.manager.controller.milvusupgrade     Starting EventSource    {"reconciler group": "milvus.io", "reconciler kind": "MilvusUpgrade", "source": "kind source: /, Kind="}
2023-02-15T07:27:26.244Z        INFO    controller-runtime.manager.controller.milvusupgrade     Starting Controller     {"reconciler group": "milvus.io", "reconciler kind": "MilvusUpgrade"}
2023-02-15T07:27:26.344Z        INFO    controller-runtime.manager.controller.milvus    Starting workers        {"reconciler group": "milvus.io", "reconciler kind": "Milvus", "worker count": 10}
2023-02-15T07:27:26.344Z        INFO    controller-runtime.manager.controller.milvuscluster     Starting workers        {"reconciler group": "milvus.io", "reconciler kind": "MilvusCluster", "worker count": 1}
2023-02-15T07:27:26.344Z        INFO    controller-runtime.manager.controller.milvusupgrade     Starting workers        {"reconciler group": "milvus.io", "reconciler kind": "MilvusUpgrade", "worker count": 1}
shell@Alicloud:~$ 

@stale
Copy link

stale bot commented Mar 17, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

@stale stale bot added the stale indicates no udpates for 30 days label Mar 17, 2023
@yanliang567 yanliang567 modified the milestones: 2.2.4, 2.2.5 Mar 21, 2023
@stale stale bot removed stale indicates no udpates for 30 days labels Mar 21, 2023
@yanliang567 yanliang567 modified the milestones: 2.2.5, 2.2.6 Apr 1, 2023
@yanliang567 yanliang567 modified the milestones: 2.2.6, 2.2.7 Apr 20, 2023
@yanliang567 yanliang567 modified the milestones: 2.2.7, 2.2.9 May 4, 2023
@yanliang567 yanliang567 modified the milestones: 2.2.9, 2.2.10 Jun 8, 2023
@yanliang567 yanliang567 modified the milestones: 2.2.10, 2.2.11 Jun 16, 2023
@yanliang567 yanliang567 modified the milestones: 2.2.11, 2.2.12 Jul 3, 2023
@stale
Copy link

stale bot commented Aug 2, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

@stale stale bot added the stale indicates no udpates for 30 days label Aug 2, 2023
@yanliang567 yanliang567 modified the milestones: 2.2.12, 2.2.13 Aug 4, 2023
@stale stale bot removed stale indicates no udpates for 30 days labels Aug 4, 2023
@stale
Copy link

stale bot commented Sep 3, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Issues or changes related a bug stale indicates no udpates for 30 days triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Development

No branches or pull requests

4 participants