[Bug]: MountVolume.SetUp failed for volume "cert" : secret "milvus-operator-webhook-cert" not found

[gaoyuan5251]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.3.1
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): CentOS 7
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

https://milvus.io/docs/v2.3.x/install_standalone-operator.md
参考该方式进行mivlus standalone安装

Expected Behavior

希望可以帮忙部署成功

Steps To Reproduce

1. kubectl apply -f cert-manager.yaml
2. kubectl apply -f milvus-operator-deployment-0.9.16.yaml

Milvus Log

Events:
Type Reason Age From Message

---

Normal Scheduled 2m12s default-scheduler Successfully assigned milvus-operator/milvus-operator-6f578f564f-hnv7l to aly-hn1-inner-risk-graph-service-svc-stg-003
Warning FailedMount 9s kubelet Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[cert kube-api-access-778nv]: timed out waiting for the condition
Warning FailedMount 4s (x9 over 2m12s) kubelet MountVolume.SetUp failed for volume "cert" : secret "milvus-operator-webhook-cert" not found

Anything else?

kubectl get secrets -A
NAMESPACE NAME TYPE DATA AGE
cert-manager cert-manager-cainjector-token-w7zs5 kubernetes.io/service-account-token 3 25m
cert-manager cert-manager-token-sh7nd kubernetes.io/service-account-token 3 25m
cert-manager cert-manager-webhook-ca Opaque 3 25m
cert-manager cert-manager-webhook-token-nwmvv kubernetes.io/service-account-token 3 25m
cert-manager default-token-ttxdp kubernetes.io/service-account-token 3 25m
default default-token-rh4kp kubernetes.io/service-account-token 3 44h
default nfs-client-provisioner-token-5mm84 kubernetes.io/service-account-token 3 28h
hll-etcd default-token-gpq5g kubernetes.io/service-account-token 3 4h15m
hll-minio default-token-8xjs2 kubernetes.io/service-account-token 3 21h
hll-minio hll-milvus-env-configuration Opaque 1 3h57m
hll-minio hll-milvus-sa-token-xm4cp kubernetes.io/service-account-token 3 21h
hll-minio hll-milvus-secret Opaque 2 3h57m
hll-minio hll-milvus-user-0 Opaque 2 3h57m
ingress-nginx default-token-wjml9 kubernetes.io/service-account-token 3 21h
ingress-nginx ingress-nginx-admission Opaque 3 21h
ingress-nginx ingress-nginx-admission-token-ghpz9 kubernetes.io/service-account-token 3 21h
ingress-nginx ingress-nginx-token-kk9w4 kubernetes.io/service-account-token 3 21h
kube-node-lease default-token-x95gd kubernetes.io/service-account-token 3 44h
kube-public default-token-fckcd kubernetes.io/service-account-token 3 44h
kube-system attachdetach-controller-token-6fjt5 kubernetes.io/service-account-token 3 44h
kube-system bootstrap-signer-token-pk7hp kubernetes.io/service-account-token 3 44h
kube-system calico-kube-controllers-token-z79bp kubernetes.io/service-account-token 3 44h
kube-system calico-node-token-rnrw5 kubernetes.io/service-account-token 3 44h
kube-system certificate-controller-token-8pp86 kubernetes.io/service-account-token 3 44h
kube-system clusterrole-aggregation-controller-token-6k8ln kubernetes.io/service-account-token 3 44h
kube-system coredns-token-lq9js kubernetes.io/service-account-token 3 44h
kube-system cronjob-controller-token-hpc92 kubernetes.io/service-account-token 3 44h
kube-system daemon-set-controller-token-bdrf4 kubernetes.io/service-account-token 3 44h
kube-system default-token-rcxbl kubernetes.io/service-account-token 3 44h
kube-system deployment-controller-token-5d4zt kubernetes.io/service-account-token 3 44h
kube-system disruption-controller-token-rxd2g kubernetes.io/service-account-token 3 44h
kube-system endpoint-controller-token-dtp9c kubernetes.io/service-account-token 3 44h
kube-system endpointslice-controller-token-rh2j6 kubernetes.io/service-account-token 3 44h
kube-system endpointslicemirroring-controller-token-sfx2n kubernetes.io/service-account-token 3 44h
kube-system ephemeral-volume-controller-token-h4bzx kubernetes.io/service-account-token 3 44h
kube-system expand-controller-token-lqvll kubernetes.io/service-account-token 3 44h
kube-system generic-garbage-collector-token-wzhsc kubernetes.io/service-account-token 3 44h
kube-system horizontal-pod-autoscaler-token-qjjcm kubernetes.io/service-account-token 3 44h
kube-system job-controller-token-6jnqs kubernetes.io/service-account-token 3 44h
kube-system kube-proxy-token-dx9p4 kubernetes.io/service-account-token 3 44h
kube-system namespace-controller-token-v52lk kubernetes.io/service-account-token 3 44h
kube-system node-controller-token-pc7p2 kubernetes.io/service-account-token 3 44h
kube-system persistent-volume-binder-token-fzkpt kubernetes.io/service-account-token 3 44h
kube-system pod-garbage-collector-token-qmbjm kubernetes.io/service-account-token 3 44h
kube-system pv-protection-controller-token-nf59v kubernetes.io/service-account-token 3 44h
kube-system pvc-protection-controller-token-ms4nf kubernetes.io/service-account-token 3 44h
kube-system replicaset-controller-token-zl4jd kubernetes.io/service-account-token 3 44h
kube-system replication-controller-token-jrhxh kubernetes.io/service-account-token 3 44h
kube-system resourcequota-controller-token-d9js8 kubernetes.io/service-account-token 3 44h
kube-system root-ca-cert-publisher-token-j29ct kubernetes.io/service-account-token 3 44h
kube-system service-account-controller-token-grlvt kubernetes.io/service-account-token 3 44h
kube-system service-controller-token-snrcx kubernetes.io/service-account-token 3 44h
kube-system statefulset-controller-token-kwf9x kubernetes.io/service-account-token 3 44h
kube-system token-cleaner-token-n7rrt kubernetes.io/service-account-token 3 44h
kube-system ttl-after-finished-controller-token-b8kdc kubernetes.io/service-account-token 3 44h
kube-system ttl-controller-token-dbnf2 kubernetes.io/service-account-token 3 44h
milvus-operator default-token-7jrt8 kubernetes.io/service-account-token 3 21m
milvus-operator milvus-operator-checker-token-975dr kubernetes.io/service-account-token 3 21m
milvus-operator milvus-operator-token-tgnph kubernetes.io/service-account-token 3 21m
minio-operator console-sa-secret kubernetes.io/service-account-token 3 22h
minio-operator console-sa-token-kr9mv kubernetes.io/service-account-token 3 22h
minio-operator default-token-tkqgh kubernetes.io/service-account-token 3 22h
minio-operator minio-operator-token-24ds2 kubernetes.io/service-account-token 3 22h

从我提供的输出来看，"milvus-operator-webhook-cert" 这个 secret 在所有命名空间中都不存在。
kubectl describe po milvus-operator-6f578f564f-hnv7l -n milvus-operator
Events:
Type Reason Age From Message

---

Normal Scheduled 4m35s default-scheduler Successfully assigned milvus-operator/milvus-operator-6f578f564f-hnv7l to aly-hn1-inner-risk-graph-service-svc-stg-003
Warning FailedMount 2m32s kubelet Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[cert kube-api-access-778nv]: timed out waiting for the condition
Warning FailedMount 25s (x10 over 4m35s) kubelet MountVolume.SetUp failed for volume "cert" : secret "milvus-operator-webhook-cert" not found
Warning FailedMount 16s kubelet Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[kube-api-access-778nv cert]: timed out waiting for the condition

[zwd1208]

@haorenfsa can you help to take a look?

[haorenfsa]

Hi @gaoyuan5251, are you using Alicloud's ACK? please refer to this issue #21961.

follow the instructions from this comment #21961 (comment)

你好 @gaoyuan5251，你是在使用Alicloud的ACK吗？ 请参考#21961 这个issue . 跟着这条评论的说明来 #21961 (comment)

[haorenfsa]

TODO： add FAQ doc for this kind of issue.
/assign

[gaoyuan5251]

TODO： add FAQ doc for this kind of issue. /assign

Hi @gaoyuan5251, are you using Alicloud's ACK? please refer to this issue #21961.

follow the instructions from this comment #21961 (comment)

你好 @gaoyuan5251，你是在使用Alicloud的ACK吗？ 请参考#21961 这个issue . 跟着这条评论的说明来 #21961 (comment)

@haorenfsa operator 按照指示可以运行，但是milvus-standalone v2.3.1 仍然不能安装/。
kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
cert-manager cert-manager-6888d6b69b-vp7ss 1/1 Running 0 20h
cert-manager cert-manager-cainjector-76f7798c9-gd78m 1/1 Running 0 20h
cert-manager cert-manager-webhook-7d4b5d8484-4dkks 1/1 Running 0 20h
default nfs-client-provisioner-6b8547fdf4-cv97z 1/1 Running 0 41h
hll-etcd etcd-0 1/1 Running 0 24h
hll-etcd etcd-1 1/1 Running 0 24h
hll-etcd etcd-2 1/1 Running 1 (24h ago) 24h
hll-minio hll-milvus-pool-0-0 2/2 Running 0 23h
hll-minio hll-milvus-pool-0-1 2/2 Running 0 23h
hll-minio hll-milvus-pool-0-2 2/2 Running 0 23h
hll-minio hll-milvus-pool-0-3 2/2 Running 0 23h
ingress-nginx ingress-nginx-admission-create-m4p76 0/1 Completed 0 41h
ingress-nginx ingress-nginx-admission-patch-h46tc 0/1 Completed 3 41h
ingress-nginx ingress-nginx-controller-8c9449dbf-6txjh 1/1 Running 0 41h
kube-system calico-kube-controllers-595b58b579-vx24b 1/1 Running 0 2d16h
kube-system calico-node-887jl 1/1 Running 0 2d16h
kube-system calico-node-dmnf6 0/1 Running 0 2d16h
kube-system calico-node-fch7c 1/1 Running 0 2d16h
kube-system calico-node-gtzkj 1/1 Running 0 2d16h
kube-system coredns-6d8c4cb4d-fbgm4 1/1 Running 0 2d16h
kube-system coredns-6d8c4cb4d-z76bc 1/1 Running 0 2d16h
kube-system etcd-aly-hn1-inner-risk-graph-service-svc-stg-001 1/1 Running 59 2d16h
kube-system kube-apiserver-aly-hn1-inner-risk-graph-service-svc-stg-001 1/1 Running 0 41h
kube-system kube-controller-manager-aly-hn1-inner-risk-graph-service-svc-stg-001 1/1 Running 5097 2d16h
kube-system kube-proxy-7klhg 1/1 Running 0 2d16h
kube-system kube-proxy-98clm 1/1 Running 0 2d16h
kube-system kube-proxy-crqs6 1/1 Running 0 2d16h
kube-system kube-proxy-l4dsx 1/1 Running 0 2d16h
kube-system kube-scheduler-aly-hn1-inner-risk-graph-service-svc-stg-001 1/1 Running 4986 2d16h
milvus-operator milvus-operator-6f578f564f-77sdm 1/1 Running 0 16m
minio-operator console-576dbcc6b7-5bbz4 1/1 Running 0 42h
minio-operator minio-operator-76f469d84b-bzz9b 1/1 Running 0 42h
minio-operator minio-operator-76f469d84b-q599p 1/1 Running 1 (41h ago) 42h

日志：
kubectl logs milvus-operator-6f578f564f-77sdm -n milvus-operator -f

2024-06-14T02:30:44.851Z ERROR controller-runtime.manager.controller.milvus Reconciler error {"reconciler group": "milvus.io", "reconciler kind": "Milvus", "name": "hll-release", "namespace": "hll-milvus-stg", "error": "Operation cannot be fulfilled on milvuses.milvus.io "hll-release": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.6/pkg/internal/controller/controller.go:214

[gaoyuan5251]

milvus-operator milvus-operator-6f578f564f-77sdm 1/1 Running 0 16m

@haorenfsa 我的milvus.yaml

apiVersion: v1
kind: Secret
metadata:
name: hll-release-minio-secret
namespace: hll-milvus
type: Opaque
stringData:
accesskey: ***
secretkey: ***

---

apiVersion: milvus.io/v1beta1
kind: Milvus
metadata:
name: hll-release
labels:
app: milvus
namespace: hll-milvus-stg

spec:
mode: standalone
config:
common:
security:
tlsEnabled: false
etcd:
rootPath: hll-release
msgChannel:
chanNamePrefix:
cluster: hll-release
minio:
bucketName: hll-milvus
rootPath: milvus/hll-release
useSSL: false
components:
image: milvusdb/milvus:v2.4.1
nodeSelector:
disktype: ssd
standalone:
replicas: 1
serviceType: LoadBalancer
dependencies:
etcd:
external: true
endpoints: ["etcd.hll-etcd.svc.cluster.local:2379"]
storage:
external: true
endpoint: *****:80
secretRef: hll-release-minio-secret
type: MinIO

[yanliang567]

so what is the new error msg? if Milvus fails to start, please help to attach the full milvus logs. please refer this doc to export the whole Milvus logs for investigation

/assign @gaoyuan5251
/unassign

[gaoyuan5251]

so what is the new error msg? if Milvus fails to start, please help to attach the full milvus logs. please refer this doc to export the whole Milvus logs for investigation

@yanliang DONE