Mons not forming quorum

[is6769]

Is this a bug report or feature request?

• Bug Report

How to reproduce it (minimal and precise):
1)initialize k8s
2)do all instructions in docs(without changing anything)

• get pods (kubectl get pod -n rook-ceph)

NAME READY STATUS RESTARTS AGE
csi-cephfsplugin-2j8np 3/3 Running 0 13m
csi-cephfsplugin-6qrtc 3/3 Running 0 13m
csi-cephfsplugin-dtjmd 3/3 Running 0 13m
csi-cephfsplugin-provisioner-8658f67749-9j49t 6/6 Running 0 13m
csi-cephfsplugin-provisioner-8658f67749-wjwdb 6/6 Running 0 13m
csi-rbdplugin-gkbgz 3/3 Running 0 13m
csi-rbdplugin-nls5l 3/3 Running 0 13m
csi-rbdplugin-nzsfh 3/3 Running 0 13m
csi-rbdplugin-provisioner-6bc6766db-cfs69 6/6 Running 0 13m
csi-rbdplugin-provisioner-6bc6766db-v6vk9 6/6 Running 0 13m
rook-ceph-crashcollector-rtest2-546879f758-294lc 0/1 Init:0/2 0 13m
rook-ceph-crashcollector-rtest3-7c584f4575-5m5p7 0/1 Init:0/2 0 3m9s
rook-ceph-mon-a-7ccf669db7-tnnz6 1/1 Running 0 14m
rook-ceph-mon-b-84d66c7868-4sp55 1/1 Running 0 3m9s
rook-ceph-operator-5ff4d5c446-s2tpc 1/1 Running 0 17m

• logs of operator(kubectl logs -n rook-ceph rook-ceph-operator-5ff4d5c446-s2tpc)

2021-04-27 17:27:37.843801 I | rookcmd: starting Rook v1.5.7 with arguments '/usr/local/bin/rook ceph operator'
2021-04-27 17:27:37.914371 I | rookcmd: flag values: --add_dir_header=false, --alsologtostderr=false, --csi-cephfs-plugin-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin.yaml, --csi-cephfs-provisioner-dep-template-path=/etc/ceph-csi/cephfs/csi-cephfsplugin-provisioner-dep.yaml, --csi-rbd-plugin-template-path=/etc/ceph-csi/rbd/csi-rbdplugin.yaml, --csi-rbd-provisioner-dep-template-path=/etc/ceph-csi/rbd/csi-rbdplugin-provisioner-dep.yaml, --enable-discovery-daemon=false, --enable-flex-driver=false, --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-flush-frequency=5s, --log-level=INFO, --log_backtrace_at=:0, --log_dir=, --log_file=, --log_file_max_size=1800, --logtostderr=true, --master=, --mon-healthcheck-interval=45s, --mon-out-timeout=10m0s, --one_output=false, --operator-image=, --service-account=, --skip_headers=false, --skip_log_headers=false, --stderrthreshold=2, --v=0, --vmodule=
2021-04-27 17:27:37.914384 I | cephcmd: starting Rook-Ceph operator
2021-04-27 17:27:41.270705 I | cephcmd: base ceph version inside the rook operator image is "ceph version 15.2.8 (bdf3eebcd22d7d0b3dd4d5501bee5bac354d5b55) octopus (stable)"
2021-04-27 17:27:41.488425 I | operator: looking for secret "rook-ceph-admission-controller"
2021-04-27 17:27:41.772114 I | operator: secret "rook-ceph-admission-controller" not found. proceeding without the admission controller
2021-04-27 17:27:41.785381 I | operator: watching all namespaces for ceph cluster CRs
2021-04-27 17:27:41.786140 I | operator: setting up the controller-runtime manager
2021-04-27 17:27:41.832131 I | ceph-cluster-controller: ConfigMap "rook-ceph-operator-config" changes detected. Updating configurations
2021-04-27 17:27:42.450995 I | ceph-cluster-controller: successfully started
2021-04-27 17:27:42.451129 I | ceph-cluster-controller: enabling hotplug orchestration
2021-04-27 17:27:42.451164 I | ceph-crashcollector-controller: successfully started
2021-04-27 17:27:42.451323 I | ceph-block-pool-controller: successfully started
2021-04-27 17:27:42.451533 I | ceph-object-store-user-controller: successfully started
2021-04-27 17:27:42.451779 I | ceph-object-realm-controller: successfully started
2021-04-27 17:27:42.451934 I | ceph-object-zonegroup-controller: successfully started
2021-04-27 17:27:42.452145 I | ceph-object-zone-controller: successfully started
2021-04-27 17:27:42.452417 I | ceph-object-controller: successfully started
2021-04-27 17:27:42.452631 I | ceph-file-controller: successfully started
2021-04-27 17:27:42.452930 I | ceph-nfs-controller: successfully started
2021-04-27 17:27:42.453322 I | operator: starting the controller-runtime manager
2021-04-27 17:28:45.517498 I | clusterdisruption-controller: create event from ceph cluster CR
2021-04-27 17:28:45.552915 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph"
2021-04-27 17:28:45.599950 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph"
2021-04-27 17:28:45.608836 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap)
2021-04-27 17:28:45.612872 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap)
2021-04-27 17:28:45.616578 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap)
2021-04-27 17:28:45.620418 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="true" (configmap)
2021-04-27 17:28:45.624201 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.2.0" (default)
2021-04-27 17:28:45.628412 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.0.1" (default)
2021-04-27 17:28:45.632009 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="k8s.gcr.io/sig-storage/csi-provisioner:v2.0.0" (default)
2021-04-27 17:28:45.635613 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="k8s.gcr.io/sig-storage/csi-attacher:v3.0.0" (default)
2021-04-27 17:28:45.644840 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.0" (default)
2021-04-27 17:28:45.649921 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default)
2021-04-27 17:28:45.814875 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default)
2021-04-27 17:28:46.011890 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default)
2021-04-27 17:28:46.421865 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config"
2021-04-27 17:28:46.421985 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.2.0"
2021-04-27 17:28:46.612744 I | ceph-cluster-controller: clusterInfo not yet found, must be a new cluster
2021-04-27 17:28:46.814992 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default)
2021-04-27 17:28:47.020307 I | ceph-cluster-controller: detecting the ceph image version for image ceph/ceph:v15.2.8...
2021-04-27 17:30:14.511667 I | ceph-cluster-controller: detected ceph image version: "15.2.8-0 octopus"
2021-04-27 17:30:14.511700 I | ceph-cluster-controller: validating ceph version from provided image
2021-04-27 17:30:14.520147 I | ceph-cluster-controller: cluster "rook-ceph": version "15.2.8-0 octopus" detected for image "ceph/ceph:v15.2.8"
2021-04-27 17:30:14.585017 I | op-mon: start running mons
2021-04-27 17:30:14.853998 I | op-mon: creating mon secrets for a new cluster
2021-04-27 17:30:14.877361 I | op-mon: existing maxMonID not found or failed to load. configmaps "rook-ceph-mon-endpoints" not found
2021-04-27 17:30:14.888514 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":[]}] data: mapping:{"node":{}} maxMonId:-1]
2021-04-27 17:30:15.088212 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2021-04-27 17:30:15.088418 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2021-04-27 17:30:16.296712 I | op-mon: targeting the mon count 3
2021-04-27 17:30:16.344793 I | op-mon: sched-mon: created canary deployment rook-ceph-mon-a-canary
2021-04-27 17:30:16.380160 I | op-mon: sched-mon: created canary deployment rook-ceph-mon-b-canary
2021-04-27 17:30:16.528358 I | op-mon: sched-mon: created canary deployment rook-ceph-mon-c-canary
2021-04-27 17:30:17.134452 I | op-mon: sched-mon: canary monitor deployment rook-ceph-mon-a-canary scheduled to rtest2
2021-04-27 17:30:17.134483 I | op-mon: assignmon: mon a assigned to node rtest2
2021-04-27 17:30:22.011490 I | op-mon: sched-mon: canary monitor deployment rook-ceph-mon-b-canary scheduled to rtest3
2021-04-27 17:30:22.011523 I | op-mon: assignmon: mon b assigned to node rtest3
2021-04-27 17:30:42.203063 I | op-mon: sched-mon: canary monitor deployment rook-ceph-mon-c-canary scheduled to rtest1
2021-04-27 17:30:42.203099 I | op-mon: assignmon: mon c assigned to node rtest1
2021-04-27 17:30:42.215750 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-a-canary"
2021-04-27 17:30:42.231592 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-b-canary"
2021-04-27 17:30:43.126786 I | op-mon: cleaning up canary monitor deployment "rook-ceph-mon-c-canary"
2021-04-27 17:30:43.195885 I | op-mon: creating mon a
2021-04-27 17:30:43.523289 I | op-mon: mon "a" endpoint is [v2:10.97.34.7:3300,v1:10.97.34.7:6789]
2021-04-27 17:30:44.266140 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.34.7:6789"]}] data:a=10.97.34.7:6789 mapping:{"node":{"a":{"Name":"rtest2","Hostname":"rtest2","Address":"192.168.1.61"},"b":{"Name":"rtest3","Hostname":"rtest3","Address":"192.168.1.62"},"c":{"Name":"rtest1","Hostname":"rtest1","Address":"192.168.1.60"}}} maxMonId:-1]
2021-04-27 17:30:44.441042 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2021-04-27 17:30:44.441710 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2021-04-27 17:30:44.502226 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2021-04-27 17:30:44.502543 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2021-04-27 17:30:44.516253 I | op-mon: 0 of 1 expected mons are ready. creating or updating deployments without checking quorum in attempt to achieve a healthy mon cluster
2021-04-27 17:30:44.613698 I | op-mon: updating maxMonID from -1 to 0 after committing mon "a"
2021-04-27 17:30:44.701118 I | op-mon: waiting for mon quorum with [a]
2021-04-27 17:30:44.833552 I | op-mon: mons running: [a]
2021-04-27 17:31:05.268357 I | op-mon: mons running: [a]
2021-04-27 17:31:25.651961 I | op-mon: mons running: [a]
2021-04-27 17:31:26.446949 I | ceph-csi: Detected ceph CSI image version: "v3.2.0"
2021-04-27 17:31:26.462549 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap)
2021-04-27 17:31:26.472049 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default)
2021-04-27 17:31:26.476715 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default)
2021-04-27 17:31:26.489218 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default)
2021-04-27 17:31:26.502711 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default)
2021-04-27 17:31:26.509640 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="" (default)
2021-04-27 17:31:26.513558 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="" (default)
2021-04-27 17:31:26.516919 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default)
2021-04-27 17:31:26.520481 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap)
2021-04-27 17:31:26.618546 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap)
2021-04-27 17:31:26.803896 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2021-04-27 17:31:26.996689 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2021-04-27 17:31:26.996720 I | ceph-csi: Kubernetes version is 1.20
2021-04-27 17:31:27.195189 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="k8s.gcr.io/sig-storage/csi-resizer:v1.0.0" (default)
2021-04-27 17:31:27.393665 I | op-k8sutil: CSI_LOG_LEVEL="" (default)
2021-04-27 17:31:27.641783 I | ceph-csi: successfully started CSI Ceph RBD
2021-04-27 17:31:27.684156 I | ceph-csi: successfully started CSI CephFS driver
2021-04-27 17:31:27.796357 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default)
2021-04-27 17:31:27.996967 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default)
2021-04-27 17:31:28.194525 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default)
2021-04-27 17:31:28.399043 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default)
2021-04-27 17:31:28.594204 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default)
2021-04-27 17:31:28.828347 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default)
2021-04-27 17:31:29.583466 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default)
2021-04-27 17:31:29.993578 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default)
2021-04-27 17:31:31.209908 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap)
W0427 17:31:31.317549       7 warnings.go:70] storage.k8s.io/v1beta1 CSIDriver is deprecated in v1.19+, unavailable in v1.22+; use storage.k8s.io/v1 CSIDriver
W0427 17:31:32.019648       7 warnings.go:70] storage.k8s.io/v1beta1 CSIDriver is deprecated in v1.19+, unavailable in v1.22+; use storage.k8s.io/v1 CSIDriver
2021-04-27 17:31:32.023076 I | ceph-csi: CSIDriver object created for driver "rook-ceph.rbd.csi.ceph.com"
2021-04-27 17:31:32.109179 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="ReadWriteOnceWithFSType" (configmap)
W0427 17:31:32.270369       7 warnings.go:70] storage.k8s.io/v1beta1 CSIDriver is deprecated in v1.19+, unavailable in v1.22+; use storage.k8s.io/v1 CSIDriver
W0427 17:31:32.379004       7 warnings.go:70] storage.k8s.io/v1beta1 CSIDriver is deprecated in v1.19+, unavailable in v1.22+; use storage.k8s.io/v1 CSIDriver
2021-04-27 17:31:32.379288 I | ceph-csi: CSIDriver object created for driver "rook-ceph.cephfs.csi.ceph.com"
2021-04-27 17:31:46.450376 I | op-mon: mons running: [a]
2021-04-27 17:31:57.105917 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-rtest2": the object has been modified; please apply your changes to the latest version and try again
2021-04-27 17:32:06.939504 I | op-mon: mon a is not yet running
2021-04-27 17:32:06.939537 I | op-mon: mons running: []
2021-04-27 17:32:27.650098 I | op-mon: mons running: [a]
2021-04-27 17:32:48.022131 I | op-mon: mons running: [a]
2021-04-27 17:33:08.368026 I | op-mon: mons running: [a]
2021-04-27 17:33:29.035792 I | op-mon: mons running: [a]
2021-04-27 17:33:49.355480 I | op-mon: mons running: [a]
2021-04-27 17:34:09.712417 I | op-mon: mons running: [a]
2021-04-27 17:34:30.096918 I | op-mon: mons running: [a]
2021-04-27 17:34:50.506947 I | op-mon: mons running: [a]
2021-04-27 17:35:10.897443 I | op-mon: mons running: [a]
2021-04-27 17:35:31.279508 I | op-mon: mons running: [a]
2021-04-27 17:35:51.660151 I | op-mon: mons running: [a]
2021-04-27 17:36:12.035631 I | op-mon: mons running: [a]
2021-04-27 17:36:32.433770 I | op-mon: mons running: [a]
2021-04-27 17:36:52.913377 I | op-mon: mons running: [a]
2021-04-27 17:37:13.428670 I | op-mon: mons running: [a]
2021-04-27 17:37:33.924985 I | op-mon: mons running: [a]
2021-04-27 17:37:54.360739 I | op-mon: mons running: [a]
2021-04-27 17:38:14.861818 I | op-mon: mons running: [a]
2021-04-27 17:38:35.313268 I | op-mon: mons running: [a]
2021-04-27 17:38:55.918124 I | op-mon: mons running: [a]
2021-04-27 17:39:16.385271 I | op-mon: mons running: [a]
2021-04-27 17:39:37.011876 I | op-mon: mons running: [a]
2021-04-27 17:39:57.642954 I | op-mon: mons running: [a]
2021-04-27 17:40:18.052350 I | op-mon: mons running: [a]
2021-04-27 17:40:38.666012 I | op-mon: mons running: [a]
2021-04-27 17:40:54.112979 E | ceph-cluster-controller: failed to reconcile. failed to reconcile cluster "rook-ceph": failed to configure local ceph cluster: failed to create cluster: failed to start ceph monitors: failed to start mon pods: failed to check mon quorum a: failed to wait for mon quorum: exceeded max retry count waiting for monitors to reach quorum
2021-04-27 17:40:55.137750 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph"
2021-04-27 17:40:55.160210 I | op-mon: parsing mon endpoints: a=10.97.34.7:6789
2021-04-27 17:40:55.176783 I | ceph-cluster-controller: detecting the ceph image version for image ceph/ceph:v15.2.8...
2021-04-27 17:41:01.926102 I | ceph-cluster-controller: detected ceph image version: "15.2.8-0 octopus"
2021-04-27 17:41:01.926130 I | ceph-cluster-controller: validating ceph version from provided image
2021-04-27 17:41:01.938621 I | op-mon: parsing mon endpoints: a=10.97.34.7:6789
2021-04-27 17:41:01.943258 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2021-04-27 17:41:01.943585 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2021-04-27 17:41:17.296059 E | ceph-cluster-controller: failed to get ceph daemons versions, this typically happens during the first cluster initialization. failed to run 'ceph versions'. timed out
. : timed out
.
2021-04-27 17:41:17.296112 I | ceph-cluster-controller: cluster "rook-ceph": version "15.2.8-0 octopus" detected for image "ceph/ceph:v15.2.8"
2021-04-27 17:41:17.434714 I | op-mon: start running mons
2021-04-27 17:41:17.446266 I | op-mon: parsing mon endpoints: a=10.97.34.7:6789
2021-04-27 17:41:17.483541 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.34.7:6789"]}] data:a=10.97.34.7:6789 mapping:{"node":{"a":{"Name":"rtest2","Hostname":"rtest2","Address":"192.168.1.61"},"b":{"Name":"rtest3","Hostname":"rtest3","Address":"192.168.1.62"},"c":{"Name":"rtest1","Hostname":"rtest1","Address":"192.168.1.60"}}} maxMonId:0]
2021-04-27 17:41:17.500738 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2021-04-27 17:41:17.501262 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2021-04-27 17:41:18.439476 I | op-mon: targeting the mon count 3
2021-04-27 17:41:49.214378 W | op-mon: failed to set Rook and/or user-defined Ceph config options before starting mons; will retry after starting mons. failed to apply default Ceph configurations: failed to set one or more Ceph configs: failed to set ceph config in the centralized mon configuration database; you may need to use the rook-config-override ConfigMap. output: timed out
. : timed out
. : failed to set ceph config in the centralized mon configuration database; you may need to use the rook-config-override ConfigMap. output: timed out
. : timed out
.
2021-04-27 17:41:49.214414 I | op-mon: creating mon b
2021-04-27 17:41:49.268255 I | op-mon: mon "a" endpoint is [v2:10.97.34.7:3300,v1:10.97.34.7:6789]
2021-04-27 17:41:49.311628 I | op-mon: mon "b" endpoint is [v2:10.104.103.34:3300,v1:10.104.103.34:6789]
2021-04-27 17:41:49.453339 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph","monitors":["10.97.34.7:6789","10.104.103.34:6789"]}] data:a=10.97.34.7:6789,b=10.104.103.34:6789 mapping:{"node":{"a":{"Name":"rtest2","Hostname":"rtest2","Address":"192.168.1.61"},"b":{"Name":"rtest3","Hostname":"rtest3","Address":"192.168.1.62"},"c":{"Name":"rtest1","Hostname":"rtest1","Address":"192.168.1.60"}}} maxMonId:0]
2021-04-27 17:41:49.475523 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2021-04-27 17:41:49.476025 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2021-04-27 17:41:50.191070 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2021-04-27 17:41:50.191388 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2021-04-27 17:41:50.225778 I | op-mon: 1 of 2 expected mon deployments exist. creating new deployment(s).
2021-04-27 17:41:50.247464 I | op-mon: deployment for mon rook-ceph-mon-a already exists. updating if needed
2021-04-27 17:41:50.304977 I | op-k8sutil: deployment "rook-ceph-mon-a" did not change, nothing to update
2021-04-27 17:41:50.499730 I | op-mon: updating maxMonID from 0 to 1 after committing mon "b"
2021-04-27 17:41:50.700870 I | op-mon: waiting for mon quorum with [a b]
2021-04-27 17:41:50.776166 E | ceph-crashcollector-controller: node reconcile failed on op "unchanged": Operation cannot be fulfilled on deployments.apps "rook-ceph-crashcollector-rtest3": the object has been modified; please apply your changes to the latest version and try again
2021-04-27 17:41:51.031276 I | op-mon: mon b is not yet running
2021-04-27 17:41:51.031308 I | op-mon: mons running: [a]
2021-04-27 17:42:11.476960 I | op-mon: mons running: [a b]
2021-04-27 17:42:31.865636 I | op-mon: mons running: [a b]
2021-04-27 17:42:52.251957 I | op-mon: mons running: [a b]
2021-04-27 17:43:12.618575 I | op-mon: mons running: [a b]
2021-04-27 17:43:33.004744 I | op-mon: mons running: [a b]
2021-04-27 17:43:53.416776 I | op-mon: mons running: [a b]
2021-04-27 17:44:13.839282 I | op-mon: mons running: [a b]
2021-04-27 17:44:34.398198 I | op-mon: mons running: [a b]
2021-04-27 17:44:54.884280 I | op-mon: mons running: [a b]
2021-04-27 17:45:15.640559 I | op-mon: mons running: [a b]
2021-04-27 17:45:36.051195 I | op-mon: mons running: [a b]
2021-04-27 17:45:56.513734 I | op-mon: mons running: [a b]
2021-04-27 17:46:16.938320 I | op-mon: mons running: [a b]
2021-04-27 17:46:37.473157 I | op-mon: mons running: [a b]
2021-04-27 17:46:57.918303 I | op-mon: mons running: [a b]
2021-04-27 17:47:18.554752 I | op-mon: mons running: [a b]
2021-04-27 17:47:39.110938 I | op-mon: mons running: [a b]
2021-04-27 17:47:59.524571 I | op-mon: mons running: [a b]
2021-04-27 17:48:19.993231 I | op-mon: mons running: [a b]
2021-04-27 17:48:40.563186 I | op-mon: mons running: [a b]
2021-04-27 17:49:01.049298 I | op-mon: mons running: [a b]
2021-04-27 17:49:21.633775 I | op-mon: mons running: [a b]
2021-04-27 17:49:42.219214 I | op-mon: mons running: [a b]
2021-04-27 17:50:02.658493 I | op-mon: mons running: [a b]
2021-04-27 17:50:23.131994 I | op-mon: mons running: [a b]

Environment:

• OS (e.g. from /etc/os-release): rhel 8.3
• Kernel (e.g. uname -a): 4.18.0-240.22.1.el8_3.x86_64
• Rook version (use rook version inside of a Rook Pod): 1.5.7
• Kubernetes version (use kubectl version): 1.20.2
• Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): bare metall

[travisn]

@is6769 The mons are not forming quorum. Most often this is a network issue between the operator and the mons. See the troubleshooting guide. If you need help troubleshooting further, please open a discussion item or join the Rook slack. Good luck!

[is6769]

problem was in cni,so i changed it from calico to flannel, and all worked