kubectl logs -n rook-ceph     rook-ceph-operator-547744ffbf-l6p4c
2022-11-23 22:44:40.593461 I | rookcmd: starting Rook v1.10.6 with arguments '/usr/local/bin/rook ceph operator'
2022-11-23 22:44:40.593654 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO, --operator-image=, --service-account=
2022-11-23 22:44:40.593668 I | cephcmd: starting Rook-Ceph operator
2022-11-23 22:44:41.159852 I | cephcmd: base ceph version inside the rook operator image is "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)"
2022-11-23 22:44:41.171852 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var)
2022-11-23 22:44:41.171885 I | operator: watching all namespaces for Ceph CRs
2022-11-23 22:44:41.172133 I | operator: setting up schemes
2022-11-23 22:44:41.176668 I | operator: setting up the controller-runtime manager
2022-11-23 22:44:41.334211 I | operator: Fetching webhook cert-manager-webhook to see if cert-manager is installed.
2022-11-23 22:44:41.359656 I | operator: failed to get cert manager
2022-11-23 22:44:41.359719 I | ceph-cluster-controller: successfully started
2022-11-23 22:44:41.359816 I | ceph-cluster-controller: enabling hotplug orchestration
2022-11-23 22:44:41.359846 I | ceph-crashcollector-controller: successfully started
2022-11-23 22:44:41.359874 I | ceph-block-pool-controller: successfully started
2022-11-23 22:44:41.359906 I | ceph-object-store-user-controller: successfully started
2022-11-23 22:44:41.359946 I | ceph-object-realm-controller: successfully started
2022-11-23 22:44:41.359968 I | ceph-object-zonegroup-controller: successfully started
2022-11-23 22:44:41.359987 I | ceph-object-zone-controller: successfully started
2022-11-23 22:44:41.360194 I | ceph-object-controller: successfully started
2022-11-23 22:44:41.360237 I | ceph-file-controller: successfully started
2022-11-23 22:44:41.360321 I | ceph-nfs-controller: successfully started
2022-11-23 22:44:41.360359 I | ceph-rbd-mirror-controller: successfully started
2022-11-23 22:44:41.360385 I | ceph-client-controller: successfully started
2022-11-23 22:44:41.360414 I | ceph-filesystem-mirror-controller: successfully started
2022-11-23 22:44:41.360444 I | operator: rook-ceph-operator-config-controller successfully started
2022-11-23 22:44:41.360464 I | ceph-csi: rook-ceph-operator-csi-controller successfully started
2022-11-23 22:44:41.360491 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started
2022-11-23 22:44:41.360536 I | ceph-bucket-topic: successfully started
2022-11-23 22:44:41.360555 I | ceph-bucket-notification: successfully started
2022-11-23 22:44:41.360575 I | ceph-bucket-notification: successfully started
2022-11-23 22:44:41.360600 I | ceph-fs-subvolumegroup-controller: successfully started
2022-11-23 22:44:41.360619 I | blockpool-rados-namespace-controller: successfully started
2022-11-23 22:44:41.362023 I | operator: starting the controller-runtime manager
2022-11-23 22:44:41.463744 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "rook-ceph-external"
2022-11-23 22:44:41.474887 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap)
2022-11-23 22:44:41.475241 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap)
2022-11-23 22:44:41.475264 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap)
2022-11-23 22:44:41.476790 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default)
2022-11-23 22:44:41.545698 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph-external"
2022-11-23 22:44:41.546244 I | op-k8sutil: ROOK_CEPH_ALLOW_LOOP_DEVICES="false" (configmap)
2022-11-23 22:44:41.546260 I | operator: rook-ceph-operator-config-controller done reconciling
2022-11-23 22:44:41.553189 E | ceph-spec: failed to update cluster condition to {Type:Connecting Status:True Reason:ClusterConnecting Message:Attempting to connect to an external Ceph cluster LastHeartbeatTime:2022-11-23 22:44:41.545752688 +0000 UTC m=+1.039745285 LastTransitionTime:2022-11-23 22:44:41.545752596 +0000 UTC m=+1.039745210}. failed to update object "rook-ceph-external/rook-ceph-external" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "rook-ceph-external": the object has been modified; please apply your changes to the latest version and try again
2022-11-23 22:44:41.557441 W | ceph-spec: waiting for connection info of the external cluster. retrying in 1m0s.
2022-11-23 22:44:41.567782 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config"
2022-11-23 22:44:41.598888 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap)
2022-11-23 22:44:41.598939 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap)
2022-11-23 22:44:41.598960 I | op-k8sutil: ROOK_CSI_ENABLE_NFS="false" (configmap)
2022-11-23 22:44:41.598975 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap)
2022-11-23 22:44:41.599005 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap)
2022-11-23 22:44:41.599024 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default)
2022-11-23 22:44:41.599039 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap)
2022-11-23 22:44:41.599116 I | op-k8sutil: CSI_GRPC_TIMEOUT_SECONDS="150" (configmap)
2022-11-23 22:44:41.599151 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default)
2022-11-23 22:44:41.599167 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default)
2022-11-23 22:44:41.599183 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default)
2022-11-23 22:44:41.599198 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default)
2022-11-23 22:44:41.599212 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default)
2022-11-23 22:44:41.599225 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default)
2022-11-23 22:44:41.599239 I | op-k8sutil: CSIADDONS_PORT="9070" (default)
2022-11-23 22:44:41.599254 I | op-k8sutil: CSIADDONS_PORT="9070" (default)
2022-11-23 22:44:41.599270 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default)
2022-11-23 22:44:41.599286 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default)
2022-11-23 22:44:41.599301 I | op-k8sutil: CSI_ENABLE_LIVENESS="false" (configmap)
2022-11-23 22:44:41.599319 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="system-node-critical" (configmap)
2022-11-23 22:44:41.599336 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="system-cluster-critical" (configmap)
2022-11-23 22:44:41.599351 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default)
2022-11-23 22:44:41.599367 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap)
2022-11-23 22:44:41.599381 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap)
2022-11-23 22:44:41.599395 I | op-k8sutil: CSI_ENABLE_NFS_SNAPSHOTTER="true" (configmap)
2022-11-23 22:44:41.599420 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap)
2022-11-23 22:44:41.599436 I | op-k8sutil: CSI_ENABLE_TOPOLOGY="false" (configmap)
2022-11-23 22:44:41.599451 I | op-k8sutil: CSI_ENABLE_ENCRYPTION="false" (configmap)
2022-11-23 22:44:41.599466 I | op-k8sutil: CSI_ENABLE_METADATA="false" (default)
2022-11-23 22:44:41.599482 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2022-11-23 22:44:41.599497 I | op-k8sutil: CSI_NFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2022-11-23 22:44:41.599522 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2022-11-23 22:44:41.599536 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap)
2022-11-23 22:44:41.599550 I | ceph-csi: Kubernetes version is 1.25
2022-11-23 22:44:41.599564 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="registry.k8s.io/sig-storage/csi-resizer:v1.6.0" (default)
2022-11-23 22:44:41.599578 I | op-k8sutil: CSI_LOG_LEVEL="" (default)
2022-11-23 22:44:41.599592 I | op-k8sutil: CSI_SIDECAR_LOG_LEVEL="" (default)
2022-11-23 22:44:41.610814 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap)
2022-11-23 22:44:41.610854 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.7.2" (default)
2022-11-23 22:44:41.610876 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1" (default)
2022-11-23 22:44:41.610894 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="registry.k8s.io/sig-storage/csi-provisioner:v3.3.0" (default)
2022-11-23 22:44:41.610978 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="registry.k8s.io/sig-storage/csi-attacher:v4.0.0" (default)
2022-11-23 22:44:41.611140 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0" (default)
2022-11-23 22:44:41.611332 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default)
2022-11-23 22:44:41.611396 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.5.0" (default)
2022-11-23 22:44:41.611414 I | op-k8sutil: CSI_TOPOLOGY_DOMAIN_LABELS="" (default)
2022-11-23 22:44:41.611489 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default)
2022-11-23 22:44:41.611651 I | op-k8sutil: ROOK_CSI_NFS_POD_LABELS="" (default)
2022-11-23 22:44:41.611771 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default)
2022-11-23 22:44:41.611899 I | op-k8sutil: CSI_CLUSTER_NAME="" (default)
2022-11-23 22:44:41.612015 I | op-k8sutil: ROOK_CSI_IMAGE_PULL_POLICY="IfNotPresent" (default)
2022-11-23 22:44:41.612131 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.7.2"
2022-11-23 22:44:41.612484 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default)
2022-11-23 22:44:41.612509 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default)
2022-11-23 22:44:45.910116 I | ceph-csi: Detected ceph CSI image version: "v3.7.2"
2022-11-23 22:44:45.922871 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default)
2022-11-23 22:44:45.922912 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default)
2022-11-23 22:44:45.922921 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default)
2022-11-23 22:44:45.922930 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default)
2022-11-23 22:44:45.923065 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default)
2022-11-23 22:44:45.923235 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default)
2022-11-23 22:44:45.923293 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default)
2022-11-23 22:44:45.923308 I | op-k8sutil: CSI_RBD_PLUGIN_VOLUME="" (default)
2022-11-23 22:44:45.923482 I | op-k8sutil: CSI_RBD_PLUGIN_VOLUME_MOUNT="" (default)
2022-11-23 22:44:46.103852 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default)
2022-11-23 22:44:46.103905 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default)
2022-11-23 22:44:46.103917 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default)
2022-11-23 22:44:46.477922 I | ceph-csi: successfully started CSI Ceph RBD driver
2022-11-23 22:44:46.477962 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default)
2022-11-23 22:44:46.477974 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default)
2022-11-23 22:44:46.477996 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default)
2022-11-23 22:44:46.478006 I | op-k8sutil: CSI_CEPHFS_PLUGIN_VOLUME="" (default)
2022-11-23 22:44:46.478014 I | op-k8sutil: CSI_CEPHFS_PLUGIN_VOLUME_MOUNT="" (default)
2022-11-23 22:44:46.535514 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default)
2022-11-23 22:44:46.535552 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default)
2022-11-23 22:44:46.535570 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default)
2022-11-23 22:44:46.729030 I | ceph-csi: successfully started CSI CephFS driver
2022-11-23 22:44:46.729066 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="File" (configmap)
2022-11-23 22:44:48.251413 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com"
2022-11-23 22:44:48.251451 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="File" (configmap)
2022-11-23 22:44:49.099899 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com"
2022-11-23 22:44:49.099933 I | ceph-csi: CSI NFS driver disabled
2022-11-23 22:44:49.099951 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists
2022-11-23 22:44:49.510866 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists
2022-11-23 22:44:52.851167 I | ceph-csi: successfully removed CSI NFS driver
2022-11-23 22:44:52.852859 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default)
2022-11-23 22:44:53.612438 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config"
2022-11-23 22:44:54.292561 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap)
2022-11-23 22:44:54.292599 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="true" (configmap)
2022-11-23 22:44:54.292609 I | op-k8sutil: ROOK_CSI_ENABLE_NFS="false" (configmap)
2022-11-23 22:44:54.292618 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap)
2022-11-23 22:44:54.292626 I | op-k8sutil: ROOK_CSI_ENABLE_GRPC_METRICS="false" (configmap)
2022-11-23 22:44:54.292636 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default)
2022-11-23 22:44:54.292644 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap)
2022-11-23 22:44:54.292669 I | op-k8sutil: CSI_GRPC_TIMEOUT_SECONDS="150" (configmap)
2022-11-23 22:44:54.292682 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default)
2022-11-23 22:44:54.292691 I | op-k8sutil: CSI_CEPHFS_GRPC_METRICS_PORT="9091" (default)
2022-11-23 22:44:54.292700 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default)
2022-11-23 22:44:54.292709 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default)
2022-11-23 22:44:54.292717 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default)
2022-11-23 22:44:54.292725 I | op-k8sutil: CSI_RBD_GRPC_METRICS_PORT="9090" (default)
2022-11-23 22:44:54.292734 I | op-k8sutil: CSIADDONS_PORT="9070" (default)
2022-11-23 22:44:54.292742 I | op-k8sutil: CSIADDONS_PORT="9070" (default)
2022-11-23 22:44:54.292750 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default)
2022-11-23 22:44:54.292758 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default)
2022-11-23 22:44:54.292766 I | op-k8sutil: CSI_ENABLE_LIVENESS="false" (configmap)
2022-11-23 22:44:54.292774 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="system-node-critical" (configmap)
2022-11-23 22:44:54.292788 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="system-cluster-critical" (configmap)
2022-11-23 22:44:54.292813 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default)
2022-11-23 22:44:54.292826 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap)
2022-11-23 22:44:54.292838 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap)
2022-11-23 22:44:54.292847 I | op-k8sutil: CSI_ENABLE_NFS_SNAPSHOTTER="true" (configmap)
2022-11-23 22:44:54.292869 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap)
2022-11-23 22:44:54.292878 I | op-k8sutil: CSI_ENABLE_TOPOLOGY="false" (configmap)
2022-11-23 22:44:54.292886 I | op-k8sutil: CSI_ENABLE_ENCRYPTION="false" (configmap)
2022-11-23 22:44:54.292894 I | op-k8sutil: CSI_ENABLE_METADATA="false" (default)
2022-11-23 22:44:54.292905 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2022-11-23 22:44:54.292918 I | op-k8sutil: CSI_NFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2022-11-23 22:44:54.292932 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2022-11-23 22:44:54.292942 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap)
2022-11-23 22:44:54.292950 I | ceph-csi: Kubernetes version is 1.25
2022-11-23 22:44:54.292961 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="registry.k8s.io/sig-storage/csi-resizer:v1.6.0" (default)
2022-11-23 22:44:54.292973 I | op-k8sutil: CSI_LOG_LEVEL="" (default)
2022-11-23 22:44:54.292984 I | op-k8sutil: CSI_SIDECAR_LOG_LEVEL="" (default)
2022-11-23 22:44:54.801066 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap)
2022-11-23 22:44:54.801117 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.7.2" (default)
2022-11-23 22:44:54.801138 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1" (default)
2022-11-23 22:44:54.801157 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="registry.k8s.io/sig-storage/csi-provisioner:v3.3.0" (default)
2022-11-23 22:44:54.801172 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="registry.k8s.io/sig-storage/csi-attacher:v4.0.0" (default)
2022-11-23 22:44:54.801187 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="registry.k8s.io/sig-storage/csi-snapshotter:v6.1.0" (default)
2022-11-23 22:44:54.801202 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default)
2022-11-23 22:44:54.801218 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.5.0" (default)
2022-11-23 22:44:54.801232 I | op-k8sutil: CSI_TOPOLOGY_DOMAIN_LABELS="" (default)
2022-11-23 22:44:54.801246 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default)
2022-11-23 22:44:54.801262 I | op-k8sutil: ROOK_CSI_NFS_POD_LABELS="" (default)
2022-11-23 22:44:54.801276 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default)
2022-11-23 22:44:54.801291 I | op-k8sutil: CSI_CLUSTER_NAME="" (default)
2022-11-23 22:44:54.801306 I | op-k8sutil: ROOK_CSI_IMAGE_PULL_POLICY="IfNotPresent" (default)
2022-11-23 22:44:54.801322 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.7.2"
2022-11-23 22:44:54.801392 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default)
2022-11-23 22:44:54.801419 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default)
2022-11-23 22:45:09.358011 I | ceph-csi: Detected ceph CSI image version: "v3.7.2"
2022-11-23 22:45:09.364424 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default)
2022-11-23 22:45:09.364456 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default)
2022-11-23 22:45:09.364467 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default)
2022-11-23 22:45:09.364475 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default)
2022-11-23 22:45:09.364518 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="" (default)
2022-11-23 22:45:09.364527 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="" (default)
2022-11-23 22:45:09.364536 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default)
2022-11-23 22:45:09.364545 I | op-k8sutil: CSI_RBD_PLUGIN_VOLUME="" (default)
2022-11-23 22:45:09.364553 I | op-k8sutil: CSI_RBD_PLUGIN_VOLUME_MOUNT="" (default)
2022-11-23 22:45:10.008496 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="" (default)
2022-11-23 22:45:10.008546 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="" (default)
2022-11-23 22:45:10.008563 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default)
2022-11-23 22:45:10.072188 I | ceph-csi: successfully started CSI Ceph RBD driver
2022-11-23 22:45:10.072243 I | op-k8sutil: CSI_CEPHFS_PLUGIN_TOLERATIONS="" (default)
2022-11-23 22:45:10.072255 I | op-k8sutil: CSI_CEPHFS_PLUGIN_NODE_AFFINITY="" (default)
2022-11-23 22:45:10.072264 I | op-k8sutil: CSI_CEPHFS_PLUGIN_RESOURCE="" (default)
2022-11-23 22:45:10.072273 I | op-k8sutil: CSI_CEPHFS_PLUGIN_VOLUME="" (default)
2022-11-23 22:45:10.072311 I | op-k8sutil: CSI_CEPHFS_PLUGIN_VOLUME_MOUNT="" (default)
2022-11-23 22:45:10.393090 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_TOLERATIONS="" (default)
2022-11-23 22:45:10.393129 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_NODE_AFFINITY="" (default)
2022-11-23 22:45:10.393164 I | op-k8sutil: CSI_CEPHFS_PROVISIONER_RESOURCE="" (default)
2022-11-23 22:45:10.871764 I | ceph-csi: successfully started CSI CephFS driver
2022-11-23 22:45:10.871810 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="File" (configmap)
2022-11-23 22:45:10.941331 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com"
2022-11-23 22:45:10.941389 I | op-k8sutil: CSI_CEPHFS_FSGROUPPOLICY="File" (configmap)
2022-11-23 22:45:10.954044 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.cephfs.csi.ceph.com"
2022-11-23 22:45:10.954078 I | ceph-csi: CSI NFS driver disabled
2022-11-23 22:45:10.954093 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists
2022-11-23 22:45:11.237345 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists
2022-11-23 22:45:11.267443 I | ceph-csi: successfully removed CSI NFS driver
2022-11-23 22:45:41.797600 W | ceph-spec: waiting for connection info of the external cluster. retrying in 1m0s.
2022-11-23 22:45:53.833361 I | ceph-spec: parsing mon endpoints: nuc9034=172.17.40.58:6789
2022-11-23 22:45:53.833533 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap)
2022-11-23 22:45:53.833553 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph-external.ceph.rook.io/bucket"
2022-11-23 22:45:53.834035 I | op-bucket-prov: successfully reconciled bucket provisioner
I1123 22:45:53.834127       1 manager.go:135] objectbucket.io/provisioner-manager "msg"="starting provisioner" "name"="rook-ceph-external.ceph.rook.io/bucket"
2022-11-23 22:46:41.808459 I | ceph-spec: parsing mon endpoints: nuc9034=172.17.40.58:6789
2022-11-23 22:46:41.808514 I | ceph-spec: found the cluster info to connect to the external cluster. will use "client.healthchecker" to check health and monitor status. mons=map[nuc9034:0xc00087a540]
2022-11-23 22:46:41.812918 I | cephclient: writing config file /var/lib/rook/rook-ceph-external/rook-ceph-external.config
2022-11-23 22:46:41.813205 I | cephclient: generated admin config in /var/lib/rook/rook-ceph-external
2022-11-23 22:46:41.813225 I | ceph-cluster-controller: external cluster identity established
2022-11-23 22:46:41.827346 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config"
2022-11-23 22:46:41.844516 I | ceph-cluster-controller: successfully updated csi config map
2022-11-23 22:46:41.844566 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph-external"
2022-11-23 22:46:41.844622 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph-external"
2022-11-23 22:46:41.844758 I | ceph-cluster-controller: reporting cluster telemetry
2022-11-23 22:46:42.668225 W | telemetry: failed to set telemetry key "rook/version". failed to set "rook/version" in the mon config-key store. output: Error EACCES: access denied: exit status 13
2022-11-23 22:46:43.402343 W | telemetry: failed to set telemetry key "rook/kubernetes/version". failed to set "rook/kubernetes/version" in the mon config-key store. output: Error EACCES: access denied: exit status 13
2022-11-23 22:46:44.071505 W | telemetry: failed to set telemetry key "rook/csi/version". failed to set "rook/csi/version" in the mon config-key store. output: Error EACCES: access denied: exit status 13
2022-11-23 22:46:44.727441 W | telemetry: failed to set telemetry key "rook/cluster/mon/max-id". failed to set "rook/cluster/mon/max-id" in the mon config-key store. output: Error EACCES: access denied: exit status 13
2022-11-23 22:46:45.369627 W | telemetry: failed to set telemetry key "rook/cluster/mon/count". failed to set "rook/cluster/mon/count" in the mon config-key store. output: Error EACCES: access denied: exit status 13
2022-11-23 22:46:46.038289 W | telemetry: failed to set telemetry key "rook/cluster/mon/allow-multiple-per-node". failed to set "rook/cluster/mon/allow-multiple-per-node" in the mon config-key store. output: Error EACCES: access denied: exit status 13
2022-11-23 22:46:46.689553 W | telemetry: failed to set telemetry key "rook/cluster/mon/pvc/enabled". failed to set "rook/cluster/mon/pvc/enabled" in the mon config-key store. output: Error EACCES: access denied: exit status 13
2022-11-23 22:46:47.366967 W | telemetry: failed to set telemetry key "rook/cluster/mon/stretch/enabled". failed to set "rook/cluster/mon/stretch/enabled" in the mon config-key store. output: Error EACCES: access denied: exit status 13
2022-11-23 22:46:48.006522 W | telemetry: failed to set telemetry key "rook/cluster/storage/device-set/count/total". failed to set "rook/cluster/storage/device-set/count/total" in the mon config-key store. output: Error EACCES: access denied: exit status 13
2022-11-23 22:46:48.653466 W | telemetry: failed to set telemetry key "rook/cluster/storage/device-set/count/portable". failed to set "rook/cluster/storage/device-set/count/portable" in the mon config-key store. output: Error EACCES: access denied: exit status 13
2022-11-23 22:46:49.307169 W | telemetry: failed to set telemetry key "rook/cluster/storage/device-set/count/non-portable". failed to set "rook/cluster/storage/device-set/count/non-portable" in the mon config-key store. output: Error EACCES: access denied: exit status 13
2022-11-23 22:46:49.948165 W | telemetry: failed to set telemetry key "rook/cluster/network/provider". failed to set "rook/cluster/network/provider" in the mon config-key store. output: Error EACCES: access denied: exit status 13
2022-11-23 22:46:50.580229 W | telemetry: failed to set telemetry key "rook/cluster/external-mode". failed to set "rook/cluster/external-mode" in the mon config-key store. output: Error EACCES: access denied: exit status 13
2022-11-23 22:47:27.477825 I | op-mon: new external mon "nuc9036" found: 172.17.40.60:6789, adding it
2022-11-23 22:47:27.477866 I | op-mon: new external mon "nuc9038" found: 172.17.40.62:6789, adding it
2022-11-23 22:47:27.477878 I | op-mon: new external mon "nuc9039" found: 172.17.40.56:6789, adding it
2022-11-23 22:47:27.477889 I | op-mon: new external mon "nuc9037" found: 172.17.40.61:6789, adding it
2022-11-23 22:47:27.509572 I | op-mon: saved mon endpoints to config map map[csi-cluster-config-json:[{"clusterID":"rook-ceph-external","monitors":["172.17.40.62:6789","172.17.40.56:6789","172.17.40.61:6789","172.17.40.58:6789","172.17.40.60:6789"],"namespace":""}] data:nuc9036=172.17.40.60:6789,nuc9038=172.17.40.62:6789,nuc9039=172.17.40.56:6789,nuc9037=172.17.40.61:6789,nuc9034=172.17.40.58:6789 mapping:{"node":{}} maxMonId:2 outOfQuorum:]
2022-11-23 22:47:27.544544 I | cephclient: writing config file /var/lib/rook/rook-ceph-external/rook-ceph-external.config
2022-11-23 22:47:27.544806 I | cephclient: generated admin config in /var/lib/rook/rook-ceph-external