failed to provision volume: context deadline exceeded #8727

stuartpb · 2021-09-16T01:59:17Z

I have two PVCs stuck waiting in the Provisioning state on my cluster. Here are the initial logs from when the rdbplugin-provisioner pod started - the first 9 failures or so coincide with the pool not being available (one of the nodes in my cluster went down during initialization, which may have delayed the pool for a while), but after that, attempts to provision start to fail with "rpc error: code = DeadlineExceeded desc = context deadline exceeded", and then after that, they start to fail with "an operation with the given Volume ID already exists":

I0916 00:40:45.633801       1 csi-provisioner.go:138] Version: v2.2.2
I0916 00:40:45.634008       1 csi-provisioner.go:161] Building kube configs for running in cluster...
I0916 00:40:48.837376       1 common.go:111] Probing CSI driver for readiness
I0916 00:40:48.850213       1 csi-provisioner.go:284] CSI driver does not support PUBLISH_UNPUBLISH_VOLUME, not watching VolumeAttachments
I0916 00:40:48.858817       1 leaderelection.go:243] attempting to acquire leader lease rook-ceph/rook-ceph-cephfs-csi-ceph-com...
I0916 00:40:48.882976       1 leaderelection.go:253] successfully acquired lease rook-ceph/rook-ceph-cephfs-csi-ceph-com
I0916 00:40:48.986593       1 controller.go:835] Starting provisioner controller rook-ceph.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-75d5d845f4-pzwfm_8e3eaf0e-6d9f-43d6-a80a-304d88607a04!
I0916 00:40:48.986630       1 clone_controller.go:66] Starting CloningProtection controller
I0916 00:40:48.986983       1 clone_controller.go:84] Started CloningProtection controller
I0916 00:40:48.986684       1 volume_store.go:97] Starting save volume queue
I0916 00:40:49.089897       1 controller.go:884] Started provisioner controller rook-ceph.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-75d5d845f4-pzwfm_8e3eaf0e-6d9f-43d6-a80a-304d88607a04!
[stuart@stushiba ~]$ kubectl -n rook-ceph logs csi-rbdplugin-provisioner-5885bc875-
csi-rbdplugin-provisioner-5885bc875-gx2d7  csi-rbdplugin-provisioner-5885bc875-l52qj  
[stuart@stushiba ~]$ kubectl -n rook-ceph logs csi-rbdplugin-provisioner-5885bc875-gx2d7 
error: a container name must be specified for pod csi-rbdplugin-provisioner-5885bc875-gx2d7, choose one of: [csi-provisioner csi-resizer csi-attacher csi-snapshotter csi-rbdplugin liveness-prometheus]
[stuart@stushiba ~]$ kubectl -n rook-ceph logs csi-rbdplugin-provisioner-5885bc875-gx2d7 csi-provisioner 
I0916 00:40:39.662855       1 csi-provisioner.go:138] Version: v2.2.2
I0916 00:40:39.663108       1 csi-provisioner.go:161] Building kube configs for running in cluster...
I0916 00:40:46.909201       1 common.go:111] Probing CSI driver for readiness
I0916 00:40:46.928922       1 csi-provisioner.go:284] CSI driver does not support PUBLISH_UNPUBLISH_VOLUME, not watching VolumeAttachments
I0916 00:40:46.933807       1 leaderelection.go:243] attempting to acquire leader lease rook-ceph/rook-ceph-rbd-csi-ceph-com...
I0916 00:40:46.960213       1 leaderelection.go:253] successfully acquired lease rook-ceph/rook-ceph-rbd-csi-ceph-com
I0916 00:40:47.063718       1 clone_controller.go:66] Starting CloningProtection controller
I0916 00:40:47.063940       1 clone_controller.go:84] Started CloningProtection controller
I0916 00:40:47.063955       1 volume_store.go:97] Starting save volume queue
I0916 00:40:47.063695       1 controller.go:835] Starting provisioner controller rook-ceph.rbd.csi.ceph.com_csi-rbdplugin-provisioner-5885bc875-gx2d7_175f1a9d-e681-4cab-b5f5-fd767abae89a!
I0916 00:40:47.164555       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 00:40:47.164876       1 controller.go:884] Started provisioner controller rook-ceph.rbd.csi.ceph.com_csi-rbdplugin-provisioner-5885bc875-gx2d7_175f1a9d-e681-4cab-b5f5-fd767abae89a!
I0916 00:40:47.164805       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 00:40:47.167828       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
I0916 00:40:47.168886       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 00:40:47.171184       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 0
I0916 00:40:47.172970       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
E0916 00:40:47.173158       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
W0916 00:40:47.173893       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 0
I0916 00:40:47.173973       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
E0916 00:40:47.174063       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:40:47.673892       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 00:40:47.674791       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 00:40:47.675096       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
I0916 00:40:47.675420       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 00:40:47.692052       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 1
E0916 00:40:47.692376       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:40:47.692136       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
W0916 00:40:47.772068       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 1
I0916 00:40:47.772144       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
E0916 00:40:47.772201       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:40:48.692903       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 00:40:48.693689       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
W0916 00:40:48.752587       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 2
I0916 00:40:48.752664       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
E0916 00:40:48.752665       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:40:48.773274       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 00:40:48.773500       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 00:40:49.001240       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 2
E0916 00:40:49.001335       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:40:49.001406       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:40:50.753826       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 00:40:50.754183       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
W0916 00:40:50.772469       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 3
E0916 00:40:50.772569       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:40:50.772647       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:40:51.002328       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 00:40:51.002615       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 00:40:51.010727       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 3
E0916 00:40:51.010826       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:40:51.010812       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:40:54.773448       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 00:40:54.773794       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
W0916 00:40:54.803125       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 4
E0916 00:40:54.803227       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:40:54.803317       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:40:55.012086       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 00:40:55.012385       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 00:40:55.025424       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 4
E0916 00:40:55.025857       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:40:55.025575       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:41:02.804229       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 00:41:02.804619       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
W0916 00:41:02.819791       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 5
E0916 00:41:02.819901       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:41:02.819893       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:41:03.026269       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 00:41:03.026685       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 00:41:03.036172       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 5
E0916 00:41:03.036282       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:41:03.036272       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:41:18.821219       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 00:41:18.821622       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
W0916 00:41:18.830322       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 6
E0916 00:41:18.830463       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:41:18.830426       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:41:19.037460       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 00:41:19.038001       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 00:41:19.043702       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 6
E0916 00:41:19.044300       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:41:19.043811       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:41:50.831676       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 00:41:50.832024       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
W0916 00:41:50.848565       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 7
E0916 00:41:50.848704       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:41:50.848783       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:41:51.044857       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 00:41:51.045203       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 00:41:51.057896       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 7
E0916 00:41:51.058005       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:41:51.057996       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": error getting secret rook-csi-rbd-provisioner in namespace rook-ceph: secrets "rook-csi-rbd-provisioner" not found
I0916 00:42:54.850128       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 00:42:54.850515       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
W0916 00:42:54.917290       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 8
E0916 00:42:54.917379       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": rpc error: code = Internal desc = pool not found: pool (double) not found in Ceph cluster
I0916 00:42:54.917438       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Internal desc = pool not found: pool (double) not found in Ceph cluster
I0916 00:42:55.059002       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 00:42:55.059395       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 00:42:55.073841       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 8
E0916 00:42:55.073951       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": rpc error: code = Internal desc = pool not found: pool (double) not found in Ceph cluster
I0916 00:42:55.073965       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Internal desc = pool not found: pool (double) not found in Ceph cluster
I0916 00:45:02.918196       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 00:45:02.918541       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
W0916 00:45:02.943320       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 9
E0916 00:45:02.943408       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": rpc error: code = Internal desc = pool not found: pool (double) not found in Ceph cluster
I0916 00:45:02.944440       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Internal desc = pool not found: pool (double) not found in Ceph cluster
I0916 00:45:03.074580       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 00:45:03.076137       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 00:45:03.088549       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 9
E0916 00:45:03.088688       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": rpc error: code = Internal desc = pool not found: pool (double) not found in Ceph cluster
I0916 00:45:03.088750       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Internal desc = pool not found: pool (double) not found in Ceph cluster
I0916 00:49:18.944040       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 00:49:18.944260       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
I0916 00:49:19.089879       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 00:49:19.090183       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 00:51:48.959014       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 10
E0916 00:51:48.959443       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0916 00:51:48.959638       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = DeadlineExceeded desc = context deadline exceeded
W0916 00:51:49.106665       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 10
E0916 00:51:49.106782       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0916 00:51:49.106871       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0916 00:55:46.976374       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 00:55:46.976386       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 00:55:46.976676       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
I0916 00:55:46.976875       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
W0916 00:55:46.989771       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 11
E0916 00:55:46.989859       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 00:55:46.989858       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
W0916 00:55:46.990560       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 11
E0916 00:55:46.990626       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 00:55:46.990707       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 00:56:48.972405       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 00:56:48.974382       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
W0916 00:56:48.986338       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 12
I0916 00:56:48.986410       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
E0916 00:56:48.986451       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 00:56:49.107068       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 00:56:49.107412       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 00:56:49.136406       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 12
E0916 00:56:49.136518       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 00:56:49.136501       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:01:48.987762       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 01:01:48.988000       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
W0916 01:01:49.002829       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 13
E0916 01:01:49.002940       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 01:01:49.002902       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 01:01:49.137384       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 01:01:49.138553       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 01:01:49.151238       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 13
E0916 01:01:49.151342       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:01:49.151323       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:06:49.004171       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 01:06:49.004477       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
W0916 01:06:49.015550       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 14
E0916 01:06:49.015654       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 01:06:49.015637       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 01:06:49.152205       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 01:06:49.152637       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 01:06:49.162534       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 14
E0916 01:06:49.163059       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:06:49.163112       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:10:46.978833       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 01:10:46.980342       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 01:10:46.981492       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
I0916 01:10:46.981560       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
W0916 01:10:46.994588       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 15
E0916 01:10:46.994901       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
W0916 01:10:46.994588       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 15
E0916 01:10:46.995247       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:10:46.995343       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:10:46.995392       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 01:11:49.016939       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 01:11:49.017251       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
W0916 01:11:49.041967       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 16
E0916 01:11:49.042118       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 01:11:49.042096       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 01:11:49.163890       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 01:11:49.164275       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 01:11:49.178065       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 16
I0916 01:11:49.178173       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
E0916 01:11:49.178211       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:16:49.042519       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 01:16:49.042827       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
W0916 01:16:49.057818       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 17
E0916 01:16:49.057943       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 01:16:49.057925       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 01:16:49.179773       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 01:16:49.180113       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 01:16:49.189030       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 17
I0916 01:16:49.189074       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
E0916 01:16:49.189155       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:21:49.058733       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 01:21:49.059169       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
W0916 01:21:49.072491       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 18
E0916 01:21:49.072609       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 01:21:49.072585       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 01:21:49.189953       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 01:21:49.190251       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 01:21:49.224118       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 18
I0916 01:21:49.224222       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
E0916 01:21:49.224281       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:25:46.977507       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 01:25:46.977600       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 01:25:46.977726       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
I0916 01:25:46.977777       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 01:25:46.991503       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 19
E0916 01:25:46.991604       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 01:25:46.991598       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
W0916 01:25:46.991747       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 19
E0916 01:25:46.991813       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:25:46.991895       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:26:49.073866       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 01:26:49.074255       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
W0916 01:26:49.089761       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 20
E0916 01:26:49.089871       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 01:26:49.089851       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 01:26:49.225089       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 01:26:49.225437       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 01:26:49.236504       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 20
E0916 01:26:49.236613       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:26:49.236593       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:31:49.090388       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 01:31:49.091457       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
W0916 01:31:49.100403       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 21
E0916 01:31:49.100501       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 01:31:49.100488       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 01:31:49.237089       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 01:31:49.237447       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 01:31:49.248863       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 21
I0916 01:31:49.248953       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
E0916 01:31:49.248992       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:36:49.101599       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 01:36:49.101993       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
W0916 01:36:49.114200       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 22
E0916 01:36:49.114314       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 01:36:49.114295       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 01:36:49.250227       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 01:36:49.250636       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 01:36:49.262780       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 22
E0916 01:36:49.262904       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:36:49.262881       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:40:46.977764       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 01:40:46.977838       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 01:40:46.978062       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
I0916 01:40:46.978116       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 01:40:46.996330       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 23
E0916 01:40:46.996434       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:40:46.996420       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
W0916 01:40:46.996330       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 23
I0916 01:40:46.996481       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
E0916 01:40:46.996523       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 01:41:49.115515       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 01:41:49.115837       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
I0916 01:41:49.129544       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
W0916 01:41:49.129319       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 24
E0916 01:41:49.130601       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 01:41:49.263819       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 01:41:49.265430       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 01:41:49.279146       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 24
E0916 01:41:49.279549       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:41:49.279274       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists

After deleting and recreating the provisioner pod, I'm still seeing the same kind of error in the new one:

I0916 00:40:40.254641       1 csi-provisioner.go:138] Version: v2.2.2
I0916 00:40:40.254807       1 csi-provisioner.go:161] Building kube configs for running in cluster...
I0916 00:40:55.905664       1 common.go:111] Probing CSI driver for readiness
I0916 00:40:55.923184       1 csi-provisioner.go:284] CSI driver does not support PUBLISH_UNPUBLISH_VOLUME, not watching VolumeAttachments
I0916 00:40:55.929077       1 leaderelection.go:243] attempting to acquire leader lease rook-ceph/rook-ceph-rbd-csi-ceph-com...
I0916 01:51:11.203227       1 leaderelection.go:253] successfully acquired lease rook-ceph/rook-ceph-rbd-csi-ceph-com
I0916 01:51:11.404491       1 controller.go:835] Starting provisioner controller rook-ceph.rbd.csi.ceph.com_csi-rbdplugin-provisioner-5885bc875-l52qj_c7813d33-5940-4343-be15-6c7d822647da!
I0916 01:51:11.404584       1 clone_controller.go:66] Starting CloningProtection controller
I0916 01:51:11.404710       1 clone_controller.go:84] Started CloningProtection controller
I0916 01:51:11.404585       1 volume_store.go:97] Starting save volume queue
I0916 01:51:11.505236       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 01:51:11.505237       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 01:51:11.509115       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
I0916 01:51:11.505557       1 controller.go:884] Started provisioner controller rook-ceph.rbd.csi.ceph.com_csi-rbdplugin-provisioner-5885bc875-l52qj_c7813d33-5940-4343-be15-6c7d822647da!
I0916 01:51:11.509715       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
W0916 01:53:41.523544       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 0
I0916 01:53:41.523592       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0916 01:53:41.523684       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": rpc error: code = DeadlineExceeded desc = context deadline exceeded
W0916 01:53:41.523707       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 0
E0916 01:53:41.523804       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0916 01:53:41.523696       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = DeadlineExceeded desc = context deadline exceeded
I0916 01:53:42.024629       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 01:53:42.024629       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 01:53:42.025129       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
I0916 01:53:42.025851       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
W0916 01:53:42.188236       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 1
W0916 01:53:42.188286       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 1
E0916 01:53:42.188346       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
E0916 01:53:42.188368       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:53:42.188331       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 01:53:42.188992       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:53:43.189559       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 01:53:43.189568       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 01:53:43.189926       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
I0916 01:53:43.190010       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
W0916 01:53:43.204898       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 2
E0916 01:53:43.205278       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
W0916 01:53:43.204897       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 2
I0916 01:53:43.205043       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
E0916 01:53:43.205374       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:53:43.205398       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 01:53:45.206103       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 01:53:45.206130       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 01:53:45.206808       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
I0916 01:53:45.207615       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
W0916 01:53:45.220338       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 3
E0916 01:53:45.220433       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
W0916 01:53:45.220457       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 3
E0916 01:53:45.220519       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:53:45.220516       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
I0916 01:53:45.220573       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
I0916 01:53:49.221518       1 controller.go:1332] provision "pomerium/redis-data-pomerium-redis-master-0" class "throwaway": started
I0916 01:53:49.221519       1 controller.go:1332] provision "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0" class "throwaway": started
I0916 01:53:49.221912       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "pomerium/redis-data-pomerium-redis-master-0"
I0916 01:53:49.222280       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "prometheus/prometheus-kps-prometheus-db-prometheus-kps-prometheus-0"
W0916 01:53:49.249755       1 controller.go:958] Retrying syncing claim "56ed064b-300e-447c-b4a3-e822de61f762", failure 4
I0916 01:53:49.249834       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pomerium", Name:"redis-data-pomerium-redis-master-0", UID:"56ed064b-300e-447c-b4a3-e822de61f762", APIVersion:"v1", ResourceVersion:"8560", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
W0916 01:53:49.249755       1 controller.go:958] Retrying syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", failure 4
I0916 01:53:49.249926       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"prometheus", Name:"prometheus-kps-prometheus-db-prometheus-kps-prometheus-0", UID:"e446ed0b-03ce-436c-a5f2-c2b91c7ae7da", APIVersion:"v1", ResourceVersion:"5961", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists
E0916 01:53:49.249877       1 controller.go:981] error syncing claim "56ed064b-300e-447c-b4a3-e822de61f762": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-56ed064b-300e-447c-b4a3-e822de61f762 already exists
E0916 01:53:49.249972       1 controller.go:981] error syncing claim "e446ed0b-03ce-436c-a5f2-c2b91c7ae7da": failed to provision volume with StorageClass "throwaway": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e446ed0b-03ce-436c-a5f2-c2b91c7ae7da already exists

The text was updated successfully, but these errors were encountered:

Madhu-1 · 2021-09-16T02:04:55Z

@stuartpb can you please paste ceph -s output to check ceph cluster is healthy and known issue with ceph library which cephcsi uses to create parallel rbd images #8696 (comment)

Madhu-1 · 2021-09-16T02:05:28Z

can you try with the ceph-csi v3.3 image and see if the issue persists?

stuartpb · 2021-09-16T07:43:18Z

ceph -s:

  cluster:
    id:     cbd096aa-3596-4bdf-85ad-f4a9055b0fe7
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum a,b,c (age 4h)
    mgr: a(active, since 6h)
    osd: 3 osds: 3 up (since 5h), 3 in (since 6h)
    rgw: 1 daemon active (1 hosts, 1 zones)
 
  data:
    pools:   9 pools, 113 pgs
    objects: 388 objects, 29 KiB
    usage:   147 MiB used, 5.5 TiB / 5.5 TiB avail
    pgs:     113 active+clean
 
  io:
    client:   1.1 KiB/s rd, 85 B/s wr, 1 op/s rd, 0 op/s wr

Madhu-1 · 2021-09-16T07:44:40Z

can you please try workaround specified at #8696 (comment)

Madhu-1 · 2021-09-21T04:30:08Z

Closing as duplicate of #8696. please feel free to reopen if it's not.

stuartpb added the bug label Sep 16, 2021

Madhu-1 mentioned this issue Sep 20, 2021

failed to provision volume with StorageClass "ceph-block": an operation with the given Volume ID pvc-ID already exists #8749

Closed

Madhu-1 closed this as completed Sep 21, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

failed to provision volume: context deadline exceeded #8727

failed to provision volume: context deadline exceeded #8727

stuartpb commented Sep 16, 2021

Madhu-1 commented Sep 16, 2021

Madhu-1 commented Sep 16, 2021

stuartpb commented Sep 16, 2021

Madhu-1 commented Sep 16, 2021

Madhu-1 commented Sep 21, 2021

failed to provision volume: context deadline exceeded #8727

failed to provision volume: context deadline exceeded #8727

Comments

stuartpb commented Sep 16, 2021

Madhu-1 commented Sep 16, 2021

Madhu-1 commented Sep 16, 2021

stuartpb commented Sep 16, 2021

Madhu-1 commented Sep 16, 2021

Madhu-1 commented Sep 21, 2021