Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failure to resize PVC after CHI recreation #1263

Closed
baznikin opened this issue Oct 27, 2023 · 5 comments
Closed

Failure to resize PVC after CHI recreation #1263

baznikin opened this issue Oct 27, 2023 · 5 comments

Comments

@baznikin
Copy link

I delete CHI and recreate it; after I tried to resize PVC but it's not happens; in operator log error

Operation cannot be fulfilled on persistentvolumeclaims "data-volume-template-chi-clickhouse-default-0-0-0": the object has been modified; please apply your changes to the latest version and try again
With clean setup of (no PVC exists at time of CHI creation) PVC resized as intendent.

Operator version: 0.21.3

Steps to reproduce:

  1. create CHI
  2. delete CHI (to check data do not deleted)
  3. recreate CHI
  4. resize PVC

Chi:

spec:
  configuration:
    clusters:
    - layout:
        replicasCount: 1
        shardsCount: 1
      name: default
      schemaPolicy:
        replica: All
        shard: All
  defaults:
    storageManagement:
      provisioner: Operator
      reclaimPolicy: Retain
    templates:
      dataVolumeClaimTemplate: data-volume-template
      serviceTemplate: default-service-template
  templates:
    volumeClaimTemplates:
    - name: data-volume-template
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 15G
  useTemplates:
  - name: secure-svc

Logs of failed state:

I1027 16:16:32.811467       1 worker.go:521] ActionPlan start---------------------------------------------:
AP item start -------------------------
modified spec items: 4
ap item path [0]:'.Templates.VolumeClaimTemplates[0].Spec.Resources.Requests["storage"].s'
ap item value[0]:'"15G"'
ap item path [1]:'.TaskID'
ap item value[1]:'"e321266b-7707-47ba-9a19-eac2b25e103e"'
ap item path [2]:'.Templates.VolumeClaimTemplates[0].Spec.Resources.Requests["storage"].i.value'
ap item value[2]:'15'
ap item path [3]:'.Templates.VolumeClaimTemplates[0].Spec.Resources.Requests["storage"].i.scale'
ap item value[3]:'9'
AP item end -------------------------
ActionPlan end---------------------------------------------
I1027 16:16:32.811573       1 worker-chi-reconciler.go:69] reconcileCHI():clickhouse/clickhouse/e321266b-7707-47ba-9a19-eac2b25e103e:ActionPlan has actions - continue reconcile
I1027 16:16:32.856488       1 worker.go:599] markReconcileStart():clickhouse/clickhouse/e321266b-7707-47ba-9a19-eac2b25e103e:reconcile started, task id: e321266b-7707-47ba-9a19-eac2b25e103e
I1027 16:16:32.977231       1 worker.go:744] FOUND host: ns:clickhouse|chi:clickhouse|clu:default|sha:0|rep:0|host:0-0
I1027 16:16:32.977308       1 worker.go:772] RemoteServersGeneratorOptions: exclude hosts: [], attributes: status: , add: true, remove: false, modify: false, found: false, excluded host(s): 
I1027 16:16:32.987385       1 worker.go:1091] updateConfigMap():clickhouse/clickhouse/e321266b-7707-47ba-9a19-eac2b25e103e:Update ConfigMap clickhouse/chi-clickhouse-common-configd
I1027 16:16:33.395964       1 worker.go:1091] updateConfigMap():clickhouse/clickhouse/e321266b-7707-47ba-9a19-eac2b25e103e:Update ConfigMap clickhouse/chi-clickhouse-common-usersd
I1027 16:16:33.827158       1 creator.go:113] CreateServiceCluster():clickhouse/clickhouse/e321266b-7707-47ba-9a19-eac2b25e103e:clickhouse/cluster-clickhouse-default
I1027 16:16:33.840218       1 worker-chi-reconciler.go:566] PDB updated clickhouse/clickhouse-default
I1027 16:16:33.840388       1 creator.go:136] CreateServiceShard():clickhouse/clickhouse/e321266b-7707-47ba-9a19-eac2b25e103e:clickhouse/shard-clickhouse-default-0
I1027 16:16:33.840498       1 worker-chi-reconciler.go:498] reconcileHost():Reconcile Host 0-0 started
I1027 16:16:34.226825       1 creator.go:593] getPodTemplate():clickhouse/clickhouse/e321266b-7707-47ba-9a19-eac2b25e103e:statefulSet chi-clickhouse-default-0-0 use default generated template
I1027 16:16:34.230340       1 worker.go:1279] getStatefulSetStatus():clickhouse/chi-clickhouse-default-0-0:cur and new StatefulSets ARE EQUAL based on labels. No StatefulSet reconcile is required for: clickhouse/chi-clickhouse-default-0-0
I1027 16:16:34.230720       1 worker.go:134] shouldForceRestartHost():Force restart is not required. Host: 0-0
I1027 16:16:34.230942       1 worker.go:978] shouldExcludeHost():The same host would not be updated host 0 shard 0 cluster default
I1027 16:16:34.398517       1 worker.go:1091] updateConfigMap():clickhouse/clickhouse/e321266b-7707-47ba-9a19-eac2b25e103e:Update ConfigMap clickhouse/chi-clickhouse-deploy-confd-default-0-0
E1027 16:16:35.399970       1 creator.go:175] updatePersistentVolumeClaim():clickhouse/data-volume-template-chi-clickhouse-default-0-0-0:unable to Update PVC err: Operation cannot be fulfilled on persistentvolumeclaims "data-volume-template-chi-clickhouse-default-0-0-0": the object has been modified; please apply your changes to the latest version and try again
E1027 16:16:35.400026       1 worker-chi-reconciler.go:802] func1():ERROR unable to reconcile PVC(clickhouse/data-volume-template-chi-clickhouse-default-0-0-0) err: Operation cannot be fulfilled on persistentvolumeclaims "data-volume-template-chi-clickhouse-default-0-0-0": the object has been modified; please apply your changes to the latest version and try again
I1027 16:16:35.400105       1 cluster.go:84] Run query on: chi-clickhouse-default-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-default-0-0.clickhouse.svc.cluster.local]
I1027 16:16:35.416949       1 worker-chi-reconciler.go:292] reconcileHostStatefulSet():Reconcile host 0-0. ClickHouse version: 23.9.2.56
I1027 16:16:35.417116       1 worker.go:134] shouldForceRestartHost():Force restart is not required. Host: 0-0
I1027 16:16:35.417135       1 worker-chi-reconciler.go:304] reconcileHostStatefulSet():Reconcile host 0-0. Reconcile StatefulSet
I1027 16:16:35.417278       1 creator.go:593] getPodTemplate():clickhouse/clickhouse/e321266b-7707-47ba-9a19-eac2b25e103e:statefulSet chi-clickhouse-default-0-0 use default generated template
I1027 16:16:35.418046       1 worker.go:1279] getStatefulSetStatus():clickhouse/chi-clickhouse-default-0-0:cur and new StatefulSets ARE EQUAL based on labels. No StatefulSet reconcile is required for: clickhouse/chi-clickhouse-default-0-0
E1027 16:16:35.996637       1 creator.go:175] updatePersistentVolumeClaim():clickhouse/data-volume-template-chi-clickhouse-default-0-0-0:unable to Update PVC err: Operation cannot be fulfilled on persistentvolumeclaims "data-volume-template-chi-clickhouse-default-0-0-0": the object has been modified; please apply your changes to the latest version and try again
E1027 16:16:35.996688       1 worker-chi-reconciler.go:802] func1():ERROR unable to reconcile PVC(clickhouse/data-volume-template-chi-clickhouse-default-0-0-0) err: Operation cannot be fulfilled on persistentvolumeclaims "data-volume-template-chi-clickhouse-default-0-0-0": the object has been modified; please apply your changes to the latest version and try again

Clean install logs:

I1027 16:37:54.553220       1 controller.go:566] ENQUEUE new ReconcileCHI cmd=update for clickhouse/clickhouse
I1027 16:37:54.553329       1 worker.go:381] worker.go:381:updateCHI():start:clickhouse/clickhouse
I1027 16:37:54.553363       1 worker.go:423] Operator IPs. Previous: 10.244.0.132 Cur: 10.244.0.132
I1027 16:37:54.553370       1 worker.go:432] Operator IPs are the same. It is restart on the same IP
I1027 16:37:54.553378       1 worker.go:440] Operator is not just started. May not be clean restart
I1027 16:37:54.555763       1 worker-chi-reconciler.go:51] worker-chi-reconciler.go:51:reconcileCHI():start:clickhouse/clickhouse
I1027 16:37:54.555804       1 worker-chi-reconciler.go:55] reconcileCHI():clickhouse/clickhouse:has ancestor, use it as a base for reconcile
I1027 16:37:54.555834       1 normalizer.go:137] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Found auto-templates num: 2
I1027 16:37:54.555848       1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Adding auto-template to list of possible templates: clickhouse/no-logs 
I1027 16:37:54.555860       1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Adding auto-template to list of possible templates: clickhouse/secure-ssl 
I1027 16:37:54.555901       1 normalizer.go:149] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Found manual-templates num: 1
I1027 16:37:54.555921       1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Apply template: clickhouse/no-logs. Selector: map[] matches labels: map[]
I1027 16:37:54.555951       1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Apply template: clickhouse/secure-ssl. Selector: map[] matches labels: map[]
I1027 16:37:54.555974       1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Apply template: /secure-svc. Selector: map[] matches labels: map[]
I1027 16:37:54.555997       1 normalizer.go:202] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Used templates count: 3
I1027 16:37:54.567241       1 worker.go:291] clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:IPs of the CHI clickhouse/clickhouse: [10.244.1.54]
I1027 16:37:54.567300       1 normalizer.go:137] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Found auto-templates num: 2
I1027 16:37:54.567317       1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Adding auto-template to list of possible templates: clickhouse/no-logs 
I1027 16:37:54.567332       1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Adding auto-template to list of possible templates: clickhouse/secure-ssl 
I1027 16:37:54.567359       1 normalizer.go:149] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Found manual-templates num: 1
I1027 16:37:54.567381       1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Apply template: clickhouse/no-logs. Selector: map[] matches labels: map[]
I1027 16:37:54.567409       1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Apply template: clickhouse/secure-ssl. Selector: map[] matches labels: map[]
I1027 16:37:54.567441       1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Apply template: /secure-svc. Selector: map[] matches labels: map[]
I1027 16:37:54.567466       1 normalizer.go:202] applyCHITemplates():clickhouse/clickhouse/1d0218be-6055-4a6b-94a7-8729b399a6c1:Used templates count: 3
I1027 16:37:54.568557       1 normalizer.go:137] applyCHITemplates():clickhouse/clickhouse:Found auto-templates num: 2
I1027 16:37:54.568577       1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse:Adding auto-template to list of possible templates: clickhouse/no-logs 
I1027 16:37:54.568596       1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse:Adding auto-template to list of possible templates: clickhouse/secure-ssl 
I1027 16:37:54.568609       1 normalizer.go:149] applyCHITemplates():clickhouse/clickhouse:Found manual-templates num: 1
I1027 16:37:54.568624       1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: clickhouse/no-logs. Selector: map[] matches labels: map[]
I1027 16:37:54.568648       1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: clickhouse/secure-ssl. Selector: map[] matches labels: map[]
I1027 16:37:54.568678       1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: /secure-svc. Selector: map[] matches labels: map[]
I1027 16:37:54.568700       1 normalizer.go:202] applyCHITemplates():clickhouse/clickhouse:Used templates count: 3
I1027 16:37:54.576822       1 worker.go:291] clickhouse/clickhouse/e8060938-721d-4df0-9685-1aeb61f7825e:IPs of the CHI clickhouse/clickhouse: [10.244.1.54]
I1027 16:37:54.576933       1 normalizer.go:137] applyCHITemplates():clickhouse/clickhouse:Found auto-templates num: 2
I1027 16:37:54.577014       1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse:Adding auto-template to list of possible templates: clickhouse/no-logs 
I1027 16:37:54.577202       1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse:Adding auto-template to list of possible templates: clickhouse/secure-ssl 
I1027 16:37:54.577231       1 normalizer.go:149] applyCHITemplates():clickhouse/clickhouse:Found manual-templates num: 1
I1027 16:37:54.577283       1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: clickhouse/no-logs. Selector: map[] matches labels: map[]
I1027 16:37:54.577447       1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: clickhouse/secure-ssl. Selector: map[] matches labels: map[]
I1027 16:37:54.577590       1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: /secure-svc. Selector: map[] matches labels: map[]
I1027 16:37:54.577721       1 normalizer.go:202] applyCHITemplates():clickhouse/clickhouse:Used templates count: 3
I1027 16:37:54.583181       1 worker.go:521] ActionPlan start---------------------------------------------:
AP item start -------------------------
modified spec items: 3
ap item path [0]:'.TaskID'
ap item value[0]:'"c14edf04-9813-4f56-8531-b93831dbbb04"'
ap item path [1]:'.Templates.VolumeClaimTemplates[0].Spec.Resources.Requests["storage"].i.value'
ap item value[1]:'15'
ap item path [2]:'.Templates.VolumeClaimTemplates[0].Spec.Resources.Requests["storage"].s'
ap item value[2]:'"15G"'
AP item end -------------------------
ActionPlan end---------------------------------------------
I1027 16:37:54.583332       1 worker-chi-reconciler.go:69] reconcileCHI():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:ActionPlan has actions - continue reconcile
I1027 16:37:54.617555       1 worker.go:599] markReconcileStart():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:reconcile started, task id: c14edf04-9813-4f56-8531-b93831dbbb04
I1027 16:37:54.709974       1 worker.go:744] FOUND host: ns:clickhouse|chi:clickhouse|clu:default|sha:0|rep:0|host:0-0
I1027 16:37:54.710036       1 worker.go:772] RemoteServersGeneratorOptions: exclude hosts: [], attributes: status: , add: true, remove: false, modify: false, found: false, excluded host(s): 
I1027 16:37:54.719919       1 worker.go:1091] updateConfigMap():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:Update ConfigMap clickhouse/chi-clickhouse-common-configd
I1027 16:37:55.172462       1 worker.go:1091] updateConfigMap():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:Update ConfigMap clickhouse/chi-clickhouse-common-usersd
I1027 16:37:55.600303       1 creator.go:113] CreateServiceCluster():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:clickhouse/cluster-clickhouse-default
I1027 16:37:55.616927       1 worker-chi-reconciler.go:566] PDB updated clickhouse/clickhouse-default
I1027 16:37:55.617044       1 creator.go:136] CreateServiceShard():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:clickhouse/shard-clickhouse-default-0
I1027 16:37:55.617104       1 worker-chi-reconciler.go:498] reconcileHost():Reconcile Host 0-0 started
I1027 16:37:56.005130       1 creator.go:593] getPodTemplate():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:statefulSet chi-clickhouse-default-0-0 use default generated template
I1027 16:37:56.006800       1 worker.go:1279] getStatefulSetStatus():clickhouse/chi-clickhouse-default-0-0:cur and new StatefulSets ARE EQUAL based on labels. No StatefulSet reconcile is required for: clickhouse/chi-clickhouse-default-0-0
I1027 16:37:56.007308       1 worker.go:134] shouldForceRestartHost():Force restart is not required. Host: 0-0
I1027 16:37:56.007337       1 worker.go:978] shouldExcludeHost():The same host would not be updated host 0 shard 0 cluster default
I1027 16:37:56.169120       1 worker.go:1091] updateConfigMap():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:Update ConfigMap clickhouse/chi-clickhouse-deploy-confd-default-0-0
I1027 16:37:57.184749       1 cluster.go:84] Run query on: chi-clickhouse-default-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-default-0-0.clickhouse.svc.cluster.local]
I1027 16:37:57.200861       1 worker-chi-reconciler.go:292] reconcileHostStatefulSet():Reconcile host 0-0. ClickHouse version: 23.9.2.56
I1027 16:37:57.201167       1 worker.go:134] shouldForceRestartHost():Force restart is not required. Host: 0-0
I1027 16:37:57.201222       1 worker-chi-reconciler.go:304] reconcileHostStatefulSet():Reconcile host 0-0. Reconcile StatefulSet
I1027 16:37:57.201375       1 creator.go:593] getPodTemplate():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:statefulSet chi-clickhouse-default-0-0 use default generated template
I1027 16:37:57.202535       1 worker.go:1279] getStatefulSetStatus():clickhouse/chi-clickhouse-default-0-0:cur and new StatefulSets ARE EQUAL based on labels. No StatefulSet reconcile is required for: clickhouse/chi-clickhouse-default-0-0
I1027 16:37:57.772604       1 creator.go:160] CreateServiceHost():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:clickhouse/chi-clickhouse-default-0-0 for Set chi-clickhouse-default-0-0
I1027 16:37:57.968558       1 worker.go:1203] updateService():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:Update Service clickhouse/chi-clickhouse-default-0-0
I1027 16:37:58.397772       1 worker.go:802] migrateTables():No need to add tables on host 0 to shard 0 in cluster default
I1027 16:37:58.397868       1 cluster.go:84] Run query on: chi-clickhouse-default-0-0.clickhouse.svc.cluster.local of [chi-clickhouse-default-0-0.clickhouse.svc.cluster.local]
I1027 16:37:58.405791       1 worker.go:887] includeHost():Include into cluster host 0 shard 0 cluster default
I1027 16:37:58.405821       1 worker.go:772] RemoteServersGeneratorOptions: exclude hosts: [], attributes: status: , add: true, remove: false, modify: false, found: false, excluded host(s): 
I1027 16:37:58.570730       1 worker.go:1091] updateConfigMap():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:Update ConfigMap clickhouse/chi-clickhouse-common-configd
I1027 16:37:59.168727       1 worker-chi-reconciler.go:537] reconcileHost():Reconcile Host 0-0 completed. ClickHouse version running: 23.9.2.56
I1027 16:37:59.603804       1 worker-chi-reconciler.go:553] reconcileHost():ProgressHostsCompleted: 1 of 1
I1027 16:38:00.000539       1 creator.go:58] CreateServiceCHI():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:clickhouse/clickhouse-clickhouse
I1027 16:38:00.168887       1 worker.go:1203] updateService():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:Update Service clickhouse/clickhouse-clickhouse
I1027 16:38:00.771738       1 worker.go:1091] updateConfigMap():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:Update ConfigMap clickhouse/chi-clickhouse-common-configd
I1027 16:38:01.203713       1 worker-chi-reconciler.go:103] reconcileCHI():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:remove items scheduled for deletion
I1027 16:38:01.605728       1 worker-deleter.go:37] clean():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:Failed to reconcile objects:
I1027 16:38:01.605769       1 worker-deleter.go:38] clean():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:Reconciled objects:
Service: clickhouse/chi-clickhouse-default-0-0
Service: clickhouse/clickhouse-clickhouse
ConfigMap: clickhouse/chi-clickhouse-common-usersd
ConfigMap: clickhouse/chi-clickhouse-deploy-confd-default-0-0
ConfigMap: clickhouse/chi-clickhouse-common-configd
PDB: clickhouse/clickhouse-default
PVC: clickhouse/data-volume-template-chi-clickhouse-default-0-0-0
StatefulSet: clickhouse/chi-clickhouse-default-0-0
I1027 16:38:02.373859       1 worker-deleter.go:41] clean():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:Existing objects:
StatefulSet: clickhouse/chi-clickhouse-default-0-0
ConfigMap: clickhouse/chi-clickhouse-common-configd
ConfigMap: clickhouse/chi-clickhouse-common-usersd
ConfigMap: clickhouse/chi-clickhouse-deploy-confd-default-0-0
Service: clickhouse/chi-clickhouse-default-0-0
Service: clickhouse/clickhouse-clickhouse
PVC: clickhouse/data-volume-template-chi-clickhouse-default-0-0-0
PDB: clickhouse/clickhouse-default
I1027 16:38:02.373935       1 worker-deleter.go:43] clean():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:Non-reconciled objects:
I1027 16:38:02.373955       1 worker-deleter.go:53] clean():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:remove items scheduled for deletion
I1027 16:38:02.799541       1 worker-deleter.go:64] worker-deleter.go:64:dropReplicas():start:clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:drop replicas based on AP
I1027 16:38:02.799584       1 worker-deleter.go:81] worker-deleter.go:81:dropReplicas():end:clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:processed replicas: 0
I1027 16:38:02.799632       1 worker.go:575] includeStopped():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:add CHI to monitoring
I1027 16:38:03.204465       1 controller.go:610] OK update watch (clickhouse/clickhouse): {"namespace":"clickhouse","name":"clickhouse","clusters":[{"name":"default","hosts":[{"name":"0-0","hostname":"chi-clickhouse-default-0-0.clickhouse.svc.cluster.local","tcpPort":9000,"tlsPort":9440,"httpPort":8123,"httpsPort":8443}]}]}
I1027 16:38:03.213330       1 worker.go:542] clickhouse/clickhouse:all IP addresses are in place
I1027 16:38:03.225662       1 normalizer.go:137] applyCHITemplates():clickhouse/clickhouse:Found auto-templates num: 2
I1027 16:38:03.225700       1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse:Adding auto-template to list of possible templates: clickhouse/no-logs 
I1027 16:38:03.225713       1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse:Adding auto-template to list of possible templates: clickhouse/secure-ssl 
I1027 16:38:03.225726       1 normalizer.go:149] applyCHITemplates():clickhouse/clickhouse:Found manual-templates num: 1
I1027 16:38:03.226347       1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: clickhouse/no-logs. Selector: map[] matches labels: map[]
I1027 16:38:03.226492       1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: clickhouse/secure-ssl. Selector: map[] matches labels: map[]
I1027 16:38:03.226595       1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: /secure-svc. Selector: map[] matches labels: map[]
I1027 16:38:03.226721       1 normalizer.go:202] applyCHITemplates():clickhouse/clickhouse:Used templates count: 3
I1027 16:38:03.367766       1 worker.go:613] clickhouse/clickhouse/833af56c-e246-4967-a07e-6ee2090469b4:IPs of the CHI-2 [10.244.1.54]
I1027 16:38:03.375655       1 normalizer.go:137] applyCHITemplates():clickhouse/clickhouse:Found auto-templates num: 2
I1027 16:38:03.375690       1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse:Adding auto-template to list of possible templates: clickhouse/no-logs 
I1027 16:38:03.375701       1 normalizer.go:139] applyCHITemplates():clickhouse/clickhouse:Adding auto-template to list of possible templates: clickhouse/secure-ssl 
I1027 16:38:03.375713       1 normalizer.go:149] applyCHITemplates():clickhouse/clickhouse:Found manual-templates num: 1
I1027 16:38:03.375904       1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: clickhouse/no-logs. Selector: map[] matches labels: map[]
I1027 16:38:03.375940       1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: clickhouse/secure-ssl. Selector: map[] matches labels: map[]
I1027 16:38:03.375964       1 normalizer.go:181] applyCHITemplates():clickhouse/clickhouse:Apply template: /secure-svc. Selector: map[] matches labels: map[]
I1027 16:38:03.375993       1 normalizer.go:202] applyCHITemplates():clickhouse/clickhouse:Used templates count: 3
I1027 16:38:03.377358       1 worker.go:617] clickhouse/clickhouse/33c5e95d-4f82-4f11-be40-7c0387663829:Update users IPS-2
I1027 16:38:03.597744       1 worker.go:638] finalizeReconcileAndMarkCompleted():clickhouse/clickhouse/c14edf04-9813-4f56-8531-b93831dbbb04:reconcile completed successfully, task id: c14edf04-9813-4f56-8531-b93831dbbb04
I1027 16:38:03.993153       1 worker-chi-reconciler.go:111] worker-chi-reconciler.go:52:reconcileCHI():end:clickhouse/clickhouse
I1027 16:38:03.993190       1 worker.go:416] worker.go:382:updateCHI():end:clickhouse/clickhouse
@prashant-shahi
Copy link

+1

@prashant-shahi
Copy link

@baznikin Were you able to resolve the issue?

@Slach
Copy link
Collaborator

Slach commented Dec 16, 2023

@sunsingerus did we look to this issue?

@jkrajniak
Copy link

Just to add, I have used a recent version of the operator from master. The change of the requested size is read without SI suffix. So, the change from 300Gi -> 350Gi results in some strange storage request size of 350.
image

@jkrajniak jkrajniak mentioned this issue Apr 12, 2024
Open
@alex-zaitsev
Copy link
Member

Let's close this in favor of #1395

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@Slach @jkrajniak @baznikin @prashant-shahi @alex-zaitsev