Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Schema isn't restored on replica after StatefullSet re-creation #1310

Closed
Dnnd opened this issue Jan 25, 2024 · 1 comment
Closed

Schema isn't restored on replica after StatefullSet re-creation #1310

Dnnd opened this issue Jan 25, 2024 · 1 comment
Assignees
@sunsingerus

Comments

@Dnnd
Copy link

Dnnd commented Jan 25, 2024

Description

I'm facing an issue with recovering data after replica loss. After StatefullSet and PVCs corresponding to a replica are deleted, and then re-created by manually specifying spec.taskID in the ClickHouseInstallation yaml file, the schema isn't restored on the re-created replica.

Setup

Versions

clickhouse-operator: 0.22.2
clickhouse-server: 23.8
clickhouse-keeper: 23.8-alpine

Steps to reproduce

  1. Assuming clickhouse-keeper is deployed to the "clickhouse-recovery" namespace, and "openebs-hostpath" storage class is used for PVC
  2. Deploy cluster with 2 replicas and 1 shard using the following config:
chi.yaml 
apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "recovery-test"
spec:
  defaults:
    templates:
      dataVolumeClaimTemplate: data-volume-template
      logVolumeClaimTemplate: log-volume-template
      podTemplate: clickhouse:23.8
  configuration:
    zookeeper:
      nodes:
        - host: clickhouse-keeper.clickhouse-recovery.svc
    clusters:
      - name: "ch-ha"
        layout:
          shardsCount: 1
          replicasCount: 3
  templates:
    volumeClaimTemplates:
      - name: data-volume-template
        spec:
          storageClassName: openebs-hostpath
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 5Gi
      - name: log-volume-template
        spec:
          storageClassName: openebs-hostpath
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1Gi

    podTemplates:
      - name: clickhouse:23.8
        spec:
          containers:
            - name: clickhouse-pod
              image: clickhouse/clickhouse-server:23.8
          resources:
            requests:
              memory: "4Gi"
              cpu: "1"
            limits:
              memory: "4Gi"
              cpu: "1"

Pods after deployment is :

NAME                                          READY   STATUS    RESTARTS   AGE
chi-recovery-test-ch-ha-0-0-0                 2/2     Running   0          26m
chi-recovery-test-ch-ha-0-1-0                 2/2     Running   0          45m
clickhouse-keeper-0                                 1/1     Running   0          27h
  1. Create a database, replicated table in the deployed cluster, and insert some data to it:
schema.sql 
create database if not exists test_db on cluster 'ch-ha' engine Atomic;

CREATE TABLE if not exists test_db.test_table_local ON CLUSTER '{cluster}' (
    `timestamp` DateTime CODEC(Delta(4), ZSTD(1)),
    `key` String CODEC(ZSTD(1)),
) ENGINE = ReplicatedMergeTree(
    '/clickhouse/{installation}/{cluster}/tables/{shard}/{database}/{table}',
    '{replica}'
) PARTITION BY toDate(timestamp)
ORDER BY (key, timestamp);


CREATE TABLE IF NOT EXISTS test_db.test_table on CLUSTER '{cluster}' AS test_db.test_table_local ENGINE = Distributed(
    '{cluster}',
    test_db,
    test_table_local,
    xxh3(key)
);

INSERT INTO test_db.test_table_local(key,timestamp)
VALUES ('key1', now()), ('key2', now());
  1. Delete StatefullSet and PVCs (to simulate an outage):
 kubectl -n clickhouse-recovery delete sts chi-recovery-test-ch-ha-0-0                                                                                                  
 kubectl -n clickhouse-recovery delete pvc data-volume-template-chi-recovery-test-ch-ha-0-0-0 log-volume-template-chi-recovery-test-ch-ha-0-0-0
  1. Execute on the running replica ('chi-recovery-test-ch-ha-0-1'):
 SYSTEM DROP REPLICA 'chi-recovery-test-ch-ha-0-0'
  1. Change spec.taskID in ClickHouseInstallation yaml file to some random string and apply it, forcing reconciliation
  2. Wait for the chi-recovery-test-ch-ha-0-0 replica to be re-created
  3. Check if test_db and test_table_local are present on chi-recovery-test-ch-ha-0-0 replica

Expected result

test_db and test_table_local are present on chi-recovery-test-ch-ha-0-0 replica, data in test_table_local is recovered from the live replica.

Actual result

Schema is not recreated, data isn't recovered
I've also observed these messages in the operator log:

I0125 08:55:09.956951       1 worker.go:898] migrateTables():No need to add tables on host 0 to shard 0 in cluster ch-ha
I0125 08:55:09.956964       1 worker.go:1038] includeHost():Include into cluster host 0 shard 0 cluster ch-ha

I0125 08:55:23.058351       1 cluster_schemer.go:238] HostCreateTables():Creating replicated objects at 0-1: [default test_db test_table test_table_local]
I0125 08:55:23.130474       1 cluster_schemer.go:245] HostCreateTables():Creating distributed objects at 0-1: [test_db]

It seems like instead of chi-recovery-test-ch-ha-0-0, operator tried to create schema on chi-recovery-test-ch-ha-0-1 ?....

Here is the full log:

Operator log 
I0125 08:54:41.905914       1 controller.go:570] ENQUEUE new ReconcileCHI cmd=update for clickhouse-recovery/recovery-test
I0125 08:54:41.905988       1 worker.go:424] worker.go:424:updateCHI():start:clickhouse-recovery/recovery-test/force-recovery-17-54
I0125 08:54:41.906025       1 worker.go:432] updateCHI():clickhouse-recovery/recovery-test/force-recovery-17-54:finalizer in place, proceed to reconcile cycle. CHI: clickhouse-recovery/recovery-test
I0125 08:54:41.906038       1 worker.go:469] Operator IPs to process CHI: recovery-test. Previous: 10.42.30.13 Cur: 10.42.30.13
I0125 08:54:41.906044       1 worker.go:478] Operator IPs are the same as on previous reconcile of the CHI: recovery-test
I0125 08:54:41.906050       1 worker.go:486] Operator is not just started. May not be clean restart
I0125 08:54:41.906733       1 worker.go:557] clickhouse-recovery/recovery-test:logCHI non-normalized yet (native) old start--------------------------------------------:
non-normalized yet (native) old
logCHI kind: ""
apiversion: ""
metadata:
    name: recovery-test
    generatename: ""
    namespace: clickhouse-recovery
    selflink: ""
    uid: 49ca13ec-137a-4890-99fd-bedcda704e3e
    resourceversion: "282625388"
    generation: 1
    creationtimestamp: "2024-01-25T08:39:15Z"
    deletiontimestamp: null
    deletiongraceperiodseconds: null
    labels: {}
    annotations:
        kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"clickhouse.altinity.com/v1","kind":"ClickHouseInstallation","metadata":{"annotations":{},"name":"recovery-test","namespace":"clickhouse-recovery"},"spec":{"configuration":{"clusters":[{"layout":{"replicasCount":2,"shardsCount":1},"name":"ch-ha"}],"zookeeper":{"nodes":[{"host":"clickhouse-keeper.clickhouse-recovery.svc"}]}},"defaults":{"templates":{"dataVolumeClaimTemplate":"data-volume-template","logVolumeClaimTemplate":"log-volume-template","podTemplate":"clickhouse:23.8"}},"templates":{"podTemplates":[{"name":"clickhouse:23.8","spec":{"containers":[{"image":"clickhouse/clickhouse-server:23.8","name":"clickhouse-pod"}],"resources":{"limits":{"cpu":"1","memory":"4Gi"},"requests":{"cpu":"1","memory":"4Gi"}}}}],"volumeClaimTemplates":[{"name":"data-volume-template","spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"5Gi"}},"storageClassName":"openebs-hostpath"}},{"name":"log-volume-template","spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}},"storageClassName":"openebs-hostpath"}}]}}}
    ownerreferences: []
    finalizers:
        - finalizer.clickhouseinstallation.altinity.com
    clustername: ""
    managedfields: []
spec:
    defaults:
        templates:
            podTemplate: clickhouse:23.8
            dataVolumeClaimTemplate: data-volume-template
            logVolumeClaimTemplate: log-volume-template
    configuration:
        zookeeper:
            nodes:
                - host: clickhouse-keeper.clickhouse-recovery.svc
        clusters:
            - name: ch-ha
              layout:
                shardsCount: 1
                replicasCount: 2
    templates:
        podTemplates:
            - name: clickhouse:23.8
              spec:
                volumes: []
                initcontainers: []
                containers:
                    - name: clickhouse-pod
                      image: clickhouse/clickhouse-server:23.8
                      command: []
                      args: []
                      workingdir: ""
                      ports: []
                      envfrom: []
                      env: []
                      resources:
                        limits: {}
                        requests: {}
                      volumemounts: []
                      volumedevices: []
                      livenessprobe: null
                      readinessprobe: null
                      startupprobe: null
                      lifecycle: null
                      terminationmessagepath: ""
                      terminationmessagepolicy: ""
                      imagepullpolicy: ""
                      securitycontext: null
                      stdin: false
                      stdinonce: false
                      tty: false
                ephemeralcontainers: []
                restartpolicy: ""
                terminationgraceperiodseconds: null
                activedeadlineseconds: null
                dnspolicy: ""
                nodeselector: {}
                serviceaccountname: ""
                deprecatedserviceaccount: ""
                automountserviceaccounttoken: null
                nodename: ""
                hostnetwork: false
                hostpid: false
                hostipc: false
                shareprocessnamespace: null
                securitycontext: null
                imagepullsecrets: []
                hostname: ""
                subdomain: ""
                affinity: null
                schedulername: ""
                tolerations: []
                hostaliases: []
                priorityclassname: ""
                priority: null
                dnsconfig: null
                readinessgates: []
                runtimeclassname: null
                enableservicelinks: null
                preemptionpolicy: null
                overhead: {}
                topologyspreadconstraints: []
                sethostnameasfqdn: null
        volumeClaimTemplates:
            - name: data-volume-template
              storagemanagement: {}
              spec:
                accessmodes:
                    - ReadWriteOnce
                selector: null
                resources:
                    limits: {}
                    requests:
                        storage:
                            format: BinarySI
                volumename: ""
                storageclassname: openebs-hostpath
                volumemode: null
                datasource: null
            - name: log-volume-template
              storagemanagement: {}
              spec:
                accessmodes:
                    - ReadWriteOnce
                selector: null
                resources:
                    limits: {}
                    requests:
                        storage:
                            format: BinarySI
                volumename: ""
                storageclassname: openebs-hostpath
                volumemode: null
                datasource: null
 end--------------------------------------------
I0125 08:54:41.907404       1 worker.go:557] clickhouse-recovery/recovery-test/force-recovery-17-54:logCHI non-normalized yet (native) new start--------------------------------------------:
non-normalized yet (native) new
logCHI kind: ClickHouseInstallation
apiversion: clickhouse.altinity.com/v1
metadata:
    name: recovery-test
    generatename: ""
    namespace: clickhouse-recovery
    selflink: ""
    uid: 49ca13ec-137a-4890-99fd-bedcda704e3e
    resourceversion: "282631348"
    generation: 2
    creationtimestamp: "2024-01-25T08:39:15Z"
    deletiontimestamp: null
    deletiongraceperiodseconds: null
    labels: {}
    annotations:
        kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"clickhouse.altinity.com/v1","kind":"ClickHouseInstallation","metadata":{"annotations":{},"name":"recovery-test","namespace":"clickhouse-recovery"},"spec":{"configuration":{"clusters":[{"layout":{"replicasCount":2,"shardsCount":1},"name":"ch-ha"}],"zookeeper":{"nodes":[{"host":"clickhouse-keeper.clickhouse-recovery.svc"}]}},"defaults":{"templates":{"dataVolumeClaimTemplate":"data-volume-template","logVolumeClaimTemplate":"log-volume-template","podTemplate":"clickhouse:23.8"}},"taskID":"force-recovery-17-54","templates":{"podTemplates":[{"name":"clickhouse:23.8","spec":{"containers":[{"image":"clickhouse/clickhouse-server:23.8","name":"clickhouse-pod"}],"resources":{"limits":{"cpu":"1","memory":"4Gi"},"requests":{"cpu":"1","memory":"4Gi"}}}}],"volumeClaimTemplates":[{"name":"data-volume-template","spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"5Gi"}},"storageClassName":"openebs-hostpath"}},{"name":"log-volume-template","spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}},"storageClassName":"openebs-hostpath"}}]}}}
    ownerreferences: []
    finalizers:
        - finalizer.clickhouseinstallation.altinity.com
    clustername: ""
    managedfields: []
spec:
    taskID: force-recovery-17-54
    defaults:
        templates:
            podTemplate: clickhouse:23.8
            dataVolumeClaimTemplate: data-volume-template
            logVolumeClaimTemplate: log-volume-template
    configuration:
        zookeeper:
            nodes:
                - host: clickhouse-keeper.clickhouse-recovery.svc
        clusters:
            - name: ch-ha
              layout:
                shardsCount: 1
                replicasCount: 2
    templates:
        podTemplates:
            - name: clickhouse:23.8
              spec:
                volumes: []
                initcontainers: []
                containers:
                    - name: clickhouse-pod
                      image: clickhouse/clickhouse-server:23.8
                      command: []
                      args: []
                      workingdir: ""
                      ports: []
                      envfrom: []
                      env: []
                      resources:
                        limits: {}
                        requests: {}
                      volumemounts: []
                      volumedevices: []
                      livenessprobe: null
                      readinessprobe: null
                      startupprobe: null
                      lifecycle: null
                      terminationmessagepath: ""
                      terminationmessagepolicy: ""
                      imagepullpolicy: ""
                      securitycontext: null
                      stdin: false
                      stdinonce: false
                      tty: false
                ephemeralcontainers: []
                restartpolicy: ""
                terminationgraceperiodseconds: null
                activedeadlineseconds: null
                dnspolicy: ""
                nodeselector: {}
                serviceaccountname: ""
                deprecatedserviceaccount: ""
                automountserviceaccounttoken: null
                nodename: ""
                hostnetwork: false
                hostpid: false
                hostipc: false
                shareprocessnamespace: null
                securitycontext: null
                imagepullsecrets: []
                hostname: ""
                subdomain: ""
                affinity: null
                schedulername: ""
                tolerations: []
                hostaliases: []
                priorityclassname: ""
                priority: null
                dnsconfig: null
                readinessgates: []
                runtimeclassname: null
                enableservicelinks: null
                preemptionpolicy: null
                overhead: {}
                topologyspreadconstraints: []
                sethostnameasfqdn: null
        volumeClaimTemplates:
            - name: data-volume-template
              storagemanagement: {}
              spec:
                accessmodes:
                    - ReadWriteOnce
                selector: null
                resources:
                    limits: {}
                    requests:
                        storage:
                            format: BinarySI
                volumename: ""
                storageclassname: openebs-hostpath
                volumemode: null
                datasource: null
            - name: log-volume-template
              storagemanagement: {}
              spec:
                accessmodes:
                    - ReadWriteOnce
                selector: null
                resources:
                    limits: {}
                    requests:
                        storage:
                            format: BinarySI
                volumename: ""
                storageclassname: openebs-hostpath
                volumemode: null
                datasource: null
 end--------------------------------------------
I0125 08:54:41.907428       1 worker-chi-reconciler.go:53] worker-chi-reconciler.go:53:reconcileCHI():start:clickhouse-recovery/recovery-test/force-recovery-17-54
I0125 08:54:41.907448       1 worker-chi-reconciler.go:59] reconcileCHI():clickhouse-recovery/recovery-test/force-recovery-17-54:Changing OLD to Normalized COMPLETED: clickhouse-recovery/recovery-test
I0125 08:54:41.907466       1 worker-chi-reconciler.go:62] reconcileCHI():clickhouse-recovery/recovery-test/force-recovery-17-54:has ancestor, use it as a base for reconcile. CHI: clickhouse-recovery/recovery-test
I0125 08:54:41.907479       1 worker-chi-reconciler.go:69] reconcileCHI():clickhouse-recovery/recovery-test/force-recovery-17-54:Normalized OLD CHI: clickhouse-recovery/recovery-test
I0125 08:54:41.907496       1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/6307791f-8ad9-4cae-bbda-7d2f15a7e741:Used templates count: 0
I0125 08:54:41.911567       1 worker.go:332] clickhouse-recovery/recovery-test/6307791f-8ad9-4cae-bbda-7d2f15a7e741:IPs of the CHI normalizer clickhouse-recovery/recovery-test: len: 1 [10.42.247.229]
I0125 08:54:41.911601       1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/6307791f-8ad9-4cae-bbda-7d2f15a7e741:Used templates count: 0
I0125 08:54:41.912106       1 worker-chi-reconciler.go:72] reconcileCHI():clickhouse-recovery/recovery-test/force-recovery-17-54:Normalized NEW CHI: clickhouse-recovery/recovery-test
I0125 08:54:41.912127       1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/force-recovery-17-54:Used templates count: 0
I0125 08:54:41.916898       1 worker.go:332] clickhouse-recovery/recovery-test/force-recovery-17-54:IPs of the CHI normalizer clickhouse-recovery/recovery-test: len: 1 [10.42.247.229]
I0125 08:54:41.916911       1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/force-recovery-17-54:Used templates count: 0
I0125 08:54:41.917744       1 worker.go:557] clickhouse-recovery/recovery-test/6307791f-8ad9-4cae-bbda-7d2f15a7e741:logCHI normalized old start--------------------------------------------:
normalized old
logCHI kind: ClickHouseInstallation
apiversion: clickhouse.altinity.com/v1
metadata:
    name: recovery-test
    generatename: ""
    namespace: clickhouse-recovery
    selflink: ""
    uid: 49ca13ec-137a-4890-99fd-bedcda704e3e
    resourceversion: "282625367"
    generation: 1
    creationtimestamp: "2024-01-25T08:39:15Z"
    deletiontimestamp: null
    deletiongraceperiodseconds: null
    labels: {}
    annotations: {}
    ownerreferences: []
    finalizers:
        - finalizer.clickhouseinstallation.altinity.com
    clustername: ""
    managedfields: []
spec:
    taskID: 6307791f-8ad9-4cae-bbda-7d2f15a7e741
    stop: "False"
    troubleshoot: "False"
    templating:
        policy: manual
    reconciling:
        policy: unspecified
        configMapPropagationTimeout: 10
        cleanup:
            unknownObjects:
                statefulSet: Delete
                pvc: Delete
                configMap: Delete
                service: Delete
            reconcileFailedObjects:
                statefulSet: Retain
                pvc: Retain
                configMap: Retain
                service: Retain
    defaults:
        replicasUseFQDN: "False"
        storageManagement: {}
        templates:
            podTemplate: clickhouse:23.8
            dataVolumeClaimTemplate: data-volume-template
            logVolumeClaimTemplate: log-volume-template
    configuration:
        zookeeper:
            nodes:
                - host: clickhouse-keeper.clickhouse-recovery.svc
                  port: 2181
        users:
            clickhouse_operator/networks/ip: '[10.42.30.13]'
            clickhouse_operator/password_sha256_hex: 716b36073a90c6fe1d445ac1af85f4777c5b7a155cea359961826a030513e448
            clickhouse_operator/profile: clickhouse_operator
            default/networks/host_regexp: (chi-recovery-test-[^.]+\d+-\d+|clickhouse\-recovery-test)\.clickhouse-recovery\.svc\.cluster\.local$
            default/networks/ip: '[::1, 127.0.0.1, 10.42.247.229, 10.42.36.127]'
            default/profile: default
            default/quota: default
        clusters:
            - name: ch-ha
              zookeeper:
                nodes:
                    - host: clickhouse-keeper.clickhouse-recovery.svc
                      port: 2181
              templates:
                podTemplate: clickhouse:23.8
                dataVolumeClaimTemplate: data-volume-template
                logVolumeClaimTemplate: log-volume-template
              schemaPolicy:
                replica: All
                shard: All
              layout:
                shardsCount: 1
                replicasCount: 2
                shards:
                    - name: "0"
                      internalReplication: "True"
                      templates:
                        podTemplate: clickhouse:23.8
                        dataVolumeClaimTemplate: data-volume-template
                        logVolumeClaimTemplate: log-volume-template
                      replicasCount: 2
                      replicas:
                        - name: 0-0
                          tcpPort: 9000
                          httpPort: 8123
                          interserverHTTPPort: 9009
                          templates:
                            podTemplate: clickhouse:23.8
                            dataVolumeClaimTemplate: data-volume-template
                            logVolumeClaimTemplate: log-volume-template
                        - name: 0-1
                          tcpPort: 9000
                          httpPort: 8123
                          interserverHTTPPort: 9009
                          templates:
                            podTemplate: clickhouse:23.8
                            dataVolumeClaimTemplate: data-volume-template
                            logVolumeClaimTemplate: log-volume-template
                replicas:
                    - name: "0"
                      templates:
                        podTemplate: clickhouse:23.8
                        dataVolumeClaimTemplate: data-volume-template
                        logVolumeClaimTemplate: log-volume-template
                      shardsCount: 1
                      shards:
                        - name: 0-0
                          tcpPort: 9000
                          httpPort: 8123
                          interserverHTTPPort: 9009
                          templates:
                            podTemplate: clickhouse:23.8
                            dataVolumeClaimTemplate: data-volume-template
                            logVolumeClaimTemplate: log-volume-template
                    - name: "1"
                      templates:
                        podTemplate: clickhouse:23.8
                        dataVolumeClaimTemplate: data-volume-template
                        logVolumeClaimTemplate: log-volume-template
                      shardsCount: 1
                      shards:
                        - name: 0-1
                          tcpPort: 9000
                          httpPort: 8123
                          interserverHTTPPort: 9009
                          templates:
                            podTemplate: clickhouse:23.8
                            dataVolumeClaimTemplate: data-volume-template
                            logVolumeClaimTemplate: log-volume-template
    templates:
        podTemplates:
            - name: clickhouse:23.8
              spec:
                volumes: []
                initcontainers: []
                containers:
                    - name: clickhouse-pod
                      image: clickhouse/clickhouse-server:23.8
                      command: []
                      args: []
                      workingdir: ""
                      ports: []
                      envfrom: []
                      env: []
                      resources:
                        limits: {}
                        requests: {}
                      volumemounts: []
                      volumedevices: []
                      livenessprobe: null
                      readinessprobe: null
                      startupprobe: null
                      lifecycle: null
                      terminationmessagepath: ""
                      terminationmessagepolicy: ""
                      imagepullpolicy: ""
                      securitycontext: null
                      stdin: false
                      stdinonce: false
                      tty: false
                ephemeralcontainers: []
                restartpolicy: ""
                terminationgraceperiodseconds: null
                activedeadlineseconds: null
                dnspolicy: ""
                nodeselector: {}
                serviceaccountname: ""
                deprecatedserviceaccount: ""
                automountserviceaccounttoken: null
                nodename: ""
                hostnetwork: false
                hostpid: false
                hostipc: false
                shareprocessnamespace: null
                securitycontext: null
                imagepullsecrets: []
                hostname: ""
                subdomain: ""
                affinity: null
                schedulername: ""
                tolerations: []
                hostaliases: []
                priorityclassname: ""
                priority: null
                dnsconfig: null
                readinessgates: []
                runtimeclassname: null
                enableservicelinks: null
                preemptionpolicy: null
                overhead: {}
                topologyspreadconstraints: []
                sethostnameasfqdn: null
        volumeClaimTemplates:
            - name: data-volume-template
              storagemanagement: {}
              spec:
                accessmodes:
                    - ReadWriteOnce
                selector: null
                resources:
                    limits: {}
                    requests:
                        storage:
                            format: BinarySI
                volumename: ""
                storageclassname: openebs-hostpath
                volumemode: null
                datasource: null
            - name: log-volume-template
              storagemanagement: {}
              spec:
                accessmodes:
                    - ReadWriteOnce
                selector: null
                resources:
                    limits: {}
                    requests:
                        storage:
                            format: BinarySI
                volumename: ""
                storageclassname: openebs-hostpath
                volumemode: null
                datasource: null
        podtemplatesindex: {}
        volumeclaimtemplatesindex: {}
 end--------------------------------------------
I0125 08:54:41.918474       1 worker.go:557] clickhouse-recovery/recovery-test/force-recovery-17-54:logCHI normalized new start--------------------------------------------:
normalized new
logCHI kind: ClickHouseInstallation
apiversion: clickhouse.altinity.com/v1
metadata:
    name: recovery-test
    generatename: ""
    namespace: clickhouse-recovery
    selflink: ""
    uid: 49ca13ec-137a-4890-99fd-bedcda704e3e
    resourceversion: "282631348"
    generation: 2
    creationtimestamp: "2024-01-25T08:39:15Z"
    deletiontimestamp: null
    deletiongraceperiodseconds: null
    labels: {}
    annotations: {}
    ownerreferences: []
    finalizers:
        - finalizer.clickhouseinstallation.altinity.com
    clustername: ""
    managedfields: []
spec:
    taskID: force-recovery-17-54
    stop: "False"
    troubleshoot: "False"
    templating:
        policy: manual
    reconciling:
        policy: unspecified
        configMapPropagationTimeout: 10
        cleanup:
            unknownObjects:
                statefulSet: Delete
                pvc: Delete
                configMap: Delete
                service: Delete
            reconcileFailedObjects:
                statefulSet: Retain
                pvc: Retain
                configMap: Retain
                service: Retain
    defaults:
        replicasUseFQDN: "False"
        storageManagement: {}
        templates:
            podTemplate: clickhouse:23.8
            dataVolumeClaimTemplate: data-volume-template
            logVolumeClaimTemplate: log-volume-template
    configuration:
        zookeeper:
            nodes:
                - host: clickhouse-keeper.clickhouse-recovery.svc
                  port: 2181
        users:
            clickhouse_operator/networks/ip: '[10.42.30.13]'
            clickhouse_operator/password_sha256_hex: 716b36073a90c6fe1d445ac1af85f4777c5b7a155cea359961826a030513e448
            clickhouse_operator/profile: clickhouse_operator
            default/networks/host_regexp: (chi-recovery-test-[^.]+\d+-\d+|clickhouse\-recovery-test)\.clickhouse-recovery\.svc\.cluster\.local$
            default/networks/ip: '[::1, 127.0.0.1, 10.42.247.229]'
            default/profile: default
            default/quota: default
        clusters:
            - name: ch-ha
              zookeeper:
                nodes:
                    - host: clickhouse-keeper.clickhouse-recovery.svc
                      port: 2181
              templates:
                podTemplate: clickhouse:23.8
                dataVolumeClaimTemplate: data-volume-template
                logVolumeClaimTemplate: log-volume-template
              schemaPolicy:
                replica: All
                shard: All
              layout:
                shardsCount: 1
                replicasCount: 2
                shards:
                    - name: "0"
                      internalReplication: "True"
                      templates:
                        podTemplate: clickhouse:23.8
                        dataVolumeClaimTemplate: data-volume-template
                        logVolumeClaimTemplate: log-volume-template
                      replicasCount: 2
                      replicas:
                        - name: 0-0
                          tcpPort: 9000
                          httpPort: 8123
                          interserverHTTPPort: 9009
                          templates:
                            podTemplate: clickhouse:23.8
                            dataVolumeClaimTemplate: data-volume-template
                            logVolumeClaimTemplate: log-volume-template
                        - name: 0-1
                          tcpPort: 9000
                          httpPort: 8123
                          interserverHTTPPort: 9009
                          templates:
                            podTemplate: clickhouse:23.8
                            dataVolumeClaimTemplate: data-volume-template
                            logVolumeClaimTemplate: log-volume-template
                replicas:
                    - name: "0"
                      templates:
                        podTemplate: clickhouse:23.8
                        dataVolumeClaimTemplate: data-volume-template
                        logVolumeClaimTemplate: log-volume-template
                      shardsCount: 1
                      shards:
                        - name: 0-0
                          tcpPort: 9000
                          httpPort: 8123
                          interserverHTTPPort: 9009
                          templates:
                            podTemplate: clickhouse:23.8
                            dataVolumeClaimTemplate: data-volume-template
                            logVolumeClaimTemplate: log-volume-template
                    - name: "1"
                      templates:
                        podTemplate: clickhouse:23.8
                        dataVolumeClaimTemplate: data-volume-template
                        logVolumeClaimTemplate: log-volume-template
                      shardsCount: 1
                      shards:
                        - name: 0-1
                          tcpPort: 9000
                          httpPort: 8123
                          interserverHTTPPort: 9009
                          templates:
                            podTemplate: clickhouse:23.8
                            dataVolumeClaimTemplate: data-volume-template
                            logVolumeClaimTemplate: log-volume-template
    templates:
        podTemplates:
            - name: clickhouse:23.8
              spec:
                volumes: []
                initcontainers: []
                containers:
                    - name: clickhouse-pod
                      image: clickhouse/clickhouse-server:23.8
                      command: []
                      args: []
                      workingdir: ""
                      ports: []
                      envfrom: []
                      env: []
                      resources:
                        limits: {}
                        requests: {}
                      volumemounts: []
                      volumedevices: []
                      livenessprobe: null
                      readinessprobe: null
                      startupprobe: null
                      lifecycle: null
                      terminationmessagepath: ""
                      terminationmessagepolicy: ""
                      imagepullpolicy: ""
                      securitycontext: null
                      stdin: false
                      stdinonce: false
                      tty: false
                ephemeralcontainers: []
                restartpolicy: ""
                terminationgraceperiodseconds: null
                activedeadlineseconds: null
                dnspolicy: ""
                nodeselector: {}
                serviceaccountname: ""
                deprecatedserviceaccount: ""
                automountserviceaccounttoken: null
                nodename: ""
                hostnetwork: false
                hostpid: false
                hostipc: false
                shareprocessnamespace: null
                securitycontext: null
                imagepullsecrets: []
                hostname: ""
                subdomain: ""
                affinity: null
                schedulername: ""
                tolerations: []
                hostaliases: []
                priorityclassname: ""
                priority: null
                dnsconfig: null
                readinessgates: []
                runtimeclassname: null
                enableservicelinks: null
                preemptionpolicy: null
                overhead: {}
                topologyspreadconstraints: []
                sethostnameasfqdn: null
        volumeClaimTemplates:
            - name: data-volume-template
              storagemanagement: {}
              spec:
                accessmodes:
                    - ReadWriteOnce
                selector: null
                resources:
                    limits: {}
                    requests:
                        storage:
                            format: BinarySI
                volumename: ""
                storageclassname: openebs-hostpath
                volumemode: null
                datasource: null
            - name: log-volume-template
              storagemanagement: {}
              spec:
                accessmodes:
                    - ReadWriteOnce
                selector: null
                resources:
                    limits: {}
                    requests:
                        storage:
                            format: BinarySI
                volumename: ""
                storageclassname: openebs-hostpath
                volumemode: null
                datasource: null
        podtemplatesindex: {}
        volumeclaimtemplatesindex: {}
 end--------------------------------------------
I0125 08:54:41.918823       1 worker.go:567] ActionPlan start---------------------------------------------:
AP item start -------------------------
removed spec items: 1
ap item path [0]:'.Configuration.Users["default/networks/ip"].vector[3]'
ap item value[0]:'"10.42.36.127"'
AP item end -------------------------
AP item start -------------------------
modified spec items: 1
ap item path [0]:'.TaskID'
ap item value[0]:'"force-recovery-17-54"'
AP item end -------------------------

ActionPlan end---------------------------------------------
I0125 08:54:41.918851       1 worker-chi-reconciler.go:83] reconcileCHI():clickhouse-recovery/recovery-test/force-recovery-17-54:ActionPlan has actions - continue reconcile
I0125 08:54:41.935295       1 worker.go:651] markReconcileStart():clickhouse-recovery/recovery-test/force-recovery-17-54:reconcile started, task id: force-recovery-17-54
I0125 08:54:42.512827       1 worker.go:806] FOUND host: ns:clickhouse-recovery|chi:recovery-test|clu:ch-ha|sha:0|rep:0|host:0-0
I0125 08:54:42.512844       1 worker.go:806] FOUND host: ns:clickhouse-recovery|chi:recovery-test|clu:ch-ha|sha:0|rep:1|host:0-1
I0125 08:54:42.512898       1 worker.go:834] RemoteServersGeneratorOptions: exclude hosts: [], attributes: status: , add: true, remove: false, modify: false, found: false, excluded host(s):
I0125 08:54:42.710480       1 worker.go:1269] updateConfigMap():clickhouse-recovery/recovery-test/force-recovery-17-54:Update ConfigMap clickhouse-recovery/chi-recovery-test-common-configd
I0125 08:54:43.517896       1 worker.go:1269] updateConfigMap():clickhouse-recovery/recovery-test/force-recovery-17-54:Update ConfigMap clickhouse-recovery/chi-recovery-test-common-usersd
I0125 08:54:44.128418       1 creator.go:113] CreateServiceCluster():clickhouse-recovery/recovery-test/force-recovery-17-54:clickhouse-recovery/cluster-recovery-test-ch-ha
I0125 08:54:44.134928       1 worker-chi-reconciler.go:716] PDB updated clickhouse-recovery/recovery-test-ch-ha
I0125 08:54:44.134949       1 worker-chi-reconciler.go:492] not found ReconcileShardsAndHostsOptionsCtxKey, use empty opts
I0125 08:54:44.134955       1 worker-chi-reconciler.go:506] starting first shard separately
I0125 08:54:44.134992       1 creator.go:136] CreateServiceShard():clickhouse-recovery/recovery-test/force-recovery-17-54:clickhouse-recovery/shard-recovery-test-ch-ha-0
I0125 08:54:44.135043       1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local]
E0125 08:54:44.138380       1 connection.go:145] QueryContext():FAILED Query(http://clickhouse_operator:***@chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local:8123/) doRequest: transport failed to send a request to ClickHouse: dial tcp: lookup chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local on 10.43.0.10:53: no such host for SQL: SELECT version()
W0125 08:54:44.138392       1 cluster.go:91] QueryAny():FAILED to run query on: chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local] skip to next. err: doRequest: transport failed to send a request to ClickHouse: dial tcp: lookup chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local on 10.43.0.10:53: no such host
E0125 08:54:44.138400       1 cluster.go:95] QueryAny():FAILED to run query on all hosts [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local]
W0125 08:54:44.138414       1 worker-chi-reconciler.go:308] getHostClickHouseVersion():Failed to get ClickHouse version on host: 0-0
I0125 08:54:44.138425       1 worker-chi-reconciler.go:621] reconcileHost():Reconcile Host start. Host: 0-0 ClickHouse version running: failed to query
I0125 08:54:44.724494       1 creator.go:589] getPodTemplate():clickhouse-recovery/recovery-test/force-recovery-17-54:statefulSet chi-recovery-test-ch-ha-0-0 use custom template: clickhouse:23.8
I0125 08:54:44.724647       1 creator.go:575] setupLogContainer():clickhouse-recovery/recovery-test/force-recovery-17-54:add log container for statefulSet chi-recovery-test-ch-ha-0-0
I0125 08:54:44.725548       1 worker.go:984] worker.go:984:excludeHost():start:exclude host start
I0125 08:54:44.725577       1 worker.go:120] shouldForceRestartHost():Host is new, no restart applicable. Host: 0-0
I0125 08:54:44.725585       1 worker.go:1125] shouldExcludeHost():Host is new, no need to exclude host 0 shard 0 cluster ch-ha
I0125 08:54:44.725589       1 worker.go:988] worker.go:985:excludeHost():end:exclude host end
I0125 08:54:44.725597       1 worker.go:1002] worker.go:1002:completeQueries():start:complete queries start
I0125 08:54:44.725611       1 worker.go:1174] shouldWaitQueries():No need to wait for queries to complete, host is a new one host 0 shard 0 cluster ch-ha
I0125 08:54:44.725615       1 worker.go:1009] worker.go:1003:completeQueries():end:complete queries end
I0125 08:54:44.910691       1 worker.go:1269] updateConfigMap():clickhouse-recovery/recovery-test/force-recovery-17-54:Update ConfigMap clickhouse-recovery/chi-recovery-test-deploy-confd-ch-ha-0-0
I0125 08:54:46.310927       1 worker-chi-reconciler.go:352] worker-chi-reconciler.go:352:reconcileHostStatefulSet():start:reconcile StatefulSet start
I0125 08:54:46.312591       1 worker-chi-reconciler.go:358] reconcileHostStatefulSet():Reconcile host 0-0. ClickHouse version: not applicable
I0125 08:54:46.312612       1 worker.go:120] shouldForceRestartHost():Host is new, no restart applicable. Host: 0-0
I0125 08:54:46.312625       1 worker-chi-reconciler.go:371] reconcileHostStatefulSet():Reconcile host 0-0. Reconcile StatefulSet
I0125 08:54:46.312703       1 creator.go:589] getPodTemplate():clickhouse-recovery/recovery-test/force-recovery-17-54:statefulSet chi-recovery-test-ch-ha-0-0 use custom template: clickhouse:23.8
I0125 08:54:46.312791       1 creator.go:575] setupLogContainer():clickhouse-recovery/recovery-test/force-recovery-17-54:add log container for statefulSet chi-recovery-test-ch-ha-0-0
I0125 08:54:46.313602       1 worker.go:1539] createStatefulSet():Create StatefulSet clickhouse-recovery/chi-recovery-test-ch-ha-0-0 - started
I0125 08:54:46.927327       1 creator.go:34] createStatefulSet()
I0125 08:54:46.927335       1 creator.go:43] Create StatefulSet clickhouse-recovery/chi-recovery-test-ch-ha-0-0
I0125 08:54:46.983937       1 worker.go:259] processReconcilePod():clickhouse-recovery/chi-recovery-test-ch-ha-0-0-0:Add Pod. clickhouse-recovery/chi-recovery-test-ch-ha-0-0-0
I0125 08:54:52.521502       1 poller.go:213] pollStatefulSet():clickhouse-recovery/chi-recovery-test-ch-ha-0-0:OK  :ObservedGeneration:1 Replicas:1 ReadyReplicas:0 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-recovery-test-ch-ha-0-0-d7556485c UpdateRevision:chi-recovery-test-ch-ha-0-0-d7556485c
I0125 08:54:56.872298       1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/force-recovery-17-54:Used templates count: 0
I0125 08:54:56.884734       1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/force-recovery-17-54:Used templates count: 0
I0125 08:54:56.885120       1 worker.go:380] clickhouse-recovery/recovery-test/force-recovery-17-54:updating endpoints for CHI-1 recovery-test
I0125 08:54:56.888017       1 worker.go:382] clickhouse-recovery/recovery-test/force-recovery-17-54:IPs of the CHI-1 update endpoints clickhouse-recovery/recovery-test: len: 2 [10.42.36.80 10.42.247.229]
I0125 08:54:56.890261       1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/force-recovery-17-54:Used templates count: 0
I0125 08:54:56.890678       1 worker.go:386] clickhouse-recovery/recovery-test/force-recovery-17-54:Update users IPS-1
I0125 08:54:56.898916       1 worker.go:1269] updateConfigMap():clickhouse-recovery/recovery-test/force-recovery-17-54:Update ConfigMap clickhouse-recovery/chi-recovery-test-common-usersd
I0125 08:55:07.542233       1 poller.go:213] pollStatefulSet():clickhouse-recovery/chi-recovery-test-ch-ha-0-0:OK  :ObservedGeneration:1 Replicas:1 ReadyReplicas:1 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-recovery-test-ch-ha-0-0-d7556485c UpdateRevision:chi-recovery-test-ch-ha-0-0-d7556485c
I0125 08:55:07.561262       1 worker.go:1556] createStatefulSet():Create StatefulSet clickhouse-recovery/chi-recovery-test-ch-ha-0-0 - completed
I0125 08:55:07.590074       1 worker-chi-reconciler.go:383] worker-chi-reconciler.go:353:reconcileHostStatefulSet():end:reconcile StatefulSet end
I0125 08:55:09.146809       1 creator.go:160] CreateServiceHost():clickhouse-recovery/recovery-test/force-recovery-17-54:clickhouse-recovery/chi-recovery-test-ch-ha-0-0 for Set chi-recovery-test-ch-ha-0-0
I0125 08:55:09.342925       1 worker.go:1407] updateService():clickhouse-recovery/recovery-test/force-recovery-17-54:Update Service clickhouse-recovery/chi-recovery-test-ch-ha-0-0
I0125 08:55:09.956874       1 worker-chi-reconciler.go:399] reconcileHostService():DONE Reconcile service of the host 0-0.
I0125 08:55:09.956951       1 worker.go:898] migrateTables():No need to add tables on host 0 to shard 0 in cluster ch-ha
I0125 08:55:09.956964       1 worker.go:1038] includeHost():Include into cluster host 0 shard 0 cluster ch-ha
I0125 08:55:09.956972       1 worker.go:834] RemoteServersGeneratorOptions: exclude hosts: [], attributes: status: , add: true, remove: false, modify: false, found: false, excluded host(s):
I0125 08:55:10.141671       1 worker.go:1269] updateConfigMap():clickhouse-recovery/recovery-test/force-recovery-17-54:Update ConfigMap clickhouse-recovery/chi-recovery-test-common-configd
I0125 08:55:11.163052       1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/force-recovery-17-54:Used templates count: 0
I0125 08:55:11.177639       1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/force-recovery-17-54:Used templates count: 0
I0125 08:55:11.178051       1 worker.go:380] clickhouse-recovery/recovery-test/force-recovery-17-54:updating endpoints for CHI-1 recovery-test
I0125 08:55:11.347896       1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local]
I0125 08:55:11.349467       1 worker-chi-reconciler.go:313] getHostClickHouseVersion():Get ClickHouse version on host: 0-0 version: 23.8.9.54
I0125 08:55:11.349493       1 worker-chi-reconciler.go:684] reconcileHost():Reconcile Host completed. Host: 0-0 ClickHouse version running: 23.8.9.54
I0125 08:55:11.941576       1 worker.go:382] clickhouse-recovery/recovery-test/force-recovery-17-54:IPs of the CHI-1 update endpoints clickhouse-recovery/recovery-test: len: 2 [10.42.36.80 10.42.247.229]
I0125 08:55:11.944386       1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/force-recovery-17-54:Used templates count: 0
I0125 08:55:11.944784       1 worker.go:386] clickhouse-recovery/recovery-test/force-recovery-17-54:Update users IPS-1
I0125 08:55:12.345129       1 worker.go:1269] updateConfigMap():clickhouse-recovery/recovery-test/force-recovery-17-54:Update ConfigMap clickhouse-recovery/chi-recovery-test-common-usersd
I0125 08:55:12.557245       1 worker-chi-reconciler.go:700] reconcileHost():ProgressHostsCompleted: 1 of 2
I0125 08:55:13.760509       1 creator.go:58] CreateServiceCHI():clickhouse-recovery/recovery-test/force-recovery-17-54:clickhouse-recovery/clickhouse-recovery-test
I0125 08:55:13.760710       1 worker.go:1359] updateService():clickhouse-recovery/recovery-test/force-recovery-17-54:reuse Port 8123 values
I0125 08:55:13.760730       1 worker.go:1359] updateService():clickhouse-recovery/recovery-test/force-recovery-17-54:reuse Port 9000 values
I0125 08:55:14.141780       1 worker.go:1407] updateService():clickhouse-recovery/recovery-test/force-recovery-17-54:Update Service clickhouse-recovery/clickhouse-recovery-test
I0125 08:55:14.957647       1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:14.962303       1 worker-chi-reconciler.go:313] getHostClickHouseVersion():Get ClickHouse version on host: 0-1 version: 23.8.9.54
I0125 08:55:14.962319       1 worker-chi-reconciler.go:621] reconcileHost():Reconcile Host start. Host: 0-1 ClickHouse version running: 23.8.9.54
I0125 08:55:15.559444       1 creator.go:589] getPodTemplate():clickhouse-recovery/recovery-test/force-recovery-17-54:statefulSet chi-recovery-test-ch-ha-0-1 use custom template: clickhouse:23.8
I0125 08:55:15.559513       1 creator.go:575] setupLogContainer():clickhouse-recovery/recovery-test/force-recovery-17-54:add log container for statefulSet chi-recovery-test-ch-ha-0-1
I0125 08:55:15.560177       1 worker.go:1508] getObjectStatusFromMetas():clickhouse-recovery/chi-recovery-test-ch-ha-0-1:cur and new objects are equal based on object version label. Update of the object is not required. Object: clickhouse-recovery/chi-recovery-test-ch-ha-0-1
I0125 08:55:15.560193       1 worker.go:984] worker.go:984:excludeHost():start:exclude host start
I0125 08:55:15.741564       1 worker.go:152] shouldForceRestartHost():Host restart is not required. Host: 0-1
I0125 08:55:15.741588       1 worker.go:1130] shouldExcludeHost():Host is the same, would not be updated, no need to exclude host 1 shard 0 cluster ch-ha
I0125 08:55:15.741595       1 worker.go:988] worker.go:985:excludeHost():end:exclude host end
I0125 08:55:15.741607       1 worker.go:1002] worker.go:1002:completeQueries():start:complete queries start
I0125 08:55:15.741624       1 worker.go:1179] shouldWaitQueries():Will wait for queries to complete according to CHOp config reconcile.host.wait.queries setting, host is not yet in the cluster host 1 shard 0 cluster ch-ha
I0125 08:55:15.741686       1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:15.746300       1 poller.go:307] pollHostContext():clickhouse-recovery/0-1-OK
I0125 08:55:15.746313       1 worker.go:1006] worker.go:1003:completeQueries():end:complete queries end
I0125 08:55:15.942155       1 worker.go:1269] updateConfigMap():clickhouse-recovery/recovery-test/force-recovery-17-54:Update ConfigMap clickhouse-recovery/chi-recovery-test-deploy-confd-ch-ha-0-1
I0125 08:55:18.948947       1 worker-chi-reconciler.go:352] worker-chi-reconciler.go:352:reconcileHostStatefulSet():start:reconcile StatefulSet start
I0125 08:55:18.948995       1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:19.043603       1 worker-chi-reconciler.go:313] getHostClickHouseVersion():Get ClickHouse version on host: 0-1 version: 23.8.9.54
I0125 08:55:19.045890       1 worker-chi-reconciler.go:358] reconcileHostStatefulSet():Reconcile host 0-1. ClickHouse version: 23.8.9.54
I0125 08:55:19.140999       1 worker.go:152] shouldForceRestartHost():Host restart is not required. Host: 0-1
I0125 08:55:19.141033       1 worker-chi-reconciler.go:371] reconcileHostStatefulSet():Reconcile host 0-1. Reconcile StatefulSet
I0125 08:55:19.141126       1 creator.go:589] getPodTemplate():clickhouse-recovery/recovery-test/force-recovery-17-54:statefulSet chi-recovery-test-ch-ha-0-1 use custom template: clickhouse:23.8
I0125 08:55:19.141222       1 creator.go:575] setupLogContainer():clickhouse-recovery/recovery-test/force-recovery-17-54:add log container for statefulSet chi-recovery-test-ch-ha-0-1
I0125 08:55:19.142000       1 worker.go:1508] getObjectStatusFromMetas():clickhouse-recovery/chi-recovery-test-ch-ha-0-1:cur and new objects are equal based on object version label. Update of the object is not required. Object: clickhouse-recovery/chi-recovery-test-ch-ha-0-1
I0125 08:55:19.142070       1 worker-chi-reconciler.go:383] worker-chi-reconciler.go:353:reconcileHostStatefulSet():end:reconcile StatefulSet end
I0125 08:55:21.544663       1 creator.go:160] CreateServiceHost():clickhouse-recovery/recovery-test/force-recovery-17-54:clickhouse-recovery/chi-recovery-test-ch-ha-0-1 for Set chi-recovery-test-ch-ha-0-1
I0125 08:55:21.741837       1 worker.go:1407] updateService():clickhouse-recovery/recovery-test/force-recovery-17-54:Update Service clickhouse-recovery/chi-recovery-test-ch-ha-0-1
I0125 08:55:22.357605       1 worker-chi-reconciler.go:399] reconcileHostService():DONE Reconcile service of the host 0-1.
I0125 08:55:22.357676       1 worker.go:915] migrateTables():Adding tables on shard/host:0/1 cluster:ch-ha
I0125 08:55:22.955996       1 cluster_schemer.go:228] cluster_schemer.go:228:HostCreateTables():start:Migrating schema objects to host 0-1
I0125 08:55:22.956073       1 cluster_schemer.go:128] shouldCreateReplicatedObjects():SchemaPolicy.Shard says we need replicated objects. Should create replicated objects for the shard: [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:22.956107       1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:22.965244       1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:22.971506       1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:22.975558       1 cluster_schemer.go:58] shouldCreateDistributedObjects():Should create distributed objects in the cluster: [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:22.975590       1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:23.025180       1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:23.054752       1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:23.058351       1 cluster_schemer.go:238] HostCreateTables():Creating replicated objects at 0-1: [default test_db test_table test_table_local]
I0125 08:55:23.130474       1 cluster_schemer.go:245] HostCreateTables():Creating distributed objects at 0-1: [test_db]
I0125 08:55:23.130869       1 cluster_schemer.go:257] cluster_schemer.go:229:HostCreateTables():end:Migrating schema objects to host 0-1
I0125 08:55:23.130900       1 worker.go:923] migrateTables():Tables added successfully on shard/host:0/1 cluster:ch-ha
I0125 08:55:23.559150       1 worker.go:1038] includeHost():Include into cluster host 1 shard 0 cluster ch-ha
I0125 08:55:23.559168       1 worker.go:834] RemoteServersGeneratorOptions: exclude hosts: [], attributes: status: , add: true, remove: false, modify: false, found: false, excluded host(s):
I0125 08:55:23.741635       1 worker.go:1269] updateConfigMap():clickhouse-recovery/recovery-test/force-recovery-17-54:Update ConfigMap clickhouse-recovery/chi-recovery-test-common-configd
I0125 08:55:24.541636       1 cluster.go:84] Run query on: chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local of [chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local]
I0125 08:55:24.542801       1 worker-chi-reconciler.go:313] getHostClickHouseVersion():Get ClickHouse version on host: 0-1 version: 23.8.9.54
I0125 08:55:24.542818       1 worker-chi-reconciler.go:684] reconcileHost():Reconcile Host completed. Host: 0-1 ClickHouse version running: 23.8.9.54
I0125 08:55:25.158705       1 worker-chi-reconciler.go:700] reconcileHost():ProgressHostsCompleted: 2 of 2
I0125 08:55:25.757534       1 worker-chi-reconciler.go:519] Starting rest of shards on workers: 1
I0125 08:55:25.942327       1 worker.go:1269] updateConfigMap():clickhouse-recovery/recovery-test/force-recovery-17-54:Update ConfigMap clickhouse-recovery/chi-recovery-test-common-configd
I0125 08:55:26.559559       1 worker-chi-reconciler.go:117] reconcileCHI():clickhouse-recovery/recovery-test/force-recovery-17-54:remove items scheduled for deletion
I0125 08:55:27.157667       1 worker-deleter.go:38] clean():clickhouse-recovery/recovery-test/force-recovery-17-54:Failed to reconcile objects:
I0125 08:55:27.157694       1 worker-deleter.go:39] clean():clickhouse-recovery/recovery-test/force-recovery-17-54:Reconciled objects:
ConfigMap: clickhouse-recovery/chi-recovery-test-deploy-confd-ch-ha-0-0
ConfigMap: clickhouse-recovery/chi-recovery-test-deploy-confd-ch-ha-0-1
ConfigMap: clickhouse-recovery/chi-recovery-test-common-configd
ConfigMap: clickhouse-recovery/chi-recovery-test-common-usersd
PDB: clickhouse-recovery/recovery-test-ch-ha
StatefulSet: clickhouse-recovery/chi-recovery-test-ch-ha-0-0
StatefulSet: clickhouse-recovery/chi-recovery-test-ch-ha-0-1
PVC: clickhouse-recovery/data-volume-template-chi-recovery-test-ch-ha-0-0-0
PVC: clickhouse-recovery/log-volume-template-chi-recovery-test-ch-ha-0-0-0
PVC: clickhouse-recovery/data-volume-template-chi-recovery-test-ch-ha-0-1-0
PVC: clickhouse-recovery/log-volume-template-chi-recovery-test-ch-ha-0-1-0
Service: clickhouse-recovery/chi-recovery-test-ch-ha-0-0
Service: clickhouse-recovery/clickhouse-recovery-test
Service: clickhouse-recovery/chi-recovery-test-ch-ha-0-1
I0125 08:55:27.943225       1 worker-deleter.go:42] clean():clickhouse-recovery/recovery-test/force-recovery-17-54:Existing objects:
ConfigMap: clickhouse-recovery/chi-recovery-test-common-usersd
ConfigMap: clickhouse-recovery/chi-recovery-test-deploy-confd-ch-ha-0-0
ConfigMap: clickhouse-recovery/chi-recovery-test-deploy-confd-ch-ha-0-1
ConfigMap: clickhouse-recovery/chi-recovery-test-common-configd
Service: clickhouse-recovery/chi-recovery-test-ch-ha-0-1
Service: clickhouse-recovery/clickhouse-recovery-test
Service: clickhouse-recovery/chi-recovery-test-ch-ha-0-0
PVC: clickhouse-recovery/data-volume-template-chi-recovery-test-ch-ha-0-0-0
PVC: clickhouse-recovery/data-volume-template-chi-recovery-test-ch-ha-0-1-0
PVC: clickhouse-recovery/log-volume-template-chi-recovery-test-ch-ha-0-0-0
PVC: clickhouse-recovery/log-volume-template-chi-recovery-test-ch-ha-0-1-0
PDB: clickhouse-recovery/recovery-test-ch-ha
StatefulSet: clickhouse-recovery/chi-recovery-test-ch-ha-0-0
StatefulSet: clickhouse-recovery/chi-recovery-test-ch-ha-0-1
I0125 08:55:27.943263       1 worker-deleter.go:44] clean():clickhouse-recovery/recovery-test/force-recovery-17-54:Non-reconciled objects:
I0125 08:55:27.943273       1 worker-deleter.go:54] clean():clickhouse-recovery/recovery-test/force-recovery-17-54:remove items scheduled for deletion
I0125 08:55:28.552630       1 worker-deleter.go:66] worker-deleter.go:66:dropReplicas():start:clickhouse-recovery/recovery-test/force-recovery-17-54:drop replicas based on AP
I0125 08:55:28.552660       1 worker-deleter.go:78] worker-deleter.go:78:dropReplicas():end:clickhouse-recovery/recovery-test/force-recovery-17-54:processed replicas: 0
I0125 08:55:28.552675       1 worker.go:628] addCHIToMonitoring():clickhouse-recovery/recovery-test/force-recovery-17-54:add CHI to monitoring
I0125 08:55:29.158949       1 controller.go:615] OK update watch (clickhouse-recovery/recovery-test): {"namespace":"clickhouse-recovery","name":"recovery-test","clusters":[{"name":"ch-ha","hosts":[{"name":"0-0","hostname":"chi-recovery-test-ch-ha-0-0.clickhouse-recovery.svc.cluster.local","tcpPort":9000,"httpPort":8123},{"name":"0-1","hostname":"chi-recovery-test-ch-ha-0-1.clickhouse-recovery.svc.cluster.local","tcpPort":9000,"httpPort":8123}]}]}
I0125 08:55:29.161007       1 worker.go:588] clickhouse-recovery/recovery-test/force-recovery-17-54:all IP addresses are in place
I0125 08:55:29.163319       1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/force-recovery-17-54:Used templates count: 0
I0125 08:55:29.163678       1 worker.go:663] clickhouse-recovery/recovery-test/force-recovery-17-54:updating endpoints for CHI-2 recovery-test
I0125 08:55:29.541088       1 worker.go:665] clickhouse-recovery/recovery-test/force-recovery-17-54:IPs of the CHI-2 finalize reconcile clickhouse-recovery/recovery-test: len: 2 [10.42.36.80 10.42.247.229]
I0125 08:55:29.543767       1 normalizer.go:203] applyCHITemplates():clickhouse-recovery/recovery-test/force-recovery-17-54:Used templates count: 0
I0125 08:55:29.544242       1 worker.go:669] clickhouse-recovery/recovery-test/force-recovery-17-54:Update users IPS-2
I0125 08:55:29.741928       1 worker.go:1269] updateConfigMap():clickhouse-recovery/recovery-test/force-recovery-17-54:Update ConfigMap clickhouse-recovery/chi-recovery-test-common-usersd
I0125 08:55:30.757498       1 worker.go:693] finalizeReconcileAndMarkCompleted():clickhouse-recovery/recovery-test/force-recovery-17-54:reconcile completed successfully, task id: force-recovery-17-54
I0125 08:55:31.358074       1 worker-chi-reconciler.go:128] worker-chi-reconciler.go:54:reconcileCHI():end:clickhouse-recovery/recovery-test/force-recovery-17-54
I0125 08:55:31.358095       1 worker.go:462] worker.go:425:updateCHI():end:clickhouse-recovery/recovery-test/force-recovery-17-54
@alex-zaitsev
Copy link
Member

Released
https://github.com/Altinity/clickhouse-operator/releases/tag/release-0.23.0

Data recovery scenarios are covered by test_036:

  • PV is deleted (PVC is not)
  • PVC is deleted
  • STS and PVC are deleted

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sunsingerus sunsingerus

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@sunsingerus @Dnnd @alex-zaitsev