Distributed and corresponding local tables are not getting created in the new Shard #308

yuzhichang · 2020-03-13T15:43:07Z

this is false according to my test .
operator version: 0.9.1

alex-zaitsev · 2020-03-13T15:47:32Z

@yuzhichang, could you describe how are you testing?

See test_013: https://github.com/Altinity/clickhouse-operator/blob/master/tests/test_operator.py

yuzhichang · 2020-03-14T01:55:02Z

My test steps:

Use operator to create a ck cluster with 3 shards, and referring to a existing zookeeper service.
The following is the manifest yaml, which is based on docs/chi-examples/08-clickhouse-version-update-02-apply-update-one.yaml:

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "version-update"
spec:
  configuration:
    zookeeper: 
      nodes:
        - host: zookeeper-0.zookeepers.zoons
          port: 2181
    clusters:
      - name: update
        templates:
          podTemplate: clickhouse:19.1.10
        layout:
          shards:
            - replicas:
                - tcpPort: 9000
                - tcpPort: 9000
                - tcpPort: 9000
                  templates:
                    podTemplate: clickhouse:19.3.7
  templates:
    podTemplates:
      - name: clickhouse:19.3.7
        spec:
          containers:
            - name: clickhouse-pod
              image: yandex/clickhouse-server:19.3.7

      - name: clickhouse:19.1.10
        spec:
          containers:
            - name: clickhouse-pod
              image: yandex/clickhouse-server:19.1.10

Get into one ck instance, create a local and distributed table on cluster. Verify these two tables are there on all three ck instances.

CREATE TABLE cdp_orders ON CLUSTER 'update' (
    order_id UInt64,
    order_time DateTime
) engine=ReplacingMergeTree()
PARTITION BY toYYYYMMDD(order_time)
ORDER BY (order_time, order_id);

CREATE TABLE dist_cdp_orders ON CLUSTER 'update' AS cdp_orders ENGINE = Distributed('update', default, cdp_orders);

Add the 4th ck instance to the manifest yaml and apply it.

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "version-update"
spec:
  configuration:
    zookeeper: 
      nodes:
        - host: zookeeper-0.zookeepers.zoons
          port: 2181
    clusters:
      - name: update
        templates:
          podTemplate: clickhouse:19.1.10
        layout:
          shards:
            - replicas:
                - tcpPort: 9000
                - tcpPort: 9000
                - tcpPort: 9000
                  templates:
                    podTemplate: clickhouse:19.3.7
            - replicas:
                - tcpPort: 9000
                  templates:
                    podTemplate: clickhouse:19.3.7

  templates:
    podTemplates:
      - name: clickhouse:19.3.7
        spec:
          containers:
            - name: clickhouse-pod
              image: yandex/clickhouse-server:19.3.7

      - name: clickhouse:19.1.10
        spec:
          containers:
            - name: clickhouse-pod
              image: yandex/clickhouse-server:19.1.10

Verify the above two tables are there on the 4th ck instance.

alex-zaitsev · 2020-03-16T06:11:38Z

I see. Looks like you closed 8123 port that is used for HTTP connections. Operator uses http client in order manage schema. So if you do not't have 8123 port it can not create schema on a new shard.

You can check operator logs -- it should be complaining about that.

yuzhichang · 2020-03-16T14:32:54Z

@alex-zaitsev I'm not aware how 8123 port is closed. I adjust the above manifest to the following and retest, but no luck:

        layout:
          shards:
            - replicas:
                - tcpPort: 9000
                  httpPort: 8123
                - tcpPort: 9000
                  httpPort: 8123
                - tcpPort: 9000
                  httpPort: 8123
                  templates:
                    podTemplate: clickhouse:19.3.7

alex-zaitsev · 2020-03-18T13:20:52Z

@yuzhichang, I think the problem is old ClickHouse version. Could you try with a newer version? Altinity Stable is currently 19.16.14.65.

yuzhichang · 2020-03-22T01:59:07Z

@alex-zaitsev I tried 19.16.14.65.

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: "version-update"
spec:
  configuration:
    zookeeper:
      nodes:
        - host: zookeeper-0.zookeepers
          port: 2181
    clusters:
      - name: update
        templates:
          podTemplate: clickhouse:19.16.14.65
        layout:
          shards:
            - replicas:
                - tcpPort: 9000
                  httpPort: 8123
            - replicas:
                - tcpPort: 9000
                  httpPort: 8123
  templates:
    podTemplates:
      - name: clickhouse:19.16.14.65
        spec:
          containers:
            - name: clickhouse-pod
              image: yandex/clickhouse-server:19.16.14.65

Operator creates table cdp_orders on the newly added 3rd shard, but not dist_cdp_orders. Here's operator log entries:

{"log":"I0322 01:34:48.387119       1 worker.go:158] updateChi(test2/version-update) - start reconcile with action plan \u003e\u003e\u003e\n","stream":"stderr","time":"2020-03-22T01:34:48.387481756Z"}
{"log":"added spec items: 2\n","stream":"stderr","time":"2020-03-22T01:34:48.387538228Z"}
{"log":"----------\n","stream":"stderr","time":"2020-03-22T01:34:48.387547442Z"}
{"log":"----- path:\n","stream":"stderr","time":"2020-03-22T01:34:48.387553929Z"}
{"log":".Configuration\n","stream":"stderr","time":"2020-03-22T01:34:48.387560427Z"}
{"log":".Clusters\n","stream":"stderr","time":"2020-03-22T01:34:48.387597446Z"}
{"log":"[0]\n","stream":"stderr","time":"2020-03-22T01:34:48.387605572Z"}
{"log":".Layout\n","stream":"stderr","time":"2020-03-22T01:34:48.387992324Z"}
{"log":".Shards\n","stream":"stderr","time":"2020-03-22T01:34:48.388006285Z"}
{"log":"[2]\n","stream":"stderr","time":"2020-03-22T01:34:48.388011079Z"}
{"log":"----- value:\n","stream":"stderr","time":"2020-03-22T01:34:48.388016467Z"}
{"log":"{ 2 %!s(int=0) true { clickhouse:19.16.14.65       } %!s(int=1) [%!s(*v1.ChiHost=\u0026{2-0 0 9000 8123 9009 { clickhouse:19.16.14.65       } {test2 version-update update 0 2 2 0 0 0 2 2-0 2 0 0 2 2 0 0 2} {6b35776e3520244558c01c9c3f0be2499e1b7792 73d42865206a76de2371ae93f33a21c234c7dca7} 0xc000740700})] {test2 version-update update %!s(int=0) 2 %!s(int=2)} %!s(*v1.ClickHouseInstallation=\u0026{{ } {version-update  test2 /apis/clickhouse.altinity.com/v1/namespaces/test2/clickhouseinstallations/version-update 1b771fb3-5f0b-43da-b2a3-5b06ee0ce1da 24309859 6 {{0 63720436914 0x2032b20}} \u003cnil\u003e \u003cnil\u003e map[] map[kubectl.kubernetes.io/last-applied-configuration:{\"apiVersion\":\"clickhouse.altinity.com/v1\",\"kind\":\"ClickHouseInstallation\",\"metadata\":{\"annotations\":{},\"name\":\"version-update\",\"namespace\":\"test2\"},\"spec\":{\"configuration\":{\"clusters\":[{\"layout\":{\"shards\":[{\"replicas\":[{\"httpPort\":8123,\"tcpPort\":9000}]},{\"replicas\":[{\"httpPort\":8123,\"tcpPort\":9000}]},{\"replicas\":[{\"httpPort\":8123,\"tcpPort\":9000}]}]},\"name\":\"update\",\"templates\":{\"podTemplate\":\"clickhouse:19.16.14.65\"}}],\"zookeeper\":{\"nodes\":[{\"host\":\"zookeeper-0.zookeepers\",\"port\":2181}]}},\"templates\":{\"podTemplates\":[{\"name\":\"clickhouse:19.16.14.65\",\"spec\":{\"containers\":[{\"image\":\"yandex/clickhouse-server:19.16.14.65\",\"name\":\"clickhouse-pod\"}]}}]}}}\n","stream":"stderr","time":"2020-03-22T01:34:48.388024517Z"}
{"log":"] [] \u003cnil\u003e [] } {false {false {} {        }} {{[{zookeeper-0.zookeepers 2181}] 0 0  } map[default/networks/host_regexp:chi-version-update-[^.]+\\d+-\\d+\\.test2.svc.cluster.local$ default/networks/ip:[::1 127.0.0.1] default/profile:default default/quota:default] map[] map[] map[] map[] [{update { 3 1 [{ 0 0 true { clickhouse:19.16.14.65       } 1 [0xc0010ce680] {test2 version-update update 0 0 0} 0xc000740700} { 1 0 true { clickhouse:19.16.14.65       } 1 [0xc0010ce9c0] {test2 version-update update 0 1 1} 0xc000740700} { 2 0 true { clickhouse:19.16.14.65       } 1 [0xc0010ceb60] {test2 version-update update 0 2 2} 0xc000740700}] [{0 { clickhouse:19.16.14.65       } 3 [0xc0010ce680 0xc0010ce9c0 0xc0010ceb60] {test2 version-update update 0 0 0} 0xc000740700}] 0xc000aae960} { clickhouse:19.16.14.65       } {test2 version-update update 0} 0xc000740700}]} {[] [{clickhouse:19.16.14.65 { []} Unspecified [] {[] [] [{clickhouse-pod yandex/clickhouse-server:19.16.14.65 [] []  [] [] [] {map[] map[]} [] [] \u003cnil\u003e \u003cnil\u003e \u003cnil\u003e    \u003cnil\u003e false false false}]  \u003cnil\u003e \u003cnil\u003e  map[]   \u003cnil\u003e  false false false \u003cnil\u003e \u003cnil\u003e []   \u003cnil\u003e  [] []  \u003cnil\u003e \u003cnil\u003e [] \u003cnil\u003e}}] [] [] map[] map[clickhouse:19.16.14.65:0xc000840a00] map[] map[]} []} {0.9.1 1 3 3 Completed 0 2 0 0 [chi-version-update-update-0-0-0 chi-version-update-update-1-0-0 chi-version-update-update-2-0-0] [chi-version-update-update-0-0.test2.svc.cluster.local chi-version-update-update-1-0.test2.svc.cluster.local chi-version-update-update-2-0.test2.svc.cluster.local] clickhouse-version-update.test2.svc.cluster.local}})}\n","stream":"stderr","time":"2020-03-22T01:34:48.388058518Z"}
{"log":"----------\n","stream":"stderr","time":"2020-03-22T01:34:48.388073453Z"}
{"log":"----- path:\n","stream":"stderr","time":"2020-03-22T01:34:48.388077996Z"}
{"log":".Configuration\n","stream":"stderr","time":"2020-03-22T01:34:48.388082466Z"}
{"log":".Clusters\n","stream":"stderr","time":"2020-03-22T01:34:48.388086748Z"}
{"log":"[0]\n","stream":"stderr","time":"2020-03-22T01:34:48.388091239Z"}
{"log":".Layout\n","stream":"stderr","time":"2020-03-22T01:34:48.388095444Z"}
{"log":".Replicas\n","stream":"stderr","time":"2020-03-22T01:34:48.388104507Z"}
{"log":"[0]\n","stream":"stderr","time":"2020-03-22T01:34:48.388137672Z"}
{"log":".Hosts\n","stream":"stderr","time":"2020-03-22T01:34:48.388152299Z"}
{"log":"[2]\n","stream":"stderr","time":"2020-03-22T01:34:48.388158489Z"}
{"log":"----- value:\n","stream":"stderr","time":"2020-03-22T01:34:48.388164068Z"}
{"log":"\u0026{2-0 %!s(int32=0) %!s(int32=9000) %!s(int32=8123) %!s(int32=9009) { clickhouse:19.16.14.65       } {test2 version-update update %!s(int=0) 2 %!s(int=2) %!s(int=0) 0 %!s(int=0) %!s(int=2) 2-0 %!s(int=2) %!s(int=0) %!s(int=0) %!s(int=2) %!s(int=2) %!s(int=0) %!s(int=0) %!s(int=2)} {6b35776e3520244558c01c9c3f0be2499e1b7792 73d42865206a76de2371ae93f33a21c234c7dca7} %!s(*v1.ClickHouseInstallation=\u0026{{ } {version-update  test2 /apis/clickhouse.altinity.com/v1/namespaces/test2/clickhouseinstallations/version-update 1b771fb3-5f0b-43da-b2a3-5b06ee0ce1da 24309859 6 {{0 63720436914 0x2032b20}} \u003cnil\u003e \u003cnil\u003e map[] map[kubectl.kubernetes.io/last-applied-configuration:{\"apiVersion\":\"clickhouse.altinity.com/v1\",\"kind\":\"ClickHouseInstallation\",\"metadata\":{\"annotations\":{},\"name\":\"version-update\",\"namespace\":\"test2\"},\"spec\":{\"configuration\":{\"clusters\":[{\"layout\":{\"shards\":[{\"replicas\":[{\"httpPort\":8123,\"tcpPort\":9000}]},{\"replicas\":[{\"httpPort\":8123,\"tcpPort\":9000}]},{\"replicas\":[{\"httpPort\":8123,\"tcpPort\":9000}]}]},\"name\":\"update\",\"templates\":{\"podTemplate\":\"clickhouse:19.16.14.65\"}}],\"zookeeper\":{\"nodes\":[{\"host\":\"zookeeper-0.zookeepers\",\"port\":2181}]}},\"templates\":{\"podTemplates\":[{\"name\":\"clickhouse:19.16.14.65\",\"spec\":{\"containers\":[{\"image\":\"yandex/clickhouse-server:19.16.14.65\",\"name\":\"clickhouse-pod\"}]}}]}}}\n","stream":"stderr","time":"2020-03-22T01:34:48.388171409Z"}
{"log":"] [] \u003cnil\u003e [] } {false {false {} {        }} {{[{zookeeper-0.zookeepers 2181}] 0 0  } map[default/networks/host_regexp:chi-version-update-[^.]+\\d+-\\d+\\.test2.svc.cluster.local$ default/networks/ip:[::1 127.0.0.1] default/profile:default default/quota:default] map[] map[] map[] map[] [{update { 3 1 [{ 0 0 true { clickhouse:19.16.14.65       } 1 [0xc0010ce680] {test2 version-update update 0 0 0} 0xc000740700} { 1 0 true { clickhouse:19.16.14.65       } 1 [0xc0010ce9c0] {test2 version-update update 0 1 1} 0xc000740700} { 2 0 true { clickhouse:19.16.14.65       } 1 [0xc0010ceb60] {test2 version-update update 0 2 2} 0xc000740700}] [{0 { clickhouse:19.16.14.65       } 3 [0xc0010ce680 0xc0010ce9c0 0xc0010ceb60] {test2 version-update update 0 0 0} 0xc000740700}] 0xc000aae960} { clickhouse:19.16.14.65       } {test2 version-update update 0} 0xc000740700}]} {[] [{clickhouse:19.16.14.65 { []} Unspecified [] {[] [] [{clickhouse-pod yandex/clickhouse-server:19.16.14.65 [] []  [] [] [] {map[] map[]} [] [] \u003cnil\u003e \u003cnil\u003e \u003cnil\u003e    \u003cnil\u003e false false false}]  \u003cnil\u003e \u003cnil\u003e  map[]   \u003cnil\u003e  false false false \u003cnil\u003e \u003cnil\u003e []   \u003cnil\u003e  [] []  \u003cnil\u003e \u003cnil\u003e [] \u003cnil\u003e}}] [] [] map[] map[clickhouse:19.16.14.65:0xc000840a00] map[] map[]} []} {0.9.1 1 3 3 Completed 0 2 0 0 [chi-version-update-update-0-0-0 chi-version-update-update-1-0-0 chi-version-update-update-2-0-0] [chi-version-update-update-0-0.test2.svc.cluster.local chi-version-update-update-1-0.test2.svc.cluster.local chi-version-update-update-2-0.test2.svc.cluster.local] clickhouse-version-update.test2.svc.cluster.local}})}\n","stream":"stderr","time":"2020-03-22T01:34:48.388195724Z"}
{"log":"----------\n","stream":"stderr","time":"2020-03-22T01:34:48.388210982Z"}
{"log":"modified spec items: 2\n","stream":"stderr","time":"2020-03-22T01:34:48.388216123Z"}
{"log":"----------\n","stream":"stderr","time":"2020-03-22T01:34:48.38822633Z"}
{"log":"----- path:\n","stream":"stderr","time":"2020-03-22T01:34:48.388249587Z"}
{"log":".Configuration\n","stream":"stderr","time":"2020-03-22T01:34:48.388255924Z"}
{"log":".Clusters\n","stream":"stderr","time":"2020-03-22T01:34:48.388268476Z"}
{"log":"[0]\n","stream":"stderr","time":"2020-03-22T01:34:48.388273702Z"}
{"log":".Layout\n","stream":"stderr","time":"2020-03-22T01:34:48.388278493Z"}
{"log":".ShardsCount\n","stream":"stderr","time":"2020-03-22T01:34:48.388283088Z"}
{"log":"----- value:\n","stream":"stderr","time":"2020-03-22T01:34:48.388287846Z"}
{"log":"%!s(int=3)\n","stream":"stderr","time":"2020-03-22T01:34:48.388292351Z"}
{"log":"----------\n","stream":"stderr","time":"2020-03-22T01:34:48.388304453Z"}
{"log":"----- path:\n","stream":"stderr","time":"2020-03-22T01:34:48.38833477Z"}
{"log":".Configuration\n","stream":"stderr","time":"2020-03-22T01:34:48.388340234Z"}
{"log":".Clusters\n","stream":"stderr","time":"2020-03-22T01:34:48.388345032Z"}
{"log":"[0]\n","stream":"stderr","time":"2020-03-22T01:34:48.38834981Z"}
{"log":".Layout\n","stream":"stderr","time":"2020-03-22T01:34:48.38836076Z"}
{"log":".Replicas\n","stream":"stderr","time":"2020-03-22T01:34:48.388371317Z"}
{"log":"[0]\n","stream":"stderr","time":"2020-03-22T01:34:48.38842513Z"}
{"log":".ShardsCount\n","stream":"stderr","time":"2020-03-22T01:34:48.388431031Z"}
{"log":"----- value:\n","stream":"stderr","time":"2020-03-22T01:34:48.388436232Z"}
{"log":"%!s(int=3)\n","stream":"stderr","time":"2020-03-22T01:34:48.388440883Z"}
{"log":"----------\n","stream":"stderr","time":"2020-03-22T01:34:48.388445879Z"}
{"log":"I0322 01:34:48.400789       1 creator.go:57] createServiceChi(test2/clickhouse-version-update)\n","stream":"stderr","time":"2020-03-22T01:34:48.400958514Z"}
{"log":"I0322 01:34:48.400831       1 creator.go:60] Reconcile Service test2/clickhouse-version-update\n","stream":"stderr","time":"2020-03-22T01:34:48.400987648Z"}
{"log":"I0322 01:34:48.400848       1 creator.go:64] Update Service test2/clickhouse-version-update\n","stream":"stderr","time":"2020-03-22T01:34:48.400994924Z"}
{"log":"I0322 01:34:48.418717       1 creator.go:37] Reconcile ConfigMap test2/chi-version-update-common-configd\n","stream":"stderr","time":"2020-03-22T01:34:48.418868862Z"}
{"log":"I0322 01:34:48.418746       1 creator.go:41] Update ConfigMap test2/chi-version-update-common-configd\n","stream":"stderr","time":"2020-03-22T01:34:48.418904283Z"}
{"log":"I0322 01:34:48.423920       1 creator.go:37] Reconcile ConfigMap test2/chi-version-update-common-usersd\n","stream":"stderr","time":"2020-03-22T01:34:48.424086985Z"}
{"log":"I0322 01:34:48.423948       1 creator.go:41] Update ConfigMap test2/chi-version-update-common-usersd\n","stream":"stderr","time":"2020-03-22T01:34:48.42411308Z"}
{"log":"I0322 01:34:48.428197       1 creator.go:103] createServiceCluster(test2/cluster-version-update-update)\n","stream":"stderr","time":"2020-03-22T01:34:48.428334727Z"}
{"log":"I0322 01:34:48.428245       1 creator.go:122] createServiceShard(test2/shard-version-update-update-0)\n","stream":"stderr","time":"2020-03-22T01:34:48.428361551Z"}
{"log":"I0322 01:34:48.428372       1 creator.go:37] Reconcile ConfigMap test2/chi-version-update-deploy-confd-update-0-0\n","stream":"stderr","time":"2020-03-22T01:34:48.428486585Z"}
{"log":"I0322 01:34:48.428391       1 creator.go:41] Update ConfigMap test2/chi-version-update-deploy-confd-update-0-0\n","stream":"stderr","time":"2020-03-22T01:34:48.428519266Z"}
{"log":"I0322 01:34:48.432483       1 creator.go:374] getPodTemplate() statefulSet chi-version-update-update-0-0 use custom template clickhouse:19.16.14.65\n","stream":"stderr","time":"2020-03-22T01:34:48.432624831Z"}
{"log":"I0322 01:34:48.440764       1 controller.go:517] Update CHI status (test2/version-update)\n","stream":"stderr","time":"2020-03-22T01:34:48.440909194Z"}
{"log":"I0322 01:34:48.460518       1 creator.go:142] createServiceHost(test2/chi-version-update-update-0-0) for Set chi-version-update-update-0-0\n","stream":"stderr","time":"2020-03-22T01:34:48.460708189Z"}
{"log":"I0322 01:34:48.460560       1 creator.go:60] Reconcile Service test2/chi-version-update-update-0-0\n","stream":"stderr","time":"2020-03-22T01:34:48.460744861Z"}
{"log":"I0322 01:34:48.460576       1 creator.go:64] Update Service test2/chi-version-update-update-0-0\n","stream":"stderr","time":"2020-03-22T01:34:48.460762296Z"}
{"log":"I0322 01:34:48.467024       1 creator.go:122] createServiceShard(test2/shard-version-update-update-1)\n","stream":"stderr","time":"2020-03-22T01:34:48.4671543Z"}
{"log":"I0322 01:34:48.467170       1 creator.go:37] Reconcile ConfigMap test2/chi-version-update-deploy-confd-update-1-0\n","stream":"stderr","time":"2020-03-22T01:34:48.467258635Z"}
{"log":"I0322 01:34:48.467193       1 creator.go:41] Update ConfigMap test2/chi-version-update-deploy-confd-update-1-0\n","stream":"stderr","time":"2020-03-22T01:34:48.467282015Z"}
{"log":"I0322 01:34:48.471567       1 creator.go:374] getPodTemplate() statefulSet chi-version-update-update-1-0 use custom template clickhouse:19.16.14.65\n","stream":"stderr","time":"2020-03-22T01:34:48.47176613Z"}
{"log":"I0322 01:34:48.477146       1 controller.go:517] Update CHI status (test2/version-update)\n","stream":"stderr","time":"2020-03-22T01:34:48.477289182Z"}
{"log":"I0322 01:34:48.495839       1 creator.go:142] createServiceHost(test2/chi-version-update-update-1-0) for Set chi-version-update-update-1-0\n","stream":"stderr","time":"2020-03-22T01:34:48.495964781Z"}
{"log":"I0322 01:34:48.495881       1 creator.go:60] Reconcile Service test2/chi-version-update-update-1-0\n","stream":"stderr","time":"2020-03-22T01:34:48.49598589Z"}
{"log":"I0322 01:34:48.495896       1 creator.go:64] Update Service test2/chi-version-update-update-1-0\n","stream":"stderr","time":"2020-03-22T01:34:48.495992858Z"}
{"log":"I0322 01:34:48.501807       1 creator.go:122] createServiceShard(test2/shard-version-update-update-2)\n","stream":"stderr","time":"2020-03-22T01:34:48.501954309Z"}
{"log":"I0322 01:34:48.501984       1 creator.go:37] Reconcile ConfigMap test2/chi-version-update-deploy-confd-update-2-0\n","stream":"stderr","time":"2020-03-22T01:34:48.502056755Z"}
{"log":"I0322 01:34:48.502142       1 creator.go:47] Create ConfigMap test2/chi-version-update-deploy-confd-update-2-0\n","stream":"stderr","time":"2020-03-22T01:34:48.502228744Z"}
{"log":"I0322 01:34:48.506055       1 creator.go:374] getPodTemplate() statefulSet chi-version-update-update-2-0 use custom template clickhouse:19.16.14.65\n","stream":"stderr","time":"2020-03-22T01:34:48.506222254Z"}
{"log":"I0322 01:34:48.506356       1 creator.go:113] Create StatefulSet test2/chi-version-update-update-2-0\n","stream":"stderr","time":"2020-03-22T01:34:48.506442104Z"}
{"log":"I0322 01:34:48.511341       1 creator.go:189] waitStatefulSetGeneration(test2/chi-version-update-update-2-0)-WAIT: object not yet created, need to wait\n","stream":"stderr","time":"2020-03-22T01:34:48.511511404Z"}
{"log":"I0322 01:34:49.949371       1 controller.go:250] endpointsInformer UpdateFunc(test2/clickhouse-version-update) IP ASSIGNED [{[{10.244.0.167  0xc00065cef0 \u0026ObjectReference{Kind:Pod,Namespace:test2,Name:chi-version-update-update-0-0-0,UID:53a63706-3833-41cd-bef2-279da5958531,APIVersion:,ResourceVersion:24306556,FieldPath:,}} {10.244.0.168  0xc00065cf00 \u0026ObjectReference{Kind:Pod,Namespace:test2,Name:chi-version-update-update-1-0-0,UID:841004f2-1726-42b3-ade8-b82e1e1886d6,APIVersion:,ResourceVersion:24306591,FieldPath:,}} {10.244.0.169  0xc00065cf10 \u0026ObjectReference{Kind:Pod,Namespace:test2,Name:chi-version-update-update-2-0-0,UID:c6db3c27-59e7-4428-a4a8-eed203ccf38b,APIVersion:,ResourceVersion:24309888,FieldPath:,}}] [] [{tcp 9000 TCP} {http 8123 TCP}]}]\n","stream":"stderr","time":"2020-03-22T01:34:49.949752724Z"}
{"log":"I0322 01:34:53.511550       1 creator.go:174] waitStatefulSetGeneration(test2/chi-version-update-update-2-0)-OK  :ObservedGeneration:1 Replicas:1 ReadyReplicas:1 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-version-update-update-2-0-7df5c69796 UpdateRevision:chi-version-update-update-2-0-7df5c69796\n","stream":"stderr","time":"2020-03-22T01:34:53.51175269Z"}
{"log":"I0322 01:34:53.511590       1 controller.go:517] Update CHI status (test2/version-update)\n","stream":"stderr","time":"2020-03-22T01:34:53.511802848Z"}
{"log":"I0322 01:34:53.532684       1 creator.go:142] createServiceHost(test2/chi-version-update-update-2-0) for Set chi-version-update-update-2-0\n","stream":"stderr","time":"2020-03-22T01:34:53.532835735Z"}
{"log":"I0322 01:34:53.532733       1 creator.go:60] Reconcile Service test2/chi-version-update-update-2-0\n","stream":"stderr","time":"2020-03-22T01:34:53.532870929Z"}
{"log":"I0322 01:34:53.532931       1 creator.go:77] Create service test2/chi-version-update-update-2-0\n","stream":"stderr","time":"2020-03-22T01:34:53.533021662Z"}
{"log":"I0322 01:34:53.537457       1 worker.go:189] Added shard 2 to cluster update\n","stream":"stderr","time":"2020-03-22T01:34:53.537585234Z"}
{"log":"I0322 01:34:53.541201       1 schemer.go:61] clickhouse-version-update.test2.svc.cluster.local\n","stream":"stderr","time":"2020-03-22T01:34:53.541317666Z"}
{"log":"I0322 01:34:53.603546       1 worker.go:336] Creating distributed objects: [default]\n","stream":"stderr","time":"2020-03-22T01:34:53.603770762Z"}
{"log":"I0322 01:34:53.610035       1 worker.go:196] Added replica 0 to shard 2 in cluster update\n","stream":"stderr","time":"2020-03-22T01:34:53.610209972Z"}
{"log":"I0322 01:34:53.613665       1 schemer.go:61] clickhouse-version-update.test2.svc.cluster.local\n","stream":"stderr","time":"2020-03-22T01:34:53.613838059Z"}
{"log":"I0322 01:34:53.754901       1 clickhouse.go:97] FAILED Query(http://clickhouse_operator:clickhouse_operator_password@clickhouse-version-update.test2.svc.cluster.local:8123/) clickhouse: Code: 195, e.displayText() = DB::Exception: Received from chi-version-update-update-2-0.test2.svc.cluster.local:9000. DB::Exception: User default is not allowed to connect from address 10.244.0.168. (version 19.16.14.65 (official build))\n","stream":"stderr","time":"2020-03-22T01:34:53.755096042Z"}
{"log":" for SQL: SELECT DISTINCT \n","stream":"stderr","time":"2020-03-22T01:34:53.75512639Z"}
{"log":"\u0009database AS name, \n","stream":"stderr","time":"2020-03-22T01:34:53.755133961Z"}
{"log":"\u0009concat('CREATE DATABASE IF NOT EXISTS ', name) AS create_db_query\n","stream":"stderr","time":"2020-03-22T01:34:53.755157193Z"}
{"log":"FROM remote('chi-version-update-update-2-0.test2.svc.cluster.local', system, tables)\n","stream":"stderr","time":"2020-03-22T01:34:53.755163528Z"}
{"log":"WHERE engine_full LIKE 'Replicated%'\n","stream":"stderr","time":"2020-03-22T01:34:53.755169917Z"}
{"log":"SETTINGS skip_unavailable_shards = 1\n","stream":"stderr","time":"2020-03-22T01:34:53.755188971Z"}
{"log":"UNION ALL\n","stream":"stderr","time":"2020-03-22T01:34:53.755195465Z"}
{"log":"SELECT DISTINCT \n","stream":"stderr","time":"2020-03-22T01:34:53.755201686Z"}
{"log":"\u0009name, \n","stream":"stderr","time":"2020-03-22T01:34:53.755208057Z"}
{"log":"\u0009replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW)', 'CREATE \\\\1 IF NOT EXISTS')\n","stream":"stderr","time":"2020-03-22T01:34:53.755222293Z"}
{"log":"FROM remote('chi-version-update-update-2-0.test2.svc.cluster.local', system, tables)\n","stream":"stderr","time":"2020-03-22T01:34:53.755262993Z"}
{"log":"WHERE engine_full LIKE 'Replicated%%'\n","stream":"stderr","time":"2020-03-22T01:34:53.75527128Z"}
{"log":"SETTINGS skip_unavailable_shards = 1\n","stream":"stderr","time":"2020-03-22T01:34:53.75527754Z"}
{"log":"I0322 01:34:53.754937       1 worker.go:320] Creating replicated objects: []\n","stream":"stderr","time":"2020-03-22T01:34:53.755284225Z"}
{"log":"I0322 01:34:53.755281       1 schemer.go:61] clickhouse-version-update.test2.svc.cluster.local\n","stream":"stderr","time":"2020-03-22T01:34:53.755406726Z"}
{"log":"I0322 01:34:53.808996       1 worker.go:324] Creating distributed objects: [default cdp_orders]\n","stream":"stderr","time":"2020-03-22T01:34:53.80917639Z"}
{"log":"I0322 01:34:53.825855       1 controller.go:517] Update CHI status (test2/version-update)\n","stream":"stderr","time":"2020-03-22T01:34:53.826029336Z"}
{"log":"I0322 01:34:53.847029       1 worker.go:225] updateChi(test2/version-update) - complete reconcile \u003c\u003c\u003c\n","stream":"stderr","time":"2020-03-22T01:34:53.847202266Z"}
{"log":"I0322 01:34:53.854975       1 worker.go:103] endpointsInformer UpdateFunc(test2/clickhouse-version-update) flushing DNS for CHI version-update\n","stream":"stderr","time":"2020-03-22T01:34:53.855149268Z"}

I noticed there's an error inside:

DB::Exception: User default is not allowed to connect from address 10.244.0.168.DB::Exception: User default is not allowed to connect from address 10.244.0.168.

alex-zaitsev · 2020-04-08T11:47:13Z

Hi @yuzhichang ,

Logs is very helpful. Operator uses following query to get Distributed tables. Does it return everything properly on your system?

		SELECT DISTINCT 
			database AS name, 
			concat('CREATE DATABASE IF NOT EXISTS ', name) AS create_db_query
		FROM 
		(
			SELECT DISTINCT database
			FROM system.tables
			WHERE engine = 'Distributed'
			SETTINGS skip_unavailable_shards = 1
			UNION ALL
			SELECT DISTINCT extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS shard_database
			FROM system.tables 
			WHERE engine = 'Distributed'
			SETTINGS skip_unavailable_shards = 1
		) 
		UNION ALL
		SELECT DISTINCT 
			name, 
			replaceRegexpOne(create_table_query, 'CREATE (TABLE|VIEW|MATERIALIZED VIEW)', 'CREATE \\1 IF NOT EXISTS')
		FROM 
		(
			SELECT 
				database, 
				name,
				2 AS order
			FROM system.tables
			WHERE engine = 'Distributed'
			SETTINGS skip_unavailable_shards = 1
			UNION ALL
			SELECT 
				extract(engine_full, 'Distributed\\([^,]+, *\'?([^,\']+)\'?, *[^,]+') AS shard_database, 
				extract(engine_full, 'Distributed\\([^,]+, [^,]+, *\'?([^,\\\')]+)') AS shard_table,
				1 AS order
			FROM system.tables
			WHERE engine = 'Distributed'
			SETTINGS skip_unavailable_shards = 1
		) 
		LEFT JOIN (select distinct database, name, create_table_query from system.tables SETTINGS skip_unavailable_shards = 1) USING (database, name)
		ORDER BY order

Error message is not good. Operator uses a dedicated user to connect, but when remote() function is invoked it goes by default user that is restricted to the cluster node but should work.

Btw, please upgrade operator to 0.9.6 (you'll have to re-install it though, see release notes: https://github.com/Altinity/clickhouse-operator/releases)

yuzhichang · 2020-04-09T03:09:48Z

Hi @alex-zaitsev. I upgraded operator to 0.9.6, destroyed my clickhouse cluster, and retested. operator created database default on the new 3rd shard, but not table cdp_orders and dist_cdp_orders.

Operator log is
80fc8c9c494d94617e25c7e5678ccbe6ff80dfee80a5a1149ef563bc1998050d-json.log.

The query on extant 1st and 2nd shard both give:

┌─name────┬─create_db_query───────────────────────┐
│ default │ CREATE DATABASE IF NOT EXISTS default │
└─────────┴───────────────────────────────────────┘
┌─name────────────┬─create_db_query──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ cdp_orders      │ CREATE TABLE IF NOT EXISTS default.cdp_orders (`order_id` UInt64, `order_time` DateTime) ENGINE = ReplacingMergeTree() PARTITION BY toYYYYMMDD(order_time) ORDER BY (order_time, order_id) SETTINGS index_granularity = 8192 │
│ dist_cdp_orders │ CREATE TABLE IF NOT EXISTS default.dist_cdp_orders (`order_id` UInt64, `order_time` DateTime) ENGINE = Distributed('update', 'default', 'cdp_orders')                                                                        │
└─────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

alex-zaitsev · 2020-04-09T10:23:49Z

@ yuzhichang, may it be something with DNS resolution? Operator uses load balancer service.
Could you please try 0.9.7 (not yet released, but docker image is already tagged)? We moved away from LoadBalancer to pod services for schema management, and it also has somewhat better logging in this regard.

yuzhichang · 2020-04-09T13:56:08Z

@alex-zaitsev, operator 0.9.7 behaves the same with 0.9.6. Here's the operator log
ad5dc9e47c90bfaaeab7a50bcab8e2a736e4e066824211d466eb0bb34a188923-json.log

alex-zaitsev · 2020-04-09T15:11:36Z

@yuzhichang , thanks. I was finally able to see the similar issue in our environment, though it is not clear why is it happening.

alex-zaitsev · 2020-06-10T15:59:29Z

@yuzhichang, we have tweaked schema creation logic in 0.9.9 (image is updated without a release). It should be much more reliable. Could you check on your side, please?

yuzhichang · 2020-06-12T09:30:20Z

This issue applies to altinity/clickhouse-operator:0.9.9. Here's operator log when expand the ck cluster to 4 shards:

I0612 09:18:59.438279       1 controller.go:256] endpointsInformer UpdateFunc(dev/clickhouse-dev) IP ASSIGNED [{[{192.168.120.210  0xc000d48f30 &ObjectReference{Kind:Pod,Namespace:dev,Name:chi-dev-cdp-3-0-0,UID:1b362a4c-5af1-4930-9bed-7c431a714714,APIVersion:,ResourceVersion:12764336,FieldPath:,}} {192.168.175.173  0xc000d48f40 &ObjectReference{Kind:Pod,Namespace:dev,Name:chi-dev-cdp-2-0-0,UID:1dc3be01-b56d-41f3-a538-daa9f6f5058f,APIVersion:,ResourceVersion:12761732,FieldPath:,}} {192.168.178.97  0xc000d48f50 &ObjectReference{Kind:Pod,Namespace:dev,Name:chi-dev-cdp-0-0-0,UID:62e67e4e-0c5b-4096-af28-40613f5c7f3d,APIVersion:,ResourceVersion:12761321,FieldPath:,}} {192.168.6.160  0xc000d48f60 &ObjectReference{Kind:Pod,Namespace:dev,Name:chi-dev-cdp-1-0-0,UID:db8b4a66-e7d9-4cd8-91dc-ec40f57ef3aa,APIVersion:,ResourceVersion:12761552,FieldPath:,}}] [] [{tcp 9000 TCP} {http 8123 TCP}]}]
I0612 09:19:01.361162       1 creator.go:101] waitStatefulSetGeneration(dev/chi-dev-cdp-3-0)-OK  :ObservedGeneration:1 Replicas:1 ReadyReplicas:1 CurrentReplicas:1 UpdatedReplicas:1 CurrentRevision:chi-dev-cdp-3-0-688fc767f UpdateRevision:chi-dev-cdp-3-0-688fc767f
I0612 09:19:01.381679       1 announcer.go:99] Create StatefulSet dev/chi-dev-cdp-3-0 - completed
I0612 09:19:01.404698       1 creator.go:144] CreateServiceHost(dev/chi-dev-cdp-3-0) for Set chi-dev-cdp-3-0
I0612 09:19:01.412764       1 announcer.go:99] Create Service dev/chi-dev-cdp-3-0
I0612 09:19:01.434089       1 announcer.go:99] Reconcile Host 3-0 completed
I0612 09:19:01.456885       1 schemer.go:231] Migrating schema objects to host 3-0
I0612 09:19:01.456936       1 schemer.go:168] Single replica in a shard. Nothing to create from
I0612 09:19:01.456947       1 schemer.go:234] Creating replica objects [] at 3-0
I0612 09:19:01.642750       1 schemer.go:238] Creating distributed objects [default] at 3-0
I0612 09:19:01.645731       1 connection.go:126] FAILED Exec(http://***:***@chi-dev-cdp-3-0.dev.svc.cluster.local:8123/) dial tcp: lookup chi-dev-cdp-3-0.dev.svc.cluster.local on 169.169.0.10:53: no such host for SQL: CREATE DATABASE IF NOT EXISTS "default"
I0612 09:19:01.645759       1 retry.go:41] FAILED attempt 1 of 10, sleep 5 sec and retry: Applying sqls
I0612 09:19:06.650188       1 connection.go:126] FAILED Exec(http://***:***@chi-dev-cdp-3-0.dev.svc.cluster.local:8123/) dial tcp: lookup chi-dev-cdp-3-0.dev.svc.cluster.local on 169.169.0.10:53: no such host for SQL: CREATE DATABASE IF NOT EXISTS "default"
I0612 09:19:06.650241       1 retry.go:41] FAILED attempt 2 of 10, sleep 10 sec and retry: Applying sqls
I0612 09:19:16.653904       1 connection.go:126] FAILED Exec(http://***:***@chi-dev-cdp-3-0.dev.svc.cluster.local:8123/) dial tcp: lookup chi-dev-cdp-3-0.dev.svc.cluster.local on 169.169.0.10:53: no such host for SQL: CREATE DATABASE IF NOT EXISTS "default"
I0612 09:19:16.653961       1 retry.go:41] FAILED attempt 3 of 10, sleep 15 sec and retry: Applying sqls
I0612 09:19:31.663979       1 retry.go:33] DONE attempt 4 of 10: Applying sqls
I0612 09:19:31.664027       1 announcer.go:99] Created schema objects on host 3-0 replica 0 to shard 3 in cluster cdp
I0612 09:19:31.693610       1 announcer.go:99] updateCHI(dev/dev) remove scheduled for deletion items
I0612 09:19:31.719734       1 announcer.go:99] updateCHI(dev/dev) update monitoring list
I0612 09:19:31.763331       1 announcer.go:99] updateCHI(dev/dev) reconcile completed
I0612 09:19:32.363791       1 controller.go:256] endpointsInformer UpdateFunc(uat/clickhouse-uat) IP ASSIGNED [{[{192.168.41.60  0xc00115c380 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-0-0-0,UID:7084339f-b160-44f7-b08e-8e21403d1376,APIVersion:,ResourceVersion:12764543,FieldPath:,}} {192.168.41.62  0xc00115c390 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-6-0-0,UID:c1ffa5db-2f67-4af1-ad89-3b37995969e1,APIVersion:,ResourceVersion:12764238,FieldPath:,}} {192.168.56.178  0xc00115c3b0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-1-0-0,UID:04037e93-71df-4e19-8349-9053acfc3361,APIVersion:,ResourceVersion:12763754,FieldPath:,}}] [{192.168.235.118  0xc00115c3c0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-4-0-0,UID:498bab80-c969-4c8e-a329-d914434c862a,APIVersion:,ResourceVersion:12764508,FieldPath:,}} {192.168.40.243  0xc00115c3d0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-2-0-0,UID:217c1425-da4e-4d38-9468-6feb12d1e7ae,APIVersion:,ResourceVersion:12764462,FieldPath:,}} {192.168.41.61  0xc00115c400 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-5-0-0,UID:78556a52-3c2f-411a-9a56-c00424de42cc,APIVersion:,ResourceVersion:12764184,FieldPath:,}} {192.168.60.182  0xc00115c410 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-3-0-0,UID:9296b403-7498-4fce-8e57-8681b387ed69,APIVersion:,ResourceVersion:12764490,FieldPath:,}}] [{tcp 9000 TCP} {http 8123 TCP}]}]
I0612 09:19:32.374988       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.60:8123: connect: connection refused
I0612 09:19:32.375030       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:19:32.375038       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:19:32.377620       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) dial tcp 192.168.56.178:8123: connect: connection refused
I0612 09:19:32.377644       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:19:32.377650       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:19:32.381603       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) dial tcp 192.168.40.243:8123: connect: connection refused
I0612 09:19:32.381629       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:19:32.381652       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:19:32.384839       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) dial tcp 192.168.60.182:8123: connect: connection refused
I0612 09:19:32.384869       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:19:32.384874       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:19:32.387299       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) dial tcp 192.168.235.118:8123: connect: connection refused
I0612 09:19:32.387320       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:19:32.387326       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:19:32.389363       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.61:8123: connect: connection refused
I0612 09:19:32.389399       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:19:32.389409       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:19:32.391482       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.62:8123: connect: connection refused
I0612 09:19:32.391503       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:19:32.391525       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:19:43.607559       1 controller.go:256] endpointsInformer UpdateFunc(uat/clickhouse-uat) IP ASSIGNED [{[{192.168.41.60  0xc0009a6970 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-0-0-0,UID:7084339f-b160-44f7-b08e-8e21403d1376,APIVersion:,ResourceVersion:12764543,FieldPath:,}} {192.168.41.61  0xc0009a6980 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-5-0-0,UID:78556a52-3c2f-411a-9a56-c00424de42cc,APIVersion:,ResourceVersion:12764617,FieldPath:,}} {192.168.41.62  0xc0009a6990 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-6-0-0,UID:c1ffa5db-2f67-4af1-ad89-3b37995969e1,APIVersion:,ResourceVersion:12764238,FieldPath:,}}] [{192.168.235.118  0xc0009a69c0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-4-0-0,UID:498bab80-c969-4c8e-a329-d914434c862a,APIVersion:,ResourceVersion:12764508,FieldPath:,}} {192.168.40.243  0xc0009a69d0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-2-0-0,UID:217c1425-da4e-4d38-9468-6feb12d1e7ae,APIVersion:,ResourceVersion:12764560,FieldPath:,}} {192.168.56.178  0xc0009a69e0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-1-0-0,UID:04037e93-71df-4e19-8349-9053acfc3361,APIVersion:,ResourceVersion:12764611,FieldPath:,}} {192.168.60.182  0xc0009a6a00 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-3-0-0,UID:9296b403-7498-4fce-8e57-8681b387ed69,APIVersion:,ResourceVersion:12764595,FieldPath:,}}] [{tcp 9000 TCP} {http 8123 TCP}]}]
I0612 09:19:43.616094       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.60:8123: connect: connection refused
I0612 09:19:43.616131       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:19:43.616139       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:19:43.618828       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) dial tcp 192.168.56.178:8123: connect: connection refused
I0612 09:19:43.618854       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:19:43.618860       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:19:43.621266       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) dial tcp 192.168.40.243:8123: connect: connection refused
I0612 09:19:43.621336       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:19:43.621364       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:19:43.624238       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) dial tcp 192.168.60.182:8123: connect: connection refused
I0612 09:19:43.624304       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:19:43.624321       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:19:43.626809       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) dial tcp 192.168.235.118:8123: connect: connection refused
I0612 09:19:43.626871       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:19:43.626893       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:19:43.630165       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.61:8123: connect: connection refused
I0612 09:19:43.630236       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:19:43.630262       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:19:43.633070       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.62:8123: connect: connection refused
I0612 09:19:43.633168       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:19:43.633216       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:20:41.245271       1 controller.go:256] endpointsInformer UpdateFunc(uat/clickhouse-uat) IP ASSIGNED [{[{192.168.235.118  0xc0004a95a0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-4-0-0,UID:498bab80-c969-4c8e-a329-d914434c862a,APIVersion:,ResourceVersion:12764922,FieldPath:,}} {192.168.41.60  0xc0004a95b0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-0-0-0,UID:7084339f-b160-44f7-b08e-8e21403d1376,APIVersion:,ResourceVersion:12764543,FieldPath:,}} {192.168.41.61  0xc0004a95c0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-5-0-0,UID:78556a52-3c2f-411a-9a56-c00424de42cc,APIVersion:,ResourceVersion:12764617,FieldPath:,}}] [{192.168.40.243  0xc0004a95d0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-2-0-0,UID:217c1425-da4e-4d38-9468-6feb12d1e7ae,APIVersion:,ResourceVersion:12764560,FieldPath:,}} {192.168.41.62  0xc0004a95e0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-6-0-0,UID:c1ffa5db-2f67-4af1-ad89-3b37995969e1,APIVersion:,ResourceVersion:12764906,FieldPath:,}} {192.168.56.178  0xc0004a95f0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-1-0-0,UID:04037e93-71df-4e19-8349-9053acfc3361,APIVersion:,ResourceVersion:12764678,FieldPath:,}} {192.168.60.182  0xc0004a9600 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-3-0-0,UID:9296b403-7498-4fce-8e57-8681b387ed69,APIVersion:,ResourceVersion:12764595,FieldPath:,}}] [{tcp 9000 TCP} {http 8123 TCP}]}]
I0612 09:20:41.255021       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.60:8123: connect: connection refused
I0612 09:20:41.255120       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:20:41.255141       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:20:41.258719       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) dial tcp 192.168.56.178:8123: connect: connection refused
I0612 09:20:41.258779       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:20:41.258795       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:20:41.261893       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) dial tcp 192.168.40.243:8123: connect: connection refused
I0612 09:20:41.261954       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:20:41.261970       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:20:41.265062       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) dial tcp 192.168.60.182:8123: connect: connection refused
I0612 09:20:41.265100       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:20:41.265107       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:20:41.267481       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) dial tcp 192.168.235.118:8123: connect: connection refused
I0612 09:20:41.267507       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:20:41.267512       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:20:41.270605       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.61:8123: connect: connection refused
I0612 09:20:41.270633       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:20:41.270640       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:20:41.272571       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.62:8123: connect: connection refused
I0612 09:20:41.272599       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:20:41.272608       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:20:50.004981       1 controller.go:256] endpointsInformer UpdateFunc(uat/clickhouse-uat) IP ASSIGNED [{[{192.168.235.118  0xc00102fac0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-4-0-0,UID:498bab80-c969-4c8e-a329-d914434c862a,APIVersion:,ResourceVersion:12764922,FieldPath:,}} {192.168.41.60  0xc00102fad0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-0-0-0,UID:7084339f-b160-44f7-b08e-8e21403d1376,APIVersion:,ResourceVersion:12764543,FieldPath:,}} {192.168.41.61  0xc00102fae0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-5-0-0,UID:78556a52-3c2f-411a-9a56-c00424de42cc,APIVersion:,ResourceVersion:12764617,FieldPath:,}} {192.168.60.182  0xc00102faf0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-3-0-0,UID:9296b403-7498-4fce-8e57-8681b387ed69,APIVersion:,ResourceVersion:12764973,FieldPath:,}}] [{192.168.40.243  0xc00102fb00 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-2-0-0,UID:217c1425-da4e-4d38-9468-6feb12d1e7ae,APIVersion:,ResourceVersion:12764560,FieldPath:,}} {192.168.41.62  0xc00102fb10 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-6-0-0,UID:c1ffa5db-2f67-4af1-ad89-3b37995969e1,APIVersion:,ResourceVersion:12764906,FieldPath:,}} {192.168.56.178  0xc00102fb20 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-1-0-0,UID:04037e93-71df-4e19-8349-9053acfc3361,APIVersion:,ResourceVersion:12764678,FieldPath:,}}] [{tcp 9000 TCP} {http 8123 TCP}]}]
I0612 09:20:50.014257       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.60:8123: connect: connection refused
I0612 09:20:50.014296       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:20:50.014322       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:20:50.017899       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) dial tcp 192.168.56.178:8123: connect: connection refused
I0612 09:20:50.017936       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:20:50.017947       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:20:50.021117       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) dial tcp 192.168.40.243:8123: connect: connection refused
I0612 09:20:50.021185       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:20:50.021204       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:20:50.024939       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) dial tcp 192.168.60.182:8123: connect: connection refused
I0612 09:20:50.024998       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:20:50.025018       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:20:50.027427       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) dial tcp 192.168.235.118:8123: connect: connection refused
I0612 09:20:50.027455       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:20:50.027463       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:20:50.030760       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.61:8123: connect: connection refused
I0612 09:20:50.030829       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:20:50.030849       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:20:50.034329       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.62:8123: connect: connection refused
I0612 09:20:50.034355       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:20:50.034363       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:21:46.175070       1 controller.go:256] endpointsInformer UpdateFunc(uat/clickhouse-uat) IP ASSIGNED [{[{192.168.235.118  0xc000a5e340 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-4-0-0,UID:498bab80-c969-4c8e-a329-d914434c862a,APIVersion:,ResourceVersion:12764922,FieldPath:,}} {192.168.41.62  0xc000a5e350 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-6-0-0,UID:c1ffa5db-2f67-4af1-ad89-3b37995969e1,APIVersion:,ResourceVersion:12765271,FieldPath:,}} {192.168.60.182  0xc000a5e360 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-3-0-0,UID:9296b403-7498-4fce-8e57-8681b387ed69,APIVersion:,ResourceVersion:12764973,FieldPath:,}}] [{192.168.40.243  0xc000a5e370 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-2-0-0,UID:217c1425-da4e-4d38-9468-6feb12d1e7ae,APIVersion:,ResourceVersion:12764560,FieldPath:,}} {192.168.41.60  0xc000a5e380 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-0-0-0,UID:7084339f-b160-44f7-b08e-8e21403d1376,APIVersion:,ResourceVersion:12765213,FieldPath:,}} {192.168.41.61  0xc000a5e390 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-5-0-0,UID:78556a52-3c2f-411a-9a56-c00424de42cc,APIVersion:,ResourceVersion:12765086,FieldPath:,}} {192.168.56.178  0xc000a5e3a0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-1-0-0,UID:04037e93-71df-4e19-8349-9053acfc3361,APIVersion:,ResourceVersion:12764678,FieldPath:,}}] [{tcp 9000 TCP} {http 8123 TCP}]}]
I0612 09:21:46.185514       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.60:8123: connect: connection refused
I0612 09:21:46.185548       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:21:46.185556       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:21:46.189136       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) dial tcp 192.168.56.178:8123: connect: connection refused
I0612 09:21:46.189211       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:21:46.189227       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:21:46.191901       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) dial tcp 192.168.40.243:8123: connect: connection refused
I0612 09:21:46.191964       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:21:46.191980       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:21:46.194447       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) dial tcp 192.168.60.182:8123: connect: connection refused
I0612 09:21:46.194484       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:21:46.194489       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:21:46.196859       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) dial tcp 192.168.235.118:8123: connect: connection refused
I0612 09:21:46.196885       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:21:46.196890       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:21:46.199053       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.61:8123: connect: connection refused
I0612 09:21:46.199093       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:21:46.199120       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:21:46.201449       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.62:8123: connect: connection refused
I0612 09:21:46.201472       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:21:46.201478       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:22:11.873873       1 controller.go:256] endpointsInformer UpdateFunc(uat/clickhouse-uat) IP ASSIGNED [{[{192.168.40.243  0xc00047b7c0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-2-0-0,UID:217c1425-da4e-4d38-9468-6feb12d1e7ae,APIVersion:,ResourceVersion:12765421,FieldPath:,}} {192.168.41.62  0xc00047b7d0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-6-0-0,UID:c1ffa5db-2f67-4af1-ad89-3b37995969e1,APIVersion:,ResourceVersion:12765271,FieldPath:,}}] [{192.168.235.118  0xc00047b7f0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-4-0-0,UID:498bab80-c969-4c8e-a329-d914434c862a,APIVersion:,ResourceVersion:12765377,FieldPath:,}} {192.168.41.60  0xc00047b810 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-0-0-0,UID:7084339f-b160-44f7-b08e-8e21403d1376,APIVersion:,ResourceVersion:12765213,FieldPath:,}} {192.168.41.61  0xc00047b820 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-5-0-0,UID:78556a52-3c2f-411a-9a56-c00424de42cc,APIVersion:,ResourceVersion:12765086,FieldPath:,}} {192.168.56.178  0xc00047b830 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-1-0-0,UID:04037e93-71df-4e19-8349-9053acfc3361,APIVersion:,ResourceVersion:12764678,FieldPath:,}} {192.168.60.182  0xc00047b850 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-3-0-0,UID:9296b403-7498-4fce-8e57-8681b387ed69,APIVersion:,ResourceVersion:12765359,FieldPath:,}}] [{tcp 9000 TCP} {http 8123 TCP}]}]
I0612 09:22:11.883610       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.60:8123: connect: connection refused
I0612 09:22:11.883648       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:22:11.883657       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:22:11.886155       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) dial tcp 192.168.56.178:8123: connect: connection refused
I0612 09:22:11.886185       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:22:11.886193       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:22:11.888650       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) dial tcp 192.168.40.243:8123: connect: connection refused
I0612 09:22:11.888676       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:22:11.888682       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:22:11.891070       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) dial tcp 192.168.60.182:8123: connect: connection refused
I0612 09:22:11.891111       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:22:11.891120       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:22:11.893306       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) dial tcp 192.168.235.118:8123: connect: connection refused
I0612 09:22:11.893331       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:22:11.893337       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:22:11.895856       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.61:8123: connect: connection refused
I0612 09:22:11.895880       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:22:11.895887       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:22:11.897840       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.62:8123: connect: connection refused
I0612 09:22:11.897866       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:22:11.897898       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:24:44.168624       1 controller.go:256] endpointsInformer UpdateFunc(uat/clickhouse-uat) IP ASSIGNED [{[{192.168.235.118  0xc000d6ad10 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-4-0-0,UID:498bab80-c969-4c8e-a329-d914434c862a,APIVersion:,ResourceVersion:12766138,FieldPath:,}} {192.168.60.182  0xc000d6ad20 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-3-0-0,UID:9296b403-7498-4fce-8e57-8681b387ed69,APIVersion:,ResourceVersion:12766217,FieldPath:,}}] [{192.168.40.243  0xc000d6ad30 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-2-0-0,UID:217c1425-da4e-4d38-9468-6feb12d1e7ae,APIVersion:,ResourceVersion:12765734,FieldPath:,}} {192.168.41.60  0xc000d6ad40 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-0-0-0,UID:7084339f-b160-44f7-b08e-8e21403d1376,APIVersion:,ResourceVersion:12765213,FieldPath:,}} {192.168.41.61  0xc000d6ad50 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-5-0-0,UID:78556a52-3c2f-411a-9a56-c00424de42cc,APIVersion:,ResourceVersion:12766185,FieldPath:,}} {192.168.41.62  0xc000d6ad60 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-6-0-0,UID:c1ffa5db-2f67-4af1-ad89-3b37995969e1,APIVersion:,ResourceVersion:12765691,FieldPath:,}} {192.168.56.178  0xc000d6ad70 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-1-0-0,UID:04037e93-71df-4e19-8349-9053acfc3361,APIVersion:,ResourceVersion:12764678,FieldPath:,}}] [{tcp 9000 TCP} {http 8123 TCP}]}]
I0612 09:24:44.177791       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.60:8123: connect: connection refused
I0612 09:24:44.177832       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:24:44.177840       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:24:44.181112       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) dial tcp 192.168.56.178:8123: connect: connection refused
I0612 09:24:44.181180       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:24:44.181198       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:24:44.184282       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) dial tcp 192.168.40.243:8123: connect: connection refused
I0612 09:24:44.184357       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:24:44.184374       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:24:44.186687       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) dial tcp 192.168.60.182:8123: connect: connection refused
I0612 09:24:44.186712       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:24:44.186717       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:24:44.189343       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) dial tcp 192.168.235.118:8123: connect: connection refused
I0612 09:24:44.189380       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:24:44.189387       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:24:44.191887       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.61:8123: connect: connection refused
I0612 09:24:44.191914       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:24:44.191921       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:24:44.194188       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.62:8123: connect: connection refused
I0612 09:24:44.194217       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:24:44.194223       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:24:54.153444       1 controller.go:256] endpointsInformer UpdateFunc(uat/clickhouse-uat) IP ASSIGNED [{[{192.168.235.118  0xc000cc2e00 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-4-0-0,UID:498bab80-c969-4c8e-a329-d914434c862a,APIVersion:,ResourceVersion:12766138,FieldPath:,}} {192.168.56.178  0xc000cc2e20 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-1-0-0,UID:04037e93-71df-4e19-8349-9053acfc3361,APIVersion:,ResourceVersion:12766273,FieldPath:,}} {192.168.60.182  0xc000cc2e30 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-3-0-0,UID:9296b403-7498-4fce-8e57-8681b387ed69,APIVersion:,ResourceVersion:12766217,FieldPath:,}}] [{192.168.40.243  0xc000cc2e90 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-2-0-0,UID:217c1425-da4e-4d38-9468-6feb12d1e7ae,APIVersion:,ResourceVersion:12765734,FieldPath:,}} {192.168.41.60  0xc000cc2ea0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-0-0-0,UID:7084339f-b160-44f7-b08e-8e21403d1376,APIVersion:,ResourceVersion:12765213,FieldPath:,}} {192.168.41.61  0xc000cc2eb0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-5-0-0,UID:78556a52-3c2f-411a-9a56-c00424de42cc,APIVersion:,ResourceVersion:12766185,FieldPath:,}} {192.168.41.62  0xc000cc2ee0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-6-0-0,UID:c1ffa5db-2f67-4af1-ad89-3b37995969e1,APIVersion:,ResourceVersion:12765691,FieldPath:,}}] [{tcp 9000 TCP} {http 8123 TCP}]}]
I0612 09:24:54.162973       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.60:8123: connect: connection refused
I0612 09:24:54.163017       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:24:54.163028       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:24:54.165604       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) dial tcp 192.168.56.178:8123: connect: connection refused
I0612 09:24:54.165635       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:24:54.165644       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:24:54.168839       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) dial tcp 192.168.40.243:8123: connect: connection refused
I0612 09:24:54.168870       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:24:54.168877       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:24:54.171448       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) dial tcp 192.168.60.182:8123: connect: connection refused
I0612 09:24:54.171479       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:24:54.171485       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:24:54.174126       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) dial tcp 192.168.235.118:8123: connect: connection refused
I0612 09:24:54.174159       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:24:54.174170       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:24:54.176560       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.61:8123: connect: connection refused
I0612 09:24:54.176591       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:24:54.176600       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:24:54.179366       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.62:8123: connect: connection refused
I0612 09:24:54.179395       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:24:54.179401       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:25:30.608203       1 controller.go:256] endpointsInformer UpdateFunc(uat/clickhouse-uat) IP ASSIGNED [{[{192.168.235.118  0xc001579ab0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-4-0-0,UID:498bab80-c969-4c8e-a329-d914434c862a,APIVersion:,ResourceVersion:12766138,FieldPath:,}} {192.168.41.62  0xc001579ac0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-6-0-0,UID:c1ffa5db-2f67-4af1-ad89-3b37995969e1,APIVersion:,ResourceVersion:12766456,FieldPath:,}} {192.168.56.178  0xc001579ad0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-1-0-0,UID:04037e93-71df-4e19-8349-9053acfc3361,APIVersion:,ResourceVersion:12766273,FieldPath:,}} {192.168.60.182  0xc001579ae0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-3-0-0,UID:9296b403-7498-4fce-8e57-8681b387ed69,APIVersion:,ResourceVersion:12766217,FieldPath:,}}] [{192.168.40.243  0xc001579af0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-2-0-0,UID:217c1425-da4e-4d38-9468-6feb12d1e7ae,APIVersion:,ResourceVersion:12765734,FieldPath:,}} {192.168.41.60  0xc001579b00 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-0-0-0,UID:7084339f-b160-44f7-b08e-8e21403d1376,APIVersion:,ResourceVersion:12765213,FieldPath:,}} {192.168.41.61  0xc001579b10 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-5-0-0,UID:78556a52-3c2f-411a-9a56-c00424de42cc,APIVersion:,ResourceVersion:12766185,FieldPath:,}}] [{tcp 9000 TCP} {http 8123 TCP}]}]
I0612 09:25:30.619403       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.60:8123: connect: connection refused
I0612 09:25:30.619440       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:25:30.619449       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:25:30.622692       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) dial tcp 192.168.56.178:8123: connect: connection refused
I0612 09:25:30.622760       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:25:30.622777       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:25:30.626421       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) dial tcp 192.168.40.243:8123: connect: connection refused
I0612 09:25:30.626486       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:25:30.626502       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:25:30.630215       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) dial tcp 192.168.60.182:8123: connect: connection refused
I0612 09:25:30.630244       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:25:30.630251       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:25:30.634108       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) dial tcp 192.168.235.118:8123: connect: connection refused
I0612 09:25:30.634168       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:25:30.634183       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:25:30.637142       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.61:8123: connect: connection refused
I0612 09:25:30.637191       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:25:30.637219       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:25:30.640045       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.62:8123: connect: connection refused
I0612 09:25:30.640080       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:25:30.640092       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:26:29.877682       1 controller.go:256] endpointsInformer UpdateFunc(uat/clickhouse-uat) IP ASSIGNED [{[{192.168.41.60  0xc001427fc0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-0-0-0,UID:7084339f-b160-44f7-b08e-8e21403d1376,APIVersion:,ResourceVersion:12766771,FieldPath:,}} {192.168.41.62  0xc001427fd0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-6-0-0,UID:c1ffa5db-2f67-4af1-ad89-3b37995969e1,APIVersion:,ResourceVersion:12766456,FieldPath:,}}] [{192.168.235.118  0xc001427fe0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-4-0-0,UID:498bab80-c969-4c8e-a329-d914434c862a,APIVersion:,ResourceVersion:12766609,FieldPath:,}} {192.168.40.243  0xc001427ff0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-2-0-0,UID:217c1425-da4e-4d38-9468-6feb12d1e7ae,APIVersion:,ResourceVersion:12765734,FieldPath:,}} {192.168.41.61  0xc000a16000 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-5-0-0,UID:78556a52-3c2f-411a-9a56-c00424de42cc,APIVersion:,ResourceVersion:12766185,FieldPath:,}} {192.168.56.178  0xc000a16010 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-1-0-0,UID:04037e93-71df-4e19-8349-9053acfc3361,APIVersion:,ResourceVersion:12766759,FieldPath:,}} {192.168.60.182  0xc000a16040 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-3-0-0,UID:9296b403-7498-4fce-8e57-8681b387ed69,APIVersion:,ResourceVersion:12766735,FieldPath:,}}] [{tcp 9000 TCP} {http 8123 TCP}]}]
I0612 09:26:29.886714       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.60:8123: connect: connection refused
I0612 09:26:29.886753       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:26:29.886761       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:26:29.889085       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) dial tcp 192.168.56.178:8123: connect: connection refused
I0612 09:26:29.889117       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:26:29.889124       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:26:29.891646       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) dial tcp 192.168.40.243:8123: connect: connection refused
I0612 09:26:29.891674       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:26:29.891681       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:26:29.893752       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) dial tcp 192.168.60.182:8123: connect: connection refused
I0612 09:26:29.893780       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:26:29.893786       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:26:29.896118       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) dial tcp 192.168.235.118:8123: connect: connection refused
I0612 09:26:29.896143       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:26:29.896149       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:26:29.898860       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.61:8123: connect: connection refused
I0612 09:26:29.898888       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:26:29.898895       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:26:29.901098       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.62:8123: connect: connection refused
I0612 09:26:29.901139       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:26:29.901161       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:31:18.591616       1 controller.go:256] endpointsInformer UpdateFunc(uat/clickhouse-uat) IP ASSIGNED [{[{192.168.235.118  0xc000a18830 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-4-0-0,UID:498bab80-c969-4c8e-a329-d914434c862a,APIVersion:,ResourceVersion:12768098,FieldPath:,}} {192.168.56.178  0xc000a18850 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-1-0-0,UID:04037e93-71df-4e19-8349-9053acfc3361,APIVersion:,ResourceVersion:12768244,FieldPath:,}}] [{192.168.40.243  0xc000a18860 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-2-0-0,UID:217c1425-da4e-4d38-9468-6feb12d1e7ae,APIVersion:,ResourceVersion:12767485,FieldPath:,}} {192.168.41.60  0xc000a18880 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-0-0-0,UID:7084339f-b160-44f7-b08e-8e21403d1376,APIVersion:,ResourceVersion:12767111,FieldPath:,}} {192.168.41.61  0xc000a18890 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-5-0-0,UID:78556a52-3c2f-411a-9a56-c00424de42cc,APIVersion:,ResourceVersion:12767968,FieldPath:,}} {192.168.41.62  0xc000a188b0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-6-0-0,UID:c1ffa5db-2f67-4af1-ad89-3b37995969e1,APIVersion:,ResourceVersion:12766906,FieldPath:,}} {192.168.60.182  0xc000a188c0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-3-0-0,UID:9296b403-7498-4fce-8e57-8681b387ed69,APIVersion:,ResourceVersion:12766824,FieldPath:,}}] [{tcp 9000 TCP} {http 8123 TCP}]}]
I0612 09:31:18.600929       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.60:8123: connect: connection refused
I0612 09:31:18.600969       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:31:18.600977       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:31:18.603397       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) dial tcp 192.168.56.178:8123: connect: connection refused
I0612 09:31:18.603456       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:31:18.603471       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:31:18.605991       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) dial tcp 192.168.40.243:8123: connect: connection refused
I0612 09:31:18.606092       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:31:18.606111       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:31:18.608508       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) dial tcp 192.168.60.182:8123: connect: connection refused
I0612 09:31:18.608570       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:31:18.608586       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:31:18.611432       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) dial tcp 192.168.235.118:8123: connect: connection refused
I0612 09:31:18.611461       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:31:18.611467       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:31:18.613693       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.61:8123: connect: connection refused
I0612 09:31:18.613739       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:31:18.613745       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:31:18.615961       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.62:8123: connect: connection refused
I0612 09:31:18.615987       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:31:18.615993       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:31:29.255270       1 controller.go:256] endpointsInformer UpdateFunc(uat/clickhouse-uat) IP ASSIGNED [{[{192.168.235.118  0xc000a18f70 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-4-0-0,UID:498bab80-c969-4c8e-a329-d914434c862a,APIVersion:,ResourceVersion:12768098,FieldPath:,}} {192.168.56.178  0xc000a18f80 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-1-0-0,UID:04037e93-71df-4e19-8349-9053acfc3361,APIVersion:,ResourceVersion:12768244,FieldPath:,}} {192.168.60.182  0xc000a18f90 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-3-0-0,UID:9296b403-7498-4fce-8e57-8681b387ed69,APIVersion:,ResourceVersion:12768298,FieldPath:,}}] [{192.168.40.243  0xc000a18fa0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-2-0-0,UID:217c1425-da4e-4d38-9468-6feb12d1e7ae,APIVersion:,ResourceVersion:12767485,FieldPath:,}} {192.168.41.60  0xc000a18fb0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-0-0-0,UID:7084339f-b160-44f7-b08e-8e21403d1376,APIVersion:,ResourceVersion:12767111,FieldPath:,}} {192.168.41.61  0xc000a18fd0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-5-0-0,UID:78556a52-3c2f-411a-9a56-c00424de42cc,APIVersion:,ResourceVersion:12767968,FieldPath:,}} {192.168.41.62  0xc000a18fe0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-6-0-0,UID:c1ffa5db-2f67-4af1-ad89-3b37995969e1,APIVersion:,ResourceVersion:12766906,FieldPath:,}}] [{tcp 9000 TCP} {http 8123 TCP}]}]
I0612 09:31:29.265219       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.60:8123: connect: connection refused
I0612 09:31:29.265270       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:31:29.265280       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:31:29.267832       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) dial tcp 192.168.56.178:8123: connect: connection refused
I0612 09:31:29.267903       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:31:29.267923       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:31:29.270045       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) dial tcp 192.168.40.243:8123: connect: connection refused
I0612 09:31:29.270124       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:31:29.270141       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:31:29.272678       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) dial tcp 192.168.60.182:8123: connect: connection refused
I0612 09:31:29.272733       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:31:29.272750       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:31:29.275269       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) dial tcp 192.168.235.118:8123: connect: connection refused
I0612 09:31:29.275311       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:31:29.275322       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:31:29.277874       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.61:8123: connect: connection refused
I0612 09:31:29.277980       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:31:29.277998       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:31:29.280150       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.62:8123: connect: connection refused
I0612 09:31:29.280204       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:31:29.280220       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:31:44.238159       1 controller.go:256] endpointsInformer UpdateFunc(uat/clickhouse-uat) IP ASSIGNED [{[{192.168.41.62  0xc001688270 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-6-0-0,UID:c1ffa5db-2f67-4af1-ad89-3b37995969e1,APIVersion:,ResourceVersion:12768378,FieldPath:,}} {192.168.56.178  0xc0016882a0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-1-0-0,UID:04037e93-71df-4e19-8349-9053acfc3361,APIVersion:,ResourceVersion:12768244,FieldPath:,}} {192.168.60.182  0xc0016882b0 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-3-0-0,UID:9296b403-7498-4fce-8e57-8681b387ed69,APIVersion:,ResourceVersion:12768298,FieldPath:,}}] [{192.168.235.118  0xc001688300 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-4-0-0,UID:498bab80-c969-4c8e-a329-d914434c862a,APIVersion:,ResourceVersion:12768320,FieldPath:,}} {192.168.40.243  0xc001688320 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-2-0-0,UID:217c1425-da4e-4d38-9468-6feb12d1e7ae,APIVersion:,ResourceVersion:12767485,FieldPath:,}} {192.168.41.60  0xc001688330 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-0-0-0,UID:7084339f-b160-44f7-b08e-8e21403d1376,APIVersion:,ResourceVersion:12767111,FieldPath:,}} {192.168.41.61  0xc001688350 &ObjectReference{Kind:Pod,Namespace:uat,Name:chi-uat-cdp-5-0-0,UID:78556a52-3c2f-411a-9a56-c00424de42cc,APIVersion:,ResourceVersion:12767968,FieldPath:,}}] [{tcp 9000 TCP} {http 8123 TCP}]}]
I0612 09:31:44.247485       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.60:8123: connect: connection refused
I0612 09:31:44.247583       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-0-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:31:44.247603       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:31:44.250312       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) dial tcp 192.168.56.178:8123: connect: connection refused
I0612 09:31:44.250388       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-1-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:31:44.250416       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:31:44.253318       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) dial tcp 192.168.40.243:8123: connect: connection refused
I0612 09:31:44.253376       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-2-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:31:44.253393       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:31:44.255931       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) dial tcp 192.168.60.182:8123: connect: connection refused
I0612 09:31:44.255987       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-3-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:31:44.256006       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:31:44.258729       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) dial tcp 192.168.235.118:8123: connect: connection refused
I0612 09:31:44.258767       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-4-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:31:44.258778       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:31:44.261291       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.61:8123: connect: connection refused
I0612 09:31:44.261318       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-5-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:31:44.261324       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls
I0612 09:31:44.263522       1 connection.go:54] FAILED Ping(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) dial tcp 192.168.41.62:8123: connect: connection refused
I0612 09:31:44.263565       1 connection.go:119] FAILED connect(http://***:***@chi-uat-cdp-6-0.uat.svc.cluster.local:8123/) for SQL: SYSTEM DROP DNS CACHE
I0612 09:31:44.263577       1 retry.go:47] FAILED single try. No retries will be made for Applying sqls

I also tried altinity/clickhouse-operator:0.10.0. All deployed ck pods loop in restarting since update operator to 0.10.0. The following is the clickhouse-server stdout:

[16:37:55]➜  clickhouse git:(master) kubectl logs -f -n uat chi-uat-cdp-0-0-0                                                                                                       [16:47:17]
Processing configuration file '/etc/clickhouse-server/config.xml'.
Merging configuration file '/etc/clickhouse-server/conf.d/macros.xml'.
Merging configuration file '/etc/clickhouse-server/conf.d/zookeeper.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/01-clickhouse-listen.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/02-clickhouse-logger.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/03-clickhouse-querylog.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/remote_servers.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/settings.xml'.
Include not found: clickhouse_remote_servers
Include not found: clickhouse_compression
Logging debug to /var/log/clickhouse-server/clickhouse-server.log
Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
Logging debug to console
2020.06.12 16:47:18.745754 [ 1 ] {} <Information> : Starting ClickHouse 20.4.5.36 with revision 54434, PID 1
2020.06.12 16:47:18.749100 [ 1 ] {} <Information> Application: starting up
2020.06.12 16:47:18.761819 [ 1 ] {} <Debug> Application: rlimit on number of file descriptors is 1048576
2020.06.12 16:47:18.761871 [ 1 ] {} <Debug> Application: Initializing DateLUT.
2020.06.12 16:47:18.761924 [ 1 ] {} <Debug> Application: Setting up /var/lib/clickhouse/tmp/ to store temporary data in it
2020.06.12 16:47:18.762344 [ 1 ] {} <Debug> Application: Configuration parameter 'interserver_http_host' doesn't exist or exists and empty. Will use 'chi-uat-cdp-0-0-0.chi-uat-cdp-0-0.uat.svc.cluster.local' as replica host.
2020.06.12 16:47:18.766673 [ 1 ] {} <Debug> ConfigReloader: Loading config '/etc/clickhouse-server/users.xml'
Processing configuration file '/etc/clickhouse-server/users.xml'.
Merging configuration file '/etc/clickhouse-server/conf.d/macros.xml'.
Merging configuration file '/etc/clickhouse-server/conf.d/zookeeper.xml'.
Merging configuration file '/etc/clickhouse-server/users.d/01-clickhouse-user.xml'.
Merging configuration file '/etc/clickhouse-server/users.d/02-clickhouse-default-profile.xml'.
Merging configuration file '/etc/clickhouse-server/users.d/profiles.xml'.
Merging configuration file '/etc/clickhouse-server/users.d/users.xml'.
Include not found: networks
Saved preprocessed configuration to '/var/lib/clickhouse//preprocessed_configs/users.xml'.
2020.06.12 16:47:21.008142 [ 1 ] {} <Information> Application: Shutting down storages.
2020.06.12 16:47:21.008268 [ 1 ] {} <Debug> Application: Shut down storages.
2020.06.12 16:47:21.009661 [ 1 ] {} <Debug> Application: Destroyed global context.
2020.06.12 16:47:22.855762 [ 1 ] {} <Error> Application: DB::Exception: Cannot parse input: expected 'eof' before: 'e+10'
2020.06.12 16:47:22.856036 [ 1 ] {} <Information> Application: shutting down
2020.06.12 16:47:22.856068 [ 1 ] {} <Debug> Application: Uninitializing subsystem: Logging Subsystem
2020.06.12 16:47:22.856393 [ 45 ] {} <Information> BaseDaemon: Stop SignalListener thread

alex-zaitsev · 2020-06-12T09:40:46Z

Hi @yuzhichang, 0.10.0 is not yet released. But it is not clear what happens to ClickHouse. Could you share your ClickHouseInstallation yaml, please?

yuzhichang · 2020-06-12T11:40:37Z

Here's my ClickHouseInstallation yaml

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"
metadata:
  name: dev
  namespace: dev
spec:
  configuration:
    clusters:
      - name: cdp
        templates:
          podTemplate: clickhouse:20.4
        layout:
          shardsCount: 3
    settings:
      logger/level: information
      query_log/flush_interval_milliseconds: 30000
      timezone: Asia/Shanghai
      max_table_size_to_drop: 0
      max_partition_size_to_drop: 0
    profiles:
      default/max_query_size: 1073741824
      default/distributed_aggregation_memory_efficient: 1
      default/max_bytes_before_external_group_by: 3000000000
      default/max_memory_usage: 7000000000
      default/max_memory_usage_for_all_queries: 8000000000
      default/joined_subquery_requires_alias: 0
      default/log_queries: 1
      default/log_query_threads: 0
    users:
      cdp_user/networks/ip:
        - "::/0"
      cdp_user/password: password
    zookeeper: 
      nodes:
        - host: zookeeper-0.zookeepers
          port: 2181
  templates:
    podTemplates:
      - name: clickhouse:20.4
        podDistribution:
          - type: ShardAntiAffinity
        spec:
          containers:
            - name: clickhouse
              image: yandex/clickhouse-server:20.4
              volumeMounts:
                - name: data-storage-vc-template
                  mountPath: /var/lib/clickhouse
                - name: log-storage-vc-template
                  mountPath: /var/log/clickhouse-server
              resources:
                requests:
                  memory: 10Gi
                cpu: 1
          nodeSelector:
            env: prod
    volumeClaimTemplates:
      - name: data-storage-vc-template
        reclaimPolicy: Retain
        spec:
          storageClassName: csi-rbd-sc
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 100Gi
      - name: log-storage-vc-template
        reclaimPolicy: Retain
        spec:
          storageClassName: csi-rbd-sc
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 2Gi

alex-zaitsev · 2020-06-12T23:02:06Z

Thanks, @yuzhichang! The bug that ClickHouse was not starting with 0.10.0 operator is fixed. Schema problem is still present sometimes, but we know what to do.

alex-zaitsev · 2020-08-19T22:40:59Z

@yuzhichang, we have finally found and fixed a race condition in the code that could result in schema not being propagated to shards. This is fixed in 0.11.0 today to be released this week.

alex-zaitsev · 2020-08-20T08:18:24Z

Could you confirm https://github.com/Altinity/clickhouse-operator/releases/tag/0.11.0 fixes the issue, please?

yuzhichang · 2020-08-24T03:36:34Z

@alex-zaitsev I confirm. Thanks a lot!

sunsingerus added the work in progress This feautre is not completed yet label Jun 23, 2020

yuzhichang closed this as completed Aug 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Distributed and corresponding local tables are not getting created in the new Shard #308

Distributed and corresponding local tables are not getting created in the new Shard #308

yuzhichang commented Mar 13, 2020 •

edited

Loading

alex-zaitsev commented Mar 13, 2020

yuzhichang commented Mar 14, 2020 •

edited

Loading

alex-zaitsev commented Mar 16, 2020

yuzhichang commented Mar 16, 2020

alex-zaitsev commented Mar 18, 2020

yuzhichang commented Mar 22, 2020 •

edited

Loading

alex-zaitsev commented Apr 8, 2020 •

edited

Loading

yuzhichang commented Apr 9, 2020

alex-zaitsev commented Apr 9, 2020

yuzhichang commented Apr 9, 2020

alex-zaitsev commented Apr 9, 2020

alex-zaitsev commented Jun 10, 2020

yuzhichang commented Jun 12, 2020 •

edited

Loading

alex-zaitsev commented Jun 12, 2020

yuzhichang commented Jun 12, 2020

alex-zaitsev commented Jun 12, 2020

alex-zaitsev commented Aug 19, 2020

alex-zaitsev commented Aug 20, 2020

yuzhichang commented Aug 24, 2020

Distributed and corresponding local tables are not getting created in the new Shard #308

Distributed and corresponding local tables are not getting created in the new Shard #308

Comments

yuzhichang commented Mar 13, 2020 • edited Loading

alex-zaitsev commented Mar 13, 2020

yuzhichang commented Mar 14, 2020 • edited Loading

alex-zaitsev commented Mar 16, 2020

yuzhichang commented Mar 16, 2020

alex-zaitsev commented Mar 18, 2020

yuzhichang commented Mar 22, 2020 • edited Loading

alex-zaitsev commented Apr 8, 2020 • edited Loading

yuzhichang commented Apr 9, 2020

alex-zaitsev commented Apr 9, 2020

yuzhichang commented Apr 9, 2020

alex-zaitsev commented Apr 9, 2020

alex-zaitsev commented Jun 10, 2020

yuzhichang commented Jun 12, 2020 • edited Loading

alex-zaitsev commented Jun 12, 2020

yuzhichang commented Jun 12, 2020

alex-zaitsev commented Jun 12, 2020

alex-zaitsev commented Aug 19, 2020

alex-zaitsev commented Aug 20, 2020

yuzhichang commented Aug 24, 2020

yuzhichang commented Mar 13, 2020 •

edited

Loading

yuzhichang commented Mar 14, 2020 •

edited

Loading

yuzhichang commented Mar 22, 2020 •

edited

Loading

alex-zaitsev commented Apr 8, 2020 •

edited

Loading

yuzhichang commented Jun 12, 2020 •

edited

Loading