Dropping a replica does not drop the table and leaves data in zookeeper

[mcgrawia]

Hi operator team,

I am using the 0.14.0 version of the operator and noticing it does not drop replicated tables when removing replicas. Here is my CHI definition (installed zookeeper separately):

apiVersion: clickhouse.altinity.com/v1
kind: ClickHouseInstallation
metadata:
  name: testlocal
  namespace: default
  labels:
    test: test
spec:
  defaults:
    templates:
      serviceTemplate: internal-only-service
      podTemplate: pod-template-with-volumes
  configuration:
    users:
      default/networks/ip: ::/0
    clusters:
      - name: local
        layout:
          shardsCount: 1
          replicasCount: 2
    zookeeper:
      nodes:
        - host: clickhouse-installation-clickhouse-zookeeper-headless
          port: 2181
  templates:
    serviceTemplates:
      - name: internal-only-service
        spec:
          type: ClusterIP
          ports:
            - name: http
              port: 8123
            - name: tcp
              port: 9000
    podTemplates:
      - name: pod-template-with-volumes
        spec:
          containers:
            - name: clickhouse
              image: yandex/clickhouse-server:21.1.2.15
              volumeMounts:
                - name: data-storage-vc-template
                  mountPath: /var/lib/clickhouse
                - name: log-storage-vc-template
                  mountPath: /var/log/clickhouse-server
    volumeClaimTemplates:
      - name: data-storage-vc-template
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi
      - name: log-storage-vc-template
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1Gi

After standing up the cluster, I create a replicated table on all replicas with:

CREATE TABLE if not exists test_scaling on cluster '{cluster}' (num UInt32)
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/test_scaling', '{replica}') ORDER BY tuple();

I then insert some data into the tables with:

insert into test_scaling
select * from system.numbers
limit 100000000;

And finally, I decrease the CHI spec.configuration.clusters[0].layout.replicasCount to 1, and reapply. After the replica is gone, I can still see the ZK data with:

select name from system.zookeeper
where path = '/clickhouse/tables/0/test_scaling/replicas';

And I see:

name
chi-testlocal-local-0-1
chi-testlocal-local-0-0

When running the operator locally, this code to drop the replica is never executed:

clickhouse-operator/pkg/model/schemer.go

Line 301 in 581cf31

func (s *Schemer) HostDeleteTables(ctx context.Context, host *chop.ChiHost) error {

I believe older version of the operator used to explicitly drop replicated tables before deleting the hosts. Is this possibly an issue with the latest version of the operator or something I may have mis-configured?

Thanks!

[mcgrawia]

Follow up:

After following the steps above, when I delete the CHI entirely (with only replica 0 left), I see these logs:

I0707 16:50:14.358010       1 announcer.go:213] deleteHost():default/testlocal:Delete host local/0-0 - started
I0707 16:50:14.469796       1 announcer.go:213] Run query on: chi-testlocal-local-0-0.default.svc.cluster.local of [chi-testlocal-local-0-0.default.svc.cluster.local]
I0707 16:50:14.493839       1 announcer.go:213] HostDeleteTables():default/testlocal:Drop tables: [test_scaling] as [DROP TABLE IF EXISTS "default"."test_scaling"]
I0707 16:50:14.883005       1 announcer.go:213] deleteTables():default/testlocal:Deleted tables on host 0-0 replica 0 to shard 0 in cluster local

So it appears the operator does delete the tables, but only when the entire CHI is deleted. Unfortunately, this means the first replica data is leaked in zookeeper, even after the CHI is deleted. When I use the zkCli, I can still see the replica 1 data remains, since it was not deleted when the replica was removed:

[zk: localhost:2181(CONNECTED) 19] ls /clickhouse/tables/0/test_scaling/replicas
[chi-testlocal-local-0-1]

[alex-zaitsev]

This is weird. We have a dedicated test that checks zookeeper contents and it passed. I now looked into the test carefully and found that the test had a bug. So thanks for the catch! Looks like it has been broken in 0.14, it works correctly before then.

Btw, in 0.15.0 version we changed the logic -- it now uses SYSTEM DROP REPLICA instead of DROP TABLE.

[mcgrawia]

Interesting, not a problem! Thanks for confirming it was actually an issue. Feel free to close this then if it's already fixed in the next version.

[alex-zaitsev]

Fixed in 0.15.0, tested in

clickhouse-operator/tests/test_operator.py

Line 748 in 0927fe5

with Then("Replica needs to be removed from the ZooKeeper as well"):