Flaky test_replicated_merge_tree_s3_zero_copy/test.py::test_drop_table

[vdimir]

Fails with

>       node.query_with_retry(
            "system sync replica test_drop_table",
            settings={"receive_timeout": 5},
            sleep_time=5,
            retry_count=10,
        )
...
>       raise Exception("Can't execute query {}".format(sql))
E       Exception: Can't execute query system sync replica test_drop_table

2023.12.07 19:39:32.734844 [ 8 ] {8c001075-0d48-4130-95af-ee4f332eb894} <Trace> default.test_drop_table (28da86af-40f3-402b-b48f-c2089d74531f): Starting up table
2023.12.07 19:39:32.737282 [ 366 ] {} <Debug> default.test_drop_table (28da86af-40f3-402b-b48f-c2089d74531f): Uncovered unexpected parts: 15_0_0_0, 11_0_0_0, 0_0_0_0. Restorable une
xpected parts: . Missing parts: . Covered unexpected parts: . Expected parts: .
2023.12.07 19:39:32.737498 [ 366 ] {} <Error> default.test_drop_table (ReplicatedMergeTreeAttachThread): Initialization failed, table will remain readonly. Error: Code: 231. DB::Exc
eption: The local set of parts of table default.test_drop_table (28da86af-40f3-402b-b48f-c2089d74531f) doesn't look like the set of parts in ZooKeeper: 31.00 rows of 31.00 total row
s in filesystem are suspicious. There are 3 uncovered unexpected parts with 31 rows (0 of them is not just-written with 0 rows), 0 missing parts (with 0 blocks), 0 covered unexpecte
d parts (with 0 rows). (TOO_MANY_UNEXPECTED_DATA_PARTS), Stack trace (when copying this message, always include the lines below):
...
29387:2023.12.07 19:40:18.112336 [ 8 ] {6f6d82e0-d53e-4ada-941f-72daddf4569f} <Error> executeQuery: Code: 242. DB::Exception: Table is in readonly mode (replica path: /test/drop_table/replicas/1). (TABLE_IS_READ_ONLY) (version 23.11.1.2703 (official build)) (from 172.16.11.1:61814) (in query: system sync replica test_drop_table), Stack trace (when copying this message, always include the lines below):

ci db

[tavplubix]

https://s3.amazonaws.com/clickhouse-test-reports/57906/e4a831be17f85e345c5c9cd8f00bd22b8e759692/integration_tests__asan__[4_4].html

[Algunenano]

Looking at the error it seems that something like this happens:

• Replica1 detaches table.
• Replica2 drops table. Since there isn't references to the blobs/nodes any more it deletes them (ZK and s3)
• Replica1 attaches table. The parts that were dropped by replica2 are no longer in ZK. TOO_MANY_UNEXPECTED_DATA_PARTS is thrown, table is kept in read-only, operations fail.