[BUG] Delete over size data, binlog stoped

[0xLeeL]

Bug Description

error log: error uploading records: HTTPDriver for http://10.8.0.1:8123 received ClickHouse error code 62 Code: 62. DB::Exception: Syntax error: failed at position 262144 ('52') (line 2, col 262143): 52. Max query size exceeded: '52'. (SYNTAX_ERROR) (version 23.7.4.5 (official build))

Steps to Reproduce

1. when mysql delete over size data, for example trucate a table exiting 5m records data

Expected Behavior

binlog sync normally

Actual Behavior

binlog sync stoped

Environment

• mysql_ch_replicator version: latest
• Operating System: ubuntu 24
• Python version: 3.11

MySQL Configuration

# Paste your MySQL configuration here (my.cnf or similar)
binlogformat is `row`

I already fixed this bug. I want a pr to fix it.

[0xLeeL]

fix code:

     def erase(self, table_name, field_name, field_values):
        if not field_values:
            return
        BATCH_SIZE = 10000
        field_value_list = list(field_values)
        for i in range(0, len(field_value_list), BATCH_SIZE):
            batch_values = field_value_list[i:i + BATCH_SIZE]
            self.do_erase(table_name, field_name, batch_values)

    def do_erase(self, table_name, field_name, field_values):
        field_name = ','.join(field_name)
        field_values = ', '.join(f'({v})' for v in field_values)
        query = DELETE_QUERY.format(**{
            'db_name': self.database,
            'table_name': table_name,
            'field_name': field_name,
            'field_values': field_values,
        })
        t1 = time.time()
        self.execute_command(query)
        t2 = time.time()
        duration = t2 - t1
        self.stats.on_event(
            table_name=table_name,
            duration=duration,
            is_insert=False,
            records=len(field_values),
        )

[bakwc]

Thanks for the report and fix. Merged, deployed in v0.0.94 release. Added config option erase_batch_size in clickhouse config section.