[BUG] Replicate new table without initial replication

[Lotuashvili]

Bug Description

I need to include a new table in config file, but the only way to do this is to delete binlog folder and run replicator again, which will replicate initial data from scratch. Keep in mind, that those tables already exists in MySQL (so the creation of those tables are not included in binlogs), but was not included in config file at the beginning (Since we didn't need reports on those tables). After developing an application, we need more and more reports, therefore more tables which has been in application from the beginning.

Steps to Reproduce

1. Run initial replication for some tables
2. Stop replication
3. Update config file and include more tables
4. Run replication
5. Update records in "new" table
6. Replicator will fail

Expected Behavior

Expected to have a separate command to run single table replication, or it would be better if it was automatic.

Actual Behavior

Problem is that it will replicate the whole database only because of one table. Using it in a production creates a massive time gap, since we have millions of records and initial replication needs 2-3 hours (and growing). Therefore, ClickHouse has old records before the initial replication is done.

[dbrepl forge 2025-11-11 17:12:31,791     INFO] running realtime replication from the position: ('mysql-bin.000424', 66937237)
[dbrepl forge 2025-11-11 17:12:32,065     INFO] set position to /home/forge/mysql-clickhouse-replicator/binlog/forge/209.bin
[dbrepl forge 2025-11-11 17:12:32,087     INFO] found transaction ('mysql-bin.000424', 66937237) inside /home/forge/mysql-clickhouse-replicator/binlog/forge/209.bin
[dbrepl forge 2025-11-11 17:12:32,092     INFO] stats: {"last_transaction": ["mysql-bin.000424", 66937625], "events_count": 1, "insert_events_count": 0, "insert_records_count": 0, "erase_events_count": 0, "erase_records_count": 0, "no_events_count": 0, "cpu_load": 0.0}
[dbrepl forge 2025-11-11 17:12:32,092     INFO] ch_stats: {"total": {"inserts": {"duration": 0.0, "events": 0, "records": 0}, "erases": {"duration": 0.0, "events": 0, "records": 0}}}
[dbrepl forge 2025-11-11 17:13:12,979    ERROR] unhandled exception
Traceback (most recent call last):
  File "/home/forge/.local/share/pipx/venvs/mysql-ch-replicator/lib/python3.12/site-packages/mysql_ch_replicator/db_replicator.py", line 216, in run
    self.run_realtime_replication()
  File "/home/forge/.local/share/pipx/venvs/mysql-ch-replicator/lib/python3.12/site-packages/mysql_ch_replicator/db_replicator.py", line 260, in run_realtime_replication
    self.realtime_replicator.run_realtime_replication()
  File "/home/forge/.local/share/pipx/venvs/mysql-ch-replicator/lib/python3.12/site-packages/mysql_ch_replicator/db_replicator_realtime.py", line 73, in run_realtime_replication
    self.handle_event(event)
  File "/home/forge/.local/share/pipx/venvs/mysql-ch-replicator/lib/python3.12/site-packages/mysql_ch_replicator/db_replicator_realtime.py", line 94, in handle_event
    event_handlers[event.event_type](event)
  File "/home/forge/.local/share/pipx/venvs/mysql-ch-replicator/lib/python3.12/site-packages/mysql_ch_replicator/db_replicator_realtime.py", line 133, in handle_insert_event
    mysql_table_structure = self.replicator.state.tables_structure[event.table_name][0]
                            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
KeyError: 'currencies'
Traceback (most recent call last):
  File "/home/forge/.local/bin/mysql_ch_replicator", line 7, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/forge/.local/share/pipx/venvs/mysql-ch-replicator/lib/python3.12/site-packages/mysql_ch_replicator/main.py", line 202, in main
    run_db_replicator(args, config)
  File "/home/forge/.local/share/pipx/venvs/mysql-ch-replicator/lib/python3.12/site-packages/mysql_ch_replicator/main.py", line 116, in run_db_replicator
    db_replicator.run()
  File "/home/forge/.local/share/pipx/venvs/mysql-ch-replicator/lib/python3.12/site-packages/mysql_ch_replicator/db_replicator.py", line 216, in run
    self.run_realtime_replication()
  File "/home/forge/.local/share/pipx/venvs/mysql-ch-replicator/lib/python3.12/site-packages/mysql_ch_replicator/db_replicator.py", line 260, in run_realtime_replication
    self.realtime_replicator.run_realtime_replication()
  File "/home/forge/.local/share/pipx/venvs/mysql-ch-replicator/lib/python3.12/site-packages/mysql_ch_replicator/db_replicator_realtime.py", line 73, in run_realtime_replication
    self.handle_event(event)
  File "/home/forge/.local/share/pipx/venvs/mysql-ch-replicator/lib/python3.12/site-packages/mysql_ch_replicator/db_replicator_realtime.py", line 94, in handle_event
    event_handlers[event.event_type](event)
  File "/home/forge/.local/share/pipx/venvs/mysql-ch-replicator/lib/python3.12/site-packages/mysql_ch_replicator/db_replicator_realtime.py", line 133, in handle_insert_event
    mysql_table_structure = self.replicator.state.tables_structure[event.table_name][0]
                            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
KeyError: 'currencies'
[runner 2025-11-11 17:13:13,433  WARNING] Process dead (exit code: 1), restarting: 

Environment

• mysql_ch_replicator version: Latest
• Operating System: Ubuntu
• Python version: 3.12

MySQL Configuration

[mysqld]
server-id = 1
log_bin = /var/log/mysql/mysql-bin.log
binlog_format = ROW
binlog_row_metadata = FULL
expire_logs_days = 2
gtid_mode = ON
enforce_gtid_consistency = ON
log_slave_updates = ON
sync_binlog = 1

Replicator Configuration

mysql:
  host: 'localhost'
  port: 3306
  user: 'clickhouse'
  password: 'xxxxxxxxx'

clickhouse:
  host: 'localhost'
  port: 8123
  user: 'default'
  password: 'xxxxxxxxxxxxxxxx'

binlog_replicator:
  data_dir: '/home/user/mysql-clickhouse-replicator/binlog/'
  records_per_file: 50000

databases: 'myapp'
tables:
  - activities
  - admins
  - balances