Mysql 8.0.14 version support Table map Event optional metaData , Sync Column

[sean-k1]

Fixed: #444
This PR depends on : #440 , #477

Description

Mysql 8.0.14 version Table map event optional meta data can be used
when the global variable binlog_row_metadata is set to 'FULL'. AND binlog_row_image is set to FULL
With this global variable value, we can retrieve column information from the binlog when a table map event occurs.

We apologize to those using MYSQL 5, but we no longer support python-mysql-replication prior to version 0.44.0.

If We sync from optional meta data, Column name Information Flag = True

Changes

Parsing Column Infromation from optional metadata

We no longer need to rely on the current moment's database table schema information.
We can sync column data at the time of event occurrence

Test

I Added Testcase mysql, mariadb version

example Sql

CREATE DATABASE test;
use test;
CREATE TABLE test4 (id int NOT NULL AUTO_INCREMENT, data VARCHAR(255), data2 VARCHAR(255), PRIMARY KEY(id));
INSERT INTO test4 (data,data2) VALUES ("Hello", "World");
UPDATE test4 SET data = "World", data2="Hello" WHERE id = 1;
DELETE FROM test4 WHERE id = 1;

dump

=== QueryEvent ===
Date: 2023-09-19T13:58:23
Log position: 101256
Event size: 85
Read bytes: 85
Schema: b'test'
Execution time: 0
Query: CREATE DATABASE test

=== QueryEvent ===
Date: 2023-09-19T13:58:23
Log position: 101530
Event size: 172
Read bytes: 172
Schema: b'test'
Execution time: 0
Query: CREATE TABLE test4 (id int NOT NULL AUTO_INCREMENT, data VARCHAR(255), data2 VARCHAR(255), PRIMARY KEY(id))

=== QueryEvent ===
Date: 2023-09-19T13:58:23
Log position: 101684
Event size: 52
Read bytes: 52
Schema: b'test'
Execution time: 0
Query: BEGIN

=== TableMapEvent ===
Date: 2023-09-19T13:58:23
Log position: 101768
Event size: 61
Read bytes: 61
Table id: 233
Schema: test
Table: test4
Columns: 3
=== OptionalMetaData ===
unsigned_column_list: [False, False, False]
default_charset_collation: 255
charset_collation: {}
column_charset: []
column_name_list: ['id', 'data', 'data2']
set_str_value_list : []
set_enum_str_value_list : []
geometry_type_list : []
simple_primary_key_list: [0]
primary_keys_with_prefix: {}
visibility_list: [True, True, True]
charset_collation_list: [255, 255]
enum_and_set_collation_list: []

=== WriteRowsEvent ===
Date: 2023-09-19T13:58:23
Log position: 101822
Event size: 31
Read bytes: 12
Table: test.test4
Affected columns: 3
Changed rows: 1
Column Name Information Flag: True
Values:
--
* id : 1
* data : Hello
* data2 : World

=== XidEvent ===
Date: 2023-09-19T13:58:23
Log position: 101853
Event size: 8
Read bytes: 8
Transaction ID: 1798

=== QueryEvent ===
Date: 2023-09-19T13:58:23
Log position: 102016
Event size: 61
Read bytes: 61
Schema: b'test'
Execution time: 0
Query: BEGIN

=== TableMapEvent ===
Date: 2023-09-19T13:58:23
Log position: 102100
Event size: 61
Read bytes: 61
Table id: 233
Schema: test
Table: test4
Columns: 3
=== OptionalMetaData ===
unsigned_column_list: [False, False, False]
default_charset_collation: 255
charset_collation: {}
column_charset: []
column_name_list: ['id', 'data', 'data2']
set_str_value_list : []
set_enum_str_value_list : []
geometry_type_list : []
simple_primary_key_list: [0]
primary_keys_with_prefix: {}
visibility_list: [True, True, True]
charset_collation_list: [255, 255]
enum_and_set_collation_list: []

=== UpdateRowsEvent ===
Date: 2023-09-19T13:58:23
Log position: 102174
Event size: 51
Read bytes: 13
Table: test.test4
Affected columns: 3
Changed rows: 1
Column Name Information Flag: True
Affected columns: 3
Values:
--
*id:1=>1
*data:Hello=>World
*data2:World=>Hello

=== XidEvent ===
Date: 2023-09-19T13:58:23
Log position: 102205
Event size: 8
Read bytes: 8
Transaction ID: 1799

=== QueryEvent ===
Date: 2023-09-19T13:58:23
Log position: 102359
Event size: 52
Read bytes: 52
Schema: b'test'
Execution time: 0
Query: BEGIN

=== TableMapEvent ===
Date: 2023-09-19T13:58:23
Log position: 102443
Event size: 61
Read bytes: 61
Table id: 233
Schema: test
Table: test4
Columns: 3
=== OptionalMetaData ===
unsigned_column_list: [False, False, False]
default_charset_collation: 255
charset_collation: {}
column_charset: []
column_name_list: ['id', 'data', 'data2']
set_str_value_list : []
set_enum_str_value_list : []
geometry_type_list : []
simple_primary_key_list: [0]
primary_keys_with_prefix: {}
visibility_list: [True, True, True]
charset_collation_list: [255, 255]
enum_and_set_collation_list: []

=== DeleteRowsEvent ===
Date: 2023-09-19T13:58:23
Log position: 102497
Event size: 31
Read bytes: 12
Table: test.test4
Affected columns: 3
Changed rows: 1
Column Name Information Flag: True
Values:
--
* id : 1
* data : World
* data2 : Hello

=== XidEvent ===
Date: 2023-09-19T13:58:23
Log position: 102528
Event size: 8
Read bytes: 8
Transaction ID: 1801

reference

https://mariadb.com/kb/en/table_map_event/#optional-metadata-block
https://dev.mysql.com/doc/dev/mysql-server/latest/classbinary__log_1_1Table__map__event.html
https://github.com/mysql/mysql-server/blob/8.0/libbinlogevents/include/rows_event.h#L389
https://github.com/mysql/mysql-server/blob/8.0/libbinlogevents/src/rows_event.cpp

[sean-k1]

@julien-duponchelle
Finally We can sync column when Event Occured!

I Also checked all Test when Mysql 8.0.23 Enviornment with github action

[antoolx]

this is great news, when will support for mysql 8 become available?

[sean-k1]

@antoolx
I don't know when the PRs will be merged, but the I think I'll release python-replicaiton-mysql as version 1.00 in the near future.

@julien-duponchelle
Can you let me know when this PR Merged?
And should I change ReadMe file Mysql 8.0.14?

[julien-duponchelle]

It's merged that's amazing.

I suggest we start working on a 1.0 release

[abdullahmafuz2020]

looking forward to the 1.0 release, very excited 🎉.
it should fix drop column issue, we are facing in our CDC process.

[sean-k1]

@abdullahmafuz2020

I'm glad this PR was helpful to you!
One thing to keep in mind, the past binlog data before setting binlog_row_metadata = "FULL", binlog_row_image = "FULL", we can't sync Column and column values

because , there is no metadata in the past binlog, please check it out!

[dongwook-chan]

@abdullahmafuz2020
As a contributor to python-mysql-replication and mentor at Korea's open source software contribution academy, this PR is part of our activities. We're studying the project's use cases. If you're using it professionally, we'd appreciate hearing about your use case. Could you please reach me via Gmail or LinkedIn?