MariaDB 10.3 support

[mia0x75]

10.2.x:

DEBU[2019-04-12 12:46:23] OnPosSynced fired with data: (mysql-bin.000001, 461), true
DEBU[2019-04-12 12:46:35] OnGTID event fired, GTID: 0-113-2        
DEBU[2019-04-12 12:46:35] OnRow fired                              
DEBU[2019-04-12 12:46:35] Data: test.t1, {"database":"test","event":{"data":{"id":1}},"event_index":1,"event_type":"insert","table":"t1","time":1555044395} 
DEBU[2019-04-12 12:46:35] OnXID event fired, (mysql-bin.000001, 665). 
DEBU[2019-04-12 12:46:35] OnPosSynced fired with data: (mysql-bin.000001, 665), false

10.3.x

DEBU[2019-04-12 12:49:58] OnPosSynced fired with data: (mysql-bin.000215, 342), true 
DEBU[2019-04-12 12:50:47] OnGTID event fired, GTID: 192-1681127-925 
DEBU[2019-04-12 12:50:47] OnXID event fired, (mysql-bin.000215, 502). 
DEBU[2019-04-12 12:50:47] OnPosSynced fired with data: (mysql-bin.000215, 502), false

[siddontang]

Hi @mia0x75

I have no time to support all the versions, can you help me investigate this?

[dveeden]

Could it be that the binlog format was set to MIXED and there were Query events instead of Row events?

[dveeden]

Testing with MariaDB 10.3.39:

podman run -p 3307:3306 -e MARIADB_ALLOW_EMPTY_ROOT_PASSWORD=1 -it mariadb:10.3 --log-bin --binlog-format=row

SET GLOBAL server_id=2;
RESET MASTER;
CREATE DATABASE test;
USE test;
CREATE TABLE t1(id INT PRIMARY KEY);
INSERT INTO t1 VALUES (1);

$ ./bin/go-mysqlbinlog -port 3307 -flavor mariadb
[2024/11/08 14:18:49] [info] binlogsyncer.go:189 create BinlogSyncer with config {ServerID:101 Flavor:mariadb Host:127.0.0.1 Port:3307 User:root Password: Localhost: Charset: SemiSyncEnabled:false RawModeEnabled:false TLSConfig:<nil> ParseTime:false TimestampStringLocation:UTC UseDecimal:true RecvBufferSize:0 HeartbeatPeriod:0s ReadTimeout:0s MaxReconnectAttempts:0 DisableRetrySync:false VerifyChecksum:false DumpCommandFlag:0 Option:<nil> Logger:0xc0000a62a0 Dialer:0x6d0d00 RowsEventDecodeFunc:<nil> TableMapOptionalMetaDecodeFunc:<nil> DiscardGTIDSet:false EventCacheCount:10240 SynchronousEventHandler:<nil>}
[2024/11/08 14:18:49] [info] binlogsyncer.go:429 begin to sync binlog from position (, 4)
[2024/11/08 14:18:49] [info] binlogsyncer.go:407 Connected to mariadb 5.5.5-10.3.39-MariaDB-1:10.3.39+maria~ubu2004-log server
[2024/11/08 14:18:49] [info] binlogsyncer.go:846 rotate to (mysqld-bin.000001, 4)
=== RotateEvent ===
Date: 1970-01-01 01:00:00
Log position: 0
Event size: 44
Position: 4
Next log name: mysqld-bin.000001

=== FormatDescriptionEvent ===
Date: 2024-11-08 14:17:21
Log position: 256
Event size: 252
Version: 4
Server version: 10.3.39-MariaDB-1:10.3.39+maria~ubu2004-log
Checksum algorithm: 1

=== MariadbGTIDListEvent ===
Date: 2024-11-08 14:17:21
Log position: 285
Event size: 29
Lists: []

=== MariadbBinLogCheckPointEvent ===
Date: 2024-11-08 14:17:21
Log position: 329
Event size: 44
Info: mysqld-bin.000001

=== MariadbGTIDEvent ===
Date: 2024-11-08 14:18:15
Log position: 371
Event size: 42
GTID: {0 2 1}
Flags: 41
CommitID: 0

=== QueryEvent ===
Date: 2024-11-08 14:18:15
Log position: 458
Event size: 87
Slave proxy ID: 9
Execution time: 0
Error code: 0
Schema: test
Query: CREATE DATABASE test

=== MariadbGTIDEvent ===
Date: 2024-11-08 14:18:21
Log position: 500
Event size: 42
GTID: {0 2 2}
Flags: 41
CommitID: 0

=== QueryEvent ===
Date: 2024-11-08 14:18:21
Log position: 602
Event size: 102
Slave proxy ID: 9
Execution time: 0
Error code: 0
Schema: test
Query: CREATE TABLE t1(id INT PRIMARY KEY)

=== MariadbGTIDEvent ===
Date: 2024-11-08 14:18:22
Log position: 644
Event size: 42
GTID: {0 2 3}
Flags: 12
CommitID: 0

=== TableMapEvent ===
Date: 2024-11-08 14:18:22
Log position: 737
Event size: 45
TableID: 19
TableID size: 6
Flags: 1
Schema: test
Table: t1
Column count: 1
Column type: 
00000000  03                                                |.|
NULL bitmap: 
00000000  00                                                |.|
Signedness bitmap: 
Default charset: []
Column charset: []
Set str value: []
Enum str value: []
Column name: []
Geometry type: []
Primary key: []
Primary key prefix: []
Enum/set default charset: []
Enum/set column charset: []
Invisible Column bitmap: 
UnsignedMap: map[int]bool(nil)
CollationMap: map[int]uint64(nil)
EnumSetCollationMap: map[int]uint64(nil)
EnumStrValueMap: map[int][]string(nil)
SetStrValueMap: map[int][]string(nil)
GeometryTypeMap: map[int]uint64(nil)
VisibilityMap: map[int]bool(nil)
Columns: 
  <n/a>  type=3    unsigned=<n/a>  null=no 

=== WriteRowsEventV1 ===
Date: 2024-11-08 14:18:22
Log position: 775
Event size: 38
TableID: 19
Flags: 1
Column count: 1
NDB data: 
Values:
--
0:1

=== XIDEvent ===
Date: 2024-11-08 14:18:22
Log position: 806
Event size: 31
XID: 13

This looks ok to me.

$ ./bin/go-canal -flavor mariadb -port 3307
[2024/11/08 14:21:10] [info] binlogsyncer.go:189 create BinlogSyncer with config {ServerID:101 Flavor:mariadb Host:127.0.0.1 Port:3307 User:root Password: Localhost: Charset:utf8 SemiSyncEnabled:false RawModeEnabled:false TLSConfig:<nil> ParseTime:false TimestampStringLocation:UTC UseDecimal:true RecvBufferSize:0 HeartbeatPeriod:1m0s ReadTimeout:1m30s MaxReconnectAttempts:0 DisableRetrySync:false VerifyChecksum:false DumpCommandFlag:0 Option:<nil> Logger:0xc0000b83c0 Dialer:0x7323a0 RowsEventDecodeFunc:0x9045c0 TableMapOptionalMetaDecodeFunc:<nil> DiscardGTIDSet:false EventCacheCount:10240 SynchronousEventHandler:<nil>}
[2024/11/08 14:21:10] [info] dump.go:171 try dump MySQL and parse
[2024/11/08 14:21:10] [info] dumper.go:294 exec mysqldump with [--host=127.0.0.1 --port=3307 --user=root --password=****** --source-data --single-transaction --skip-lock-tables --compact --skip-opt --quick --no-create-info --skip-extended-insert --skip-tz-utc --hex-blob --default-character-set=utf8 --column-statistics=0 --databases test]
mysqldump: [Warning] Using a password on the command line interface can be insecure.
insert test.t1 [[1]]
[2024/11/08 14:21:10] [info] dump.go:187 dump MySQL and parse OK, use 0.01 seconds, start binlog replication at (mysqld-bin.000001, 806)
[2024/11/08 14:21:10] [info] binlogsyncer.go:429 begin to sync binlog from position (mysqld-bin.000001, 806)
[2024/11/08 14:21:10] [info] binlogsyncer.go:407 Connected to mariadb 5.5.5-10.3.39-MariaDB-1:10.3.39+maria~ubu2004-log server
[2024/11/08 14:21:10] [info] sync.go:22 start sync binlog at binlog file (mysqld-bin.000001, 806)
[2024/11/08 14:21:10] [info] binlogsyncer.go:846 rotate to (mysqld-bin.000001, 806)
[2024/11/08 14:21:10] [info] sync.go:59 received fake rotate event, next log name is mysqld-bin.000001

This also has the insert test.t1 [[1]].

So looks like this might have been fixed already?
@lance6716 @atercattus please verify and close

[lance6716]

can not reproduce