DM could not read mariabinlog (unsupport type 140 in binlog)

[kzinas-adv]

What did you do?

Tidb dm fails to read mariadb bin log
dmctl_task_status.txt

It starts as expected at 0-84533484-1054899238 and goes to next
(LogPos:0x20e899de)
decimal log pos 552114654 from bin-log
Data lengh 43940 characters data in that column json, data type BLOB

MariaDB SLAVE READ ONLY [database]> select sum(char_length(data)) from table where start_time = '2021-10-25 20:00:00' and title = 'TES TEST TEST' and game_id = 1303 and tstamp = '2021-10-25 20:03:36'\G
*************************** 1. row ***************************
sum(char_length(data)): 43940

What did you expect to see?

Smoth tidb dm repliction flow

What did you see instead?

               Err: unsupport type 140 in binlog and don't know how to handle",
                          "RawCause": "",

"Workaround": "Please check if the binlog file could be parsed by mysqlbinlog."

Versions of the cluster

DM version (run dmctl -V or dm-worker -V or dm-master -V):

[root@adv-dev-dm-test-mst01 tmp]# tiup dm --version
Starting component `dm`: /root/.tiup/components/dm/v1.6.1/tiup-dm --version
tiup version 1.6.1 tiup
Go Version: go1.17.2
Git Ref: v1.6.1
GitHash: 1f6ca2b18e5c96c075efb4c08084d6dc8659050b
[root@adv-dev-dm-test-mst01 tmp]# 

[root@adv-dev-dm-test-mst01 tmp]# /opt/tidb/dm/deploy-test/dm-master-8261/bin/dm-master/dm-master -V
Release Version: v2.0.7
Git Commit Hash: cd46eee3f1d00d6c78888e4a38daf80f29c69266
Git Branch: heads/refs/tags/v2.0.7
UTC Build Time: 2021-09-29 07:40:17
Go Version: go version go1.16.4 linux/amd64

[root@adv-dev-dm-test02 backup]# /opt/tidb/dm/deploy-test/dm-worker-8262/bin/dm-worker/dm-worker -V
Release Version: v2.0.7
Git Commit Hash: cd46eee3f1d00d6c78888e4a38daf80f29c69266
Git Branch: heads/refs/tags/v2.0.7
UTC Build Time: 2021-09-29 07:40:08
Go Version: go version go1.16.4 linux/amd64

Upstream MySQL/MariaDB server version:

MariaDB-server-10.3.31-1.el8.x86_64

How did you deploy DM: tiup or manually?

tiup

current status of DM cluster (execute query-status in dmctl)

(paste current status of DM cluster here)

[lance6716]

Thanks for reporting, currently we can't handle every type of MariaDB.

As you said "Data lengh 43940 characters data in that column json, data type BLOB", I don't understand what's the column type of that table (JSON or BLOB?). Can you post the table structure?

[kzinas-adv]

I found issue, it was compressed columns in the source DB. Issue not only with tables which used in dm replication, but with any compressed column in the source bin-log.