Consistent table metadata and DDL

Problem

(In incremental stage only)

dtle will have table structure (metadata) (e.g. column names, types) stored in memory (src & dst). The metadata will be utilized when

• [src] filtering rows by 'where' predicate
• [dst] constructing DML
• and more...

Src MySQL Task

The metadata for some table is queried from src mysqld when a DDL on the table is encountered

Consider the case

mysqld	dtle
DDL1 executed
some DML
DDL2 executed
	process DDL1
	- get metadata from mysqld
	- but the data is after DDL2
	process some DML with inconsistent metadata
more DML
	process DDL2
	...

Dst MySQL Task

The metadata for some table is queried from dst mysqld when a DDL on the table has been replayed. Nothing is wrong as far as considered

Dst Kafka Task

On dest side (Kafka), the metadata is passed from src MySQL task, which might be inconsistent to the table structure at the time of row change.

Possible solution

• Run a helper mysqld, run only DDL and query table structure from it
• Parse DDL and calculate table structure change
  • like debezium

Also consider

• e.mysqlContext.ReplicateDoDb: user assigned db.tb
• vs extractor.replicateDoDb: db.tb exists when creating the job. for full-copy.
• NewMySQLReader()
• Crash safe

sqls for getting metadata used by dtle

• ShowCreateTable|ShowCreateView(): show create table %s.%s
  • only for full copying: create the table on dest.
• GetTableColumns(): show columns from %s.%s
  • queried every time for a DDL
• ApplyColumnTypes(): select * from information_schema.columns where table_schema=? and table_name=?
  • queried every time for a DDL
• validateTableTriggers()
  • only at initial inspection
• getCandidateUniqueKeys()
  • only at initial inspection
  • used only for full copying

The libmysqld embedded server library is deprecated as of MySQL 5.7.19 and will be removed in MySQL 8.0.

https://mysqlserverteam.com/mysql-8-0-retiring-support-for-libmysqld/

Limitation

• If you start a increment-only job (e.g. by assigning GTID), you should make sure current table structure matches the one of assigned
  • aka no significant DDL after the GTID

test case 1

1. Start a job copying a321 full schema
2. Execute these quickly (paste to bin/mysql at once)

create table a321.a (id int primary key auto_increment);
insert into a321.a values (0);
drop table a321.a;

3. repeat step 2 several times.
   • Before the fix, it should crash in crash in binlog_reader #267.
   • After the fix it won't crash.

case 2

Background: If a DDL is executed with an error result, it will be still recorded in binlog. The error DDL should be replayed on dst side for its side effect (in native replication), and compare evt.ErrorCode.

Reference: http://blog.itpub.net/29773961/viewspace-2150907/

I cannot reproduce such a case with mysql 5.7.

d1041f1