Skip to content

Conversation

@tuteng
Copy link
Member

@tuteng tuteng commented Dec 31, 2018

Motivation

Optimization of mysql binlog to pulsar use canal

Modifications

Add MessageUtils for parse columns
Support query of presto sql

Result

INSERT

mysql:

MariaDB [aaa]> insert into users321(name, extra) values('xxxddxxxmm', 'ddd');

python consumer:

{u'timestamp': u'2018-12-31 00:31:24', u'message': u'[{"data":null,"database":"","es":1546234283000,"id":58,"isDdl":false,"mysqlType":null,"old":null,"sql":"insert into users321(name, extra) values(\'xxxddxxxmm\', \'ddd\')","sqlType":null,"table":"","ts":1546234284669,"type":"QUERY"},{"data":[{"isKey":"1","isNull":"0","index":"0","mysqlType":"int(11)","columnName":"id","columnValue":"34","updated":"1"},{"isKey":"0","isNull":"0","index":"1","mysqlType":"varchar(50)","columnName":"name","columnValue":"xxxddxxxmm","updated":"1"},{"isKey":"0","isNull":"0","index":"2","mysqlType":"varchar(50)","columnName":"extra","columnValue":"ddd","updated":"1"}],"database":"aaa","es":1546234283000,"id":58,"isDdl":false,"mysqlType":null,"old":null,"sql":"","sqlType":null,"table":"users321","ts":1546234284670,"type":"INSERT"}]', u'id': 58}

presto sql:

presto> select id, timestamp, message from pulsar."public/default".my_topic_test where id=58;
 id |      timestamp      |
----+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 58 | 2018-12-31 00:31:24 | [{"data":null,"database":"","es":1546234283000,"id":58,"isDdl":false,"mysqlType":null,"old":null,"sql":"insert into users321(name, extra) values('xxxddxxxmm', 'ddd')","sqlType"

@sijie sijie added type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages area/connector labels Dec 31, 2018
List<FlatMessage> flatMessages = FlatMessage.messageConverter(message);
long batchId = message.getId();
SimpleCanalConnector con = (SimpleCanalConnector) connector;
con.isLazyParseEntry();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not very familiar with Canal. does isLazyParseEntry() changing the state of the connector, so the con will parse entries lazily?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line of code is useless. It should be generated automatically when changing files. I didn't notice it. Now I have removed it. Please review the code again.

@sijie sijie added this to the 2.3.0 milestone Jan 2, 2019
@sijie sijie merged commit bbd736d into apache:master Jan 2, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/connector type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants