DM task configuration optimization

[amyangfei]

DM task configuration optimization

Abstract

In this project we will focus on the optimization of DM task configuration and try to achieve some improvement in DM usability. The optimization includs the following items:

• Extend the wildcard matching ability used in table-routing and binlog event filter.
• Unify the matching strategy used in table-routing, binlog event filter and black and white list.
• Improve existing configuration related tools to make them more useful.

Problem Statement

• The wildcard matching strategy used in table-routing and binlog event filter supports * and ?, we want to extend them to support more matching rules, including the [...] matching rule.
  • Wildcard varies in different domains, we should first determine which wildcard regulation will be used. In this project we suggest to use the most common syntax, which contains *, ? and [...].
  • As we use a trie tree to store matching rules, it is easy to implement positive matching but it is difficult to implement the negative matching, which often in the [!...] format, this will be a bonus of this problem.
• There exists three table/binlog matching features in DM, table routing, black and white list and binlog event filter. The matching rules used in these three features are not unified, which often makes user confused and difficult to use.
• Many data replication scenarios have large scale of tables and complicated replication requirements, in which cases user have to write a long, complicated task config.
• We have already developed some config related tools, including DM-portal (already released) and DM-simulator (PR in review). However the DM-portal is not up to date according to the recent config refactor. We need to find out whether DM-portal is a proper way to solve the configuration automatic generation problem, if it is the correct way, we should make the config strategy generated by it up to date, otherwise we could try to come up with a better solution.

Success Criteria

• Extend the wildcard matching strategy, support [...] type matching in everywhere we use wildcard matching in DM.
• Support wildcard in black and white list feature, but still keep the original regular expression way.
• Make DM-simulator come true.

TODO list

• Extend wildcard matching, support [...] type matching.
• Extend wildcard matching, support negative [...] type matching.
• Support wildcard matching in black and white list feature.
• Plot the table router in DM-simulator using dot graphs
• Push current DM-simulator PR merged

Difficulty

Medium

Score

2400

Mentor(s)

@amyangfei

Recommended Skills

DM operation

References

• DM souce code reading part 7 (Chinese version)
• DM configuration optimization document, part 1
• DM-portal source
• DM-portal user manual

[lzmhhh123]

@amyangfei Hello, yangfei. @lamxTyler @winoros and I would like to take this issue. Look forward to your help.