onlineddl: support regular expressions rules for PT/GH-OST #2139

wentaojin · 2021-09-15T11:19:04Z

What problem does this PR solve?

Solve the problem of custom PT/GH-OST format

This feature is added because some users may modify the pt/gh-ost code because they are worried about the business mistakenly deleting the table, which may cause the format to be inconsistent with the Dm, which affects synchronization，the feature can solve it

close #2123

What is changed and how it works?

Add DM config，if values is null, dm will keep default , if values is not null , match according to specific value

shadow-table-rule: [] # 支持正则，默认 ^_(.+)_(?:new|gho)$ 匹配多个具体值后加 (.*)
trash-table-rule: [] # 支持正则, 默认 ^_(.+)_(?:ghc|del|old)$ 匹配多个具体值后加 (.*)

ti-chi-bot · 2021-09-15T11:19:06Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

Ehco1996
glorv

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

syncer/online-ddl-tools/online_ddl.go

dm/config/subtask.go

syncer/syncer.go

Ehco1996 · 2021-09-16T09:23:42Z

Please fix the integration test error on CI as well 🤣

This is because you have changed the config of the task and can be fixed by referring to this commit

Note that in your case you should not use diff -i to ignore the inconsistency, but rather set shadow-table-rule and trash-table-rule in task.yaml and check that they are euqal after export

glorv · 2021-09-22T03:05:16Z

dm/config/subtask.go

+	ShadowTableRule []string `yaml:"shadow-table-rule" toml:"shadow-table-rule" json:"shadow-table-rule"`
+	TrashTableRule  []string `yaml:"trash-table-rule" toml:"trash-table-rule" json:"trash-table-rule"`


Suggested change

ShadowTableRule []string `yaml:"shadow-table-rule" toml:"shadow-table-rule" json:"shadow-table-rule"`

TrashTableRule []string `yaml:"trash-table-rule" toml:"trash-table-rule" json:"trash-table-rule"`

ShadowTableRules []string `yaml:"shadow-table-rules" toml:"shadow-table-rules" json:"shadow-table-rules"`

TrashTableRules []string `yaml:"trash-table-rules" toml:"trash-table-rules" json:"trash-table-rules"`

glorv · 2021-09-22T03:05:43Z

dm/config/task.go

@@ -302,6 +302,10 @@ type TaskConfig struct {
 	MySQLInstances []*MySQLInstance `yaml:"mysql-instances" toml:"mysql-instances" json:"mysql-instances"`

 	OnlineDDL bool `yaml:"online-ddl" toml:"online-ddl" json:"online-ddl"`
+	// pt/gh-ost name rule,support regex
+	ShadowTableRule []string `yaml:"shadow-table-rule" toml:"shadow-table-rule" json:"shadow-table-rule"`


glorv · 2021-09-22T03:08:37Z

pkg/terror/error_list.go

@@ -877,6 +879,10 @@ var (
 	ErrConfigGenTableRouter         = New(codeConfigGenTableRouter, ClassConfig, ScopeInternal, LevelHigh, "generate table router error", "Please check the `routes` config in task configuration file.")
 	ErrConfigGenColumnMapping       = New(codeConfigGenColumnMapping, ClassConfig, ScopeInternal, LevelHigh, "generate column mapping error", "Please check the `column-mappings` config in task configuration file.")
 	ErrConfigInvalidChunkFileSize   = New(codeConfigInvalidChunkFileSize, ClassConfig, ScopeInternal, LevelHigh, "invalid `chunk-filesize` %v", "Please check the `chunk-filesize` config in task configuration file.")
+	ErrConfigOnlineDDLRegexCompile  = New(codeConfigOnlineDDLRegexCompile, ClassConfig, ScopeInternal, LevelHigh,


I think we can merge this two error code together, the only reason that regexp compile fail is that the syntax is not correct.

dm/config/subtask.go

Ehco1996

rest LGTM

Ehco1996 · 2021-09-22T05:03:58Z

dm/config/subtask.go

+	adjustedRules := make([]string, 0, len(rules))
+	for _, r := range rules {
+		if !strings.HasPrefix(r, "^") {
+			r = "^" + r


perhaps it would be better to just return the error here rather than adjust re ((add "^" and "$") ) for the user

I think this way is simpler for users. They need to check the error message and rewrite task.yaml if we return an error here. Waiting for other reviewers' suggestions...

It is ok if noted in document.

syncer/syncer_test.go

lichunzhu · 2021-09-22T08:35:32Z

/run-all-tests
https://ci.pingcap.net/blue/organizations/jenkins/dm_ghpr_new_test/detail/dm_ghpr_new_test/2632/pipeline

lichunzhu · 2021-09-23T04:37:45Z

/run-unit-tests

lichunzhu · 2021-09-23T05:24:52Z

/merge

ti-chi-bot · 2021-09-23T05:24:54Z

This pull request has been accepted and is ready to merge.

Commit hash: 5850fb4

)

ti-chi-bot · 2021-09-23T05:27:22Z

In response to a cherrypick label: new pull request created: #2156.

…2156)

) (pingcap#2156)

regex pt/gh-ost

b2cecc6

ti-chi-bot added the first-time-contributor label Sep 15, 2021

ti-chi-bot requested review from GMHDBJD and lance6716 September 15, 2021 11:19

ti-chi-bot added the size/L label Sep 15, 2021

glorv added this to the v2.0.7 milestone Sep 15, 2021

lichunzhu reviewed Sep 16, 2021

View reviewed changes

syncer/online-ddl-tools/online_ddl.go Show resolved Hide resolved

lichunzhu reviewed Sep 16, 2021

View reviewed changes

regex pt/gh-ost opti

acd5c31

ti-chi-bot added the needs-rebase label Sep 17, 2021

merge master and resolve conflicts

dbf67a4

ti-chi-bot removed the needs-rebase label Sep 18, 2021

lichunzhu added 7 commits September 18, 2021 17:01

address comments, fix CI

bbe783d

simplify some code

b8c53bd

fix integration test

7b644fb

fix

8e7fcc2

fix

868efb2

Merge branch 'master' of https://github.com/pingcap/dm

297b21b

fix lint

eb26ac3

glorv reviewed Sep 22, 2021

View reviewed changes

lichunzhu added 2 commits September 22, 2021 11:31

address comments

2ea5f77

fix ut

2e0f6c6

Ehco1996 reviewed Sep 22, 2021

View reviewed changes

dm/config/subtask.go Outdated Show resolved Hide resolved

address comments and fix tests

dc06aa9

Ehco1996 reviewed Sep 22, 2021

View reviewed changes

fix tests

630bb97

glorv approved these changes Sep 22, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 One reviewer already commented LGTM label Sep 22, 2021

update terror

c4bb7aa

Ehco1996 approved these changes Sep 22, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Two reviewers already commented LGTM, ready for merge and removed status/LGT1 One reviewer already commented LGTM labels Sep 22, 2021

Ehco1996 and others added 4 commits September 23, 2021 07:31

Merge branch 'master' into master

965dbda

fix upgrade test

874b253

Merge branch 'master' of https://github.com/wentaojin/dm

f4a5e39

fix new-relay

5850fb4

ti-chi-bot added the status/can-merge label Sep 23, 2021

ti-chi-bot merged commit 48d4830 into pingcap:master Sep 23, 2021

ti-chi-bot pushed a commit to ti-chi-bot/dm that referenced this pull request Sep 23, 2021

onlineddl: support regular expressions rules for PT/GH-OST (pingcap#2139

a24f5f5

)

ti-chi-bot mentioned this pull request Sep 23, 2021

onlineddl: support regular expressions rules for PT/GH-OST (#2139) #2156

Merged

ti-chi-bot added a commit that referenced this pull request Sep 23, 2021

onlineddl: support regular expressions rules for PT/GH-OST (#2139) (#…

9721c47

…2156)

lichunzhu pushed a commit to lichunzhu/dm that referenced this pull request Sep 23, 2021

onlineddl: support regular expressions rules for PT/GH-OST (pingcap#2139

19bfc7c

) (pingcap#2156)

lichunzhu mentioned this pull request Sep 23, 2021

.*: cherry-pick #2139 and update change log for v2.0.7 #2161

Merged

ti-chi-bot pushed a commit that referenced this pull request Sep 26, 2021

.*: cherry-pick #2139 and update change log for v2.0.7 (#2161)

ea66a07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

onlineddl: support regular expressions rules for PT/GH-OST #2139

onlineddl: support regular expressions rules for PT/GH-OST #2139

wentaojin commented Sep 15, 2021 •

edited by Ehco1996

Loading

ti-chi-bot commented Sep 15, 2021 •

edited

Loading

Ehco1996 commented Sep 16, 2021 •

edited

Loading

glorv Sep 22, 2021

glorv Sep 22, 2021

glorv Sep 22, 2021

Ehco1996 left a comment

Ehco1996 Sep 22, 2021

lichunzhu Sep 22, 2021 •

edited

Loading

okJiang Sep 23, 2021

lichunzhu commented Sep 22, 2021

lichunzhu commented Sep 23, 2021

lichunzhu commented Sep 23, 2021

ti-chi-bot commented Sep 23, 2021

ti-chi-bot commented Sep 23, 2021

		ShadowTableRule []string `yaml:"shadow-table-rule" toml:"shadow-table-rule" json:"shadow-table-rule"`
		TrashTableRule []string `yaml:"trash-table-rule" toml:"trash-table-rule" json:"trash-table-rule"`

onlineddl: support regular expressions rules for PT/GH-OST #2139

onlineddl: support regular expressions rules for PT/GH-OST #2139

Conversation

wentaojin commented Sep 15, 2021 • edited by Ehco1996 Loading

What problem does this PR solve?

What is changed and how it works?

ti-chi-bot commented Sep 15, 2021 • edited Loading

Ehco1996 commented Sep 16, 2021 • edited Loading

glorv Sep 22, 2021

Choose a reason for hiding this comment

glorv Sep 22, 2021

Choose a reason for hiding this comment

glorv Sep 22, 2021

Choose a reason for hiding this comment

Ehco1996 left a comment

Choose a reason for hiding this comment

Ehco1996 Sep 22, 2021

Choose a reason for hiding this comment

lichunzhu Sep 22, 2021 • edited Loading

Choose a reason for hiding this comment

okJiang Sep 23, 2021

Choose a reason for hiding this comment

lichunzhu commented Sep 22, 2021

lichunzhu commented Sep 23, 2021

lichunzhu commented Sep 23, 2021

ti-chi-bot commented Sep 23, 2021

ti-chi-bot commented Sep 23, 2021

wentaojin commented Sep 15, 2021 •

edited by Ehco1996

Loading

ti-chi-bot commented Sep 15, 2021 •

edited

Loading

Ehco1996 commented Sep 16, 2021 •

edited

Loading

lichunzhu Sep 22, 2021 •

edited

Loading