data migration: add data migration best practices #10550
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
[REVIEW NOTIFICATION] This pull request has been approved by:
To complete the pull request process, please ask the reviewers in the list to review by filling The full list of commands accepted by this bot can be found here. Reviewer can indicate their review by submitting an approval review. |
ti-chi-bot
added
the
do-not-merge/work-in-progress
ti-chi-bot
added
missing-translation-status
hfxsd
added
area/best-practices
ti-chi-bot
added
the
size/M
ti-chi-bot
added
size/L
ti-chi-bot
removed
the
do-not-merge/work-in-progress
hfxsd
commented
Sep 30, 2022
hfxsd
commented
Sep 30, 2022
hfxsd
commented
Sep 30, 2022
hfxsd
commented
Sep 30, 2022
hfxsd
commented
Sep 30, 2022
hfxsd
commented
Sep 30, 2022
hfxsd
commented
Sep 30, 2022
hfxsd
commented
Sep 30, 2022
qiancai
reviewed
Sep 30, 2022
dm/dm-best-practices.md
Outdated
|
|
||
| #### Character sets and collations | ||
|
|
||
| Since TiDB v6.0.0, the new framework for collations is used by default. If you want TiDB to support utf8_general_ci, utf8mb4_general_ci, utf8_unicode_ci, utf8mb4_unicode_ci, gbk_chinese_ci and gbk_bin, you need explicitly declare it when creating the cluster by setting the value of `new_collations_enabled_on_first_bootstrap` to `true`. For more information, see [New framework for collations](/character-set-and-collation.md#new-framework-for-collations). |
There was a problem hiding this comment.
new_collations_enabled_on_first_bootstrap is true by default since v6.0.0, so there is no need to explicitly declare it?
hfxsd
commented
Sep 30, 2022
qiancai
reviewed
Sep 30, 2022
dm/dm-best-practices.md
Outdated
|
|
||
| - When you use DM to perform long-time data replication, sometimes the migration task is blocked for a long time due to various reasons. If you enable the relay log, you can effectively deal with the problem of upstream Binlog being recycled due to the blocking of the migration task. | ||
|
|
||
| There are some restrictions using the relay log. DM supports high availability. When a DM-worker fails, it will try to promote an idle DM-worker instance to a working instance. If the upstream Binlog does not contain the necessary migration logs, it may cause interruption. You need to intervene manually to copy the relay log to the new DM-worker node as soon as possible, and modify the corresponding relay meta file. For details, see [Troubleshooting](/dm/dm-error-handling.md#the-relay-unit-throws-error-event-from--in--diff-from-passed-in-event--or-a-migration-task-is-interrupted-with-failing-to-get-or-parse-binlog-errors-like-get-binlog-error-error-1236-hy000-and-binlog-checksum-mismatch-data-may-be-corrupted-returned). |
There was a problem hiding this comment.
Suggested change
| There are some restrictions using the relay log. DM supports high availability. When a DM-worker fails, it will try to promote an idle DM-worker instance to a working instance. If the upstream Binlog does not contain the necessary migration logs, it may cause interruption. You need to intervene manually to copy the relay log to the new DM-worker node as soon as possible, and modify the corresponding relay meta file. For details, see [Troubleshooting](/dm/dm-error-handling.md#the-relay-unit-throws-error-event-from--in--diff-from-passed-in-event--or-a-migration-task-is-interrupted-with-failing-to-get-or-parse-binlog-errors-like-get-binlog-error-error-1236-hy000-and-binlog-checksum-mismatch-data-may-be-corrupted-returned). | |
| There are some restrictions on using the relay log. DM supports high availability. When a DM-worker fails, it will try to promote an idle DM-worker instance to a working instance. If the upstream binlogs do not contain the necessary migration logs, it may cause interruption. You need to intervene manually to copy the relay log to the new DM-worker node as soon as possible, and modify the corresponding relay meta file. For details, see [Troubleshooting](/dm/dm-error-handling.md#the-relay-unit-throws-error-event-from--in--diff-from-passed-in-event--or-a-migration-task-is-interrupted-with-failing-to-get-or-parse-binlog-errors-like-get-binlog-error-error-1236-hy000-and-binlog-checksum-mismatch-data-may-be-corrupted-returned). |
dm/dm-best-practices.md
Outdated
|
|
||
| #### Use PT-osc/GH-ost in upstream | ||
|
|
||
| In daily MySQL operation and maintenance, usually you use tools such as PT-osc/GH-ost to change the schema online to minimize impact on the business. However, the whole process will be logged to MySQL Binlog. Migrating such data to TiDB downstream will result in a lot of write operations, which is neither efficient nor economical. |
There was a problem hiding this comment.
Suggested change
| In daily MySQL operation and maintenance, usually you use tools such as PT-osc/GH-ost to change the schema online to minimize impact on the business. However, the whole process will be logged to MySQL Binlog. Migrating such data to TiDB downstream will result in a lot of write operations, which is neither efficient nor economical. | |
| In daily MySQL operation and maintenance, usually you use tools such as PT-osc/GH-ost to change the schema online to minimize impact on the business. However, the whole process will be logged to MySQL Binlog. Migrating such data to TiDB downstream will result in a lot of unnecessary write operations, which is neither efficient nor economical. |
dm/dm-best-practices.md
Outdated
|
|
||
| ## Best practices during migration | ||
|
|
||
| This section introduces how to troubleshoot problems you encounter during migration. |
There was a problem hiding this comment.
Suggested change
| This section introduces how to troubleshoot problems you encounter during migration. | |
| This section introduces how to troubleshoot problems you might encounter during migration. |
dm/dm-best-practices.md
Outdated
|
|
||
| ### Interrupted migration task due to failed DDL | ||
|
|
||
| DM supports skipping or replacing DDL statements that cause the migration task to interrupt. For details, see [Handle Failed DDL Statements](/dm/handle-failed-ddl-statements.md#usage-examples). |
There was a problem hiding this comment.
Suggested change
| DM supports skipping or replacing DDL statements that cause the migration task to interrupt. For details, see [Handle Failed DDL Statements](/dm/handle-failed-ddl-statements.md#usage-examples). | |
| DM supports skipping or replacing DDL statements that cause a migration task to interrupt. For details, see [Handle Failed DDL Statements](/dm/handle-failed-ddl-statements.md#usage-examples). |
Co-authored-by: Grace Cai <qqzczy@126.com>
Frank945946
approved these changes
Sep 30, 2022
|
/merge |
|
@hfxsd: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
|
/merge |
|
This pull request has been accepted and is ready to merge. Commit hash: e87f109
|
ti-chi-bot
added
the
status/can-merge
12 tasks
|
In response to a cherrypick label: new pull request created: #10619. |
ti-chi-bot
added a commit
that referenced
this pull request
Sep 30, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is changed, added or deleted? (Required)
Add data migration best practices.
Which TiDB version(s) do your changes apply to? (Required)
Tips for choosing the affected version(s):
By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.
For details, see tips for choosing the affected versions.
What is the related PR or file link(s)?
Do your changes match any of the following descriptions?