Conflict data resolution enhancement during data import GA #17276
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ti-chi-bot
bot
added
missing-translation-status
lyzx2001
changed the title
qiancai
added
v8.1
ti-chi-bot
bot
removed
the
missing-translation-status
lyzx2001
commented
Apr 19, 2024
|
/cc @Frank945946 |
|
@qiancai: GitHub didn't allow me to request PR reviews from the following users: Frank945946. Note that only pingcap members and repo collaborators can review this PR, and authors cannot review their own PRs. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Frank945946
approved these changes
Apr 19, 2024
| # - "": in the physical import mode, TiDB Lightning does not detect or handle conflicting data. If the source file contains conflicting primary or unique key records, the subsequent step reports an error. In the logical import mode, TiDB Lightning converts the "" strategy to the "error" strategy for processing. | ||
| # - "error": when detecting conflicting primary or unique key records in the imported data, TiDB Lightning terminates the import and reports an error. | ||
| # - "replace": when encountering conflicting primary or unique key records, TiDB Lightning retains the latest data and overwrites the old data. | ||
| # The conflicting data are recorded in the `lightning_task_info.conflict_error_v2` table (recording conflicting data detected by post-import conflict detection in the physical import mode) and the `conflict_records` table (recording conflicting data detected by preprocess conflict detection in both logical and physical import modes) of the target TiDB cluster. | ||
| # If you turn on both preprocess and post-import conflict detection in physical import mode, the conflicting data can be checked in `lightning_task_info.conflict_view` view. |
There was a problem hiding this comment.
Suggested change
| # If you turn on both preprocess and post-import conflict detection in physical import mode, the conflicting data can be checked in `lightning_task_info.conflict_view` view. | |
| # If you set Conflic.strategy="replace" in physical import mode, the conflicting data can be checked in `lightning_task_info.conflict_view` view. |
| # You can manually insert the correct records into the target table based on your application requirements. Note that the target TiKV must be v5.2.0 or later versions. | ||
| # - "ignore": when encountering conflicting primary or unique key records, TiDB Lightning retains the old data and ignores the new data. This option can only be used in the logical import mode. | ||
| strategy = "" | ||
| # Controls whether to enable preprocess conflict detection, which checks conflicts in data before importing it to TiDB. In scenarios where the ratio of conflict records is greater than or equal to 1%, it is recommended to enable preprocess conflict detection for better performance in conflict detection. In other scenarios, it is recommended to disable it. The default value is false, indicating that TiDB Lightning only checks conflicts after the import. If you set it to true, TiDB Lightning checks conflicts both before and after the import. This parameter is experimental, and it can be used only in the physical import mode. | ||
| # Controls whether to enable preprocess conflict detection, which checks conflicts in data before importing it to TiDB. In scenarios where the ratio of conflict records is greater than or equal to 1%, it is recommended to enable preprocess conflict detection for better performance in conflict detection. In other scenarios, it is recommended to disable it. The default value is false, indicating that TiDB Lightning only checks conflicts after the import. If you set it to true, TiDB Lightning checks conflicts both before and after the import. This parameter can be used only in the physical import mode. |
There was a problem hiding this comment.
Suggested change
| # Controls whether to enable preprocess conflict detection, which checks conflicts in data before importing it to TiDB. In scenarios where the ratio of conflict records is greater than or equal to 1%, it is recommended to enable preprocess conflict detection for better performance in conflict detection. In other scenarios, it is recommended to disable it. The default value is false, indicating that TiDB Lightning only checks conflicts after the import. If you set it to true, TiDB Lightning checks conflicts both before and after the import. This parameter can be used only in the physical import mode. | |
| # Controls whether to enable preprocess conflict detection, which checks conflicts in data before importing it to TiDB. In scenarios where the ratio of conflict records is greater than or equal to 1%, it is recommended to set "precheck-conflict-before-import"=true for better performance in conflict detection. In other scenarios, it is recommended to disable it. The default value is false, indicating that TiDB Lightning only checks conflicts during import. If you set it to true, TiDB Lightning checks conflicts both before and during the import. This parameter can be used only in the physical import mode. |
| # - "": TiDB Lightning does not detect or handle conflicting data. If the source file contains conflicting primary or unique key records, the subsequent step reports an error. | ||
| # - "error": when detecting conflicting primary or unique key records in the imported data, TiDB Lightning terminates the import and reports an error. | ||
| # - "replace": when encountering conflicting primary or unique key records, TiDB Lightning retains the latest data and overwrites the old data. | ||
| # The conflicting data are recorded in the `lightning_task_info.conflict_error_v2` table (recording conflicting data detected by post-import conflict detection) and the `conflict_records` table (recording conflicting data detected by preprocess conflict detection) of the target TiDB cluster. | ||
| # If you turn on both preprocess and post-import conflict detection in physical import mode, the conflicting data can be checked in `lightning_task_info.conflict_view` view. |
There was a problem hiding this comment.
Suggested change
| # If you turn on both preprocess and post-import conflict detection in physical import mode, the conflicting data can be checked in `lightning_task_info.conflict_view` view. | |
| # If you set Conflic.strategy="replace" in physical import mode, the conflicting data can be checked in `lightning_task_info.conflict_view` view. |
| # You can manually insert the correct records into the target table based on your application requirements. Note that the target TiKV must be v5.2.0 or later versions. | ||
| strategy = "" | ||
| # Controls whether to enable preprocess conflict detection, which checks conflicts in data before importing it to TiDB. In scenarios where the ratio of conflict records is greater than or equal to 1%, it is recommended to enable preprocess conflict detection for better performance in conflict detection. In other scenarios, it is recommended to disable it. The default value is false, indicating that TiDB Lightning only checks conflicts after the import. If you set it to true, TiDB Lightning checks conflicts both before and after the import. This parameter is experimental. | ||
| # Controls whether to enable preprocess conflict detection, which checks conflicts in data before importing it to TiDB. In scenarios where the ratio of conflict records is greater than or equal to 1%, it is recommended to enable preprocess conflict detection for better performance in conflict detection. In other scenarios, it is recommended to disable it. The default value is false, indicating that TiDB Lightning only checks conflicts after the import. If you set it to true, TiDB Lightning checks conflicts both before and after the import. |
There was a problem hiding this comment.
Suggested change
| # Controls whether to enable preprocess conflict detection, which checks conflicts in data before importing it to TiDB. In scenarios where the ratio of conflict records is greater than or equal to 1%, it is recommended to enable preprocess conflict detection for better performance in conflict detection. In other scenarios, it is recommended to disable it. The default value is false, indicating that TiDB Lightning only checks conflicts after the import. If you set it to true, TiDB Lightning checks conflicts both before and after the import. | |
| # Controls whether to enable preprocess conflict detection, which checks conflicts in data before importing it to TiDB. In scenarios where the ratio of conflict records is greater than or equal to 1%, it is recommended to set "precheck-conflict-before-import"=true for better performance in conflict detection. In other scenarios, it is recommended to disable it. The default value is false, indicating that TiDB Lightning only checks conflicts during import. If you set it to true, TiDB Lightning checks conflicts both before and during the import. This parameter can be used only in the physical import mode. |
|
@Frank945946: adding LGTM is restricted to approvers and reviewers in OWNERS files. In response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Oreoxmt
added
the
type/compatibility-or-feature-change
qiancai
reviewed
Apr 25, 2024
| # - "": in the physical import mode, TiDB Lightning does not detect or handle conflicting data. If the source file contains conflicting primary or unique key records, the subsequent step reports an error. In the logical import mode, TiDB Lightning converts the "" strategy to the "error" strategy for processing. | ||
| # - "error": when detecting conflicting primary or unique key records in the imported data, TiDB Lightning terminates the import and reports an error. | ||
| # - "replace": when encountering conflicting primary or unique key records, TiDB Lightning retains the latest data and overwrites the old data. | ||
| # The conflicting data are recorded in the `lightning_task_info.conflict_error_v2` table (recording conflicting data detected by post-import conflict detection in the physical import mode) and the `conflict_records` table (recording conflicting data detected by preprocess conflict detection in both logical and physical import modes) of the target TiDB cluster. | ||
| # If you set `conflict.strategy = "replace"` in physical import mode, the conflicting data can be checked in `lightning_task_info.conflict_view` view. |
There was a problem hiding this comment.
Suggested change
| # If you set `conflict.strategy = "replace"` in physical import mode, the conflicting data can be checked in `lightning_task_info.conflict_view` view. | |
| # If you set `conflict.strategy = "replace"` in physical import mode, the conflicting data can be checked in the `lightning_task_info.conflict_view` view. |
There was a problem hiding this comment.
Is "conflict_view" a table? If yes, can we use "the lightning_task_info.conflict_view table" instead?
There was a problem hiding this comment.
No, "conflict_view" is a view
| # threshold = 10000 | ||
| # Controls the maximum number of records in the `conflict_records` table. The default value is 10000. In the physical import mode, if the strategy is "replace", the conflict records that are overwritten are recorded. | ||
| # In the logical import mode, if the strategy is "ignore", the conflict records that are ignored are recorded; if the strategy is "replace", the conflict records are not recorded. | ||
| # Starting from v8.1.0, max-record-rows will be assigned the value of threshold, regardless the user input. max-record-rows will be deprecated in the future. | ||
| # max-record-rows = 100 |
There was a problem hiding this comment.
Suggested change
| # max-record-rows = 100 | |
| # max-record-rows = 1000 |
update it according to L141
| # - "": TiDB Lightning does not detect or handle conflicting data. If the source file contains conflicting primary or unique key records, the subsequent step reports an error. | ||
| # - "error": when detecting conflicting primary or unique key records in the imported data, TiDB Lightning terminates the import and reports an error. | ||
| # - "replace": when encountering conflicting primary or unique key records, TiDB Lightning retains the latest data and overwrites the old data. | ||
| # The conflicting data are recorded in the `lightning_task_info.conflict_error_v2` table (recording conflicting data detected by post-import conflict detection) and the `conflict_records` table (recording conflicting data detected by preprocess conflict detection) of the target TiDB cluster. | ||
| # If you set `conflict.strategy = "replace"` in physical import mode, the conflicting data can be checked in `lightning_task_info.conflict_view` view. |
There was a problem hiding this comment.
Suggested change
| # If you set `conflict.strategy = "replace"` in physical import mode, the conflicting data can be checked in `lightning_task_info.conflict_view` view. | |
| # If you set `conflict.strategy = "replace"` in physical import mode, the conflicting data can be checked in the `lightning_task_info.conflict_view` view. |
| # precheck-conflict-before-import = false | ||
| # threshold = 9223372036854775807 | ||
| # threshold = 10000 | ||
| # max-record-rows = 100 |
There was a problem hiding this comment.
Suggested change
| # max-record-rows = 100 | |
| # max-record-rows = 1000 |
qiancai
reviewed
Apr 25, 2024
qiancai
reviewed
Apr 25, 2024
Co-authored-by: Grace Cai <qqzczy@126.com>
qiancai
reviewed
Apr 26, 2024
qiancai
reviewed
Apr 26, 2024
qiancai
reviewed
Apr 26, 2024
qiancai
reviewed
Apr 26, 2024
Co-authored-by: Grace Cai <qqzczy@126.com>
17 tasks
lyzx2001
commented
Apr 28, 2024
| # threshold = 9223372036854775807 | ||
| # Controls the maximum number of records in the `conflict_records` table. The default value is 100. In the physical import mode, if the strategy is "replace", the conflict records that are overwritten are recorded. In the logical import mode, if the strategy is "ignore", the conflict records that are ignored are recorded; if the strategy is "replace", the conflict records are not recorded. | ||
| # max-record-rows = 100 | ||
| # Controls the maximum number of conflict errors that can be handled when strategy is "replace" or "ignore". You can set it only when strategy is "replace" or "ignore". The default value is 10000. If you set it to be a much larger value, it is possible that the import will have performance degradation or fail due to process being killed or other potential errors. |
qiancai
reviewed
Apr 29, 2024
Co-authored-by: Grace Cai <qqzczy@126.com>
Co-authored-by: Grace Cai <qqzczy@126.com>
qiancai
reviewed
May 6, 2024
Co-authored-by: Grace Cai <qqzczy@126.com>
qiancai
added
translation/done
lyzx2001
commented
May 6, 2024
Co-authored-by: Luo Yangzhixin <rep_laureline@hotmail.com>
Oreoxmt
approved these changes
May 7, 2024
Oreoxmt
approved these changes
May 14, 2024
|
@Oreoxmt: Your lgtm message is repeated, so it is ignored. In response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
qiancai
approved these changes
May 14, 2024
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: qiancai The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
First-time contributors' checklist
What is changed, added or deleted? (Required)
Which TiDB version(s) do your changes apply to? (Required)
Tips for choosing the affected version(s):
By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.
For details, see tips for choosing the affected versions.
What is the related PR or file link(s)?
Do your changes match any of the following descriptions?