Skip to content
This repository has been archived by the owner on Aug 21, 2023. It is now read-only.

dump: always split TiDB v3.* tables through tidb rowid to save TiDB's memory #301

Merged
merged 24 commits into from
Jul 13, 2021

Conversation

lichunzhu
Copy link
Contributor

What problem does this PR solve?

fix #295, an extension of #273
In TiDB v3.x:

  1. table INFORMATION_SCHEMA.TIKV_REGION_STATUS doesn't have table_name/table_schema.
  2. doesn't have SHOW TABLE REGIONS statement.
  3. table INFORMATION_SCHEMA.PARTITIONS doesn't actually have data.
  4. doesn't have tidb_decode_key function.
mysql> show create table information_schema.TIKV_REGION_STATUS;
+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table              | Create Table                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| TIKV_REGION_STATUS | CREATE TABLE `TIKV_REGION_STATUS` (
  `REGION_ID` bigint(21) unsigned DEFAULT NULL,
  `START_KEY` text DEFAULT NULL,
  `END_KEY` text DEFAULT NULL,
  `EPOCH_CONF_VER` bigint(21) unsigned DEFAULT NULL,
  `EPOCH_VERSION` bigint(21) unsigned DEFAULT NULL,
  `WRITTEN_BYTES` bigint(21) unsigned DEFAULT NULL,
  `READ_BYTES` bigint(21) unsigned DEFAULT NULL,
  `APPROXIMATE_SIZE` bigint(21) unsigned DEFAULT NULL,
  `APPROXIMATE_KEYS` bigint(21) unsigned DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin |
+--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

What is changed and how it works?

Split chunks by PKIsHandle for TiDB v3.*. Make sure TiDB won't OOM through splitting tables by rowids.
We do some staff by our own to solve the problem that tidb lacks some info.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
    Test dump with v3.0.0. The result is looks expected.

Related changes

  • Need to cherry-pick to the release branch

Release note

  • Always split TiDB v3.* tables through tidb rowid to save TiDB's memory.

@lichunzhu
Copy link
Contributor Author

@kennytm PTAL

}

// for sorting
type byIntValue []int64

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about using sort.Slice ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

@@ -0,0 +1,990 @@
4,"","7480000000000000FF0500000000000000F8"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's region info read from TiDB. This is built by creating a table, inserting data and select * from information_schema.tikv_region_status. These three columns are region_id, start_key, end_key separately.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh it's test fixtures

@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • lance6716

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added the status/LGT1 One reviewer approved (LGTM1) label Jul 13, 2021
@lichunzhu
Copy link
Contributor Author

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 62c52a8

@ti-chi-bot ti-chi-bot merged commit ef07f40 into pingcap:master Jul 13, 2021
ti-chi-bot pushed a commit to ti-chi-bot/dumpling that referenced this pull request Jul 13, 2021
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #306.

@lichunzhu lichunzhu deleted the fixSplitRegionTiDB3 branch July 13, 2021 07:04
ti-chi-bot pushed a commit to ti-chi-bot/dumpling that referenced this pull request Jul 13, 2021
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #307.

@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #308.

tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
tisonkun pushed a commit to tisonkun/dumpling that referenced this pull request Oct 20, 2021
tisonkun pushed a commit to tisonkun/tidb that referenced this pull request Oct 20, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Use dumpling --chunk-by-region=true in tidb v3, got Unknown column ERROR
6 participants