Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick 4 PRs related to tikv-importer to release-2.1 #4369

Merged
merged 4 commits into from Mar 14, 2019

Conversation

Projects
None yet
4 participants
@kennytm
Copy link
Contributor

commented Mar 13, 2019

Signed-off-by: Lonng chris@lonng.org
Signed-off-by: kennytm kennytm@gmail.com

What have you changed? (mandatory)

Cherry-picks the following 4 PRs for 2.1.6 release:

  • #4347 (increase default region-split-size to 512 MiB)
  • #4348 (store intermediate SST files on disk instead of memory)
  • #4350 (restrict memory usage by RocksDB)
  • #4352 (wait after split before scatter)

What are the type of the changes? (mandatory)

  • Improvement (change which is an improvement to an existing feature)
  • Bug fix (change which fixes an issue)

How has this PR been tested? (mandatory)

Tested manually using a data set with size 6.5 TB

Does this PR affect documentation (docs) update? (mandatory)

No

Does this PR affect tidb-ansible update? (mandatory)

Yes (will be filed separately, together with Lightning changes)

Refer to a related PR or issue link (optional)

See above

Benchmark result if necessary (optional)

Add a few positive/negative examples (optional)

kennytm added some commits Mar 12, 2019

import: increase default region-split-size to 512 MiB (#4347)
This makes the SST size to upload larger, which reduces the number of SST
files needed to ingested, and improve overall speed.

Signed-off-by: kennytm <kennytm@gmail.com>
import: store the intermediate SST files on disk instead of memory (#…
…4348)

* import: store the intermediate SST files on disk instead of memory

The bottle-neck of Importer is in SST ingestion on the TiKV side, not
disk I/O. Storing these SST in RAM will just increase the chance Importer
getting killed by OOM.

Signed-off-by: kennytm <kennytm@gmail.com>
import: fix #4257, restrict memory usage by RocksDB (#4350)
* import: put index and filter in block cache to restrict memory usage

Signed-off-by: Lonng <chris@lonng.org>
Signed-off-by: kennytm <kennytm@gmail.com>
import: wait after split before scatter (#4352)
Signed-off-by: kennytm <kennytm@gmail.com>

@kennytm kennytm force-pushed the kennytm:importer-cherry-pick-batch-1 branch from 3cbc72e to b207a99 Mar 13, 2019

@kennytm

This comment has been minimized.

Copy link
Contributor Author

commented Mar 13, 2019

@overvenus
Copy link
Contributor

left a comment

👍

@huachaohuang huachaohuang merged commit 229c423 into tikv:release-2.1 Mar 14, 2019

2 checks passed

DCO All commits are signed off!
Details
idc-jenkins-ci/test Jenkins job succeeded.
Details

@kennytm kennytm deleted the kennytm:importer-cherry-pick-batch-1 branch Mar 14, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.