*: merging tidb-lightning to release-4.0 #746
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ingcap#132) * restore: write index kvs and data kvs to seperate engine * restore: rename to * restore: use single engine file to store index * restore: make index engine limited by table concurrency * restore: revert some changes * restore: modify checkpoint proto * restore: implement checkpoint for index engine file * tests: add failpoint for CheckpointStatusIndexImported * address comment * address comment * address comment
* loader: recognize CSV files * *: generalize the parser interface * mydump: added a CSV parser * mydump,restore: enable CSV parser * tests: added integration test for CSV * mydumper: added test case for empty CSV * config: improved description * restore: fixed a compile error on Go 1.12
update tidb version to latest release2.1, and add a test case for issue pingcap/tidb#9532
…ingcap#142) Currently, cleanup script in integration test does not work, if the developer wants to re-run integration test in his/her local environment, he/she has to clean /tmp/lightning_test_result manually
Test `checkpoint_error_destroy` in integration got error randomly, both in local environment and JenkinsCI. Lightning could exit before all checkpoints are saved, as the `WaitGroup` is not always added before this line https://github.com/pingcap/tidb-lightning/blob/master/lightning/restore/restore.go#L204 returns 1. After `RestoreController.Run` is returned, no more messages will be sent to `rc.saveCpCh`, so we can close it safely and ensure all `saveCp` are consumed 2. remove some unused code
…" (pingcap#152) This reverts commit 852c5e9.
* restore: add local checksum log * Update lightning/restore/restore.go Co-Authored-By: lonng <chris@lonng.org>
use gofail to control lightning to kill itself after one chunk is imported
…ingcap#145) * *: parse the data source directly into data and skip the KV encoder This skips the more complex pingcap/parser, and speeds up parsing speed by 50%. We have also refactored the KV delivery mechanism to use channels directly, and revamped metrics: - Make the metrics about engines into its own `engines` counter. The `tables` counter is exclusively about tables now. - Removed `block_read_seconds`, `block_read_bytes`, `block_encode_seconds` since the concept of "block" no longer applies. Replaced by the equivalents named `row_***`. - Removed `chunk_parser_read_row_seconds` for being overlapping with `row_read_seconds`. - Changed `block_deliver_bytes` into a histogram vec, with kind=index or kind=data. Introduced `block_deliver_kv_pairs`. * tests,restore: prevent spurious error in checkpoint_chunks test Only kill Lightning if the whole chunk is imported exactly. The chunk checkpoint may be recorded before a chunk is fully written, and this will hit the failpoint more than 5 times. * kv: use composed interface to simplify some types * kv: properly handle the SQL mode * common: disable IsContextCanceledError() when log level = debug This helps debugging some mysterious cancellation where the log is inhibited. Added IsReallyContextCanceledError() for code logic affected by error type. * restore: made some log more detailed * restore: made the SlowDownImport failpoint apply to index engines too * restore: do not open a write stream when there are no KV pairs to send * tests: ensure we drop the checkpoints DB before re-run * mydump: fixed various off-by-one errors in the CSV parser * *: rename `!IsContextCanceledError` to `ShouldLogError` * *: addressed comments * restore: zero the checksums and column permutations on initialization * *: addressed comments * tests: add back a missing license header * tests: improve a comment.
…gcap#158 (pingcap#159) * tests: fix a test failure due to conflict between pingcap#145 and pingcap#158 * restore: apply the row count limit to failpoint KillIfImportedChunk too
Put the large table in the front of the slice which can avoid large table take a long time to import and block small table to release index worker.
* config: remove deprecated -compact, -switch-mode flags from tidb-lightning They can still be called from tidb-lightning-ctl. * config: search also conf/tidb-lightning.toml for default -config path Fallback to standard default if -config is not supplied * config: provide command line arguments for some common options * config: stop searching for tidb-lightning.toml if --config is unspecified
* tests: fix existing test failure * mydump: fixed conversion of integers into bits We need to create a special branch for integers, since casting 123 and '123' into BIT type behave differently. Also fixed handling of 0x/0b bit strings since Ragel doesn't recognize '+' in a regex -_-. * mydump: store description of `token` in an array instead of switch cases * tests: test behavior of integers for ENUM and SET types as well
* *: use pingcap/log (zap) for logging
Some redundant logs (e.g. logging about the same thing inside and outside
a function) are removed.
The {QueryRow,Transact,Exec}WithRetry functions are revamped to include
the logger.
* common,config: addressed comments
* *: addressed comments
* restore,verification: addressed comments
* kv: fix handling of column default values * if the column is AUTO_INCREMENT, fill in with row_id (assume it is missing for the entire table instead of just a few values) * if the column has DEFAULT, fill in that value * tests: ensure DEFAULT CURRENT_TIMESTAMP works
Ensures table + index <= the default max-open-engine which is 8.
* config: added [[routes]] config * mydump, restore: support table routing * tests: added test case for table routing * mydump: ensure rerouted schemas will not be created * config: allows routes to be case-sensitive * restore: replace CREATE TABLE -> IF NOT EXISTS using tidb/parser * mydump/loader_test: add unit test for route() and refactor * tests: TiDB doesn't support `DECIMAL(20, 0)`. * tests: workaround pingcap/parser#310
The character is too exotic and breaks TiDB and some old git.
…ss (pingcap#178) If no files are completely imported within the first 5 minutes, we get `finished == 0` and the logger will try to log a nil field and crashes.
Signed-off-by: Neil Shen <overvenus@gmail.com>
Signed-off-by: Neil Shen <overvenus@gmail.com>
Signed-off-by: Neil Shen <overvenus@gmail.com>
Signed-off-by: Neil Shen <overvenus@gmail.com>
Signed-off-by: Neil Shen <overvenus@gmail.com>
overvenus
force-pushed
the
release-4.0-merging
branch
from
9725af7
to
b28c1eb
Compare
Signed-off-by: Neil Shen <overvenus@gmail.com>
overvenus
force-pushed
the
release-4.0-merging
branch
from
b28c1eb
to
45831c6
Compare
|
/lgtm |
|
/lgtm |
ti-srebot
previously approved these changes
Mar 2, 2021
overvenus
changed the base branch from
subtree-lightning-release-4.0
to
release-4.0
|
|
|
/lgtm |
ti-srebot
approved these changes
Mar 3, 2021
3pointer
approved these changes
Mar 3, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
This PR merges TiDB-Lightning into release-4.0 branch, a cherry-pick of #665.
To facilitate review, it targets subtree-lightning-release-4.0 temporary, will targets to and squash merges into release-4.0 after LGTM2.
Check List
Tests
Code changes
Side effects
Release Note