-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
*: update format #7
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
@@ -12,4 +12,4 @@ test: | |||
cargo test | |||
|
|||
format: | |||
rustfmt --write-mode overwrite src/*.rs | |||
rustfmt --write-mode overwrite src/lib.rs src/bin/server.rs |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should not process format manually
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
using *.rs
is not ok?
@siddontang PTAL |
birdstorm
added a commit
to birdstorm/tikv
that referenced
this pull request
Oct 23, 2019
solotzg
referenced
this pull request
in pingcap/tidb-engine-ext
Nov 15, 2019
birdstorm
added a commit
to birdstorm/tikv
that referenced
this pull request
Nov 19, 2019
birdstorm
added a commit
to birdstorm/tikv
that referenced
this pull request
Feb 15, 2020
Signed-off-by: birdstorm <samuelwyf@hotmail.com>
brson
pushed a commit
to brson/tikv
that referenced
this pull request
Apr 13, 2020
Signed-off-by: birdstorm <samuelwyf@hotmail.com>
BugenZhao
pushed a commit
to BugenZhao/tikv
that referenced
this pull request
Aug 31, 2021
fix precisions & add bit support
mornyx
added a commit
to mornyx/tikv
that referenced
this pull request
Dec 7, 2021
# This is the 1st commit message: Add latency stats for kv get and kv batch get Signed-off-by: mornyx <mornyx.z@gmail.com> # This is the commit message #2: tikv_util: upgrade `gag` from 0.1 to 1.0 (tikv#11277) ref tikv#11276 Signed-off-by: kennytm <kennytm@gmail.com> # This is the commit message tikv#3: codec: add compile time assert to guarantee safety (tikv#11357) * codec: add compile time assert to guarantee safety close tikv#11356. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * add more explanation. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#4: rafstore: fix commit state check in apply thread (tikv#11401) * fix apply check & close tikv#11396 Signed-off-by: gengliqi <gengliqiii@gmail.com> * make format Signed-off-by: gengliqi <gengliqiii@gmail.com> * format Signed-off-by: gengliqi <gengliqiii@gmail.com> # This is the commit message tikv#5: backup: create external storage per-backup reqeust. (tikv#11113) * backup: create external storage per-backup reqeust. fixed tikv#11112 Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: make clippy happy Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: fix build Signed-off-by: yujuncen <yujuncen@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#6: Remove duplicate enum and macro about numberic_enum_mod (tikv#11347) * remove duplicated and useless `remove duplicated and useless` close tikv#11297 Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * remove duplicate macro rocksdb_enum_mod Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * fix format problem Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * resolve discussion Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * fix test error Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * move macro `numeric_enum_serializing_mod` to a proper location Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * move `perf_level_serde` of mod config in engine trait to mod perf_context Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * Add test about `rate-limiter-mode` and fix compile error in integration test Signed-off-by: yuqi1129 <yuqi4733@gmail.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#7: ctl: detect raft db correctly (tikv#11395) Close tikv#11393. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#8: Support to iterate table properties for KvEngine (tikv#11388) * support to iterate table properties for KvEngine * close tikv#11387 Signed-off-by: qupeng <qupeng@pingcap.com> # This is the commit message tikv#9: gc: Fix GC scan effectiveness to avoid OOM (tikv#11416) * close tikv#11410 limit the pending task for gc worker Signed-off-by: Connor1996 <zbk602423539@gmail.com> * avoid using prefix seek Signed-off-by: Connor1996 <zbk602423539@gmail.com> * rename test Signed-off-by: Connor1996 <zbk602423539@gmail.com> * address comment Signed-off-by: Connor1996 <zbk602423539@gmail.com> * fix build Signed-off-by: Connor1996 <zbk602423539@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#10: fix a panic about parsing memory.limit_in_bytes (tikv#11354) * handle int overflow for memory.limit_in_bytes clsoe tikv#11353. Signed-off-by: qupeng <qupeng@pingcap.com> * close tikv#11353 Signed-off-by: qupeng <qupeng@pingcap.com> * address comments Signed-off-by: qupeng <qupeng@pingcap.com> # This is the commit message tikv#11: build: fix make docker on Linux (tikv#11349) * fix make docker fix tikv#11312 Signed-off-by: tabokie <xy.tao@outlook.com> * fix cargo Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#12: engine_rocks: ignore NoSpace error during flush or compaction (tikv#11225) * ignore NoSpace error during flush or compaction fix tikv#11224 Signed-off-by: tabokie <xy.tao@outlook.com> * use constant Signed-off-by: tabokie <xy.tao@outlook.com> * fix typo Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#13: gc: set pending capacity for gc worker (tikv#11429) Signed-off-by: Connor1996 <zbk602423539@gmail.com> # This is the commit message tikv#14: raftstore: support synchronous write (tikv#11418) * support sync write & ref tikv#11165 Signed-off-by: gengliqi <gengliqiii@gmail.com> * add tests Signed-off-by: gengliqi <gengliqiii@gmail.com> * change default store-io-pool-size to 0 Signed-off-by: gengliqi <gengliqiii@gmail.com> * remove unnecessary test Signed-off-by: gengliqi <gengliqiii@gmail.com> * merge raftstore: separate read write ready(tikv#10592) Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix test Signed-off-by: gengliqi <gengliqiii@gmail.com> * make clippy happy Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comment Signed-off-by: gengliqi <gengliqiii@gmail.com> * remove offset in NormalFsm Signed-off-by: gengliqi <gengliqiii@gmail.com> * make format Signed-off-by: gengliqi <gengliqiii@gmail.com> # This is the commit message tikv#15: metrics: add Raft Engine panel (tikv#11415) Signed-off-by: tabokie <xy.tao@outlook.com> # This is the commit message tikv#16: resource-metering: add some metrics (tikv#11414) * resource-metering: add some metrics Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * more log Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * polish description, ref tikv#11436 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#17: metrics: add asynchronous write io panels (tikv#11408) * update metrics name Signed-off-by: gengliqi <gengliqiii@gmail.com> * update tikv-details & close tikv#10540 Signed-off-by: gengliqi <gengliqiii@gmail.com> * rename Raft GC to Raft Log GC Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comment Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix datasource Signed-off-by: gengliqi <gengliqiii@gmail.com> * add more metrics Signed-off-by: gengliqi <gengliqiii@gmail.com> * update metrics Signed-off-by: gengliqi <gengliqiii@gmail.com> * update Signed-off-by: gengliqi <gengliqiii@gmail.com> # This is the commit message tikv#18: storage: fix reverse scan check memory locks (tikv#11447) close tikv#11440 Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> # This is the commit message tikv#19: test: make test_sending_fail_with_net_error more statable (tikv#11366) * sending count should deregister on time Signed-off-by: bufferflies <1045931706@qq.com> * make test statable Signed-off-by: bufferflies <1045931706@qq.com> * remove unused change Signed-off-by: bufferflies <1045931706@qq.com> * remove unused change Signed-off-by: bufferflies <1045931706@qq.com> * close tikv#11365 Signed-off-by: bufferflies <1045931706@qq.com> * reformat annotation Signed-off-by: bufferflies <1045931706@qq.com> * add send once filter Signed-off-by: bufferflies <1045931706@qq.com> * lint pass Signed-off-by: bufferflies <1045931706@qq.com> * lint pass Signed-off-by: bufferflies <1045931706@qq.com> * replace sendOnceMsgFilter Signed-off-by: bufferflies <1045931706@qq.com> * remove unnecessary change Signed-off-by: bufferflies <1045931706@qq.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#20: raftstore: increase batch for raftlog-gc-worker (tikv#11405) * increase batch Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * do not change seek Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix fmt Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix test Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix number overflow Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * increase batch Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * revert time Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * set last_compacted_idx for split region Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * log large range gc task Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * address comment Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * close tikv#11404 Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> Co-authored-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#21: Make tikv-server fully generic over kv engines (tikv#10945) * server: Allow flow_info_receiver to be None Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Move init_flow_receiver call into init_raw_engines Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract debug service creation to CreateKvEngine::start_debug_service Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract backup service creation to CreateKvEngine::start_backup_service Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract rocks cfg controller registration to CreateKvEngine::register_kv_config Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract create_raftstore_compaction_listener to free function Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract create_kv_engine to CreateKvEngine trait Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Parameterize EngineResourceInfo over EK Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Parameterize TikvServer over EK Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Document CreateKvEngine trait Signed-off-by: Brian Anderson <andersrb@gmail.com> * clippy Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract raft engine creation to CreateRaftEngine::create_raft_engine Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract CreateRaftEngine::register_raft_config method Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract CreateRaftEngine::create_engine_infos method Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Unspecialize init_raw_engines method Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract setup code to generic run_server function Signed-off-by: Brian Anderson <andersrb@gmail.com> * clippy ref tikv#6402 Signed-off-by: Brian Anderson <andersrb@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#22: Revert "Make tikv-server fully generic over kv engines (tikv#10945)" (tikv#11462) This reverts commit df2e921. Ref tikv#11461. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> # This is the commit message tikv#23: raftstore: flush waterfall metrics when using synchronous write (tikv#11451) Signed-off-by: gengliqi <gengliqiii@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#24: fix one disk full unstable test case on assert condition. (tikv#11391) close tikv#11389 Signed-off-by: tier-cap <zhengxiaojin@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#25: resolved_ts: return early after regions already had quorum (tikv#11352) * close tikv#11351 Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Zixiong Liu <liuzixiong@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * make clippy happy Signed-off-by: 5kbpers <tangminghua@pingcap.com> * close tikv#11400 Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: qupeng <qupeng@pingcap.com> Co-authored-by: Zixiong Liu <liuzixiong@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#26: support to get approximate keys and size for a given range from UserCollectedProperties (tikv#11465) Signed-off-by: qupeng <qupeng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#27: copr: add paging API support for streaming-like processling (tikv#11300) * copr response with range Signed-off-by: you06 <you1474600@gmail.com> * switch to new kvproto Signed-off-by: you06 <you1474600@gmail.com> * fix bugs... Signed-off-by: you06 <you1474600@gmail.com> * format Signed-off-by: you06 <you1474600@gmail.com> * remove unsed field Signed-off-by: you06 <you1474600@gmail.com> * add comments Signed-off-by: you06 <you1474600@gmail.com> * update kvproto Signed-off-by: you06 <you1474600@gmail.com> * fix bench Signed-off-by: you06 <you1474600@gmail.com> * update cargo.lock Signed-off-by: you06 <you1474600@gmail.com> * update comment Signed-off-by: you06 <you1474600@gmail.com> * add test for copr paging Signed-off-by: you06 <you1474600@gmail.com> * close tikv#11448 Signed-off-by: you06 <you1474600@gmail.com> * Update components/tidb_query_executors/src/runner.rs Co-authored-by: Lei Zhao <zlwgx1023@gmail.com> Signed-off-by: you06 <you1474600@gmail.com> * check range in response Signed-off-by: you06 <you1474600@gmail.com> * format code Signed-off-by: you06 <you1474600@gmail.com> * add test for reverse scan Signed-off-by: you06 <you1474600@gmail.com> * add test for multi ranges Signed-off-by: you06 <you1474600@gmail.com> * test drained with multi ranges Signed-off-by: you06 <you1474600@gmail.com> * fix clippy Signed-off-by: you06 <you1474600@gmail.com> * fix lint Signed-off-by: you06 <you1474600@gmail.com> * paging by all record count Signed-off-by: you06 <you1474600@gmail.com> * format Signed-off-by: you06 <you1474600@gmail.com> Co-authored-by: Lei Zhao <zlwgx1023@gmail.com> # This is the commit message tikv#28: encryption: support seek for encryption writer (tikv#10929) * support seek for encryption writer Signed-off-by: tabokie <xy.tao@outlook.com> * minor fix, ref: tikv#11119 Signed-off-by: tabokie <xy.tao@outlook.com> * fix error from merge Signed-off-by: tabokie <xy.tao@outlook.com> * fight format checker, ref tikv#11119 Signed-off-by: tabokie <xy.tao@outlook.com> * add wrappers Signed-off-by: tabokie <xy.tao@outlook.com> * remove pub quantifiers and fix lint Signed-off-by: tabokie <xy.tao@outlook.com> * fix name export Signed-off-by: tabokie <xy.tao@outlook.com> * revert naming changes Signed-off-by: tabokie <xy.tao@outlook.com> * address comment on unit test Signed-off-by: tabokie <xy.tao@outlook.com> * fix test and inline wrapper calls Signed-off-by: tabokie <xy.tao@outlook.com> * fix comments Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#29: server: remove register PD process (tikv#10985) * remove register PD process Signed-off-by: Ryan Leung <rleungx@gmail.com> * resolve conflicts close tikv#11472 Signed-off-by: Ryan Leung <rleungx@gmail.com> # This is the commit message tikv#30: engine: provide customized file builder to raft engine (tikv#10937) * open raft engine with customized file builder ref tikv#11119 Signed-off-by: tabokie <xy.tao@outlook.com> * overhaul configurations Signed-off-by: tabokie <xy.tao@outlook.com> * update default config and address comment Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: qupeng <qupeng@pingcap.com> # This is the commit message tikv#31: *: update some dependencies (tikv#11458) close tikv#10997, ref tikv#11276 Signed-off-by: tabokie <xy.tao@outlook.com> # This is the commit message tikv#32: coprocessor: add `to_binary` and `from_binary` function (tikv#11397) Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#33: server: fix IOType configure for recving snapshot (tikv#11122) * fix IOType for recving snapshot Signed-off-by: MuZhou233 <muzhou233@outlook.com> * Set io_type before Write::write_all. close tikv#10659 Signed-off-by: MuZhou233 <MuZhou233@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#34: raftstore: add pessimistic lock table to peer (tikv#11459) ref tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#35: engine: fix backward compatibility issue of raft-engine.recovery-mode (tikv#11490) * add test case Signed-off-by: tabokie <xy.tao@outlook.com> * update raft-engine, fix tikv#11489 Signed-off-by: tabokie <xy.tao@outlook.com> * handle timeout error Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: qupeng <qupeng@pingcap.com> # This is the commit message tikv#36: metrics: add duration composition panel (tikv#11491) * metrics: add duration composition pannel Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> * close tikv#11492 Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#37: *: support read-through lock (tikv#11238) * storage: get/batch_get support read-through lock Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * storage: scan support read-through lock Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * coprocessor: support read-through lock Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * don't check access_locks when checking memory locks Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * fix clippy close tikv#11402 Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * update kvproto and add some comments Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * test desc scan Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#38: storage: support raw value encode for API V2 (tikv#11139) * implement api v2 raw value Signed-off-by: Andy Lok <andylokandy@hotmail.com> * storage: support raw value encode for API V2 ref tikv#10938 close tikv#11041 Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix clippy Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix clippy Signed-off-by: Andy Lok <andylokandy@hotmail.com> * remove println Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * add test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * disable test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test again Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test again Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test again Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix overflow Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix overflow Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve comment Signed-off-by: Andy Lok <andylokandy@hotmail.com> * add case Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve comment Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix clippy Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#39: *: replace snapshot `CbContext` with `SnapshotExt` (tikv#11471) ref tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> # This is the commit message tikv#40: Add comments Signed-off-by: mornyx <mornyx.z@gmail.com> # This is the commit message tikv#41: Extract one stats struct for KvGet/KvBatchGet Signed-off-by: mornyx <mornyx.z@gmail.com> # This is the commit message tikv#42: raftstore: relax merge result check (tikv#11478) * add test case Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * raftstore: relax merge result check Close tikv#11475. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * fix unstable transfer leader case Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Liqi Geng <gengliqiii@gmail.com> # This is the commit message tikv#43: engine: move raft engine purge job to a dedicated thread (tikv#11497) * update raft engine and implement batch_gc, ref tikv#11119 Signed-off-by: tabokie <xy.tao@outlook.com> * introduce purge-worker Signed-off-by: tabokie <xy.tao@outlook.com> * clean up old worker Signed-off-by: tabokie <xy.tao@outlook.com> * update raft engine Signed-off-by: tabokie <xy.tao@outlook.com> # This is the commit message tikv#44: Update max_ts for Prewrite and AcquirePessimisticLock with should_not_exist flag. (tikv#11488) * update max_ts for prewrite with should_not_exist flag Signed-off-by: dwangxxx <dwang_slam@163.com> * close tikv#11315 Signed-off-by: dwangxxx <dwang_slam@163.com> * close tikv#11315 Signed-off-by: dwangxxx <dwang_slam@163.com> * fix unit test Signed-off-by: dwangxxx <dwang_slam@163.com> * fix format Signed-off-by: dwangxxx <dwang_slam@163.com> * fix test Signed-off-by: dwangxxx <dwang_slam@163.com> * add test. Signed-off-by: dwangxxx <dwang_slam@163.com> * fix clippy error Signed-off-by: dwangxxx <dwang_slam@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#45: Rename KvGetLatencyStats Signed-off-by: mornyx <mornyx.z@gmail.com> # This is the commit message tikv#46: Rename Signed-off-by: mornyx <mornyx.z@gmail.com> # This is the commit message tikv#47: *: simplify pull request template (tikv#11504) We only needs to explain what's change. Problem description should be traced as issue. This PR also polishes contributing guide to explain the new flow. Close tikv#11503. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: qupeng <qupeng@pingcap.com> # This is the commit message tikv#48: cdc: introduce TsFilter into incremental scan (tikv#11385) * close tikv#11384 Signed-off-by: qupeng <qupeng@pingcap.com> # This is the commit message tikv#49: metric: add stale read count metric (tikv#11051) Signed-off-by: linning <linningde25@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#50: storage:change configuration storage.max-key-size default value from 4096 to 8192 (tikv#11331) close tikv#11241 Signed-off-by: TonsnakeLin <lpbgytong@163.com> # This is the commit message tikv#51: storage: make it cheaper to clone Scheduler (tikv#11501) closes tikv#11500 Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#52: raftclient: count term and index in estimated size (tikv#11493) * add test case Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * count term and index tag If there are many entries in a message, the estimated size of message can be way smaller than the actual size. This PR fixes the error by also counting index and term in estimation. It also remove the hard limit as the estimation is closed enough. Close tikv#9714. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#53: scanner/forward: move to next key if accesses a lock with delete type (tikv#11544) close tikv#11541 Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> # This is the commit message tikv#54: *: add configuration for in-memory pessimistic lock (tikv#11481) ref tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#55: backup: fix background panic in `endpoint::tests::test_adjust_thread_pool_size` (tikv#11556) * backup: fix tikv#11548 Signed-off-by: yujuncen <yujuncen@pingcap.com> # This is the commit message tikv#56: tikv_util: reimplement poll_future_notify (tikv#11550) The current implementation is very easy to get deadlock if the future is woken immediately during polling. This PR resolves the issue by maintaining states instead. Note, if the future is implemented in the wrong way that always notifying the future without checking if it has readiness, the new implementation can run into dead loop. Close tikv#11549. Co-authored-by: qupeng <qupeng@pingcap.com> # This is the commit message tikv#57: fix taking rootfs as filesystem by mistake (tikv#11554) * fix adding rootfs as filesystem by mistake Signed-off-by: onlyacat <huangzhe8263@gmail.com> * reformat by lints Signed-off-by: onlyacat <huangzhe8263@gmail.com> * close tikv#11552 Signed-off-by: onlyacat <huangzhe8263@gmail.com> # This is the commit message tikv#58: raftstore: set term when catch up logs for merge (tikv#11545) * add test case Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * raftstore: set term when catch up logs for merge Prepare merge may includes logs from different terms, which may be even larger than the term of a follower that is lag behind. So when catching up logs by commit merge, its term should also be set to get a correct metadata. Close tikv#11526. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * add log for becoming follower Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * correct commit log check Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
ti-chi-bot
added a commit
that referenced
this pull request
Feb 24, 2022
ref #7, close pingcap/tidb#32506 Signed-off-by: yisaer <disxiaofei@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
CalvinNeo
added a commit
to pingcap/tidb-engine-ext
that referenced
this pull request
May 18, 2022
* raftstore: support dynamically scaling pool size (#11159) (#11213) close #11159 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * storage: reduce extra next/pre in row_scan (#11625) * reduce extra next/pre in row_scan close https://github.com/tikv/tikv/issues/11572 Signed-off-by: Jayice <1185430411@qq.com> * reduce extra next/pre in row_scan close #11572 Signed-off-by: Jayice <1185430411@qq.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: remove some unnecessary dependencies (#11592) close #11593 Signed-off-by: Ryan Leung <rleungx@gmail.com> Co-authored-by: Yilin Chen <sticnarf@gmail.com> * raftstore: make threads of snap-generator pool configurable (#11274) * add snap-generator-pool-size for configurable snapshot generator thread pool Signed-off-by: iosmanthus <myosmanthustree@gmail.com> * rename config macro Signed-off-by: iosmanthus <myosmanthustree@gmail.com> Co-authored-by: Andy Lok <andylokandy@hotmail.com> * ctl:add prompt for ctl when required param is not specified (#11639) * add prompt for ctl close #11514 Signed-off-by: Jayice <1185430411@qq.com> * format code close #11514 Signed-off-by: Jayice <1185430411@qq.com> * format code close #11514 Signed-off-by: Jayice <1185430411@qq.com> * format code close #11514 Signed-off-by: Jayice <1185430411@qq.com> * optimize duplicate codes:pd check close #11514 Signed-off-by: Jayice <1185430411@qq.com> * move pd check in get_pd_rpc_client close #11514 Signed-off-by: Jayice <1185430411@qq.com> * simplify the codes check host|data-dir close #11514 Signed-off-by: Jayice <1185430411@qq.com> * ctl: validate column family name when scanning data using ctl (#11269) * ctl: should check column name validation when scan data using ctl close #11268 Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * fix format problem Signed-off-by: yuqi1129 <yuqi4733@gmail.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com> * txn: Do assertions on data for mutations to reduce potential data inconsistency (#10977) * Add mutation checking logic Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix build Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix usages of mutation Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Update kvproto and renaming Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Add a simple test Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix incorrect assertion Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Support check existance during pessimistic lock Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Add assertion level support Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Add tests Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Add more test cases Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Add test for check_existence of pessimistic_lock request Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix build Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * ref #11542 Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Add a metric for counting different way of checking assertion, for reference about the performance impact Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * metrics: fix memory metrics (#11656) Signed-off-by: Shuaipeng Yu <jackysp@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * coprocesser: refactor encoding and add gbk_chinese_ci map (#11661) * close #11660 Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com> * close #11660 Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * storage: Add API V2 check for RawKV and TxnKV requests (txn part) (#11216) * wip Signed-off-by: pingyu <yuping@pingcap.com> * wip Signed-off-by: pingyu <yuping@pingcap.com> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <yuping@pingcap.com> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <yuping@pingcap.com> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <yuping@pingcap.com> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <yuping@pingcap.com> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <yuping@pingcap.com> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <yuping@pingcap.com> * fix enable_ttl Signed-off-by: andylokandy <andylokandy@hotmail.com> * simplify test Signed-off-by: andylokandy <andylokandy@hotmail.com> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <yuping@pingcap.com> Co-authored-by: andylokandy <andylokandy@hotmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * cdc: load uninlined value more effectively (#11615) Signed-off-by: qupeng <qupeng@pingcap.com> * cdc: load old value with prefix seek if possible (#11643) * close #11642 Signed-off-by: qupeng <qupeng@pingcap.com> * a more comment Signed-off-by: qupeng <qupeng@pingcap.com> * ctl: split into multiple modules (#11658) * ctl: split into multiple modules ref #10938 Signed-off-by: andylokandy <andylokandy@hotmail.com> * refactor if let to match Signed-off-by: andylokandy <andylokandy@hotmail.com> * address comment Signed-off-by: andylokandy <andylokandy@hotmail.com> * address comment Signed-off-by: andylokandy <andylokandy@hotmail.com> * address comment Signed-off-by: andylokandy <andylokandy@hotmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * Fix build issues on non-linux platforms (#11671) * close #11676 Signed-off-by: devillove084 <786537003@qq.com> * all: Migrate from std:: types to primitives. (#11677) * Deprecation is planned for MIN,MAX,EPSLION,INFINITY,NEG_INFINITY of the (i|u|f)(8|16|32|64) types in standard. ref #11678 ```release-note NONE ``` Signed-off-by: Harold Dost <harolddost@gmail.com> * metrics alert: fix apply cpu alert (#11672) Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * resrc_mtr: introduce scanned keys statistics (#11085) close #11690 Signed-off-by: mornyx <mornyx.z@gmail.com> * file_system: support collect the io flow by proc (#11667) * file_system: support collect the read flow by proc Signed-off-by: MuZhou233 <muzhou233@outlook.com> * fix lint Signed-off-by: MuZhou233 <muzhou233@outlook.com> * fix clippy Signed-off-by: MuZhou233 <muzhou233@outlook.com> * Delete unused content & Adjust copyright text Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Add bench_fetch_io_bytes Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * fix lint Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Integrate thread_io Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * record bytes in request Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * fix logical bug Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Update test write_bytes Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Delete unused & Rename func Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Add test for read & Update test for write Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Add buffered thread io Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Replace DashMap with ThreadLocal Signed-off-by: MuZhou233 <muzhou233@outlook.com> * Rename variables and function & Remove unused call Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Change fetch logic. close #10867 Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Add AtomicIOBytes Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Split ThreadIOSentinel Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * add THREAD_IO_TOTAL for metrics bytes fetcher Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Adjust code style & fix Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * improve write_bytes test Signed-off-by: MuZhou233 <muzhou233@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: check epoch and validness of lock table when writing pessimistic locks (#11591) ref #11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * rsmeter: introduce datasink (#11688) * add datasink trait Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * RAII removes on_reporter_closing Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fmt Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix build ref #11691 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * datasink -> data_sink Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fmt Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix build Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * clippy: Fixing up clippy errors. (#11193) * clippy: Fixing up clippy errors. * Fix crate references * Fix anonymous lifetimes references. ref #4301 Signed-off-by: Harold Dost <harolddost@gmail.com> * Test Revert of Iterations ref #4301 Signed-off-by: Harold Dost <harolddost@gmail.com> * clippy: Enable check for rust-2018-idioms ref #4301 Signed-off-by: Harold Dost <harolddost@gmail.com> * rsmeter: add centralized place to construct protobuf data (#11696) ref #11691 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * tikv_util: update procfs to 0.12.0 (#11703) Signed-off-by: linning <linningde25@gmail.com> * Use generic for api version to reduce runtime branching (#11687) * Use generic for api version to reduce runtime branching ref #10938 Signed-off-by: andylokandy <andylokandy@hotmail.com> * fix test Signed-off-by: andylokandy <andylokandy@hotmail.com> * add test Signed-off-by: andylokandy <andylokandy@hotmail.com> * fix clippy Signed-off-by: andylokandy <andylokandy@hotmail.com> * address comment Signed-off-by: andylokandy <andylokandy@hotmail.com> * fmt Signed-off-by: andylokandy <andylokandy@hotmail.com> * address comment Signed-off-by: andylokandy <andylokandy@hotmail.com> * address comment Signed-off-by: andylokandy <andylokandy@hotmail.com> * rename vars Signed-off-by: andylokandy <andylokandy@hotmail.com> * improve vars Signed-off-by: andylokandy <andylokandy@hotmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * sst_importer: check api version when importing (#11664) * sst_importer: check api version when importing ref #10938 Signed-off-by: Peng Guanwen <pg999w@outlook.com> * Fix bugs Signed-off-by: Peng Guanwen <pg999w@outlook.com> * fix bug of Iterator::{scan,scan_cf} Signed-off-by: Peng Guanwen <pg999w@outlook.com> * remove redundant log Signed-off-by: Peng Guanwen <pg999w@outlook.com> * Show the entire key Signed-off-by: Peng Guanwen <pg999w@outlook.com> * Fix lint issue Signed-off-by: Peng Guanwen <pg999w@outlook.com> * resolve conflict Signed-off-by: andylokandy <andylokandy@hotmail.com> Co-authored-by: Andy Lok <andylokandy@hotmail.com> * backup: pipeline scan and save step (#11528) * br: sperate io and scan threads Signed-off-by: Yu Juncen <yu745514916@live.com> * br/stream: make clippy happy Signed-off-by: yujuncen <yujuncen@pingcap.com> * br/stream: fix tests Signed-off-by: Yu Juncen <yu745514916@live.com> * backup: fix a dummy bug Signed-off-by: Yu Juncen <yu745514916@live.com> * backup: add config of io-threads Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: ref #11350: some minior change Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: added some metrics Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: better tuning concurrency, and added a metric Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: added some slow log and metrics Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: address comments Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: fix build Signed-off-by: yujuncen <yujuncen@pingcap.com> * Revert "backup: fix build" This reverts commit 74537be56c410d0f91af2f48b7e65356ab53720e. Signed-off-by: yujuncen <yujuncen@pingcap.com> * Revert "backup: address comments" This reverts commit 77d75756028bea3496f878f0ad8bbc562f5d00aa. Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: address comments Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: always set IO type to Export Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup:make clippy happy Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: address comments Signed-off-by: yujuncen <yujuncen@pingcap.com> * metrics: fix the metrics cannot be displayed (#11710) ref #11662 Signed-off-by: Ryan Leung <rleungx@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * backup: return api-version to BR when backup (#11704) * Update kvproto Signed-off-by: Peng Guanwen <pg999w@outlook.com> * Return api-version for br ref #10938 Signed-off-by: Peng Guanwen <pg999w@outlook.com> * Reformat code Signed-off-by: Peng Guanwen <pg999w@outlook.com> * Update components/external_storage/export/src/export.rs Signed-off-by: Peng Guanwen <pg999w@outlook.com> Co-authored-by: Andy Lok <andylokandy@hotmail.com> * format code Signed-off-by: Peng Guanwen <pg999w@outlook.com> Co-authored-by: Andy Lok <andylokandy@hotmail.com> * file_system: limit thread_io target os (#11715) * limit thread_io target os Signed-off-by: MuZhou233 <muzhou233@outlook.com> * limit thread_io target os. typo. close #11698 Signed-off-by: MuZhou233 <muzhou233@outlook.com> Co-authored-by: Yilin Chen <sticnarf@gmail.com> * file_system: bypass file allocate implementation in fs2 (#11700) Close #10688 Patch fs2 to https://github.com/tabokie/fs2-rs/tree/tikv. In which, `posix_fallocate` is replaced with `fallocate` (https://github.com/danburkert/fs2-rs/pull/42). * bypass file allocate implementation in fs2 Signed-off-by: tabokie <xy.tao@outlook.com> * metrics: remove min legend of some Grafana panels (#11487) Minimum legend isn't very useful, and it occupied valuable visual real estate. Remove some min legends from Grafana metrics, except for the following: - Level 0 chance - Number files at each level - File Count - Entry Count - Allocator Stats - Encrypted files - Encryption meta files size Ref #11119 Signed-off-by: tabokie <xy.tao@outlook.com> * coprocessor: not return rows when there is no input for simple aggregation (#11708) * not return rows when there is no input for simple aggregation Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn> * close #11735, and address comments Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn> * backup: reduce the default thread pool size of backup, enable auto tune by default (#11699) * backup: clamp auto tune values Signed-off-by: yujuncen <yujuncen@pingcap.com> * ref #11000: change the default backup threads. Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: run tasks in background Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: run rustfmt Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: set remain threads to 20% of vcpu Signed-off-by: yujuncen <yujuncen@pingcap.com> * br-stream: fix build Signed-off-by: yujuncen <yujuncen@pingcap.com> * Backup: add S3 metrics && add s3_multi_part_size config (#11666) * s3: add request metrics for s3 storage Signed-off-by: 3pointer <luancheng@pingcap.com> * s3: add grafana json Signed-off-by: 3pointer <luancheng@pingcap.com> * br: add config for s3 multi part upload Signed-off-by: 3pointer <luancheng@pingcap.com> * update comment Signed-off-by: 3pointer <luancheng@pingcap.com> * update comment Signed-off-by: 3pointer <luancheng@pingcap.com> * address comment Signed-off-by: 3pointer <luancheng@pingcap.com> * hidden the new config Signed-off-by: 3pointer <luancheng@pingcap.com> * only hidden config in config-template.toml Signed-off-by: 3pointer <luancheng@pingcap.com> * address comment Signed-off-by: 3pointer <luancheng@pingcap.com> * close #11727 && format Signed-off-by: 3pointer <luancheng@pingcap.com> * fix Signed-off-by: 3pointer <luancheng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * rsmeter: support multiple datasinks (#11707) * rsmeter: support multiple datasinks Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * wrap config notifier Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * add unit tests Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * polish ref #11691 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * address comments Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix test Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix outdated comments Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * engine: properly estimate store size (#11728) * properly estimate store size, ref #11119 Signed-off-by: tabokie <xy.tao@outlook.com> * address comments Signed-off-by: tabokie <xy.tao@outlook.com> * implement global config client for tikv (#11685) * implement global config client close #11686 Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * implement unit test for global config in pd client Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * resolved formatting issues Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * impl pd client asyncly Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * according to reviewer's opinion removing store method implementation Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * Merge branch 'master' of https://github.com/tikv/tikv into global_conf Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * write test case for watch global config when grpc server is closed. Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * resove issues Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * clippy Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * watch global config rentry future in test deadlock just removed Signed-off-by: lemonhx <lemonhx@lemonhx.tech> Co-authored-by: Zhenchi <zhongzc_arch@outlook.com> * storage: skip scanning lock when using rc (#11701) * skip scanning lock when using rc, ref #11485 Signed-off-by: you06 <you1474600@gmail.com> * format code Signed-off-by: you06 <you1474600@gmail.com> * add test Signed-off-by: you06 <you1474600@gmail.com> * address comment Signed-off-by: you06 <you1474600@gmail.com> * add an integration test Signed-off-by: you06 <you1474600@gmail.com> * format code Signed-off-by: you06 <you1474600@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: pushdown substring to tikv (#11494) * corp: pushdown substring to tikv Signed-off-by: guo-shaoge <shaoge1994@163.com> * fix case. close #11495 Signed-off-by: guo-shaoge <shaoge1994@163.com> * fix comment Signed-off-by: guo-shaoge <shaoge1994@163.com> * rsmeter: distinguish between collectors and observers within recorder (#11712) * rsmeter: support multiple datasinks Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * wrap config notifier Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * add unit tests Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * polish ref #11691 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * rsmeter: distinguish between collectors and observers within recorder Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fmt & try to resolve #11689 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * add unit tests ref #11691 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * retrigger test Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * address comments Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix test Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix outdated comments Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * Update components/resource_metering/src/recorder/sub_recorder/mod.rs Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Yexiang Zhang <mornyx.z@gmail.com> * address comment Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * remove enabled Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Co-authored-by: Wenxuan <breezewish@pingcap.com> Co-authored-by: Yexiang Zhang <mornyx.z@gmail.com> * config: validate online configurable thread pools (tikv#11159) (#11714) ref #11159 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * res_meter: upgrade kvproto (#11749) * Upgrade kvproto close #11748 Signed-off-by: mornyx <mornyx.z@gmail.com> * Fix test compilation Signed-off-by: mornyx <mornyx.z@gmail.com> * Fix test compilation Signed-off-by: mornyx <mornyx.z@gmail.com> * Fix test compilation Signed-off-by: mornyx <mornyx.z@gmail.com> * Use master kvproto Signed-off-by: mornyx <mornyx.z@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * rsmeter: add pubsub datasink (#11719) * rsmeter: add pubsub datasink close #11691 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix build Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * Update components/resource_metering/src/reporter/pubsub.rs Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Wenxuan <hi@breeswish.org> * fix build Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Wenxuan <hi@breeswish.org> * status_server: Support online config update for configuration hosting platform (#11693) close #11692 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * res_meter: optimize top k (#11753) * Keep top k on recorder Signed-off-by: mornyx <mornyx.z@gmail.com> * Modify comments Signed-off-by: mornyx <mornyx.z@gmail.com> * Fix pd.rs Signed-off-by: mornyx <mornyx.z@gmail.com> * Refactor Signed-off-by: mornyx <mornyx.z@gmail.com> * Remove import Signed-off-by: mornyx <mornyx.z@gmail.com> * max_resource_groups 200 -> 100 Signed-off-by: mornyx <mornyx.z@gmail.com> * Fix ut Signed-off-by: mornyx <mornyx.z@gmail.com> * Use iterator to avoid clone Signed-off-by: mornyx <mornyx.z@gmail.com> * Optimize close #11754 Signed-off-by: mornyx <mornyx.z@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftclient: delay flush (#11705) * raftclient: delay flush Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * remove raftstore wait Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * Ref #11309. tikv#11310 introduces delay in raftstore thread to reduce the flush rate. It can only reduce flush rate from one thread. If there are many raftstore threads or IO threads, messages can still be flush frequently. This PR reduces the flush rate by introducing delay in raft client. So delay will work at connection level and achieve the maximun batch affect. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * remove useless field Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * address comment Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * fix unstable raw get case Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Liqi Geng <gengliqiii@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * cloud: support azure blob storage as external storage backend (#11732) * support azure storage blob for external storage Signed-off-by: Leavrth <jianjun.liao@outlook.com> * modify the priority of the azure login methods Signed-off-by: Leavrth <jianjun.liao@outlook.com> * fix feature match branch Signed-off-by: Leavrth <jianjun.liao@outlook.com> * implement token update retriable Signed-off-by: Leavrth <jianjun.liao@outlook.com> * fix some problems Signed-off-by: Leavrth <jianjun.liao@outlook.com> * close #11731 Signed-off-by: Leavrth <jianjun.liao@outlook.com> * rustfmt Signed-off-by: Leavrth <jianjun.liao@outlook.com> * make clippy success Signed-off-by: Leavrth <jianjun.liao@outlook.com> * simplify uploader Signed-off-by: Leavrth <jianjun.liao@outlook.com> * commit some suggestions Signed-off-by: Leavrth <jianjun.liao@outlook.com> * coprocesser: update encoding lib (#11506) Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * coprocesser: implement upper and lower function on GBK charset (#11756) Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * logger: support archiving and rotation (#11651) (#11657) * logger: support archiving and rotation (#11651) close #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: add testcases to test dt_from_file_name (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: simplify DateTime format (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * diagnostics: refactor (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: run clippy for integration tests (#11768) * run clippy --fix with nightly-2021-12-17 Signed-off-by: tabokie <xy.tao@outlook.com> * manual clippy Signed-off-by: tabokie <xy.tao@outlook.com> * format and clippy Signed-off-by: tabokie <xy.tao@outlook.com> * enable default features when running clippy, ref #11119 Signed-off-by: tabokie <xy.tao@outlook.com> * fix build Signed-off-by: tabokie <xy.tao@outlook.com> * revert box_collection lint Signed-off-by: tabokie <xy.tao@outlook.com> * Update tests/integrations/raftstore/test_bootstrap.rs Signed-off-by: Xinye Tao <xy.tao@outlook.com> Co-authored-by: Lei Zhao <zlwgx1023@gmail.com> Co-authored-by: Lei Zhao <zlwgx1023@gmail.com> * logger: fix config-template's log unit (#11651) (#11777) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: improve testcase (#11651) (#11778) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * raftstore: propose in-memory pessimistic locks before leader transfer (#11577) * *: check epoch and validness of lock table when writing pessimistic locks ref #11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * raftstore: propose in-memory pessimistic locks before leader transfer ref #11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * mark delete in scheduler and delete after apply Signed-off-by: Yilin Chen <sticnarf@gmail.com> * Add tests for two different orders between proposing locks and write command Signed-off-by: Yilin Chen <sticnarf@gmail.com> * do not clear pessimistic locks before transfer leader Signed-off-by: Yilin Chen <sticnarf@gmail.com> * delete memory locks in the apply thread and add more term and version check Signed-off-by: Yilin Chen <sticnarf@gmail.com> * remove potentially incorrect assert Signed-off-by: Yilin Chen <sticnarf@gmail.com> * check term and version when reading the lock table Signed-off-by: Yilin Chen <sticnarf@gmail.com> * revert a log change Signed-off-by: Yilin Chen <sticnarf@gmail.com> * change term when becoming follower Signed-off-by: Yilin Chen <sticnarf@gmail.com> * fix lint Signed-off-by: Yilin Chen <sticnarf@gmail.com> * do not check version when reading locks (temporarily) Signed-off-by: Yilin Chen <sticnarf@gmail.com> * fix another test Signed-off-by: Yilin Chen <sticnarf@gmail.com> * only treat MsgTransferLeader with context with valid msg Signed-off-by: Yilin Chen <sticnarf@gmail.com> * fix a tiny comment error Signed-off-by: Yilin Chen <sticnarf@gmail.com> * make clippy happy Signed-off-by: Yilin Chen <sticnarf@gmail.com> * res_meter: ignore the read keys test temporarily (#11779) ref #11765 Signed-off-by: mornyx <mornyx.z@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * logger: change critical to fatal (#11651) (#11780) * logger: change critical to fatal (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: fix test_parse_log_level (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * update CHANGELOG.md (#11790) * update from tikv-server v5.0.1(20210423) from https://github.com/tikv/tikv/releases Signed-off-by: xiejiandong <xiejiandong@pingcap.com> * update CHANGELOG.md close #11167 Signed-off-by: xiejiandong <xiejiandong@pingcap.com> * config: relax the bounds of online configurable worker number (#11651) (#11798) close #11776 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: invoke logger_compatible_adjust before run_tikv (#11651) (#11792) close #11789 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: unify thread related native APIs (#11785) * *: unify thread related native APIs Now there are duplicated codes calling syscalls to fetch thread informations. This PR provides unified APIs and remove duplicated codes. Close #11784. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * address comment Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * declare platform compatability Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * Fix method on_tick will not update metric when enable_auto_tune is false (#11791) * Fix method on_tick will not update metric when enable_auto_tune is false ref #11787 Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * resolve format problem in discussion Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * resolve format problem in discussion again Signed-off-by: yuqi1129 <yuqi4733@gmail.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: add a new tick to renew lease in advance (#6427) * renew lease in raft tick Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check lease before sending read index Signed-off-by: 5kbpers <tangminghua@pingcap.com> * renew lease on raft tick to cover lease till next tick Signed-off-by: linning <linningde25@gmail.com> * refactor Signed-off-by: linning <linningde25@gmail.com> * don't use inspect_lease Signed-off-by: linning <linningde25@gmail.com> * fix ci Signed-off-by: linning <linningde25@gmail.com> * add test Signed-off-by: linning <linningde25@gmail.com> * remove unneed comment Signed-off-by: linning <linningde25@gmail.com> * close #5388 Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check pending_read_count before hibernating Signed-off-by: 5kbpers <tangminghua@pingcap.com> * increase the range to check lease expired Signed-off-by: 5kbpers <tangminghua@pingcap.com> * renew leader lease after not hibernate Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix hibernate states metrics Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check writes and pre_read_index before proposing read index Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * use a individual tick to renewing lease Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix condition Signed-off-by: 5kbpers <tangminghua@pingcap.com> * stablize test_node_renew_lease Signed-off-by: 5kbpers <tangminghua@pingcap.com> * cleanup unused changes Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add config item for interval of tick Signed-off-by: 5kbpers <tangminghua@pingcap.com> * rename tick Signed-off-by: 5kbpers <tangminghua@pingcap.com> * disable the tick for test_inconsistent_configuration Signed-off-by: 5kbpers <tangminghua@pingcap.com> * change deafault value of check_leader_lease_interval Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * disable the tick for test_read_hibernated_region Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * count renew lease read index into raft metrics Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * disable the tick for test_renew_lease Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix lint error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add a 100ms buffer for renew_bound Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * make clippy happy Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: linning <linningde25@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * cloud: Add retry for azure blob server busy error (#11813) * add retry for azure blob server busy error Signed-off-by: Leavrth <jianjun.liao@outlook.com> * close#11812 Signed-off-by: Leavrth <jianjun.liao@outlook.com> * use to_string Signed-off-by: Leavrth <jianjun.liao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * tikv_util: make cgroup parsing more robust (#11786) * make cgroup parsing more robust Signed-off-by: tabokie <xy.tao@outlook.com> * fix error message Signed-off-by: tabokie <xy.tao@outlook.com> * parse cpu quota in floats Signed-off-by: tabokie <xy.tao@outlook.com> * *: make it compile on macos (#11825) * *: make it compile on macos close #11823 Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * remove extra comment Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * resolved_ts: add more metrics (#11809) * add metrics Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add slow log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * remove duplicate metric Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add resolved ts metrics for leader peers Signed-off-by: 5kbpers <tangminghua@pingcap.com> * close #11767 Signed-off-by: 5kbpers <tangminghua@pingcap.com> * init endpoint with store meta Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * rsmeter: fix perf regression caused by arc swap (#11833) * rsmeter: fix perf regression caused by arc swap Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * use atomic cell instead Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * debug assert Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Wenxuan <breezewish@pingcap.com> * load base split: more accurate sample for large key ranges batch (#11039) * more accurate sample for large key ranges Signed-off-by: lhy1024 <admin@liudos.us> * update Signed-off-by: lhy1024 <admin@liudos.us> * ref #11521 Signed-off-by: lhy1024 <admin@liudos.us> * address comment Signed-off-by: lhy1024 <admin@liudos.us> * fix test Signed-off-by: lhy1024 <admin@liudos.us> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftclient: enable delay only in high load (#11772) and improve CPU efficiency under high load. But it turns out this can lead to significant regression in normal load. So this PR adds CPU stats and only enabling delay in high load. This is also the same strategy used in v4.x. Close #11769. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * tikv-ctl: fix tikv-ctl's output is incomplete on calling process::exit (#11231) Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com> Co-authored-by: Connor <zbk602423539@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: skip deleting snapshot files in peer pending_remove is true (#11782) * raftstore: skip deleting snapshot files in peer when the peer is pending removal and the snapshot is being applied and canceled -- close #11746 This is to avoid the potential panic when the snapshot files are deleted, but the peer's status (Tombstone) is not persisted in disk due to tikv crash. Signed-off-by: tonyxuqqi <tonyxuqi@outlook.com> * address code review feedback -- close #11746 Signed-off-by: qi.xu <tonxuqi@outlook.com> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <tonxuqi@outlook.com> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <tonxuqi@outlook.com> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <tonxuqi@outlook.com> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <tonxuqi@outlook.com> Co-authored-by: qi.xu <tonxuqi@outlook.com> * load_statis: fix compilation on macos (#11851) close #11772 Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * config: override rocksdb config when flow control enabled (#11840) Signed-off-by: Connor1996 <zbk602423539@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * rocksdb: consider env sharing when configuring background threads (#11760) #9649 increased the default setting of kvdb background flushes (3 for 8c and 4 for 16c). This PR takes another approach: use smaller concurrency for individual dbs, but configure the global thread pool size with the sum of kvdb and raftdb concurrency. This way we won't over-configure flush workers when raft-engine is enabled. * consider env sharing when configuring background threads ref #11119 Signed-off-by: tabokie <xy.tao@outlook.com> * server/config: keep compatible using option (#11862) * server/config: keep compatible using option Using `Option` for `heavy_load_wait_duration` to keep compatible with versions prior to v5.3. Close #11861. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * fix format Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * fix clippy Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: zhouqiang <zhouqiang.cl@gmail.com> * raftstore: fix missing workers' stop on shutdown (#11864) ref #11159 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: move in-memory pessimistic locks to split regions (#11655) ref #11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: Remove the entrance of enabling io snooper. (#11394) * *: Remove the entrance of enabling io snooper. close #10867 Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * fix Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * fix test Signed-off-by: MuZhou233 <muzhou233@outlook.com> * fix test Signed-off-by: MuZhou233 <muzhou233@outlook.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: change PeerTicks to enum (#11849) * raftstore: change PeerTicks to enum close #11848 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * use bool array for registry Signed-off-by: Yilin Chen <sticnarf@gmail.com> * rename PeerTicks to PeerTick Signed-off-by: Yilin Chen <sticnarf@gmail.com> * add associated const for number of PeerTick types Signed-off-by: Yilin Chen <sticnarf@gmail.com> * change naming Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: check memory limit when inserting in-memory pessimistic locks (#11841) * *: check memory limit when inserting in-memory pessimistic locks ref #11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * fix merged upstream Signed-off-by: Yilin Chen <sticnarf@gmail.com> * fix test Signed-off-by: Yilin Chen <sticnarf@gmail.com> * fix typo Signed-off-by: Yilin Chen <sticnarf@gmail.com> * *: fix invalid failpoint caused by typo (#11709) * fix invalid failpoint caused by typo close #11734 Signed-off-by: Ryan Leung <rleungx@gmail.com> * address the comment Signed-off-by: Ryan Leung <rleungx@gmail.com> * address the comment Signed-off-by: Ryan Leung <rleungx@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: Extract significant router (#11750) * ref#11409 add significant router Signed-off-by: Connor1996 <zbk602423539@gmail.com> * make format Signed-off-by: Connor1996 <zbk602423539@gmail.com> * address comment Signed-off-by: Connor1996 <zbk602423539@gmail.com> * raftstore: move scan delete to raft log gc worker (#11853) * raftstore: move scan delete to raft log gc worker When clearing raft metas, raftstore will scan raft logs and delete them one by one. Seeking can be slow if there are a lot of tombstone keys. This PR moves the operation to raft log gc worker to reduce the impact. The final solution should be also moving remaining IO operations to async write IO threads. Close #10210. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * address comment Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * speed up destroy Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * fix compile Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * further speed up Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * revert test case configuration Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * address comment Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * address comment Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * doc: update new rules for linking issue and commit message (#11832) close #11097, close #11831 Signed-off-by: zhangyangyu <angwerzx@126.com> Co-authored-by: Mini256 <minianter@foxmail.com> Co-authored-by: Mini256 <minianter@foxmail.com> Co-authored-by: Xiaoguang Sun <sunxiaoguang@users.noreply.github.com> * *: update rust toolchain to 2022-01-07 (#11875) * update rust toolchain to 2022-01-17 Signed-off-by: tabokie <xy.tao@outlook.com> * address comment: clean up unnecessary `as_ref()` Signed-off-by: tabokie <xy.tao@outlook.com> * try fixing the underflow Signed-off-by: tabokie <xy.tao@outlook.com> * mute underflow warning for raft metrics Signed-off-by: tabokie <xy.tao@outlook.com> * clean up unused data members Signed-off-by: tabokie <xy.tao@outlook.com> * format Signed-off-by: tabokie <xy.tao@outlook.com> * step back to 2022-01-07 Signed-off-by: tabokie <xy.tao@outlook.com> * readme: change images based on github theme (#11795) Signed-off-by: sloorush <aarush.bhatt@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: don't remove other peer's read delegate (#11882) * don't remove read delegate besides peer_destroy Signed-off-by: linning <linningde25@gmail.com> * add test case Signed-off-by: linning <linningde25@gmail.com> * make clippy happy Signed-off-by: linning <linningde25@gmail.com> * address comment Signed-off-by: linning <linningde25@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * logger: use eprintln! if the logger is not initialized (#11869) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: fix the calculation of total column size in analyze (#11884) Signed-off-by: xuyifan <xuyifangreeneyes@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * engine: update raft-engine for data consistency fix (#11885) Fix https://github.com/tikv/raft-engine/issues/142. When encountering this bug, TiKV will panic with message "applied index > max(commit index, recorded commit index)" after restart. * update raft-engine Signed-off-by: tabokie <xy.tao@outlook.com> * update one more commit Signed-off-by: tabokie <xy.tao@outlook.com> * check docker build during clippy, ref #11312 (#11819) Signed-off-by: tabokie <xy.tao@outlook.com> * gc_worker: fix incorrect scheduled_tasks counting (#11904) * gc_worker: fix incorrect scheduled_tasks counting close #11903 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * remove check_is_busy totally Signed-off-by: Yilin Chen <sticnarf@gmail.com> * do not use wildcard match in error handling Signed-off-by: Yilin Chen <sticnarf@gmail.com> * raftstore: Introduce raft log fetcher (#11900) * ref#11409 introduce raft log fetcher Signed-off-by: Connor1996 <zbk602423539@gmail.com> * update kvproto Signed-off-by: Connor1996 <zbk602423539@gmail.com> * address comment Signed-off-by: Connor1996 <zbk602423539@gmail.com> * fix test build Signed-off-by: Connor1996 <zbk602423539@gmail.com> * fix clippy Signed-off-by: Connor1996 <zbk602423539@gmail.com> * limit capacity Signed-off-by: Connor1996 <zbk602423539@gmail.com> * update kvproto Signed-off-by: Connor1996 <zbk602423539@gmail.com> * call stop on worker Signed-off-by: Connor1996 <zbk602423539@gmail.com> * rename worker Signed-off-by: Connor1996 <zbk602423539@gmail.com> * status_server: add pprof flamegraph response header (#10951) * tweak(status_server): add pprof flamegraph response header Signed-off-by: Suhaha <jklopsdfw@gmail.com> * test(status_server): add Content-Type asset to test_pprof_profile_service Signed-off-by: Suhaha <jklopsdfw@gmail.com> * tweak(status_server): add pprof flamegraph response header Signed-off-by: Suhaha <jklopsdfw@gmail.com> * tweak(status_server): add pprof flamegraph response header close #11917 Signed-off-by: Suhaha <jklopsdfw@gmail.com> * close #11917 Signed-off-by: Suhaha <jklopsdfw@gmail.com> Co-authored-by: goroutine <ngaut@users.noreply.github.com> Co-authored-by: Connor <zbk602423539@gmail.com> * raftstore: renew leader lease in advance when handle read request (#9307) * renew lease advance Signed-off-by: linning <linningde25@gmail.com> * add log Signed-off-by: linning <linningde25@gmail.com> * make clippy happy Signed-off-by: linning <linningde25@gmail.com> * ref #11579 Signed-off-by: 5kbpers <tangminghua@pingcap.com> * reset raft tick Signed-off-by: 5kbpers <tangminghua@pingcap.com> * set has ready Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add renew_leader_lease_advance_duration config Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix panic Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * disable renewing for test_lease_unsafe_during_leader_transfers Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: 5kbpers <tangminghua@pingcap.com> * pprof: ignore cpu profiling on non-x86 arch (#11925) * Disable cpu profiling on non-x86_64 arch Signed-off-by: mornyx <mornyx.z@gmail.com> * Fix warns Signed-off-by: mornyx <mornyx.z@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * github: add new pr requirement for linking issue and commit message (#11887) Signed-off-by: zhangyangyu <angwerzx@126.com> Co-authored-by: Xiaoguang Sun <sunxiaoguang@users.noreply.github.com> * engine: enable raft engine by default (#11928) * enable raft engine by default and synchronize docs Signed-off-by: tabokie <xy.tao@outlook.com> * update raft engine Signed-off-by: tabokie <xy.tao@outlook.com> * update raft-engine and fix tests Signed-off-by: tabokie <xy.tao@outlook.com> * gc_worker: Limit the key range to scan for GcKeys tasks (#11922) close tikv/tikv#11752, close tikv/tikv#11902 Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: TransferLeader support multiple target peers (#11063) ref tikv/tikv#822, ref tikv/tikv#4214, close tikv/tikv#10602 Signed-off-by: MrCroxx <mrcroxx@outlook.com> * raftstore: remove the leaders field of `StoreMeta` (#11934) close tikv/tikv#11933 Remove `StoreMeta.leaders` ``` ### Related changes ### Check List <!--REMOVE the items that are not applicable--> Tests <!-- At least one of them must be included. --> - Manual test (add detailed scripts or steps below) `cargo check` ### Release note <!-- bugfixes or new feature need a release note --> ```release-note None Signed-off-by: linning <linningde25@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: Fetch raft log in async manner (#11409) close tikv/tikv#10408, close tikv/tikv#11320 Signed-off-by: Connor1996 <zbk602423539@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: add quarter function (#11935) ref tikv/tikv#5751 Signed-off-by: zhongyong jin <AT-Fieldless@outlook.com> * Enable full debug info for dev and test (#11949) ref tikv/tikv#5049, ref tikv/tikv#5572, close tikv/tikv#5572 ``` ### Related changes - PR to update `pingcap/docs`/`pingcap/docs-cn`: - PR to update `pingcap/tidb-ansible`: - Need to cherry-pick to the release branch --> ### Check List <!--REMOVE the items that are not applicable--> Tests <!-- At least one of them must be included. --> - Manual test (add detailed scripts or steps below) - No code Manual test: Linux box with 8 cores (3:50 -> 4:00): ``` # Before this PR: $ make clean cargo clean rm -rf bin dist $ time make build cargo build --no-default-features --features "jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 ... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 3m 50s real 3m50,487s user 38m11,859s sys 3m9,540s # After this PR: $ git diff diff --git a/Cargo.toml b/Cargo.toml index 71f5329d3..67cb9d183 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -267,7 +267,7 @@ default-members = ["cmd/tikv-server", "cmd/tikv-ctl"] [profile.dev] opt-level = 0 -debug = 1 # required for line numbers in tests, see tikv #5049 +debug = true codegen-units = 4 lto = false incremental = true @@ -293,7 +293,7 @@ codegen-units = 4 [profile.test] opt-level = 0 -debug = 1 # enable line numbers by default for easy test debugging +debug = true codegen-units = 16 lto = false incremental = true mattias@msig:~/repos/tikv$ time make clean cargo clean rm -rf bin dist real 0m0,975s user 0m0,148s sys 0m0,828s mattias@msig:~/repos/tikv$ time make build cargo build --no-default-features --features "jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 ... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 4m 00s real 4m0,201s user 39m45,037s sys 3m16,397s ``` Macbook Air M1: ``` # Before: % time make build cargo build --no-default-features --features " jemalloc test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 .... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 4m 01s make build 1107.42s user 116.20s system 506% cpu 4:01.46 total # After: % time make build cargo build --no-default-features --features " jemalloc test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 .... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 4m 10s make build 1179.39s user 120.74s system 518% cpu 4:10.98 total ``` Side effects - Performance regression, Only when building, to the benefit of full debug info by default. ### Release note <!-- bugfixes or new feature need a release note --> ```release-note None Signed-off-by: Mattias Jonsson <mjonss@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * correct a metric about compaction filter (#11938) correct a metric about compaction filter Signed-off-by: qupeng <qupeng@pingcap.com> * raft: Fix possible panic on entries fetched callback (#11961) close tikv/tikv#11951 Signed-off-by: Connor1996 <zbk602423539@gmail.com> * split_controller: refine the sample function and add some comments (#11952) Signed-off-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * metrics: fix grid position in TiKV Details (#11936) ref tikv/tikv#11119 Fix grid positions in TiKV Details dashboard. Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * cdc: capture old value from txn layer dynamically (#11896) close tikv/tikv#10091 Signed-off-by: hi-rustin <rustin.liu@gmail.com> Signed-off-by: qupeng <qupeng@pingcap.com> Co-authored-by: hi-rustin <rustin.liu@gmail.com> * split_controller: reorganize the structs and add more comments (#11967) Reorganize the structs and add more comments. Signed-off-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: support Bit column push downBit column (#11968) close tikv/tikv#11893, ref tikv/tikv#11893, ref pingcap/tidb#31884 Signed-off-by: xiejiandong <xiejiandong@pingcap.com> * copr: support push down mod, sysdate to TiKV (#11970) close tikv/tikv#11916, ref tikv/tikv#11916 Signed-off-by: xiejiandong <xiejiandong@pingcap.com> * raftstore: propose in-memory pessimistic locks before prepare merge (#11758) ref tikv/tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * stats_monitor: reformat stats_monitor tick (#11972) Signed-off-by: lhy1024 <admin@liudos.us> * copr: support extract scalar value from bit (#11980) close tikv/tikv#11893 Signed-off-by: yisaer <disxiaofei@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * diagnosis: support get cpu time of threads for macOS (#11978) close tikv/tikv#11977 Add the variant for macOS into `components/tikv_util/src/sys/thread.rs` Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Co-authored-by: Jay <BusyJay@users.noreply.github.com> * rust: update toolchain to fix missing rls (#11954) close tikv/tikv#11953 Signed-off-by: Connor1996 <zbk602423539@gmail.com> Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Co-authored-by: Yilin Chen <sticnarf@gmail.com> * cdc: separate resolved region outliers (#11991) Separate broadcasing outlier regions and normal regions, so 1) downstreams know where they should send resolve lock requests, and 2) resolved ts of normal regions does not fallback. close pingcap/tiflow#4516 close pingcap/tiflow#4311 ref pingcap/tiflow#4146 Signed-off-by: Neil Shen <overvenus@gmail.com> * *: collect key ranges for the read request in ResourceMeteringTag (#11995) close tikv/tikv#11988 *: collect key ranges for the read request in ResourceMeteringTag Signed-off-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * split_controller: move sample_threshold check out of the split_key iteration (#11986) split_controller: move sample_threshold check out of the split_key iteration Signed-off-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: support insertutf8 and lowerutf8 (#11987) ref tikv/tikv#5751 Signed-off-by: yisaer <disxiaofei@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: support rest greatest/least functions (#12003) ref tikv/tikv#5751 Signed-off-by: yisaer <disxiaofei@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: force compact with gentleness (#12006) * force compact with gentleness Signed-off-by: tabokie <xy.tao@outlook.com> * address comment Signed-off-by: tabokie <xy.tao@outlook.com> * split_controller: refine the LOAD_BASE_SPLIT_EVENT metrics label definitions (#12010) split_controller: refine the LOAD_BASE_SPLIT_EVENT metrics label definitions Signed-off-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * split_controller: fix the incorrect sampled key ranges number check (#12013) close tikv/tikv#12012 split_controller: fix the incorrect sampled key ranges number check Signed-off-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: Support adjusting max_inflight_msgs dynamically (#11866) close tikv/tikv#11865 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: fix missing flag solvement for bit column in decoding (#12016) ref tikv/tikv#7, close pingcap/tidb#32506 Signed-off-by: yisaer <disxiaofei@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * file_system: overhaul proc-based io tracing utilities (#11873) ref tikv/tikv#10867, fix tikv/tikv#11775 Signed-off-by: tabokie <xy.tao@outlook.com> * raftstore: Wait for the apply index equals to the commit index (#11716) ref tikv/tikv#10483 Signed-off-by: v01dstar <yang.zhang@pingcap.com> * copr: revert 11968/11980/12016 (#12030) ref tikv/tikv#11968, ref tikv/tikv#11980, ref tikv/tikv#12016, ref pingcap/tidb#32506 Signed-off-by: yisaer <disxiaofei@163.com> * build: fix a Mac build issue (#12027) ref tikv/tikv#10867, ref tikv/tikv#11873 Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: Extra physical table id column (#11931) close tikv/tikv#11888 Added EXTRA_PHYSICAL_TABLE_ID_COL_ID support, to support TiDB's table partition dynamic prune mode, where a single request includes multiple partitions, but when Pessimistic lock (like `SELECT FOR UPDATE`) or ongoing transaction (having something in the tidb session local transaction buffer) each row needs its partition id / physical table ID to either lock that row or to check against the transaction buffer. Signed-off-by: Mattias Jonsson <mjonss@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: hotfix panic from tokio-timer (#12004) close tikv/tikv#11940, ref tikv/tikv#11940 *: hotfix panic from tokio-timer by enlarging the length of the level vector Signed-off-by: you06 <you1474600@gmail.com> * build: add arm64 check on SSE in makefile (#12035) close tikv/tikv#12034 Signed-off-by: Jin Dong <djdongjin95@gmail.com> Co-authored-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * replication mode: sync state to pd (#11751) Signed-off-by: disksing <i@disksing.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: revert pessimistic locks status when failing to propose PrepareMerge (#11985) ref tikv/tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * copr: support push bit column down (#12037) ref pingcap/tidb#30738 Signed-off-by: yisaer <disxiaofei@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * split_controller: add more sample function test cases (#12058) split_controller: add more sample function test cases Signed-off-by: JmPotato <ghzpotato@gmail.com> * copr: fix greatest/least time/date args type (#12056) Signed-off-by: yisaer <disxiaofei@163.com> * cdc: advancing resolved ts correctly for clusters with tiflash (#12050) close pingcap/tiflow#4461 cdc: advancing resolved ts correctly for clusters with tiflash Signed-off-by: qupeng <qupeng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: bump master branch version to v6.0.0-alpha (#12077) close tikv/tikv#12075 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * raftstore: check uninitialized destroy for merge (#12055) close tikv/tikv#12048 When a peer is destroyed without being initialized, it will store itself to peer list and the region epoch is missing. In merge if such state is detected, it should abort merging instead of panicking. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: Add TruncateManager and TruncateWorker (#11553) ref tikv/rfcs#81 Signed-off-by: longfangsong <longfangsong@icloud.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * TiKV supports buckets (#11763) ref tikv/tikv#11759 Signed-off-by: qi.xu <tonxuqi@outlook.com> Co-authored-by: qi.xu <tonxuqi@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * api_version: Codec for RawKV key (#12036) ref tikv/tikv#11965 Key of RawKV is encoded as `user-key + memcomparable-padding + timestamp` in API v2. Signed-off-by: pingyu <yuping@pingcap.com> Co-authored-by: Andy Lok <andylokandy@hotmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: fix some typos (#12066) Signed-off-by: cuishuang <imcusg@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: fix stale message cause panic (#12054) close tikv/tikv#12023 raftstore: fix stale message cause panic Signed-off-by: linning <linningde25@gmail.com> * raftstore: ignore async fetch result when the peer is pending removed (#12038) close tikv/tikv#11973, close tikv/tikv#12026 Signed-off-by: Connor1996 <zbk602423539@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: remove part about wechat in doc (#12101) close tikv/tikv#12100 Signed-off-by: jackwener <jakevingoo@gmail.com> * engine: upgrade raft engine (#12095) ref tikv/tikv#165, ref tikv/raft-engine#165, ref tikv/tikv#11119 Signed-off-by: Randy <ztelur@gmail.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com> * *: Fix possible undefined behavior for transmuting vec (#12096) close tikv/tikv#12070 Signed-off-by: Connor1996 <zbk602423539@gm…
CalvinNeo
added a commit
to pingcap/tidb-engine-ext
that referenced
this pull request
May 30, 2022
* storage: Add API V2 check for RawKV and TxnKV requests (txn part) (#11216) * wip Signed-off-by: pingyu <yuping@pingcap.com> * wip Signed-off-by: pingyu <yuping@pingcap.com> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <yuping@pingcap.com> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <yuping@pingcap.com> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <yuping@pingcap.com> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <yuping@pingcap.com> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <yuping@pingcap.com> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <yuping@pingcap.com> * fix enable_ttl Signed-off-by: andylokandy <andylokandy@hotmail.com> * simplify test Signed-off-by: andylokandy <andylokandy@hotmail.com> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <yuping@pingcap.com> Co-authored-by: andylokandy <andylokandy@hotmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * cdc: load uninlined value more effectively (#11615) Signed-off-by: qupeng <qupeng@pingcap.com> * cdc: load old value with prefix seek if possible (#11643) * close #11642 Signed-off-by: qupeng <qupeng@pingcap.com> * a more comment Signed-off-by: qupeng <qupeng@pingcap.com> * ctl: split into multiple modules (#11658) * ctl: split into multiple modules ref #10938 Signed-off-by: andylokandy <andylokandy@hotmail.com> * refactor if let to match Signed-off-by: andylokandy <andylokandy@hotmail.com> * address comment Signed-off-by: andylokandy <andylokandy@hotmail.com> * address comment Signed-off-by: andylokandy <andylokandy@hotmail.com> * address comment Signed-off-by: andylokandy <andylokandy@hotmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * Fix build issues on non-linux platforms (#11671) * close #11676 Signed-off-by: devillove084 <786537003@qq.com> * all: Migrate from std:: types to primitives. (#11677) * Deprecation is planned for MIN,MAX,EPSLION,INFINITY,NEG_INFINITY of the (i|u|f)(8|16|32|64) types in standard. ref #11678 ```release-note NONE ``` Signed-off-by: Harold Dost <harolddost@gmail.com> * metrics alert: fix apply cpu alert (#11672) Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * resrc_mtr: introduce scanned keys statistics (#11085) close #11690 Signed-off-by: mornyx <mornyx.z@gmail.com> * file_system: support collect the io flow by proc (#11667) * file_system: support collect the read flow by proc Signed-off-by: MuZhou233 <muzhou233@outlook.com> * fix lint Signed-off-by: MuZhou233 <muzhou233@outlook.com> * fix clippy Signed-off-by: MuZhou233 <muzhou233@outlook.com> * Delete unused content & Adjust copyright text Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Add bench_fetch_io_bytes Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * fix lint Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Integrate thread_io Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * record bytes in request Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * fix logical bug Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Update test write_bytes Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Delete unused & Rename func Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Add test for read & Update test for write Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Add buffered thread io Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Replace DashMap with ThreadLocal Signed-off-by: MuZhou233 <muzhou233@outlook.com> * Rename variables and function & Remove unused call Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Change fetch logic. close #10867 Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Add AtomicIOBytes Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Split ThreadIOSentinel Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * add THREAD_IO_TOTAL for metrics bytes fetcher Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Adjust code style & fix Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * improve write_bytes test Signed-off-by: MuZhou233 <muzhou233@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: check epoch and validness of lock table when writing pessimistic locks (#11591) ref #11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * rsmeter: introduce datasink (#11688) * add datasink trait Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * RAII removes on_reporter_closing Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fmt Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix build ref #11691 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * datasink -> data_sink Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fmt Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix build Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * clippy: Fixing up clippy errors. (#11193) * clippy: Fixing up clippy errors. * Fix crate references * Fix anonymous lifetimes references. ref #4301 Signed-off-by: Harold Dost <harolddost@gmail.com> * Test Revert of Iterations ref #4301 Signed-off-by: Harold Dost <harolddost@gmail.com> * clippy: Enable check for rust-2018-idioms ref #4301 Signed-off-by: Harold Dost <harolddost@gmail.com> * rsmeter: add centralized place to construct protobuf data (#11696) ref #11691 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * tikv_util: update procfs to 0.12.0 (#11703) Signed-off-by: linning <linningde25@gmail.com> * Use generic for api version to reduce runtime branching (#11687) * Use generic for api version to reduce runtime branching ref #10938 Signed-off-by: andylokandy <andylokandy@hotmail.com> * fix test Signed-off-by: andylokandy <andylokandy@hotmail.com> * add test Signed-off-by: andylokandy <andylokandy@hotmail.com> * fix clippy Signed-off-by: andylokandy <andylokandy@hotmail.com> * address comment Signed-off-by: andylokandy <andylokandy@hotmail.com> * fmt Signed-off-by: andylokandy <andylokandy@hotmail.com> * address comment Signed-off-by: andylokandy <andylokandy@hotmail.com> * address comment Signed-off-by: andylokandy <andylokandy@hotmail.com> * rename vars Signed-off-by: andylokandy <andylokandy@hotmail.com> * improve vars Signed-off-by: andylokandy <andylokandy@hotmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * sst_importer: check api version when importing (#11664) * sst_importer: check api version when importing ref #10938 Signed-off-by: Peng Guanwen <pg999w@outlook.com> * Fix bugs Signed-off-by: Peng Guanwen <pg999w@outlook.com> * fix bug of Iterator::{scan,scan_cf} Signed-off-by: Peng Guanwen <pg999w@outlook.com> * remove redundant log Signed-off-by: Peng Guanwen <pg999w@outlook.com> * Show the entire key Signed-off-by: Peng Guanwen <pg999w@outlook.com> * Fix lint issue Signed-off-by: Peng Guanwen <pg999w@outlook.com> * resolve conflict Signed-off-by: andylokandy <andylokandy@hotmail.com> Co-authored-by: Andy Lok <andylokandy@hotmail.com> * backup: pipeline scan and save step (#11528) * br: sperate io and scan threads Signed-off-by: Yu Juncen <yu745514916@live.com> * br/stream: make clippy happy Signed-off-by: yujuncen <yujuncen@pingcap.com> * br/stream: fix tests Signed-off-by: Yu Juncen <yu745514916@live.com> * backup: fix a dummy bug Signed-off-by: Yu Juncen <yu745514916@live.com> * backup: add config of io-threads Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: ref #11350: some minior change Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: added some metrics Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: better tuning concurrency, and added a metric Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: added some slow log and metrics Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: address comments Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: fix build Signed-off-by: yujuncen <yujuncen@pingcap.com> * Revert "backup: fix build" This reverts commit 74537be56c410d0f91af2f48b7e65356ab53720e. Signed-off-by: yujuncen <yujuncen@pingcap.com> * Revert "backup: address comments" This reverts commit 77d75756028bea3496f878f0ad8bbc562f5d00aa. Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: address comments Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: always set IO type to Export Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup:make clippy happy Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: address comments Signed-off-by: yujuncen <yujuncen@pingcap.com> * metrics: fix the metrics cannot be displayed (#11710) ref #11662 Signed-off-by: Ryan Leung <rleungx@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * backup: return api-version to BR when backup (#11704) * Update kvproto Signed-off-by: Peng Guanwen <pg999w@outlook.com> * Return api-version for br ref #10938 Signed-off-by: Peng Guanwen <pg999w@outlook.com> * Reformat code Signed-off-by: Peng Guanwen <pg999w@outlook.com> * Update components/external_storage/export/src/export.rs Signed-off-by: Peng Guanwen <pg999w@outlook.com> Co-authored-by: Andy Lok <andylokandy@hotmail.com> * format code Signed-off-by: Peng Guanwen <pg999w@outlook.com> Co-authored-by: Andy Lok <andylokandy@hotmail.com> * file_system: limit thread_io target os (#11715) * limit thread_io target os Signed-off-by: MuZhou233 <muzhou233@outlook.com> * limit thread_io target os. typo. close #11698 Signed-off-by: MuZhou233 <muzhou233@outlook.com> Co-authored-by: Yilin Chen <sticnarf@gmail.com> * file_system: bypass file allocate implementation in fs2 (#11700) Close #10688 Patch fs2 to https://github.com/tabokie/fs2-rs/tree/tikv. In which, `posix_fallocate` is replaced with `fallocate` (https://github.com/danburkert/fs2-rs/pull/42). * bypass file allocate implementation in fs2 Signed-off-by: tabokie <xy.tao@outlook.com> * metrics: remove min legend of some Grafana panels (#11487) Minimum legend isn't very useful, and it occupied valuable visual real estate. Remove some min legends from Grafana metrics, except for the following: - Level 0 chance - Number files at each level - File Count - Entry Count - Allocator Stats - Encrypted files - Encryption meta files size Ref #11119 Signed-off-by: tabokie <xy.tao@outlook.com> * coprocessor: not return rows when there is no input for simple aggregation (#11708) * not return rows when there is no input for simple aggregation Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn> * close #11735, and address comments Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn> * backup: reduce the default thread pool size of backup, enable auto tune by default (#11699) * backup: clamp auto tune values Signed-off-by: yujuncen <yujuncen@pingcap.com> * ref #11000: change the default backup threads. Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: run tasks in background Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: run rustfmt Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: set remain threads to 20% of vcpu Signed-off-by: yujuncen <yujuncen@pingcap.com> * br-stream: fix build Signed-off-by: yujuncen <yujuncen@pingcap.com> * Backup: add S3 metrics && add s3_multi_part_size config (#11666) * s3: add request metrics for s3 storage Signed-off-by: 3pointer <luancheng@pingcap.com> * s3: add grafana json Signed-off-by: 3pointer <luancheng@pingcap.com> * br: add config for s3 multi part upload Signed-off-by: 3pointer <luancheng@pingcap.com> * update comment Signed-off-by: 3pointer <luancheng@pingcap.com> * update comment Signed-off-by: 3pointer <luancheng@pingcap.com> * address comment Signed-off-by: 3pointer <luancheng@pingcap.com> * hidden the new config Signed-off-by: 3pointer <luancheng@pingcap.com> * only hidden config in config-template.toml Signed-off-by: 3pointer <luancheng@pingcap.com> * address comment Signed-off-by: 3pointer <luancheng@pingcap.com> * close #11727 && format Signed-off-by: 3pointer <luancheng@pingcap.com> * fix Signed-off-by: 3pointer <luancheng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * rsmeter: support multiple datasinks (#11707) * rsmeter: support multiple datasinks Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * wrap config notifier Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * add unit tests Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * polish ref #11691 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * address comments Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix test Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix outdated comments Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * engine: properly estimate store size (#11728) * properly estimate store size, ref #11119 Signed-off-by: tabokie <xy.tao@outlook.com> * address comments Signed-off-by: tabokie <xy.tao@outlook.com> * implement global config client for tikv (#11685) * implement global config client close #11686 Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * implement unit test for global config in pd client Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * resolved formatting issues Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * impl pd client asyncly Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * according to reviewer's opinion removing store method implementation Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * Merge branch 'master' of https://github.com/tikv/tikv into global_conf Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * write test case for watch global config when grpc server is closed. Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * resove issues Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * clippy Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * watch global config rentry future in test deadlock just removed Signed-off-by: lemonhx <lemonhx@lemonhx.tech> Co-authored-by: Zhenchi <zhongzc_arch@outlook.com> * storage: skip scanning lock when using rc (#11701) * skip scanning lock when using rc, ref #11485 Signed-off-by: you06 <you1474600@gmail.com> * format code Signed-off-by: you06 <you1474600@gmail.com> * add test Signed-off-by: you06 <you1474600@gmail.com> * address comment Signed-off-by: you06 <you1474600@gmail.com> * add an integration test Signed-off-by: you06 <you1474600@gmail.com> * format code Signed-off-by: you06 <you1474600@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: pushdown substring to tikv (#11494) * corp: pushdown substring to tikv Signed-off-by: guo-shaoge <shaoge1994@163.com> * fix case. close #11495 Signed-off-by: guo-shaoge <shaoge1994@163.com> * fix comment Signed-off-by: guo-shaoge <shaoge1994@163.com> * rsmeter: distinguish between collectors and observers within recorder (#11712) * rsmeter: support multiple datasinks Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * wrap config notifier Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * add unit tests Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * polish ref #11691 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * rsmeter: distinguish between collectors and observers within recorder Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fmt & try to resolve #11689 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * add unit tests ref #11691 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * retrigger test Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * address comments Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix test Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix outdated comments Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * Update components/resource_metering/src/recorder/sub_recorder/mod.rs Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Yexiang Zhang <mornyx.z@gmail.com> * address comment Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * remove enabled Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Co-authored-by: Wenxuan <breezewish@pingcap.com> Co-authored-by: Yexiang Zhang <mornyx.z@gmail.com> * config: validate online configurable thread pools (tikv#11159) (#11714) ref #11159 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * res_meter: upgrade kvproto (#11749) * Upgrade kvproto close #11748 Signed-off-by: mornyx <mornyx.z@gmail.com> * Fix test compilation Signed-off-by: mornyx <mornyx.z@gmail.com> * Fix test compilation Signed-off-by: mornyx <mornyx.z@gmail.com> * Fix test compilation Signed-off-by: mornyx <mornyx.z@gmail.com> * Use master kvproto Signed-off-by: mornyx <mornyx.z@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * rsmeter: add pubsub datasink (#11719) * rsmeter: add pubsub datasink close #11691 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix build Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * Update components/resource_metering/src/reporter/pubsub.rs Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Wenxuan <hi@breeswish.org> * fix build Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Wenxuan <hi@breeswish.org> * status_server: Support online config update for configuration hosting platform (#11693) close #11692 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * res_meter: optimize top k (#11753) * Keep top k on recorder Signed-off-by: mornyx <mornyx.z@gmail.com> * Modify comments Signed-off-by: mornyx <mornyx.z@gmail.com> * Fix pd.rs Signed-off-by: mornyx <mornyx.z@gmail.com> * Refactor Signed-off-by: mornyx <mornyx.z@gmail.com> * Remove import Signed-off-by: mornyx <mornyx.z@gmail.com> * max_resource_groups 200 -> 100 Signed-off-by: mornyx <mornyx.z@gmail.com> * Fix ut Signed-off-by: mornyx <mornyx.z@gmail.com> * Use iterator to avoid clone Signed-off-by: mornyx <mornyx.z@gmail.com> * Optimize close #11754 Signed-off-by: mornyx <mornyx.z@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftclient: delay flush (#11705) * raftclient: delay flush Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * remove raftstore wait Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * Ref #11309. tikv#11310 introduces delay in raftstore thread to reduce the flush rate. It can only reduce flush rate from one thread. If there are many raftstore threads or IO threads, messages can still be flush frequently. This PR reduces the flush rate by introducing delay in raft client. So delay will work at connection level and achieve the maximun batch affect. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * remove useless field Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * address comment Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * fix unstable raw get case Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Liqi Geng <gengliqiii@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * cloud: support azure blob storage as external storage backend (#11732) * support azure storage blob for external storage Signed-off-by: Leavrth <jianjun.liao@outlook.com> * modify the priority of the azure login methods Signed-off-by: Leavrth <jianjun.liao@outlook.com> * fix feature match branch Signed-off-by: Leavrth <jianjun.liao@outlook.com> * implement token update retriable Signed-off-by: Leavrth <jianjun.liao@outlook.com> * fix some problems Signed-off-by: Leavrth <jianjun.liao@outlook.com> * close #11731 Signed-off-by: Leavrth <jianjun.liao@outlook.com> * rustfmt Signed-off-by: Leavrth <jianjun.liao@outlook.com> * make clippy success Signed-off-by: Leavrth <jianjun.liao@outlook.com> * simplify uploader Signed-off-by: Leavrth <jianjun.liao@outlook.com> * commit some suggestions Signed-off-by: Leavrth <jianjun.liao@outlook.com> * coprocesser: update encoding lib (#11506) Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * coprocesser: implement upper and lower function on GBK charset (#11756) Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * logger: support archiving and rotation (#11651) (#11657) * logger: support archiving and rotation (#11651) close #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: add testcases to test dt_from_file_name (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: simplify DateTime format (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * diagnostics: refactor (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: run clippy for integration tests (#11768) * run clippy --fix with nightly-2021-12-17 Signed-off-by: tabokie <xy.tao@outlook.com> * manual clippy Signed-off-by: tabokie <xy.tao@outlook.com> * format and clippy Signed-off-by: tabokie <xy.tao@outlook.com> * enable default features when running clippy, ref #11119 Signed-off-by: tabokie <xy.tao@outlook.com> * fix build Signed-off-by: tabokie <xy.tao@outlook.com> * revert box_collection lint Signed-off-by: tabokie <xy.tao@outlook.com> * Update tests/integrations/raftstore/test_bootstrap.rs Signed-off-by: Xinye Tao <xy.tao@outlook.com> Co-authored-by: Lei Zhao <zlwgx1023@gmail.com> Co-authored-by: Lei Zhao <zlwgx1023@gmail.com> * logger: fix config-template's log unit (#11651) (#11777) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: improve testcase (#11651) (#11778) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * raftstore: propose in-memory pessimistic locks before leader transfer (#11577) * *: check epoch and validness of lock table when writing pessimistic locks ref #11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * raftstore: propose in-memory pessimistic locks before leader transfer ref #11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * mark delete in scheduler and delete after apply Signed-off-by: Yilin Chen <sticnarf@gmail.com> * Add tests for two different orders between proposing locks and write command Signed-off-by: Yilin Chen <sticnarf@gmail.com> * do not clear pessimistic locks before transfer leader Signed-off-by: Yilin Chen <sticnarf@gmail.com> * delete memory locks in the apply thread and add more term and version check Signed-off-by: Yilin Chen <sticnarf@gmail.com> * remove potentially incorrect assert Signed-off-by: Yilin Chen <sticnarf@gmail.com> * check term and version when reading the lock table Signed-off-by: Yilin Chen <sticnarf@gmail.com> * revert a log change Signed-off-by: Yilin Chen <sticnarf@gmail.com> * change term when becoming follower Signed-off-by: Yilin Chen <sticnarf@gmail.com> * fix lint Signed-off-by: Yilin Chen <sticnarf@gmail.com> * do not check version when reading locks (temporarily) Signed-off-by: Yilin Chen <sticnarf@gmail.com> * fix another test Signed-off-by: Yilin Chen <sticnarf@gmail.com> * only treat MsgTransferLeader with context with valid msg Signed-off-by: Yilin Chen <sticnarf@gmail.com> * fix a tiny comment error Signed-off-by: Yilin Chen <sticnarf@gmail.com> * make clippy happy Signed-off-by: Yilin Chen <sticnarf@gmail.com> * res_meter: ignore the read keys test temporarily (#11779) ref #11765 Signed-off-by: mornyx <mornyx.z@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * logger: change critical to fatal (#11651) (#11780) * logger: change critical to fatal (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: fix test_parse_log_level (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * update CHANGELOG.md (#11790) * update from tikv-server v5.0.1(20210423) from https://github.com/tikv/tikv/releases Signed-off-by: xiejiandong <xiejiandong@pingcap.com> * update CHANGELOG.md close #11167 Signed-off-by: xiejiandong <xiejiandong@pingcap.com> * config: relax the bounds of online configurable worker number (#11651) (#11798) close #11776 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: invoke logger_compatible_adjust before run_tikv (#11651) (#11792) close #11789 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: unify thread related native APIs (#11785) * *: unify thread related native APIs Now there are duplicated codes calling syscalls to fetch thread informations. This PR provides unified APIs and remove duplicated codes. Close #11784. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * address comment Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * declare platform compatability Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * Fix method on_tick will not update metric when enable_auto_tune is false (#11791) * Fix method on_tick will not update metric when enable_auto_tune is false ref #11787 Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * resolve format problem in discussion Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * resolve format problem in discussion again Signed-off-by: yuqi1129 <yuqi4733@gmail.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: add a new tick to renew lease in advance (#6427) * renew lease in raft tick Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check lease before sending read index Signed-off-by: 5kbpers <tangminghua@pingcap.com> * renew lease on raft tick to cover lease till next tick Signed-off-by: linning <linningde25@gmail.com> * refactor Signed-off-by: linning <linningde25@gmail.com> * don't use inspect_lease Signed-off-by: linning <linningde25@gmail.com> * fix ci Signed-off-by: linning <linningde25@gmail.com> * add test Signed-off-by: linning <linningde25@gmail.com> * remove unneed comment Signed-off-by: linning <linningde25@gmail.com> * close #5388 Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check pending_read_count before hibernating Signed-off-by: 5kbpers <tangminghua@pingcap.com> * increase the range to check lease expired Signed-off-by: 5kbpers <tangminghua@pingcap.com> * renew leader lease after not hibernate Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix hibernate states metrics Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check writes and pre_read_index before proposing read index Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * use a individual tick to renewing lease Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix condition Signed-off-by: 5kbpers <tangminghua@pingcap.com> * stablize test_node_renew_lease Signed-off-by: 5kbpers <tangminghua@pingcap.com> * cleanup unused changes Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add config item for interval of tick Signed-off-by: 5kbpers <tangminghua@pingcap.com> * rename tick Signed-off-by: 5kbpers <tangminghua@pingcap.com> * disable the tick for test_inconsistent_configuration Signed-off-by: 5kbpers <tangminghua@pingcap.com> * change deafault value of check_leader_lease_interval Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * disable the tick for test_read_hibernated_region Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * count renew lease read index into raft metrics Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * disable the tick for test_renew_lease Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix lint error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add a 100ms buffer for renew_bound Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * make clippy happy Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: linning <linningde25@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * cloud: Add retry for azure blob server busy error (#11813) * add retry for azure blob server busy error Signed-off-by: Leavrth <jianjun.liao@outlook.com> * close#11812 Signed-off-by: Leavrth <jianjun.liao@outlook.com> * use to_string Signed-off-by: Leavrth <jianjun.liao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * tikv_util: make cgroup parsing more robust (#11786) * make cgroup parsing more robust Signed-off-by: tabokie <xy.tao@outlook.com> * fix error message Signed-off-by: tabokie <xy.tao@outlook.com> * parse cpu quota in floats Signed-off-by: tabokie <xy.tao@outlook.com> * *: make it compile on macos (#11825) * *: make it compile on macos close #11823 Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * remove extra comment Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * resolved_ts: add more metrics (#11809) * add metrics Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add slow log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * remove duplicate metric Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add resolved ts metrics for leader peers Signed-off-by: 5kbpers <tangminghua@pingcap.com> * close #11767 Signed-off-by: 5kbpers <tangminghua@pingcap.com> * init endpoint with store meta Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * rsmeter: fix perf regression caused by arc swap (#11833) * rsmeter: fix perf regression caused by arc swap Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * use atomic cell instead Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * debug assert Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Wenxuan <breezewish@pingcap.com> * load base split: more accurate sample for large key ranges batch (#11039) * more accurate sample for large key ranges Signed-off-by: lhy1024 <admin@liudos.us> * update Signed-off-by: lhy1024 <admin@liudos.us> * ref #11521 Signed-off-by: lhy1024 <admin@liudos.us> * address comment Signed-off-by: lhy1024 <admin@liudos.us> * fix test Signed-off-by: lhy1024 <admin@liudos.us> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftclient: enable delay only in high load (#11772) and improve CPU efficiency under high load. But it turns out this can lead to significant regression in normal load. So this PR adds CPU stats and only enabling delay in high load. This is also the same strategy used in v4.x. Close #11769. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * tikv-ctl: fix tikv-ctl's output is incomplete on calling process::exit (#11231) Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com> Co-authored-by: Connor <zbk602423539@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: skip deleting snapshot files in peer pending_remove is true (#11782) * raftstore: skip deleting snapshot files in peer when the peer is pending removal and the snapshot is being applied and canceled -- close #11746 This is to avoid the potential panic when the snapshot files are deleted, but the peer's status (Tombstone) is not persisted in disk due to tikv crash. Signed-off-by: tonyxuqqi <tonyxuqi@outlook.com> * address code review feedback -- close #11746 Signed-off-by: qi.xu <tonxuqi@outlook.com> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <tonxuqi@outlook.com> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <tonxuqi@outlook.com> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <tonxuqi@outlook.com> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <tonxuqi@outlook.com> Co-authored-by: qi.xu <tonxuqi@outlook.com> * load_statis: fix compilation on macos (#11851) close #11772 Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * config: override rocksdb config when flow control enabled (#11840) Signed-off-by: Connor1996 <zbk602423539@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * rocksdb: consider env sharing when configuring background threads (#11760) #9649 increased the default setting of kvdb background flushes (3 for 8c and 4 for 16c). This PR takes another approach: use smaller concurrency for individual dbs, but configure the global thread pool size with the sum of kvdb and raftdb concurrency. This way we won't over-configure flush workers when raft-engine is enabled. * consider env sharing when configuring background threads ref #11119 Signed-off-by: tabokie <xy.tao@outlook.com> * server/config: keep compatible using option (#11862) * server/config: keep compatible using option Using `Option` for `heavy_load_wait_duration` to keep compatible with versions prior to v5.3. Close #11861. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * fix format Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * fix clippy Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: zhouqiang <zhouqiang.cl@gmail.com> * raftstore: fix missing workers' stop on shutdown (#11864) ref #11159 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: move in-memory pessimistic locks to split regions (#11655) ref #11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: Remove the entrance of enabling io snooper. (#11394) * *: Remove the entrance of enabling io snooper. close #10867 Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * fix Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * fix test Signed-off-by: MuZhou233 <muzhou233@outlook.com> * fix test Signed-off-by: MuZhou233 <muzhou233@outlook.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: change PeerTicks to enum (#11849) * raftstore: change PeerTicks to enum close #11848 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * use bool array for registry Signed-off-by: Yilin Chen <sticnarf@gmail.com> * rename PeerTicks to PeerTick Signed-off-by: Yilin Chen <sticnarf@gmail.com> * add associated const for number of PeerTick types Signed-off-by: Yilin Chen <sticnarf@gmail.com> * change naming Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: check memory limit when inserting in-memory pessimistic locks (#11841) * *: check memory limit when inserting in-memory pessimistic locks ref #11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * fix merged upstream Signed-off-by: Yilin Chen <sticnarf@gmail.com> * fix test Signed-off-by: Yilin Chen <sticnarf@gmail.com> * fix typo Signed-off-by: Yilin Chen <sticnarf@gmail.com> * *: fix invalid failpoint caused by typo (#11709) * fix invalid failpoint caused by typo close #11734 Signed-off-by: Ryan Leung <rleungx@gmail.com> * address the comment Signed-off-by: Ryan Leung <rleungx@gmail.com> * address the comment Signed-off-by: Ryan Leung <rleungx@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: Extract significant router (#11750) * ref#11409 add significant router Signed-off-by: Connor1996 <zbk602423539@gmail.com> * make format Signed-off-by: Connor1996 <zbk602423539@gmail.com> * address comment Signed-off-by: Connor1996 <zbk602423539@gmail.com> * raftstore: move scan delete to raft log gc worker (#11853) * raftstore: move scan delete to raft log gc worker When clearing raft metas, raftstore will scan raft logs and delete them one by one. Seeking can be slow if there are a lot of tombstone keys. This PR moves the operation to raft log gc worker to reduce the impact. The final solution should be also moving remaining IO operations to async write IO threads. Close #10210. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * address comment Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * speed up destroy Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * fix compile Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * further speed up Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * revert test case configuration Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * address comment Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * address comment Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * doc: update new rules for linking issue and commit message (#11832) close #11097, close #11831 Signed-off-by: zhangyangyu <angwerzx@126.com> Co-authored-by: Mini256 <minianter@foxmail.com> Co-authored-by: Mini256 <minianter@foxmail.com> Co-authored-by: Xiaoguang Sun <sunxiaoguang@users.noreply.github.com> * *: update rust toolchain to 2022-01-07 (#11875) * update rust toolchain to 2022-01-17 Signed-off-by: tabokie <xy.tao@outlook.com> * address comment: clean up unnecessary `as_ref()` Signed-off-by: tabokie <xy.tao@outlook.com> * try fixing the underflow Signed-off-by: tabokie <xy.tao@outlook.com> * mute underflow warning for raft metrics Signed-off-by: tabokie <xy.tao@outlook.com> * clean up unused data members Signed-off-by: tabokie <xy.tao@outlook.com> * format Signed-off-by: tabokie <xy.tao@outlook.com> * step back to 2022-01-07 Signed-off-by: tabokie <xy.tao@outlook.com> * readme: change images based on github theme (#11795) Signed-off-by: sloorush <aarush.bhatt@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: don't remove other peer's read delegate (#11882) * don't remove read delegate besides peer_destroy Signed-off-by: linning <linningde25@gmail.com> * add test case Signed-off-by: linning <linningde25@gmail.com> * make clippy happy Signed-off-by: linning <linningde25@gmail.com> * address comment Signed-off-by: linning <linningde25@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * logger: use eprintln! if the logger is not initialized (#11869) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: fix the calculation of total column size in analyze (#11884) Signed-off-by: xuyifan <xuyifangreeneyes@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * engine: update raft-engine for data consistency fix (#11885) Fix https://github.com/tikv/raft-engine/issues/142. When encountering this bug, TiKV will panic with message "applied index > max(commit index, recorded commit index)" after restart. * update raft-engine Signed-off-by: tabokie <xy.tao@outlook.com> * update one more commit Signed-off-by: tabokie <xy.tao@outlook.com> * check docker build during clippy, ref #11312 (#11819) Signed-off-by: tabokie <xy.tao@outlook.com> * gc_worker: fix incorrect scheduled_tasks counting (#11904) * gc_worker: fix incorrect scheduled_tasks counting close #11903 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * remove check_is_busy totally Signed-off-by: Yilin Chen <sticnarf@gmail.com> * do not use wildcard match in error handling Signed-off-by: Yilin Chen <sticnarf@gmail.com> * raftstore: Introduce raft log fetcher (#11900) * ref#11409 introduce raft log fetcher Signed-off-by: Connor1996 <zbk602423539@gmail.com> * update kvproto Signed-off-by: Connor1996 <zbk602423539@gmail.com> * address comment Signed-off-by: Connor1996 <zbk602423539@gmail.com> * fix test build Signed-off-by: Connor1996 <zbk602423539@gmail.com> * fix clippy Signed-off-by: Connor1996 <zbk602423539@gmail.com> * limit capacity Signed-off-by: Connor1996 <zbk602423539@gmail.com> * update kvproto Signed-off-by: Connor1996 <zbk602423539@gmail.com> * call stop on worker Signed-off-by: Connor1996 <zbk602423539@gmail.com> * rename worker Signed-off-by: Connor1996 <zbk602423539@gmail.com> * status_server: add pprof flamegraph response header (#10951) * tweak(status_server): add pprof flamegraph response header Signed-off-by: Suhaha <jklopsdfw@gmail.com> * test(status_server): add Content-Type asset to test_pprof_profile_service Signed-off-by: Suhaha <jklopsdfw@gmail.com> * tweak(status_server): add pprof flamegraph response header Signed-off-by: Suhaha <jklopsdfw@gmail.com> * tweak(status_server): add pprof flamegraph response header close #11917 Signed-off-by: Suhaha <jklopsdfw@gmail.com> * close #11917 Signed-off-by: Suhaha <jklopsdfw@gmail.com> Co-authored-by: goroutine <ngaut@users.noreply.github.com> Co-authored-by: Connor <zbk602423539@gmail.com> * raftstore: renew leader lease in advance when handle read request (#9307) * renew lease advance Signed-off-by: linning <linningde25@gmail.com> * add log Signed-off-by: linning <linningde25@gmail.com> * make clippy happy Signed-off-by: linning <linningde25@gmail.com> * ref #11579 Signed-off-by: 5kbpers <tangminghua@pingcap.com> * reset raft tick Signed-off-by: 5kbpers <tangminghua@pingcap.com> * set has ready Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add renew_leader_lease_advance_duration config Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix panic Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * disable renewing for test_lease_unsafe_during_leader_transfers Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: 5kbpers <tangminghua@pingcap.com> * pprof: ignore cpu profiling on non-x86 arch (#11925) * Disable cpu profiling on non-x86_64 arch Signed-off-by: mornyx <mornyx.z@gmail.com> * Fix warns Signed-off-by: mornyx <mornyx.z@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * github: add new pr requirement for linking issue and commit message (#11887) Signed-off-by: zhangyangyu <angwerzx@126.com> Co-authored-by: Xiaoguang Sun <sunxiaoguang@users.noreply.github.com> * engine: enable raft engine by default (#11928) * enable raft engine by default and synchronize docs Signed-off-by: tabokie <xy.tao@outlook.com> * update raft engine Signed-off-by: tabokie <xy.tao@outlook.com> * update raft-engine and fix tests Signed-off-by: tabokie <xy.tao@outlook.com> * gc_worker: Limit the key range to scan for GcKeys tasks (#11922) close tikv/tikv#11752, close tikv/tikv#11902 Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: TransferLeader support multiple target peers (#11063) ref tikv/tikv#822, ref tikv/tikv#4214, close tikv/tikv#10602 Signed-off-by: MrCroxx <mrcroxx@outlook.com> * raftstore: remove the leaders field of `StoreMeta` (#11934) close tikv/tikv#11933 Remove `StoreMeta.leaders` ``` ### Related changes ### Check List <!--REMOVE the items that are not applicable--> Tests <!-- At least one of them must be included. --> - Manual test (add detailed scripts or steps below) `cargo check` ### Release note <!-- bugfixes or new feature need a release note --> ```release-note None Signed-off-by: linning <linningde25@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: Fetch raft log in async manner (#11409) close tikv/tikv#10408, close tikv/tikv#11320 Signed-off-by: Connor1996 <zbk602423539@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: add quarter function (#11935) ref tikv/tikv#5751 Signed-off-by: zhongyong jin <AT-Fieldless@outlook.com> * Enable full debug info for dev and test (#11949) ref tikv/tikv#5049, ref tikv/tikv#5572, close tikv/tikv#5572 ``` ### Related changes - PR to update `pingcap/docs`/`pingcap/docs-cn`: - PR to update `pingcap/tidb-ansible`: - Need to cherry-pick to the release branch --> ### Check List <!--REMOVE the items that are not applicable--> Tests <!-- At least one of them must be included. --> - Manual test (add detailed scripts or steps below) - No code Manual test: Linux box with 8 cores (3:50 -> 4:00): ``` # Before this PR: $ make clean cargo clean rm -rf bin dist $ time make build cargo build --no-default-features --features "jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 ... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 3m 50s real 3m50,487s user 38m11,859s sys 3m9,540s # After this PR: $ git diff diff --git a/Cargo.toml b/Cargo.toml index 71f5329d3..67cb9d183 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -267,7 +267,7 @@ default-members = ["cmd/tikv-server", "cmd/tikv-ctl"] [profile.dev] opt-level = 0 -debug = 1 # required for line numbers in tests, see tikv #5049 +debug = true codegen-units = 4 lto = false incremental = true @@ -293,7 +293,7 @@ codegen-units = 4 [profile.test] opt-level = 0 -debug = 1 # enable line numbers by default for easy test debugging +debug = true codegen-units = 16 lto = false incremental = true mattias@msig:~/repos/tikv$ time make clean cargo clean rm -rf bin dist real 0m0,975s user 0m0,148s sys 0m0,828s mattias@msig:~/repos/tikv$ time make build cargo build --no-default-features --features "jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 ... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 4m 00s real 4m0,201s user 39m45,037s sys 3m16,397s ``` Macbook Air M1: ``` # Before: % time make build cargo build --no-default-features --features " jemalloc test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 .... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 4m 01s make build 1107.42s user 116.20s system 506% cpu 4:01.46 total # After: % time make build cargo build --no-default-features --features " jemalloc test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 .... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 4m 10s make build 1179.39s user 120.74s system 518% cpu 4:10.98 total ``` Side effects - Performance regression, Only when building, to the benefit of full debug info by default. ### Release note <!-- bugfixes or new feature need a release note --> ```release-note None Signed-off-by: Mattias Jonsson <mjonss@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * correct a metric about compaction filter (#11938) correct a metric about compaction filter Signed-off-by: qupeng <qupeng@pingcap.com> * raft: Fix possible panic on entries fetched callback (#11961) close tikv/tikv#11951 Signed-off-by: Connor1996 <zbk602423539@gmail.com> * split_controller: refine the sample function and add some comments (#11952) Signed-off-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * metrics: fix grid position in TiKV Details (#11936) ref tikv/tikv#11119 Fix grid positions in TiKV Details dashboard. Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * cdc: capture old value from txn layer dynamically (#11896) close tikv/tikv#10091 Signed-off-by: hi-rustin <rustin.liu@gmail.com> Signed-off-by: qupeng <qupeng@pingcap.com> Co-authored-by: hi-rustin <rustin.liu@gmail.com> * split_controller: reorganize the structs and add more comments (#11967) Reorganize the structs and add more comments. Signed-off-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: support Bit column push downBit column (#11968) close tikv/tikv#11893, ref tikv/tikv#11893, ref pingcap/tidb#31884 Signed-off-by: xiejiandong <xiejiandong@pingcap.com> * copr: support push down mod, sysdate to TiKV (#11970) close tikv/tikv#11916, ref tikv/tikv#11916 Signed-off-by: xiejiandong <xiejiandong@pingcap.com> * raftstore: propose in-memory pessimistic locks before prepare merge (#11758) ref tikv/tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * stats_monitor: reformat stats_monitor tick (#11972) Signed-off-by: lhy1024 <admin@liudos.us> * copr: support extract scalar value from bit (#11980) close tikv/tikv#11893 Signed-off-by: yisaer <disxiaofei@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * diagnosis: support get cpu time of threads for macOS (#11978) close tikv/tikv#11977 Add the variant for macOS into `components/tikv_util/src/sys/thread.rs` Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Co-authored-by: Jay <BusyJay@users.noreply.github.com> * rust: update toolchain to fix missing rls (#11954) close tikv/tikv#11953 Signed-off-by: Connor1996 <zbk602423539@gmail.com> Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Co-authored-by: Yilin Chen <sticnarf@gmail.com> * cdc: separate resolved region outliers (#11991) Separate broadcasing outlier regions and normal regions, so 1) downstreams know where they should send resolve lock requests, and 2) resolved ts of normal regions does not fallback. close pingcap/tiflow#4516 close pingcap/tiflow#4311 ref pingcap/tiflow#4146 Signed-off-by: Neil Shen <overvenus@gmail.com> * *: collect key ranges for the read request in ResourceMeteringTag (#11995) close tikv/tikv#11988 *: collect key ranges for the read request in ResourceMeteringTag Signed-off-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * split_controller: move sample_threshold check out of the split_key iteration (#11986) split_controller: move sample_threshold check out of the split_key iteration Signed-off-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: support insertutf8 and lowerutf8 (#11987) ref tikv/tikv#5751 Signed-off-by: yisaer <disxiaofei@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: support rest greatest/least functions (#12003) ref tikv/tikv#5751 Signed-off-by: yisaer <disxiaofei@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: force compact with gentleness (#12006) * force compact with gentleness Signed-off-by: tabokie <xy.tao@outlook.com> * address comment Signed-off-by: tabokie <xy.tao@outlook.com> * split_controller: refine the LOAD_BASE_SPLIT_EVENT metrics label definitions (#12010) split_controller: refine the LOAD_BASE_SPLIT_EVENT metrics label definitions Signed-off-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * split_controller: fix the incorrect sampled key ranges number check (#12013) close tikv/tikv#12012 split_controller: fix the incorrect sampled key ranges number check Signed-off-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: Support adjusting max_inflight_msgs dynamically (#11866) close tikv/tikv#11865 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: fix missing flag solvement for bit column in decoding (#12016) ref tikv/tikv#7, close pingcap/tidb#32506 Signed-off-by: yisaer <disxiaofei@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * file_system: overhaul proc-based io tracing utilities (#11873) ref tikv/tikv#10867, fix tikv/tikv#11775 Signed-off-by: tabokie <xy.tao@outlook.com> * raftstore: Wait for the apply index equals to the commit index (#11716) ref tikv/tikv#10483 Signed-off-by: v01dstar <yang.zhang@pingcap.com> * copr: revert 11968/11980/12016 (#12030) ref tikv/tikv#11968, ref tikv/tikv#11980, ref tikv/tikv#12016, ref pingcap/tidb#32506 Signed-off-by: yisaer <disxiaofei@163.com> * build: fix a Mac build issue (#12027) ref tikv/tikv#10867, ref tikv/tikv#11873 Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: Extra physical table id column (#11931) close tikv/tikv#11888 Added EXTRA_PHYSICAL_TABLE_ID_COL_ID support, to support TiDB's table partition dynamic prune mode, where a single request includes multiple partitions, but when Pessimistic lock (like `SELECT FOR UPDATE`) or ongoing transaction (having something in the tidb session local transaction buffer) each row needs its partition id / physical table ID to either lock that row or to check against the transaction buffer. Signed-off-by: Mattias Jonsson <mjonss@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: hotfix panic from tokio-timer (#12004) close tikv/tikv#11940, ref tikv/tikv#11940 *: hotfix panic from tokio-timer by enlarging the length of the level vector Signed-off-by: you06 <you1474600@gmail.com> * build: add arm64 check on SSE in makefile (#12035) close tikv/tikv#12034 Signed-off-by: Jin Dong <djdongjin95@gmail.com> Co-authored-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * replication mode: sync state to pd (#11751) Signed-off-by: disksing <i@disksing.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: revert pessimistic locks status when failing to propose PrepareMerge (#11985) ref tikv/tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * copr: support push bit column down (#12037) ref pingcap/tidb#30738 Signed-off-by: yisaer <disxiaofei@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * split_controller: add more sample function test cases (#12058) split_controller: add more sample function test cases Signed-off-by: JmPotato <ghzpotato@gmail.com> * copr: fix greatest/least time/date args type (#12056) Signed-off-by: yisaer <disxiaofei@163.com> * cdc: advancing resolved ts correctly for clusters with tiflash (#12050) close pingcap/tiflow#4461 cdc: advancing resolved ts correctly for clusters with tiflash Signed-off-by: qupeng <qupeng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: bump master branch version to v6.0.0-alpha (#12077) close tikv/tikv#12075 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * raftstore: check uninitialized destroy for merge (#12055) close tikv/tikv#12048 When a peer is destroyed without being initialized, it will store itself to peer list and the region epoch is missing. In merge if such state is detected, it should abort merging instead of panicking. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: Add TruncateManager and TruncateWorker (#11553) ref tikv/rfcs#81 Signed-off-by: longfangsong <longfangsong@icloud.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * TiKV supports buckets (#11763) ref tikv/tikv#11759 Signed-off-by: qi.xu <tonxuqi@outlook.com> Co-authored-by: qi.xu <tonxuqi@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * api_version: Codec for RawKV key (#12036) ref tikv/tikv#11965 Key of RawKV is encoded as `user-key + memcomparable-padding + timestamp` in API v2. Signed-off-by: pingyu <yuping@pingcap.com> Co-authored-by: Andy Lok <andylokandy@hotmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: fix some typos (#12066) Signed-off-by: cuishuang <imcusg@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: fix stale message cause panic (#12054) close tikv/tikv#12023 raftstore: fix stale message cause panic Signed-off-by: linning <linningde25@gmail.com> * raftstore: ignore async fetch result when the peer is pending removed (#12038) close tikv/tikv#11973, close tikv/tikv#12026 Signed-off-by: Connor1996 <zbk602423539@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: remove part about wechat in doc (#12101) close tikv/tikv#12100 Signed-off-by: jackwener <jakevingoo@gmail.com> * engine: upgrade raft engine (#12095) ref tikv/tikv#165, ref tikv/raft-engine#165, ref tikv/tikv#11119 Signed-off-by: Randy <ztelur@gmail.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com> * *: Fix possible undefined behavior for transmuting vec (#12096) close tikv/tikv#12070 Signed-off-by: Connor1996 <zbk602423539@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * github: Remove the pingcap/tidb-ansible from the PR template (#12102) close tikv/tikv#12103 Signed-off-by: hi-rustin <rustin.liu@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: extract reschedule to inject a fail point (#12108) Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * raftstore: reactivate in-memory pessimistic locking when leader transfer fails (#11883) ref tikv/tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: check memory locks for replica read only on the leader (#12115) close tikv/tikv#12109 Consider a reader sends read index to the leader it supposed to be, but when the leader receives the read index message, it has stepped down to a follower. Without this commit, a peer will check the locks and re-fill the read index context with the result no matter what role it is. So, when the read index request is redirected to the new leader again, it no longer carries the context of lock checking. This commmit changes to only do the check when the peer is a leader. Then, the read index request will remain unchanged before being redirected to the leader. If the lease of the leader expires, it is still safe to check on the uncertained leader. If the heartbeat check passes and no new leader is elected, then the check works fine. If there is a new leader, when the old leader becomes a follower, it will clear its pending read index. Then, the reader has to resend the read index again to the correct leader. So, generally it is safe as long as we guarantee the check only happens on the leader. Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * apiv2 rawkv: logical deletion flag (#12090) ref tikv/tikv#11965 Signed-off-by: haojinming <jinming.hao@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: add projection (#10689) close tikv/tikv#12114 Signed-off-by: ichn-hu <zfhu16@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Co-authored-by: Alex Chi <iskyzh@gmail.com> * raftstore: adjust raft max msg size dynamically (#12018) close tikv/tikv#12017 Signed-off-by: glorv <glorvs@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: update deps (#12098) - Upgrade grpcio to 0.10 - Upgrade openssl to 1.1.1m - Remove dependency on prost Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * tests: make raftstore tests about transactions more stable (#12122) close tikv/tikv#12120 A new fail point "after_propose" is added to insert callback from the test. Then we can ensure we release other key fail points after they take effect. The other change is to retry request automatically to avoid stale command errors. Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * metrics: support multi k8s in grafana dashboards (#12008) close tikv/tikv#12104 Signed-off-by: just1900 <legendj228@gmail.com> * storage: set in-memory pessimistic lock feature gate to 6.0.0 (#12078) ref tikv/tikv#11452 The in-memory pessimistic lock feature should be only enabled after all TiKV instances have been upgraded to the new version. Otherwise, transferring leader may be blocked because of the new protocol and procedure when in-memory pessimistic locks exist. So, we need to use a feature gate to make sure this feature is only enabled after all TiKVs have been upgraded to the new version. Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore,cdc: pass leader transferee to cdc observer (#12124) Signed-off-by: qupeng <qupeng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * config: simplify handling of nullable configurations (#12133) Support deriving `O…
CalvinNeo
added a commit
to pingcap/tidb-engine-ext
that referenced
this pull request
May 30, 2022
…roxy (#72) * ctl: validate column family name when scanning data using ctl (#11269) * ctl: should check column name validation when scan data using ctl close #11268 Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * fix format problem Signed-off-by: yuqi1129 <yuqi4733@gmail.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com> * txn: Do assertions on data for mutations to reduce potential data inconsistency (#10977) * Add mutation checking logic Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix build Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix usages of mutation Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Update kvproto and renaming Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Add a simple test Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix incorrect assertion Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Support check existance during pessimistic lock Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Add assertion level support Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Add tests Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Add more test cases Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Add test for check_existence of pessimistic_lock request Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Fix build Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * ref #11542 Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> * Add a metric for counting different way of checking assertion, for reference about the performance impact Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * metrics: fix memory metrics (#11656) Signed-off-by: Shuaipeng Yu <jackysp@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * coprocesser: refactor encoding and add gbk_chinese_ci map (#11661) * close #11660 Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com> * close #11660 Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * storage: Add API V2 check for RawKV and TxnKV requests (txn part) (#11216) * wip Signed-off-by: pingyu <yuping@pingcap.com> * wip Signed-off-by: pingyu <yuping@pingcap.com> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <yuping@pingcap.com> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <yuping@pingcap.com> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <yuping@pingcap.com> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <yuping@pingcap.com> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <yuping@pingcap.com> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <yuping@pingcap.com> * fix enable_ttl Signed-off-by: andylokandy <andylokandy@hotmail.com> * simplify test Signed-off-by: andylokandy <andylokandy@hotmail.com> * ref #10974: Add API V2 check for RawKV and TxnKV requests (txn) Signed-off-by: pingyu <yuping@pingcap.com> Co-authored-by: andylokandy <andylokandy@hotmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * cdc: load uninlined value more effectively (#11615) Signed-off-by: qupeng <qupeng@pingcap.com> * cdc: load old value with prefix seek if possible (#11643) * close #11642 Signed-off-by: qupeng <qupeng@pingcap.com> * a more comment Signed-off-by: qupeng <qupeng@pingcap.com> * ctl: split into multiple modules (#11658) * ctl: split into multiple modules ref #10938 Signed-off-by: andylokandy <andylokandy@hotmail.com> * refactor if let to match Signed-off-by: andylokandy <andylokandy@hotmail.com> * address comment Signed-off-by: andylokandy <andylokandy@hotmail.com> * address comment Signed-off-by: andylokandy <andylokandy@hotmail.com> * address comment Signed-off-by: andylokandy <andylokandy@hotmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * Fix build issues on non-linux platforms (#11671) * close #11676 Signed-off-by: devillove084 <786537003@qq.com> * all: Migrate from std:: types to primitives. (#11677) * Deprecation is planned for MIN,MAX,EPSLION,INFINITY,NEG_INFINITY of the (i|u|f)(8|16|32|64) types in standard. ref #11678 ```release-note NONE ``` Signed-off-by: Harold Dost <harolddost@gmail.com> * metrics alert: fix apply cpu alert (#11672) Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * resrc_mtr: introduce scanned keys statistics (#11085) close #11690 Signed-off-by: mornyx <mornyx.z@gmail.com> * file_system: support collect the io flow by proc (#11667) * file_system: support collect the read flow by proc Signed-off-by: MuZhou233 <muzhou233@outlook.com> * fix lint Signed-off-by: MuZhou233 <muzhou233@outlook.com> * fix clippy Signed-off-by: MuZhou233 <muzhou233@outlook.com> * Delete unused content & Adjust copyright text Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Add bench_fetch_io_bytes Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * fix lint Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Integrate thread_io Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * record bytes in request Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * fix logical bug Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Update test write_bytes Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Delete unused & Rename func Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Add test for read & Update test for write Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Add buffered thread io Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Replace DashMap with ThreadLocal Signed-off-by: MuZhou233 <muzhou233@outlook.com> * Rename variables and function & Remove unused call Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Change fetch logic. close #10867 Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Add AtomicIOBytes Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Split ThreadIOSentinel Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * add THREAD_IO_TOTAL for metrics bytes fetcher Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * Adjust code style & fix Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * improve write_bytes test Signed-off-by: MuZhou233 <muzhou233@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: check epoch and validness of lock table when writing pessimistic locks (#11591) ref #11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * rsmeter: introduce datasink (#11688) * add datasink trait Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * RAII removes on_reporter_closing Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fmt Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix build ref #11691 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * datasink -> data_sink Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fmt Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix build Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * clippy: Fixing up clippy errors. (#11193) * clippy: Fixing up clippy errors. * Fix crate references * Fix anonymous lifetimes references. ref #4301 Signed-off-by: Harold Dost <harolddost@gmail.com> * Test Revert of Iterations ref #4301 Signed-off-by: Harold Dost <harolddost@gmail.com> * clippy: Enable check for rust-2018-idioms ref #4301 Signed-off-by: Harold Dost <harolddost@gmail.com> * rsmeter: add centralized place to construct protobuf data (#11696) ref #11691 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * tikv_util: update procfs to 0.12.0 (#11703) Signed-off-by: linning <linningde25@gmail.com> * Use generic for api version to reduce runtime branching (#11687) * Use generic for api version to reduce runtime branching ref #10938 Signed-off-by: andylokandy <andylokandy@hotmail.com> * fix test Signed-off-by: andylokandy <andylokandy@hotmail.com> * add test Signed-off-by: andylokandy <andylokandy@hotmail.com> * fix clippy Signed-off-by: andylokandy <andylokandy@hotmail.com> * address comment Signed-off-by: andylokandy <andylokandy@hotmail.com> * fmt Signed-off-by: andylokandy <andylokandy@hotmail.com> * address comment Signed-off-by: andylokandy <andylokandy@hotmail.com> * address comment Signed-off-by: andylokandy <andylokandy@hotmail.com> * rename vars Signed-off-by: andylokandy <andylokandy@hotmail.com> * improve vars Signed-off-by: andylokandy <andylokandy@hotmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * sst_importer: check api version when importing (#11664) * sst_importer: check api version when importing ref #10938 Signed-off-by: Peng Guanwen <pg999w@outlook.com> * Fix bugs Signed-off-by: Peng Guanwen <pg999w@outlook.com> * fix bug of Iterator::{scan,scan_cf} Signed-off-by: Peng Guanwen <pg999w@outlook.com> * remove redundant log Signed-off-by: Peng Guanwen <pg999w@outlook.com> * Show the entire key Signed-off-by: Peng Guanwen <pg999w@outlook.com> * Fix lint issue Signed-off-by: Peng Guanwen <pg999w@outlook.com> * resolve conflict Signed-off-by: andylokandy <andylokandy@hotmail.com> Co-authored-by: Andy Lok <andylokandy@hotmail.com> * backup: pipeline scan and save step (#11528) * br: sperate io and scan threads Signed-off-by: Yu Juncen <yu745514916@live.com> * br/stream: make clippy happy Signed-off-by: yujuncen <yujuncen@pingcap.com> * br/stream: fix tests Signed-off-by: Yu Juncen <yu745514916@live.com> * backup: fix a dummy bug Signed-off-by: Yu Juncen <yu745514916@live.com> * backup: add config of io-threads Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: ref #11350: some minior change Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: added some metrics Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: better tuning concurrency, and added a metric Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: added some slow log and metrics Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: address comments Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: fix build Signed-off-by: yujuncen <yujuncen@pingcap.com> * Revert "backup: fix build" This reverts commit 74537be56c410d0f91af2f48b7e65356ab53720e. Signed-off-by: yujuncen <yujuncen@pingcap.com> * Revert "backup: address comments" This reverts commit 77d75756028bea3496f878f0ad8bbc562f5d00aa. Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: address comments Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: always set IO type to Export Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup:make clippy happy Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: address comments Signed-off-by: yujuncen <yujuncen@pingcap.com> * metrics: fix the metrics cannot be displayed (#11710) ref #11662 Signed-off-by: Ryan Leung <rleungx@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * backup: return api-version to BR when backup (#11704) * Update kvproto Signed-off-by: Peng Guanwen <pg999w@outlook.com> * Return api-version for br ref #10938 Signed-off-by: Peng Guanwen <pg999w@outlook.com> * Reformat code Signed-off-by: Peng Guanwen <pg999w@outlook.com> * Update components/external_storage/export/src/export.rs Signed-off-by: Peng Guanwen <pg999w@outlook.com> Co-authored-by: Andy Lok <andylokandy@hotmail.com> * format code Signed-off-by: Peng Guanwen <pg999w@outlook.com> Co-authored-by: Andy Lok <andylokandy@hotmail.com> * file_system: limit thread_io target os (#11715) * limit thread_io target os Signed-off-by: MuZhou233 <muzhou233@outlook.com> * limit thread_io target os. typo. close #11698 Signed-off-by: MuZhou233 <muzhou233@outlook.com> Co-authored-by: Yilin Chen <sticnarf@gmail.com> * file_system: bypass file allocate implementation in fs2 (#11700) Close #10688 Patch fs2 to https://github.com/tabokie/fs2-rs/tree/tikv. In which, `posix_fallocate` is replaced with `fallocate` (https://github.com/danburkert/fs2-rs/pull/42). * bypass file allocate implementation in fs2 Signed-off-by: tabokie <xy.tao@outlook.com> * metrics: remove min legend of some Grafana panels (#11487) Minimum legend isn't very useful, and it occupied valuable visual real estate. Remove some min legends from Grafana metrics, except for the following: - Level 0 chance - Number files at each level - File Count - Entry Count - Allocator Stats - Encrypted files - Encryption meta files size Ref #11119 Signed-off-by: tabokie <xy.tao@outlook.com> * coprocessor: not return rows when there is no input for simple aggregation (#11708) * not return rows when there is no input for simple aggregation Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn> * close #11735, and address comments Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn> * backup: reduce the default thread pool size of backup, enable auto tune by default (#11699) * backup: clamp auto tune values Signed-off-by: yujuncen <yujuncen@pingcap.com> * ref #11000: change the default backup threads. Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: run tasks in background Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: run rustfmt Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: set remain threads to 20% of vcpu Signed-off-by: yujuncen <yujuncen@pingcap.com> * br-stream: fix build Signed-off-by: yujuncen <yujuncen@pingcap.com> * Backup: add S3 metrics && add s3_multi_part_size config (#11666) * s3: add request metrics for s3 storage Signed-off-by: 3pointer <luancheng@pingcap.com> * s3: add grafana json Signed-off-by: 3pointer <luancheng@pingcap.com> * br: add config for s3 multi part upload Signed-off-by: 3pointer <luancheng@pingcap.com> * update comment Signed-off-by: 3pointer <luancheng@pingcap.com> * update comment Signed-off-by: 3pointer <luancheng@pingcap.com> * address comment Signed-off-by: 3pointer <luancheng@pingcap.com> * hidden the new config Signed-off-by: 3pointer <luancheng@pingcap.com> * only hidden config in config-template.toml Signed-off-by: 3pointer <luancheng@pingcap.com> * address comment Signed-off-by: 3pointer <luancheng@pingcap.com> * close #11727 && format Signed-off-by: 3pointer <luancheng@pingcap.com> * fix Signed-off-by: 3pointer <luancheng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * rsmeter: support multiple datasinks (#11707) * rsmeter: support multiple datasinks Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * wrap config notifier Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * add unit tests Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * polish ref #11691 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * address comments Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix test Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix outdated comments Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * engine: properly estimate store size (#11728) * properly estimate store size, ref #11119 Signed-off-by: tabokie <xy.tao@outlook.com> * address comments Signed-off-by: tabokie <xy.tao@outlook.com> * implement global config client for tikv (#11685) * implement global config client close #11686 Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * implement unit test for global config in pd client Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * resolved formatting issues Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * impl pd client asyncly Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * according to reviewer's opinion removing store method implementation Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * Merge branch 'master' of https://github.com/tikv/tikv into global_conf Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * write test case for watch global config when grpc server is closed. Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * resove issues Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * clippy Signed-off-by: lemonhx <lemonhx@lemonhx.tech> * watch global config rentry future in test deadlock just removed Signed-off-by: lemonhx <lemonhx@lemonhx.tech> Co-authored-by: Zhenchi <zhongzc_arch@outlook.com> * storage: skip scanning lock when using rc (#11701) * skip scanning lock when using rc, ref #11485 Signed-off-by: you06 <you1474600@gmail.com> * format code Signed-off-by: you06 <you1474600@gmail.com> * add test Signed-off-by: you06 <you1474600@gmail.com> * address comment Signed-off-by: you06 <you1474600@gmail.com> * add an integration test Signed-off-by: you06 <you1474600@gmail.com> * format code Signed-off-by: you06 <you1474600@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: pushdown substring to tikv (#11494) * corp: pushdown substring to tikv Signed-off-by: guo-shaoge <shaoge1994@163.com> * fix case. close #11495 Signed-off-by: guo-shaoge <shaoge1994@163.com> * fix comment Signed-off-by: guo-shaoge <shaoge1994@163.com> * rsmeter: distinguish between collectors and observers within recorder (#11712) * rsmeter: support multiple datasinks Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * wrap config notifier Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * add unit tests Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * polish ref #11691 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * rsmeter: distinguish between collectors and observers within recorder Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fmt & try to resolve #11689 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * add unit tests ref #11691 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * retrigger test Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * address comments Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix test Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix outdated comments Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * Update components/resource_metering/src/recorder/sub_recorder/mod.rs Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Yexiang Zhang <mornyx.z@gmail.com> * address comment Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * remove enabled Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Co-authored-by: Wenxuan <breezewish@pingcap.com> Co-authored-by: Yexiang Zhang <mornyx.z@gmail.com> * config: validate online configurable thread pools (tikv#11159) (#11714) ref #11159 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * res_meter: upgrade kvproto (#11749) * Upgrade kvproto close #11748 Signed-off-by: mornyx <mornyx.z@gmail.com> * Fix test compilation Signed-off-by: mornyx <mornyx.z@gmail.com> * Fix test compilation Signed-off-by: mornyx <mornyx.z@gmail.com> * Fix test compilation Signed-off-by: mornyx <mornyx.z@gmail.com> * Use master kvproto Signed-off-by: mornyx <mornyx.z@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * rsmeter: add pubsub datasink (#11719) * rsmeter: add pubsub datasink close #11691 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * fix build Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * Update components/resource_metering/src/reporter/pubsub.rs Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Wenxuan <hi@breeswish.org> * fix build Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Wenxuan <hi@breeswish.org> * status_server: Support online config update for configuration hosting platform (#11693) close #11692 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * res_meter: optimize top k (#11753) * Keep top k on recorder Signed-off-by: mornyx <mornyx.z@gmail.com> * Modify comments Signed-off-by: mornyx <mornyx.z@gmail.com> * Fix pd.rs Signed-off-by: mornyx <mornyx.z@gmail.com> * Refactor Signed-off-by: mornyx <mornyx.z@gmail.com> * Remove import Signed-off-by: mornyx <mornyx.z@gmail.com> * max_resource_groups 200 -> 100 Signed-off-by: mornyx <mornyx.z@gmail.com> * Fix ut Signed-off-by: mornyx <mornyx.z@gmail.com> * Use iterator to avoid clone Signed-off-by: mornyx <mornyx.z@gmail.com> * Optimize close #11754 Signed-off-by: mornyx <mornyx.z@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftclient: delay flush (#11705) * raftclient: delay flush Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * remove raftstore wait Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * Ref #11309. tikv#11310 introduces delay in raftstore thread to reduce the flush rate. It can only reduce flush rate from one thread. If there are many raftstore threads or IO threads, messages can still be flush frequently. This PR reduces the flush rate by introducing delay in raft client. So delay will work at connection level and achieve the maximun batch affect. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * remove useless field Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * address comment Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * fix unstable raw get case Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Liqi Geng <gengliqiii@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * cloud: support azure blob storage as external storage backend (#11732) * support azure storage blob for external storage Signed-off-by: Leavrth <jianjun.liao@outlook.com> * modify the priority of the azure login methods Signed-off-by: Leavrth <jianjun.liao@outlook.com> * fix feature match branch Signed-off-by: Leavrth <jianjun.liao@outlook.com> * implement token update retriable Signed-off-by: Leavrth <jianjun.liao@outlook.com> * fix some problems Signed-off-by: Leavrth <jianjun.liao@outlook.com> * close #11731 Signed-off-by: Leavrth <jianjun.liao@outlook.com> * rustfmt Signed-off-by: Leavrth <jianjun.liao@outlook.com> * make clippy success Signed-off-by: Leavrth <jianjun.liao@outlook.com> * simplify uploader Signed-off-by: Leavrth <jianjun.liao@outlook.com> * commit some suggestions Signed-off-by: Leavrth <jianjun.liao@outlook.com> * coprocesser: update encoding lib (#11506) Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * coprocesser: implement upper and lower function on GBK charset (#11756) Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * logger: support archiving and rotation (#11651) (#11657) * logger: support archiving and rotation (#11651) close #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: add testcases to test dt_from_file_name (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: simplify DateTime format (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: do a little refactoring (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * diagnostics: refactor (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: run clippy for integration tests (#11768) * run clippy --fix with nightly-2021-12-17 Signed-off-by: tabokie <xy.tao@outlook.com> * manual clippy Signed-off-by: tabokie <xy.tao@outlook.com> * format and clippy Signed-off-by: tabokie <xy.tao@outlook.com> * enable default features when running clippy, ref #11119 Signed-off-by: tabokie <xy.tao@outlook.com> * fix build Signed-off-by: tabokie <xy.tao@outlook.com> * revert box_collection lint Signed-off-by: tabokie <xy.tao@outlook.com> * Update tests/integrations/raftstore/test_bootstrap.rs Signed-off-by: Xinye Tao <xy.tao@outlook.com> Co-authored-by: Lei Zhao <zlwgx1023@gmail.com> Co-authored-by: Lei Zhao <zlwgx1023@gmail.com> * logger: fix config-template's log unit (#11651) (#11777) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: improve testcase (#11651) (#11778) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * raftstore: propose in-memory pessimistic locks before leader transfer (#11577) * *: check epoch and validness of lock table when writing pessimistic locks ref #11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * raftstore: propose in-memory pessimistic locks before leader transfer ref #11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * mark delete in scheduler and delete after apply Signed-off-by: Yilin Chen <sticnarf@gmail.com> * Add tests for two different orders between proposing locks and write command Signed-off-by: Yilin Chen <sticnarf@gmail.com> * do not clear pessimistic locks before transfer leader Signed-off-by: Yilin Chen <sticnarf@gmail.com> * delete memory locks in the apply thread and add more term and version check Signed-off-by: Yilin Chen <sticnarf@gmail.com> * remove potentially incorrect assert Signed-off-by: Yilin Chen <sticnarf@gmail.com> * check term and version when reading the lock table Signed-off-by: Yilin Chen <sticnarf@gmail.com> * revert a log change Signed-off-by: Yilin Chen <sticnarf@gmail.com> * change term when becoming follower Signed-off-by: Yilin Chen <sticnarf@gmail.com> * fix lint Signed-off-by: Yilin Chen <sticnarf@gmail.com> * do not check version when reading locks (temporarily) Signed-off-by: Yilin Chen <sticnarf@gmail.com> * fix another test Signed-off-by: Yilin Chen <sticnarf@gmail.com> * only treat MsgTransferLeader with context with valid msg Signed-off-by: Yilin Chen <sticnarf@gmail.com> * fix a tiny comment error Signed-off-by: Yilin Chen <sticnarf@gmail.com> * make clippy happy Signed-off-by: Yilin Chen <sticnarf@gmail.com> * res_meter: ignore the read keys test temporarily (#11779) ref #11765 Signed-off-by: mornyx <mornyx.z@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * logger: change critical to fatal (#11651) (#11780) * logger: change critical to fatal (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: fix test_parse_log_level (#11651) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * update CHANGELOG.md (#11790) * update from tikv-server v5.0.1(20210423) from https://github.com/tikv/tikv/releases Signed-off-by: xiejiandong <xiejiandong@pingcap.com> * update CHANGELOG.md close #11167 Signed-off-by: xiejiandong <xiejiandong@pingcap.com> * config: relax the bounds of online configurable worker number (#11651) (#11798) close #11776 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> * logger: invoke logger_compatible_adjust before run_tikv (#11651) (#11792) close #11789 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: unify thread related native APIs (#11785) * *: unify thread related native APIs Now there are duplicated codes calling syscalls to fetch thread informations. This PR provides unified APIs and remove duplicated codes. Close #11784. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * address comment Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * declare platform compatability Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * Fix method on_tick will not update metric when enable_auto_tune is false (#11791) * Fix method on_tick will not update metric when enable_auto_tune is false ref #11787 Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * resolve format problem in discussion Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * resolve format problem in discussion again Signed-off-by: yuqi1129 <yuqi4733@gmail.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: add a new tick to renew lease in advance (#6427) * renew lease in raft tick Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check lease before sending read index Signed-off-by: 5kbpers <tangminghua@pingcap.com> * renew lease on raft tick to cover lease till next tick Signed-off-by: linning <linningde25@gmail.com> * refactor Signed-off-by: linning <linningde25@gmail.com> * don't use inspect_lease Signed-off-by: linning <linningde25@gmail.com> * fix ci Signed-off-by: linning <linningde25@gmail.com> * add test Signed-off-by: linning <linningde25@gmail.com> * remove unneed comment Signed-off-by: linning <linningde25@gmail.com> * close #5388 Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check pending_read_count before hibernating Signed-off-by: 5kbpers <tangminghua@pingcap.com> * increase the range to check lease expired Signed-off-by: 5kbpers <tangminghua@pingcap.com> * renew leader lease after not hibernate Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix hibernate states metrics Signed-off-by: 5kbpers <tangminghua@pingcap.com> * check writes and pre_read_index before proposing read index Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix test Signed-off-by: 5kbpers <tangminghua@pingcap.com> * use a individual tick to renewing lease Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix condition Signed-off-by: 5kbpers <tangminghua@pingcap.com> * stablize test_node_renew_lease Signed-off-by: 5kbpers <tangminghua@pingcap.com> * cleanup unused changes Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add config item for interval of tick Signed-off-by: 5kbpers <tangminghua@pingcap.com> * rename tick Signed-off-by: 5kbpers <tangminghua@pingcap.com> * disable the tick for test_inconsistent_configuration Signed-off-by: 5kbpers <tangminghua@pingcap.com> * change deafault value of check_leader_lease_interval Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * disable the tick for test_read_hibernated_region Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * count renew lease read index into raft metrics Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * disable the tick for test_renew_lease Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix lint error Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add a 100ms buffer for renew_bound Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * make clippy happy Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: linning <linningde25@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * cloud: Add retry for azure blob server busy error (#11813) * add retry for azure blob server busy error Signed-off-by: Leavrth <jianjun.liao@outlook.com> * close#11812 Signed-off-by: Leavrth <jianjun.liao@outlook.com> * use to_string Signed-off-by: Leavrth <jianjun.liao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * tikv_util: make cgroup parsing more robust (#11786) * make cgroup parsing more robust Signed-off-by: tabokie <xy.tao@outlook.com> * fix error message Signed-off-by: tabokie <xy.tao@outlook.com> * parse cpu quota in floats Signed-off-by: tabokie <xy.tao@outlook.com> * *: make it compile on macos (#11825) * *: make it compile on macos close #11823 Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * remove extra comment Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * resolved_ts: add more metrics (#11809) * add metrics Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add slow log Signed-off-by: 5kbpers <tangminghua@pingcap.com> * remove duplicate metric Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add resolved ts metrics for leader peers Signed-off-by: 5kbpers <tangminghua@pingcap.com> * close #11767 Signed-off-by: 5kbpers <tangminghua@pingcap.com> * init endpoint with store meta Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * rsmeter: fix perf regression caused by arc swap (#11833) * rsmeter: fix perf regression caused by arc swap Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * use atomic cell instead Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * debug assert Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Wenxuan <breezewish@pingcap.com> * load base split: more accurate sample for large key ranges batch (#11039) * more accurate sample for large key ranges Signed-off-by: lhy1024 <admin@liudos.us> * update Signed-off-by: lhy1024 <admin@liudos.us> * ref #11521 Signed-off-by: lhy1024 <admin@liudos.us> * address comment Signed-off-by: lhy1024 <admin@liudos.us> * fix test Signed-off-by: lhy1024 <admin@liudos.us> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftclient: enable delay only in high load (#11772) and improve CPU efficiency under high load. But it turns out this can lead to significant regression in normal load. So this PR adds CPU stats and only enabling delay in high load. This is also the same strategy used in v4.x. Close #11769. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * tikv-ctl: fix tikv-ctl's output is incomplete on calling process::exit (#11231) Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com> Co-authored-by: Connor <zbk602423539@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: skip deleting snapshot files in peer pending_remove is true (#11782) * raftstore: skip deleting snapshot files in peer when the peer is pending removal and the snapshot is being applied and canceled -- close #11746 This is to avoid the potential panic when the snapshot files are deleted, but the peer's status (Tombstone) is not persisted in disk due to tikv crash. Signed-off-by: tonyxuqqi <tonyxuqi@outlook.com> * address code review feedback -- close #11746 Signed-off-by: qi.xu <tonxuqi@outlook.com> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <tonxuqi@outlook.com> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <tonxuqi@outlook.com> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <tonxuqi@outlook.com> * address code review feedback 2 -- close #11746 Signed-off-by: qi.xu <tonxuqi@outlook.com> Co-authored-by: qi.xu <tonxuqi@outlook.com> * load_statis: fix compilation on macos (#11851) close #11772 Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * config: override rocksdb config when flow control enabled (#11840) Signed-off-by: Connor1996 <zbk602423539@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * rocksdb: consider env sharing when configuring background threads (#11760) #9649 increased the default setting of kvdb background flushes (3 for 8c and 4 for 16c). This PR takes another approach: use smaller concurrency for individual dbs, but configure the global thread pool size with the sum of kvdb and raftdb concurrency. This way we won't over-configure flush workers when raft-engine is enabled. * consider env sharing when configuring background threads ref #11119 Signed-off-by: tabokie <xy.tao@outlook.com> * server/config: keep compatible using option (#11862) * server/config: keep compatible using option Using `Option` for `heavy_load_wait_duration` to keep compatible with versions prior to v5.3. Close #11861. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * fix format Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * fix clippy Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: zhouqiang <zhouqiang.cl@gmail.com> * raftstore: fix missing workers' stop on shutdown (#11864) ref #11159 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: move in-memory pessimistic locks to split regions (#11655) ref #11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: Remove the entrance of enabling io snooper. (#11394) * *: Remove the entrance of enabling io snooper. close #10867 Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * fix Signed-off-by: MuZhou233 <MuZhou233@outlook.com> * fix test Signed-off-by: MuZhou233 <muzhou233@outlook.com> * fix test Signed-off-by: MuZhou233 <muzhou233@outlook.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: change PeerTicks to enum (#11849) * raftstore: change PeerTicks to enum close #11848 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * use bool array for registry Signed-off-by: Yilin Chen <sticnarf@gmail.com> * rename PeerTicks to PeerTick Signed-off-by: Yilin Chen <sticnarf@gmail.com> * add associated const for number of PeerTick types Signed-off-by: Yilin Chen <sticnarf@gmail.com> * change naming Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: check memory limit when inserting in-memory pessimistic locks (#11841) * *: check memory limit when inserting in-memory pessimistic locks ref #11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * fix merged upstream Signed-off-by: Yilin Chen <sticnarf@gmail.com> * fix test Signed-off-by: Yilin Chen <sticnarf@gmail.com> * fix typo Signed-off-by: Yilin Chen <sticnarf@gmail.com> * *: fix invalid failpoint caused by typo (#11709) * fix invalid failpoint caused by typo close #11734 Signed-off-by: Ryan Leung <rleungx@gmail.com> * address the comment Signed-off-by: Ryan Leung <rleungx@gmail.com> * address the comment Signed-off-by: Ryan Leung <rleungx@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: Extract significant router (#11750) * ref#11409 add significant router Signed-off-by: Connor1996 <zbk602423539@gmail.com> * make format Signed-off-by: Connor1996 <zbk602423539@gmail.com> * address comment Signed-off-by: Connor1996 <zbk602423539@gmail.com> * raftstore: move scan delete to raft log gc worker (#11853) * raftstore: move scan delete to raft log gc worker When clearing raft metas, raftstore will scan raft logs and delete them one by one. Seeking can be slow if there are a lot of tombstone keys. This PR moves the operation to raft log gc worker to reduce the impact. The final solution should be also moving remaining IO operations to async write IO threads. Close #10210. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * address comment Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * speed up destroy Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * fix compile Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * further speed up Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * revert test case configuration Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * address comment Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * address comment Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * doc: update new rules for linking issue and commit message (#11832) close #11097, close #11831 Signed-off-by: zhangyangyu <angwerzx@126.com> Co-authored-by: Mini256 <minianter@foxmail.com> Co-authored-by: Mini256 <minianter@foxmail.com> Co-authored-by: Xiaoguang Sun <sunxiaoguang@users.noreply.github.com> * *: update rust toolchain to 2022-01-07 (#11875) * update rust toolchain to 2022-01-17 Signed-off-by: tabokie <xy.tao@outlook.com> * address comment: clean up unnecessary `as_ref()` Signed-off-by: tabokie <xy.tao@outlook.com> * try fixing the underflow Signed-off-by: tabokie <xy.tao@outlook.com> * mute underflow warning for raft metrics Signed-off-by: tabokie <xy.tao@outlook.com> * clean up unused data members Signed-off-by: tabokie <xy.tao@outlook.com> * format Signed-off-by: tabokie <xy.tao@outlook.com> * step back to 2022-01-07 Signed-off-by: tabokie <xy.tao@outlook.com> * readme: change images based on github theme (#11795) Signed-off-by: sloorush <aarush.bhatt@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: don't remove other peer's read delegate (#11882) * don't remove read delegate besides peer_destroy Signed-off-by: linning <linningde25@gmail.com> * add test case Signed-off-by: linning <linningde25@gmail.com> * make clippy happy Signed-off-by: linning <linningde25@gmail.com> * address comment Signed-off-by: linning <linningde25@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * logger: use eprintln! if the logger is not initialized (#11869) ref #11651 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: fix the calculation of total column size in analyze (#11884) Signed-off-by: xuyifan <xuyifangreeneyes@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * engine: update raft-engine for data consistency fix (#11885) Fix https://github.com/tikv/raft-engine/issues/142. When encountering this bug, TiKV will panic with message "applied index > max(commit index, recorded commit index)" after restart. * update raft-engine Signed-off-by: tabokie <xy.tao@outlook.com> * update one more commit Signed-off-by: tabokie <xy.tao@outlook.com> * check docker build during clippy, ref #11312 (#11819) Signed-off-by: tabokie <xy.tao@outlook.com> * gc_worker: fix incorrect scheduled_tasks counting (#11904) * gc_worker: fix incorrect scheduled_tasks counting close #11903 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * remove check_is_busy totally Signed-off-by: Yilin Chen <sticnarf@gmail.com> * do not use wildcard match in error handling Signed-off-by: Yilin Chen <sticnarf@gmail.com> * raftstore: Introduce raft log fetcher (#11900) * ref#11409 introduce raft log fetcher Signed-off-by: Connor1996 <zbk602423539@gmail.com> * update kvproto Signed-off-by: Connor1996 <zbk602423539@gmail.com> * address comment Signed-off-by: Connor1996 <zbk602423539@gmail.com> * fix test build Signed-off-by: Connor1996 <zbk602423539@gmail.com> * fix clippy Signed-off-by: Connor1996 <zbk602423539@gmail.com> * limit capacity Signed-off-by: Connor1996 <zbk602423539@gmail.com> * update kvproto Signed-off-by: Connor1996 <zbk602423539@gmail.com> * call stop on worker Signed-off-by: Connor1996 <zbk602423539@gmail.com> * rename worker Signed-off-by: Connor1996 <zbk602423539@gmail.com> * status_server: add pprof flamegraph response header (#10951) * tweak(status_server): add pprof flamegraph response header Signed-off-by: Suhaha <jklopsdfw@gmail.com> * test(status_server): add Content-Type asset to test_pprof_profile_service Signed-off-by: Suhaha <jklopsdfw@gmail.com> * tweak(status_server): add pprof flamegraph response header Signed-off-by: Suhaha <jklopsdfw@gmail.com> * tweak(status_server): add pprof flamegraph response header close #11917 Signed-off-by: Suhaha <jklopsdfw@gmail.com> * close #11917 Signed-off-by: Suhaha <jklopsdfw@gmail.com> Co-authored-by: goroutine <ngaut@users.noreply.github.com> Co-authored-by: Connor <zbk602423539@gmail.com> * raftstore: renew leader lease in advance when handle read request (#9307) * renew lease advance Signed-off-by: linning <linningde25@gmail.com> * add log Signed-off-by: linning <linningde25@gmail.com> * make clippy happy Signed-off-by: linning <linningde25@gmail.com> * ref #11579 Signed-off-by: 5kbpers <tangminghua@pingcap.com> * reset raft tick Signed-off-by: 5kbpers <tangminghua@pingcap.com> * set has ready Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * add renew_leader_lease_advance_duration config Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix panic Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * address comment Signed-off-by: 5kbpers <tangminghua@pingcap.com> * disable renewing for test_lease_unsafe_during_leader_transfers Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: 5kbpers <tangminghua@pingcap.com> * pprof: ignore cpu profiling on non-x86 arch (#11925) * Disable cpu profiling on non-x86_64 arch Signed-off-by: mornyx <mornyx.z@gmail.com> * Fix warns Signed-off-by: mornyx <mornyx.z@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * github: add new pr requirement for linking issue and commit message (#11887) Signed-off-by: zhangyangyu <angwerzx@126.com> Co-authored-by: Xiaoguang Sun <sunxiaoguang@users.noreply.github.com> * engine: enable raft engine by default (#11928) * enable raft engine by default and synchronize docs Signed-off-by: tabokie <xy.tao@outlook.com> * update raft engine Signed-off-by: tabokie <xy.tao@outlook.com> * update raft-engine and fix tests Signed-off-by: tabokie <xy.tao@outlook.com> * gc_worker: Limit the key range to scan for GcKeys tasks (#11922) close tikv/tikv#11752, close tikv/tikv#11902 Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: TransferLeader support multiple target peers (#11063) ref tikv/tikv#822, ref tikv/tikv#4214, close tikv/tikv#10602 Signed-off-by: MrCroxx <mrcroxx@outlook.com> * raftstore: remove the leaders field of `StoreMeta` (#11934) close tikv/tikv#11933 Remove `StoreMeta.leaders` ``` ### Related changes ### Check List <!--REMOVE the items that are not applicable--> Tests <!-- At least one of them must be included. --> - Manual test (add detailed scripts or steps below) `cargo check` ### Release note <!-- bugfixes or new feature need a release note --> ```release-note None Signed-off-by: linning <linningde25@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: Fetch raft log in async manner (#11409) close tikv/tikv#10408, close tikv/tikv#11320 Signed-off-by: Connor1996 <zbk602423539@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: add quarter function (#11935) ref tikv/tikv#5751 Signed-off-by: zhongyong jin <AT-Fieldless@outlook.com> * Enable full debug info for dev and test (#11949) ref tikv/tikv#5049, ref tikv/tikv#5572, close tikv/tikv#5572 ``` ### Related changes - PR to update `pingcap/docs`/`pingcap/docs-cn`: - PR to update `pingcap/tidb-ansible`: - Need to cherry-pick to the release branch --> ### Check List <!--REMOVE the items that are not applicable--> Tests <!-- At least one of them must be included. --> - Manual test (add detailed scripts or steps below) - No code Manual test: Linux box with 8 cores (3:50 -> 4:00): ``` # Before this PR: $ make clean cargo clean rm -rf bin dist $ time make build cargo build --no-default-features --features "jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 ... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 3m 50s real 3m50,487s user 38m11,859s sys 3m9,540s # After this PR: $ git diff diff --git a/Cargo.toml b/Cargo.toml index 71f5329d3..67cb9d183 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -267,7 +267,7 @@ default-members = ["cmd/tikv-server", "cmd/tikv-ctl"] [profile.dev] opt-level = 0 -debug = 1 # required for line numbers in tests, see tikv #5049 +debug = true codegen-units = 4 lto = false incremental = true @@ -293,7 +293,7 @@ codegen-units = 4 [profile.test] opt-level = 0 -debug = 1 # enable line numbers by default for easy test debugging +debug = true codegen-units = 16 lto = false incremental = true mattias@msig:~/repos/tikv$ time make clean cargo clean rm -rf bin dist real 0m0,975s user 0m0,148s sys 0m0,828s mattias@msig:~/repos/tikv$ time make build cargo build --no-default-features --features "jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 ... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 4m 00s real 4m0,201s user 39m45,037s sys 3m16,397s ``` Macbook Air M1: ``` # Before: % time make build cargo build --no-default-features --features " jemalloc test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 .... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 4m 01s make build 1107.42s user 116.20s system 506% cpu 4:01.46 total # After: % time make build cargo build --no-default-features --features " jemalloc test-engines-rocksdb cloud-aws cloud-gcp cloud-azure" Compiling libc v0.2.106 .... Compiling server v0.0.1 (tikv/components/server) Finished dev [unoptimized + debuginfo] target(s) in 4m 10s make build 1179.39s user 120.74s system 518% cpu 4:10.98 total ``` Side effects - Performance regression, Only when building, to the benefit of full debug info by default. ### Release note <!-- bugfixes or new feature need a release note --> ```release-note None Signed-off-by: Mattias Jonsson <mjonss@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * correct a metric about compaction filter (#11938) correct a metric about compaction filter Signed-off-by: qupeng <qupeng@pingcap.com> * raft: Fix possible panic on entries fetched callback (#11961) close tikv/tikv#11951 Signed-off-by: Connor1996 <zbk602423539@gmail.com> * split_controller: refine the sample function and add some comments (#11952) Signed-off-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * metrics: fix grid position in TiKV Details (#11936) ref tikv/tikv#11119 Fix grid positions in TiKV Details dashboard. Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * cdc: capture old value from txn layer dynamically (#11896) close tikv/tikv#10091 Signed-off-by: hi-rustin <rustin.liu@gmail.com> Signed-off-by: qupeng <qupeng@pingcap.com> Co-authored-by: hi-rustin <rustin.liu@gmail.com> * split_controller: reorganize the structs and add more comments (#11967) Reorganize the structs and add more comments. Signed-off-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: support Bit column push downBit column (#11968) close tikv/tikv#11893, ref tikv/tikv#11893, ref pingcap/tidb#31884 Signed-off-by: xiejiandong <xiejiandong@pingcap.com> * copr: support push down mod, sysdate to TiKV (#11970) close tikv/tikv#11916, ref tikv/tikv#11916 Signed-off-by: xiejiandong <xiejiandong@pingcap.com> * raftstore: propose in-memory pessimistic locks before prepare merge (#11758) ref tikv/tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * stats_monitor: reformat stats_monitor tick (#11972) Signed-off-by: lhy1024 <admin@liudos.us> * copr: support extract scalar value from bit (#11980) close tikv/tikv#11893 Signed-off-by: yisaer <disxiaofei@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * diagnosis: support get cpu time of threads for macOS (#11978) close tikv/tikv#11977 Add the variant for macOS into `components/tikv_util/src/sys/thread.rs` Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Co-authored-by: Jay <BusyJay@users.noreply.github.com> * rust: update toolchain to fix missing rls (#11954) close tikv/tikv#11953 Signed-off-by: Connor1996 <zbk602423539@gmail.com> Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Co-authored-by: Yilin Chen <sticnarf@gmail.com> * cdc: separate resolved region outliers (#11991) Separate broadcasing outlier regions and normal regions, so 1) downstreams know where they should send resolve lock requests, and 2) resolved ts of normal regions does not fallback. close pingcap/tiflow#4516 close pingcap/tiflow#4311 ref pingcap/tiflow#4146 Signed-off-by: Neil Shen <overvenus@gmail.com> * *: collect key ranges for the read request in ResourceMeteringTag (#11995) close tikv/tikv#11988 *: collect key ranges for the read request in ResourceMeteringTag Signed-off-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * split_controller: move sample_threshold check out of the split_key iteration (#11986) split_controller: move sample_threshold check out of the split_key iteration Signed-off-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: support insertutf8 and lowerutf8 (#11987) ref tikv/tikv#5751 Signed-off-by: yisaer <disxiaofei@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: support rest greatest/least functions (#12003) ref tikv/tikv#5751 Signed-off-by: yisaer <disxiaofei@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: force compact with gentleness (#12006) * force compact with gentleness Signed-off-by: tabokie <xy.tao@outlook.com> * address comment Signed-off-by: tabokie <xy.tao@outlook.com> * split_controller: refine the LOAD_BASE_SPLIT_EVENT metrics label definitions (#12010) split_controller: refine the LOAD_BASE_SPLIT_EVENT metrics label definitions Signed-off-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * split_controller: fix the incorrect sampled key ranges number check (#12013) close tikv/tikv#12012 split_controller: fix the incorrect sampled key ranges number check Signed-off-by: JmPotato <ghzpotato@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: Support adjusting max_inflight_msgs dynamically (#11866) close tikv/tikv#11865 Signed-off-by: Wenbo Zhang <ethercflow@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: fix missing flag solvement for bit column in decoding (#12016) ref tikv/tikv#7, close pingcap/tidb#32506 Signed-off-by: yisaer <disxiaofei@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * file_system: overhaul proc-based io tracing utilities (#11873) ref tikv/tikv#10867, fix tikv/tikv#11775 Signed-off-by: tabokie <xy.tao@outlook.com> * raftstore: Wait for the apply index equals to the commit index (#11716) ref tikv/tikv#10483 Signed-off-by: v01dstar <yang.zhang@pingcap.com> * copr: revert 11968/11980/12016 (#12030) ref tikv/tikv#11968, ref tikv/tikv#11980, ref tikv/tikv#12016, ref pingcap/tidb#32506 Signed-off-by: yisaer <disxiaofei@163.com> * build: fix a Mac build issue (#12027) ref tikv/tikv#10867, ref tikv/tikv#11873 Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: Extra physical table id column (#11931) close tikv/tikv#11888 Added EXTRA_PHYSICAL_TABLE_ID_COL_ID support, to support TiDB's table partition dynamic prune mode, where a single request includes multiple partitions, but when Pessimistic lock (like `SELECT FOR UPDATE`) or ongoing transaction (having something in the tidb session local transaction buffer) each row needs its partition id / physical table ID to either lock that row or to check against the transaction buffer. Signed-off-by: Mattias Jonsson <mjonss@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: hotfix panic from tokio-timer (#12004) close tikv/tikv#11940, ref tikv/tikv#11940 *: hotfix panic from tokio-timer by enlarging the length of the level vector Signed-off-by: you06 <you1474600@gmail.com> * build: add arm64 check on SSE in makefile (#12035) close tikv/tikv#12034 Signed-off-by: Jin Dong <djdongjin95@gmail.com> Co-authored-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * replication mode: sync state to pd (#11751) Signed-off-by: disksing <i@disksing.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: revert pessimistic locks status when failing to propose PrepareMerge (#11985) ref tikv/tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * copr: support push bit column down (#12037) ref pingcap/tidb#30738 Signed-off-by: yisaer <disxiaofei@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * split_controller: add more sample function test cases (#12058) split_controller: add more sample function test cases Signed-off-by: JmPotato <ghzpotato@gmail.com> * copr: fix greatest/least time/date args type (#12056) Signed-off-by: yisaer <disxiaofei@163.com> * cdc: advancing resolved ts correctly for clusters with tiflash (#12050) close pingcap/tiflow#4461 cdc: advancing resolved ts correctly for clusters with tiflash Signed-off-by: qupeng <qupeng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: bump master branch version to v6.0.0-alpha (#12077) close tikv/tikv#12075 Signed-off-by: Yilin Chen <sticnarf@gmail.com> * raftstore: check uninitialized destroy for merge (#12055) close tikv/tikv#12048 When a peer is destroyed without being initialized, it will store itself to peer list and the region epoch is missing. In merge if such state is detected, it should abort merging instead of panicking. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: Add TruncateManager and TruncateWorker (#11553) ref tikv/rfcs#81 Signed-off-by: longfangsong <longfangsong@icloud.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * TiKV supports buckets (#11763) ref tikv/tikv#11759 Signed-off-by: qi.xu <tonxuqi@outlook.com> Co-authored-by: qi.xu <tonxuqi@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * api_version: Codec for RawKV key (#12036) ref tikv/tikv#11965 Key of RawKV is encoded as `user-key + memcomparable-padding + timestamp` in API v2. Signed-off-by: pingyu <yuping@pingcap.com> Co-authored-by: Andy Lok <andylokandy@hotmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: fix some typos (#12066) Signed-off-by: cuishuang <imcusg@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * raftstore: fix stale message cause panic (#12054) close tikv/tikv#12023 raftstore: fix stale message cause panic Signed-off-by: linning <linningde25@gmail.com> * raftstore: ignore async fetch result when the peer is pending removed (#12038) close tikv/tikv#11973, close tikv/tikv#12026 Signed-off-by: Connor1996 <zbk602423539@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: remove part about wechat in doc (#12101) close tikv/tikv#12100 Signed-off-by: jackwener <jakevingoo@gmail.com> * engine: upgrade raft engine (#12095) ref tikv/tikv#165, ref tikv/raft-engine#165, ref tikv/tikv#11119 Signed-off-by: Randy <ztelur@gmail.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com> * *: Fix possible undefined behavior for transmuting vec (#12096) close tikv/tikv#12070 Signed-off-by: Connor1996 <zbk602423539@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * github: Remove the pingcap/tidb-ansible from the PR template (#12102) close tikv/tikv#12103 Signed-off-by: hi-rustin <rustin.liu@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * copr: extract reschedule to inject a fail point (#12108) Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * raftstore: reactivate in-memory pessimistic locking when leader transfer fails (#11883) ref tikv/tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> * *: check memory locks for replica read only on the leader (#12115) close tikv/tikv#12109 Consider a reader sends read index to the leader it supposed to be, but when the leader receives the read index message, it has stepped down to a follower. Without this commit, a peer will check the locks and re-fill the read index context with the result no matter what role it is. So, when the read index request is redirected to the new leader again, it no longer carries the context of lock checking. This commmit changes to only do the check when the peer is a leader. Then, the read index request will remain unchanged before being redirected to the leader. If the lease of the leader expires, it is still safe to check on the uncertained leader. If the heartbeat check passes and no new leader is elected, then the check works fine. If there is a new leader, when the old leader becomes a follower, it will clear its pending read index. Then, the reader has to resend th…
WangLe1321
pushed a commit
to WangLe1321/tikv
that referenced
this pull request
Jun 23, 2022
br: remove metrics after task stopped
coocood
added a commit
that referenced
this pull request
Jun 23, 2022
WangLe1321
added a commit
to WangLe1321/tikv
that referenced
this pull request
Jun 23, 2022
commit 900dd03 Merge: 9fda729 8eceb1f Author: Zak Zhao <57036248+joccau@users.noreply.github.com> Date: Thu Jun 23 12:12:48 2022 +0800 Merge pull request tikv#9 from joccau/fix-limit-fluent log-backup: limit concurrent fluent when write kv-files to storage commit 8eceb1f Author: joccau <zak.zhao@pingcap.com> Date: Thu Jun 23 11:21:20 2022 +0800 move local variables to const Signed-off-by: joccau <zak.zhao@pingcap.com> commit 4465085 Author: joccau <zak.zhao@pingcap.com> Date: Wed Jun 22 16:40:48 2022 +0800 add unit test Signed-off-by: joccau <zak.zhao@pingcap.com> commit 8ef93a1 Author: joccau <zak.zhao@pingcap.com> Date: Wed Jun 22 00:37:46 2022 +0800 limit the fluent when write kv file, ref issues#12869 Signed-off-by: joccau <zak.zhao@pingcap.com> commit 9fda729 Merge: d219d84 acc06c3 Author: Zak Zhao <57036248+joccau@users.noreply.github.com> Date: Wed Jun 15 17:28:24 2022 +0800 Merge pull request tikv#8 from YuJuncen/update-max-ts log_backup: update max ts commit acc06c3 Author: Yu Juncen <yujuncen@pingcap.com> Date: Wed Jun 15 16:37:33 2022 +0800 *: update max ts Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit d219d84 Merge: baf35ea e6d1dfe Author: Zak Zhao <57036248+joccau@users.noreply.github.com> Date: Wed Jun 15 13:50:32 2022 +0800 Merge pull request tikv#7 from WangLe1321/feat/br_stream_metric br: remove metrics after task stopped commit e6d1dfe Author: WangLe1321 <wangle1321@163.com> Date: Tue Jun 14 17:55:47 2022 +0800 br: modify to recieve returned value commit 9b1c101 Author: WangLe1321 <wangle1321@163.com> Date: Tue Jun 14 17:47:20 2022 +0800 feat: check result after remove metric commit 2619864 Author: WangLe1321 <wangle1321@163.com> Date: Tue Jun 14 15:11:38 2022 +0800 br: remove metrics when task stopped commit baf35ea Merge: b1d493c 4b6ee85 Author: Zak Zhao <57036248+joccau@users.noreply.github.com> Date: Mon Jun 13 10:55:23 2022 +0800 Merge pull request tikv#6 from joccau/error-safe-point log-backup: update flush time if succeeding in do_flush commit 4b6ee85 Author: joccau <zak.zhao@pingcap.com> Date: Sat Jun 11 17:38:47 2022 +0800 update flush time if succeeding in do_flush, ref issue#12802 Signed-off-by: joccau <zak.zhao@pingcap.com> commit b1d493c Merge: 0e445f8 c9b5f06 Author: Zak Zhao <57036248+joccau@users.noreply.github.com> Date: Sat Jun 11 13:51:02 2022 +0800 Merge pull request tikv#5 from joccau/cherry-pick-12770 log_backup: adapt the enhanced checkpoint model commit c9b5f06 Author: Yu Juncen <yujuncen@pingcap.com> Date: Thu Jun 9 11:58:33 2022 +0800 fix build && make clippy happy Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 7b83eec Author: Yu Juncen <yujuncen@pingcap.com> Date: Mon Jun 6 15:46:26 2022 +0800 fix build && make clippy happy Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 228cf35 Author: Yu Juncen <yujuncen@pingcap.com> Date: Mon Jun 6 11:07:46 2022 +0800 Squashed commit of the following: commit dca2e4efec59a47299d56c9469164f12eef978b7 Author: Yu Juncen <yujuncen@pingcap.com> Date: Thu Jun 2 18:12:13 2022 +0800 make slash_etc support mvcc Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 0ead359 Author: Yu Juncen <yujuncen@pingcap.com> Date: Tue May 31 19:13:43 2022 +0800 *: make Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 751f0fa Author: Yu Juncen <yujuncen@pingcap.com> Date: Tue May 31 14:04:33 2022 +0800 make 2-phase flushing Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit a0981dc Author: Yu Juncen <yujuncen@pingcap.com> Date: Tue May 31 13:07:16 2022 +0800 make initial scanning sync with region operator Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 5b091ea Author: Yu Juncen <yujuncen@pingcap.com> Date: Mon May 30 13:44:39 2022 +0800 WIP: register region checkpoint, keep checkpoint until flush Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit c0fa5dd Author: Yu Juncen <yujuncen@pingcap.com> Date: Sat May 28 20:50:16 2022 +0800 added wait group Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit cf2c4ff Author: Yu Juncen <yujuncen@pingcap.com> Date: Sat May 28 19:58:03 2022 +0800 Squashed commit of the following: commit 09e9aa53ffdf320a956409b7b220ff1f68283b47 Author: Yu Juncen <yujuncen@pingcap.com> Date: Sat May 28 13:01:49 2022 +0800 basic change Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 505bcc7c548165a55ac22c5258ee39f0ab7a8937 Merge: ac005ba42 93fecdc Author: Yu Juncen <yujuncen@pingcap.com> Date: Fri May 27 11:59:00 2022 +0800 Merge branch 'master' of https://github.com/tikv/tikv into checkpoint-v2 commit ac005ba423289bf656dc27aff7087a34a0f43400 Author: Yu Juncen <yujuncen@pingcap.com> Date: Mon May 23 18:17:05 2022 +0800 WIP Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 4d0efff84f61818b1403b7d83a4be8e285889b79 Author: Yu Juncen <yujuncen@pingcap.com> Date: Fri May 20 19:00:12 2022 +0800 WIP: mark electing regions Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 10b4acb Author: Yu Juncen <yujuncen@pingcap.com> Date: Fri May 20 13:02:26 2022 +0800 using checkpoint v2 Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 929ea27 Merge: e336ec6 4703705 Author: Yu Juncen <yujuncen@pingcap.com> Date: Thu May 19 21:14:12 2022 +0800 Merge branch 'master' of https://github.com/tikv/tikv into inflight-detect Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit e336ec6 Author: Yu Juncen <yujuncen@pingcap.com> Date: Thu May 19 21:12:50 2022 +0800 wait longer for flushing· Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit b5abdf8 Merge: 4c600ac 8c02d7b Author: Yu Juncen <yujuncen@pingcap.com> Date: Thu May 19 17:48:24 2022 +0800 Merge branch 'master' of https://github.com/tikv/tikv into inflight-detect commit 4c600ac Author: Yu Juncen <yujuncen@pingcap.com> Date: Thu May 19 17:18:14 2022 +0800 fix? test Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit d9430da Author: Yu Juncen <yujuncen@pingcap.com> Date: Thu May 19 16:00:22 2022 +0800 fix? test Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 33c8b8d Merge: 41c6e5a 500c4d6 Author: Yu Juncen <yujuncen@pingcap.com> Date: Thu May 19 14:12:18 2022 +0800 Merge branch 'inflight-detect' of https://github.com/yujuncen/tikv into inflight-detect commit 41c6e5a Author: Yu Juncen <yujuncen@pingcap.com> Date: Thu May 19 14:12:03 2022 +0800 fix status transform Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 500c4d6 Merge: 4cf0b17 486761a Author: 山岚 <36239017+YuJuncen@users.noreply.github.com> Date: Thu May 19 13:30:49 2022 +0800 Merge branch 'master' into inflight-detect commit 4cf0b17 Author: Yu Juncen <yujuncen@pingcap.com> Date: Thu May 19 01:05:37 2022 +0800 fix bad safepoint location Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit fbaf668 Merge: b390173 9c78e8b Author: Yu Juncen <yujuncen@pingcap.com> Date: Thu May 19 01:04:43 2022 +0800 Merge branch 'master' of https://github.com/tikv/tikv into inflight-detect commit b390173 Author: Yu Juncen <yujuncen@pingcap.com> Date: Wed May 18 22:53:39 2022 +0800 added test Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit f05af56 Author: Yu Juncen <yujuncen@pingcap.com> Date: Wed May 18 18:01:08 2022 +0800 fix inf waiting Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 363d264 Author: Yu Juncen <yujuncen@pingcap.com> Date: Wed May 18 16:47:11 2022 +0800 don't advance checkpoint ts if there are inflight StartObserve Signed-off-by: Yu Juncen <yujuncen@pingcap.com> Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit cc70212 Author: Yu Juncen <yujuncen@pingcap.com> Date: Tue May 31 14:27:34 2022 +0800 Squashed commit of the following: commit 605ec49e3686352903fb8627213f1faafe446a96 Author: Yu Juncen <yujuncen@pingcap.com> Date: Sat May 28 19:56:30 2022 +0800 fix build Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit ce046e0 Merge: 9c6d827 4facb62 Author: Yu Juncen <yujuncen@pingcap.com> Date: Sat May 28 19:51:09 2022 +0800 Merge branch 'master' of https://github.com/tikv/tikv into refactor-region-op Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 9c6d827 Author: Yu Juncen <yujuncen@pingcap.com> Date: Mon May 16 18:59:53 2022 +0800 remove file on drop Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit d3408c1 Author: Yu Juncen <yujuncen@pingcap.com> Date: Mon May 16 17:54:01 2022 +0800 set checkpoint to global checkpoint once start Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit c209007 Author: Yu Juncen <yujuncen@pingcap.com> Date: Mon May 16 17:10:32 2022 +0800 fix tests Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 1569fa6 Author: Yu Juncen <yujuncen@pingcap.com> Date: Mon May 16 15:09:10 2022 +0800 refactor region op Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 09edd39 Author: Yu Juncen <yujuncen@pingcap.com> Date: Sun May 15 14:08:19 2022 +0800 fix scan pool name Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit ac31fb4 Author: Yu Juncen <yujuncen@pingcap.com> Date: Sun May 15 14:01:24 2022 +0800 tweaking metrics Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit e11eb3d Author: Yu Juncen <yujuncen@pingcap.com> Date: Sun May 15 13:25:57 2022 +0800 add some metrics - Task Status - Internal Task (Actor Messages) Handling time Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit a9d2bfa Author: Yu Juncen <yujuncen@pingcap.com> Date: Sat May 14 23:09:59 2022 +0800 address comments Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 9bc5605 Merge: 3faeeb2 744d688 Author: 山岚 <36239017+YuJuncen@users.noreply.github.com> Date: Sat May 14 16:25:50 2022 +0800 Merge branch 'master' into fix-test commit 3faeeb2 Author: Yu Juncen <yujuncen@pingcap.com> Date: Sat May 14 15:47:41 2022 +0800 check observer version when accepting new events. For log backup, it is ok to repeat data, but when lock repeated when observe changing, it may make lock failed to be unlocked. Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit b746aa1 Author: Yu Juncen <yujuncen@pingcap.com> Date: Sat May 14 11:07:03 2022 +0800 make clippy happy Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 3fec0d7 Merge: ef44a4d de208b5 Author: Yu Juncen <yujuncen@pingcap.com> Date: Sat May 14 10:38:43 2022 +0800 Merge branch 'master' of https://github.com/tikv/tikv into fix-test Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit ef44a4d Author: Yu Juncen <yujuncen@pingcap.com> Date: Fri May 13 23:35:56 2022 +0800 add delay when start Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 647810f Author: Yu Juncen <yujuncen@pingcap.com> Date: Fri May 13 14:54:44 2022 +0800 make test more stable && handle epoch not match Signed-off-by: Yu Juncen <yujuncen@pingcap.com> Signed-off-by: Yu Juncen <yujuncen@pingcap.com> Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 1eca46b Author: Yu Juncen <yujuncen@pingcap.com> Date: Thu Jun 2 23:45:08 2022 +0800 make foramt Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 94d6e0b Author: Yu Juncen <yujuncen@pingcap.com> Date: Thu Jun 2 20:18:14 2022 +0800 implement checkpoint v2 Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit 0e445f8 Merge: 835436c 1698814 Author: Zak Zhao <57036248+joccau@users.noreply.github.com> Date: Sat Jun 11 12:19:25 2022 +0800 Merge pull request tikv#4 from joccau/cherry-pick-12712 security, server: fix running local test (tikv#12712) commit 1698814 Author: 山岚 <36239017+YuJuncen@users.noreply.github.com> Date: Wed Jun 1 18:36:28 2022 +0800 security, server: fix running local test (tikv#12712) close tikv#12711 Signed-off-by: Yu Juncen <yujuncen@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> commit 835436c Merge: a4580f8 621c9de Author: Zak Zhao <57036248+joccau@users.noreply.github.com> Date: Sat Jun 11 12:11:26 2022 +0800 Merge pull request tikv#3 from joccau/cherry-pick-12757 br: log-backup: fix bug about restoring point at TiCloud with KMS tikv#12757 commit a4580f8 Merge: d769651 4a29d32 Author: Zak Zhao <57036248+joccau@users.noreply.github.com> Date: Sat Jun 11 12:11:08 2022 +0800 Merge pull request tikv#2 from joccau/cherry-pick-12710 br: log backup: disable test log by default (tikv#12710) commit d769651 Merge: c26134b ceb547e Author: Zak Zhao <57036248+joccau@users.noreply.github.com> Date: Sat Jun 11 12:10:35 2022 +0800 Merge pull request tikv#1 from joccau/cherry-pick-12640 br: log-backup: adapt user-defined TLS config (tikv#12640) commit 621c9de Author: Zak Zhao <57036248+joccau@users.noreply.github.com> Date: Tue Jun 7 18:16:30 2022 +0800 log-backup: fix bug about restoring point at TiCloud with KMS (tikv#12757) close tikv#12750, close tikv#12751 Signed-off-by: joccau <zak.zhao@pingcap.com> commit 4a29d32 Author: 山岚 <36239017+YuJuncen@users.noreply.github.com> Date: Wed Jun 1 00:42:27 2022 +0800 log backup: disable test log by default (tikv#12710) close tikv#12709 Signed-off-by: Yu Juncen <yujuncen@pingcap.com> commit ceb547e Author: 山岚 <36239017+YuJuncen@users.noreply.github.com> Date: Mon May 30 16:52:27 2022 +0800 log-backup: adapt user-defined TLS config (tikv#12640) close tikv#12668 log-backup: adapt user-defined TLS config Signed-off-by: Yu Juncen <yujuncen@pingcap.com> Co-authored-by: kennytm <kennytm@gmail.com> Signed-off-by: WangLe1321 <wangle1321@163.com>
Leavrth
added a commit
to Leavrth/tikv
that referenced
this pull request
Aug 2, 2023
* filter out default_cf keys without committed write_cf Signed-off-by: Leavrth <jianjun.liao@outlook.com> * fix some bugs Signed-off-by: Leavrth <jianjun.liao@outlook.com> --------- Signed-off-by: Leavrth <jianjun.liao@outlook.com>
mittalrishabh
pushed a commit
to mittalrishabh/tikv
that referenced
this pull request
May 8, 2024
…#15937) (tikv#7) close tikv#15919 A "stale peer" refers to a peer that still exists on a TiKV node but has been removed from the raft group, typically through a confchange operation. TiKV performs regular checks and validations on its peers to ensure that no such stale peer exists. However, the current stale peer check is not enough when dealing with uninitialized stale peers that its region has been merged. These uninitialized stale peers are left indefinitely, consuming CPU, memory, and blocking resolved ts. This commit introduces an in-memory state for peers whose raft messages create uninitialized stale peers. The stale peer then sends a MsgCheckStalePeer message to the corresponding peer, validating whether it should be removed. Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.