Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UB in TsSet conversions #12070

Closed
dtolnay opened this issue Mar 2, 2022 · 4 comments · Fixed by #12096
Closed

UB in TsSet conversions #12070

dtolnay opened this issue Mar 2, 2022 · 4 comments · Fixed by #12096

Comments

@dtolnay
Copy link

dtolnay commented Mar 2, 2022

let ts = unsafe { ::std::mem::transmute::<Vec<u64>, Vec<TimeStamp>>(ts) };

let ts = unsafe { ::std::mem::transmute::<Vec<u64>, Vec<TimeStamp>>(ts) };

This unsafe cast is UB because Vec<u64> and Vec<TimeStamp> don't have the same layout in general. For example a Rust implementation using profile-guided optimization might put the vec's fields in a different order for the two types, due to different access patterns across all the places those 2 different instantiations are used (e.g. one might store the len first while the other might store the capacity first, because len() is called more often on one than the other).

@sticnarf
Copy link
Contributor

sticnarf commented Mar 3, 2022

Thanks for pointing this out.

I find .into_iter().map().collect() can be optimized out if they really have the same layout:

https://godbolt.org/z/an9jxa7v8

I believe we can and should use a safe way to implement it here. But as you say the behavior is actually not guaranteed. Do you have any suggestion for confirming the optimization so we won't get performance regression unexpectedly?

@BusyJay
Copy link
Member

BusyJay commented Mar 3, 2022

How about

let (ptr, len, cap) = ts.into_raw_parts();
let ts = unsafe { Vec::from_raw_parts(ptr as _, len, cap) };

@Connor1996
Copy link
Member

Thanks! We notice that the Clippy complains about it after upgrading to recent rust nightly. So are the layout of the two types promised to be the same for the previous rust version? @dtolnay

@sticnarf
Copy link
Contributor

sticnarf commented Mar 3, 2022

Thanks! We notice that the Clippy complains about it after upgrading to recent rust nightly. So are the layout of the two types promised to be the same for the previous rust version? @dtolnay

I believe not. It's never guaranteed to be the same. It's just clippy that adds the lints recently.

Connor1996 added a commit to Connor1996/tikv that referenced this issue Mar 4, 2022
Signed-off-by: Connor1996 <zbk602423539@gmail.com>
ti-chi-bot added a commit that referenced this issue Mar 7, 2022
close #12070

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
CalvinNeo added a commit to pingcap/tidb-engine-ext that referenced this issue May 18, 2022
* raftstore: support dynamically scaling pool size (#11159) (#11213)

close #11159

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* storage: reduce extra next/pre in row_scan (#11625)

* reduce extra next/pre in row_scan
close https://github.com/tikv/tikv/issues/11572
Signed-off-by: Jayice <1185430411@qq.com>

* reduce extra next/pre in row_scan
close #11572
Signed-off-by: Jayice <1185430411@qq.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: remove some unnecessary dependencies (#11592)

close #11593

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: Yilin Chen <sticnarf@gmail.com>

* raftstore: make threads of snap-generator pool configurable (#11274)

* add snap-generator-pool-size for configurable snapshot generator thread pool

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

* rename config macro

Signed-off-by: iosmanthus <myosmanthustree@gmail.com>

Co-authored-by: Andy Lok <andylokandy@hotmail.com>

* ctl:add prompt for ctl when required param is not specified (#11639)

* add prompt for ctl
close #11514
Signed-off-by: Jayice <1185430411@qq.com>

* format code
close #11514
Signed-off-by: Jayice <1185430411@qq.com>

* format code
close #11514
Signed-off-by: Jayice <1185430411@qq.com>

* format code
close #11514
Signed-off-by: Jayice <1185430411@qq.com>

* optimize duplicate codes:pd check
close #11514
Signed-off-by: Jayice <1185430411@qq.com>

* move pd check in get_pd_rpc_client
close #11514
Signed-off-by: Jayice <1185430411@qq.com>

* simplify the codes check host|data-dir
close #11514
Signed-off-by: Jayice <1185430411@qq.com>

* ctl: validate column family name when scanning data using ctl (#11269)

* ctl: should check column name validation when scan data using ctl

close #11268
Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* fix format problem

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>

* txn: Do assertions on data for mutations to reduce potential data inconsistency (#10977)

* Add mutation checking logic

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix build

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix usages of mutation

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Update kvproto and renaming

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add a simple test

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix incorrect assertion

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Support check existance during pessimistic lock

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add assertion level support

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add tests

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add more test cases

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add test for check_existence of pessimistic_lock request

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix build

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* ref #11542

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add a metric for counting different way of checking assertion, for reference about the performance impact

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* metrics: fix memory metrics (#11656)

Signed-off-by: Shuaipeng Yu <jackysp@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* coprocesser: refactor encoding and add gbk_chinese_ci map (#11661)

* close #11660

Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com>

* close #11660

Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* storage: Add API V2 check for RawKV and TxnKV requests (txn part) (#11216)

* wip

Signed-off-by: pingyu <yuping@pingcap.com>

* wip

Signed-off-by: pingyu <yuping@pingcap.com>

* ref #10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref #10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref #10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref #10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref #10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref #10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* fix enable_ttl

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* simplify test

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* ref #10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

Co-authored-by: andylokandy <andylokandy@hotmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* cdc: load uninlined value more effectively (#11615)

Signed-off-by: qupeng <qupeng@pingcap.com>

* cdc: load old value with prefix seek if possible (#11643)

* close #11642

Signed-off-by: qupeng <qupeng@pingcap.com>

* a more comment

Signed-off-by: qupeng <qupeng@pingcap.com>

* ctl: split into multiple modules (#11658)

* ctl: split into multiple modules

ref #10938

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* refactor if let to match

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* Fix build issues on non-linux platforms (#11671)

* close #11676

Signed-off-by: devillove084 <786537003@qq.com>

* all: Migrate from std:: types to primitives. (#11677)

* Deprecation is planned for MIN,MAX,EPSLION,INFINITY,NEG_INFINITY of
  the (i|u|f)(8|16|32|64) types in standard.

ref #11678

```release-note
NONE
```

Signed-off-by: Harold Dost <harolddost@gmail.com>

* metrics alert: fix apply cpu alert (#11672)

Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* resrc_mtr: introduce scanned keys statistics (#11085)

close #11690

Signed-off-by: mornyx <mornyx.z@gmail.com>

* file_system: support collect the io flow by proc (#11667)

* file_system: support collect the read flow by proc

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* fix lint

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* fix clippy

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* Delete unused content & Adjust copyright text

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Add bench_fetch_io_bytes

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* fix lint

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Integrate thread_io

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* record bytes in request

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* fix logical bug

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Update test write_bytes

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Delete unused & Rename func

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Add test for read & Update test for write

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Add buffered thread io

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Replace DashMap with ThreadLocal

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* Rename variables and function & Remove unused call

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Change fetch logic. close #10867

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Add AtomicIOBytes

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Split ThreadIOSentinel

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* add THREAD_IO_TOTAL for metrics bytes fetcher

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Adjust code style & fix

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* improve write_bytes test

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: check epoch and validness of lock table when writing pessimistic locks (#11591)

ref #11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* rsmeter: introduce datasink (#11688)

* add datasink trait

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* RAII removes on_reporter_closing

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fmt

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix build

ref #11691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* datasink -> data_sink

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fmt

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix build

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* clippy: Fixing up clippy errors. (#11193)

* clippy: Fixing up clippy errors.

* Fix crate references
* Fix anonymous lifetimes references.

ref #4301
Signed-off-by: Harold Dost <harolddost@gmail.com>

* Test Revert of Iterations

ref #4301

Signed-off-by: Harold Dost <harolddost@gmail.com>

* clippy: Enable check for rust-2018-idioms

ref #4301

Signed-off-by: Harold Dost <harolddost@gmail.com>

* rsmeter: add centralized place to construct protobuf data (#11696)

ref #11691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* tikv_util: update procfs to 0.12.0 (#11703)

Signed-off-by: linning <linningde25@gmail.com>

* Use generic for api version to reduce runtime branching (#11687)

* Use generic for api version to reduce runtime branching

ref #10938

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* fix test

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* add test

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* fix clippy

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* fmt

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* rename vars

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* improve vars

Signed-off-by: andylokandy <andylokandy@hotmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* sst_importer: check api version when importing (#11664)

* sst_importer: check api version when importing

ref #10938

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Fix bugs

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* fix bug of Iterator::{scan,scan_cf}

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* remove redundant log

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Show the entire key

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Fix lint issue

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* resolve conflict

Signed-off-by: andylokandy <andylokandy@hotmail.com>

Co-authored-by: Andy Lok <andylokandy@hotmail.com>

* backup: pipeline scan and save step (#11528)

* br: sperate io and scan threads

Signed-off-by: Yu Juncen <yu745514916@live.com>

* br/stream: make clippy happy

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: fix tests

Signed-off-by: Yu Juncen <yu745514916@live.com>

* backup: fix a dummy bug

Signed-off-by: Yu Juncen <yu745514916@live.com>

* backup: add config of io-threads

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: ref #11350: some minior change

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: added some metrics

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: better tuning concurrency, and added a metric

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: added some slow log and metrics

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: address comments

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: fix build

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* Revert "backup: fix build"

This reverts commit 74537be56c410d0f91af2f48b7e65356ab53720e.

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* Revert "backup: address comments"

This reverts commit 77d75756028bea3496f878f0ad8bbc562f5d00aa.

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: address comments

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: always set IO type to Export

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup:make clippy happy

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: address comments

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* metrics: fix the metrics cannot be displayed (#11710)

ref #11662

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* backup: return api-version to BR when backup (#11704)

* Update kvproto

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Return api-version for br

ref #10938

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Reformat code

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Update components/external_storage/export/src/export.rs

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

Co-authored-by: Andy Lok <andylokandy@hotmail.com>

* format code

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

Co-authored-by: Andy Lok <andylokandy@hotmail.com>

* file_system: limit thread_io target os (#11715)

* limit thread_io target os

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* limit thread_io target os. typo. close #11698

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

Co-authored-by: Yilin Chen <sticnarf@gmail.com>

* file_system: bypass file allocate implementation in fs2 (#11700)

Close #10688

Patch fs2 to https://github.com/tabokie/fs2-rs/tree/tikv. In which, `posix_fallocate` is replaced with `fallocate` (https://github.com/danburkert/fs2-rs/pull/42).

* bypass file allocate implementation in fs2

Signed-off-by: tabokie <xy.tao@outlook.com>

* metrics: remove min legend of some Grafana panels (#11487)

Minimum legend isn't very useful, and it occupied valuable visual real estate.

Remove some min legends from Grafana metrics, except for the following:
- Level 0 chance
- Number files at each level
- File Count
- Entry Count
- Allocator Stats
- Encrypted files
- Encryption meta files size

Ref #11119

Signed-off-by: tabokie <xy.tao@outlook.com>

* coprocessor: not return rows when there is no input for simple aggregation (#11708)

* not return rows when there is no input for simple aggregation

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

* close #11735, and address comments

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

* backup: reduce the default thread pool size of backup, enable auto tune by default (#11699)

* backup: clamp auto tune values

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* ref #11000: change the default backup threads.

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: run tasks in background

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: run rustfmt

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: set remain threads to 20% of vcpu

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br-stream: fix build

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* Backup: add S3 metrics && add s3_multi_part_size config  (#11666)

* s3: add request metrics for s3 storage

Signed-off-by: 3pointer <luancheng@pingcap.com>

* s3: add grafana json

Signed-off-by: 3pointer <luancheng@pingcap.com>

* br: add config for s3 multi part upload

Signed-off-by: 3pointer <luancheng@pingcap.com>

* update comment

Signed-off-by: 3pointer <luancheng@pingcap.com>

* update comment

Signed-off-by: 3pointer <luancheng@pingcap.com>

* address comment

Signed-off-by: 3pointer <luancheng@pingcap.com>

* hidden the new config

Signed-off-by: 3pointer <luancheng@pingcap.com>

* only hidden config in config-template.toml

Signed-off-by: 3pointer <luancheng@pingcap.com>

* address comment

Signed-off-by: 3pointer <luancheng@pingcap.com>

* close #11727 && format

Signed-off-by: 3pointer <luancheng@pingcap.com>

* fix

Signed-off-by: 3pointer <luancheng@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* rsmeter: support multiple datasinks (#11707)

* rsmeter: support multiple datasinks

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* wrap config notifier

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* add unit tests

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* polish
ref #11691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix test

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix outdated comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* engine: properly estimate store size (#11728)

* properly estimate store size, ref #11119

Signed-off-by: tabokie <xy.tao@outlook.com>

* address comments

Signed-off-by: tabokie <xy.tao@outlook.com>

* implement global config client for tikv (#11685)

* implement global config client

close #11686

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* implement unit test for global config in pd client

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* resolved formatting issues

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* impl pd client asyncly

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* according to reviewer's opinion
removing store method implementation

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* Merge branch 'master' of https://github.com/tikv/tikv into global_conf

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* write test case for watch global config when
grpc server is closed.

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* resove issues

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* clippy

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* watch global config rentry future in test deadlock just removed

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

Co-authored-by: Zhenchi <zhongzc_arch@outlook.com>

* storage: skip scanning lock when using rc (#11701)

* skip scanning lock when using rc, ref #11485

Signed-off-by: you06 <you1474600@gmail.com>

* format code

Signed-off-by: you06 <you1474600@gmail.com>

* add test

Signed-off-by: you06 <you1474600@gmail.com>

* address comment

Signed-off-by: you06 <you1474600@gmail.com>

* add an integration test

Signed-off-by: you06 <you1474600@gmail.com>

* format code

Signed-off-by: you06 <you1474600@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: pushdown substring to tikv (#11494)

* corp: pushdown substring to tikv

Signed-off-by: guo-shaoge <shaoge1994@163.com>

* fix case. close #11495

Signed-off-by: guo-shaoge <shaoge1994@163.com>

* fix comment

Signed-off-by: guo-shaoge <shaoge1994@163.com>

* rsmeter: distinguish between collectors and observers within recorder  (#11712)

* rsmeter: support multiple datasinks

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* wrap config notifier

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* add unit tests

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* polish
ref #11691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* rsmeter: distinguish between collectors and observers within recorder

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fmt & try to resolve #11689

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* add unit tests
ref #11691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* retrigger test

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix test

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix outdated comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* Update components/resource_metering/src/recorder/sub_recorder/mod.rs

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Yexiang Zhang <mornyx.z@gmail.com>

* address comment

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* remove enabled

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Co-authored-by: Wenxuan <breezewish@pingcap.com>
Co-authored-by: Yexiang Zhang <mornyx.z@gmail.com>

* config: validate online configurable thread pools (tikv#11159) (#11714)

ref #11159

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* res_meter: upgrade kvproto (#11749)

* Upgrade kvproto

close #11748

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Fix test compilation

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Fix test compilation

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Fix test compilation

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Use master kvproto

Signed-off-by: mornyx <mornyx.z@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* rsmeter: add pubsub datasink  (#11719)

* rsmeter: add pubsub datasink
close #11691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix build

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* Update components/resource_metering/src/reporter/pubsub.rs

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Wenxuan <hi@breeswish.org>

* fix build

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Wenxuan <hi@breeswish.org>

* status_server: Support online config update for configuration hosting platform (#11693)

close #11692

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* res_meter: optimize top k (#11753)

* Keep top k on recorder

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Modify comments

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Fix pd.rs

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Refactor

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Remove import

Signed-off-by: mornyx <mornyx.z@gmail.com>

* max_resource_groups 200 -> 100

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Fix ut

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Use iterator to avoid clone

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Optimize

close #11754

Signed-off-by: mornyx <mornyx.z@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftclient: delay flush (#11705)

* raftclient: delay flush

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* remove raftstore wait

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* Ref #11309.

tikv#11310 introduces delay in raftstore thread to reduce the flush rate.
It can only reduce flush rate from one thread. If there are many
raftstore threads or IO threads, messages can still be flush frequently.
This PR reduces the flush rate by introducing delay in raft client. So
delay will work at connection level and achieve the maximun batch
affect.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* remove useless field

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* address comment

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* fix unstable raw get case

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Liqi Geng <gengliqiii@gmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* cloud: support azure blob storage as external storage backend (#11732)

* support azure storage blob for external storage

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* modify the priority of the azure login methods

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* fix feature match branch

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* implement token update retriable

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* fix some problems

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* close #11731

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* rustfmt

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* make clippy success

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* simplify uploader

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* commit some suggestions

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* coprocesser: update encoding lib (#11506)

Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* coprocesser: implement upper and lower function on GBK charset (#11756)

Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* logger: support archiving and rotation (#11651) (#11657)

* logger: support archiving and rotation (#11651)

close #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: add testcases to test dt_from_file_name (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: simplify DateTime format (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: do a little refactoring (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: do a little refactoring (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: do a little refactoring (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: do a little refactoring (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: do a little refactoring (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* diagnostics: refactor  (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: run clippy for integration tests (#11768)

* run clippy --fix with nightly-2021-12-17

Signed-off-by: tabokie <xy.tao@outlook.com>

* manual clippy

Signed-off-by: tabokie <xy.tao@outlook.com>

* format and clippy

Signed-off-by: tabokie <xy.tao@outlook.com>

* enable default features when running clippy, ref #11119

Signed-off-by: tabokie <xy.tao@outlook.com>

* fix build

Signed-off-by: tabokie <xy.tao@outlook.com>

* revert box_collection lint

Signed-off-by: tabokie <xy.tao@outlook.com>

* Update tests/integrations/raftstore/test_bootstrap.rs

Signed-off-by: Xinye Tao <xy.tao@outlook.com>

Co-authored-by: Lei Zhao <zlwgx1023@gmail.com>

Co-authored-by: Lei Zhao <zlwgx1023@gmail.com>

* logger: fix config-template's log unit (#11651) (#11777)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: improve testcase (#11651) (#11778)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* raftstore: propose in-memory pessimistic locks before leader transfer (#11577)

* *: check epoch and validness of lock table when writing pessimistic locks

ref #11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* raftstore: propose in-memory pessimistic locks before leader transfer

ref #11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* mark delete in scheduler and delete after apply

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* Add tests for two different orders between proposing locks and write command

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* do not clear pessimistic locks before transfer leader

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* delete memory locks in the apply thread and add more term and version check

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* remove potentially incorrect assert

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* check term and version when reading the lock table

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* revert a log change

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* change term when becoming follower

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* fix lint

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* do not check version when reading locks (temporarily)

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* fix another test

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* only treat MsgTransferLeader with context with valid msg

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* fix a tiny comment error

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* make clippy happy

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* res_meter: ignore the read keys test temporarily (#11779)

ref #11765

Signed-off-by: mornyx <mornyx.z@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* logger: change critical to fatal (#11651) (#11780)

* logger: change critical to fatal (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: fix test_parse_log_level (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* update CHANGELOG.md (#11790)

* update from tikv-server v5.0.1(20210423)

from https://github.com/tikv/tikv/releases

Signed-off-by: xiejiandong <xiejiandong@pingcap.com>

* update CHANGELOG.md close #11167

Signed-off-by: xiejiandong <xiejiandong@pingcap.com>

* config: relax the bounds of online configurable worker number (#11651) (#11798)

close #11776

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: invoke logger_compatible_adjust before run_tikv (#11651) (#11792)

close #11789

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: unify thread related native APIs (#11785)

* *: unify thread related native APIs

Now there are duplicated codes calling syscalls to fetch thread
informations. This PR provides unified APIs and remove duplicated codes.

Close #11784.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* address comment

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* declare platform compatability

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* Fix method on_tick will not update metric when enable_auto_tune is false (#11791)

* Fix method on_tick will not update metric when enable_auto_tune is false
ref #11787
Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* resolve format problem in discussion

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* resolve format problem in discussion again

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: add a new tick to renew lease in advance (#6427)

* renew lease in raft tick

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check lease before sending read index

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* renew lease on raft tick to cover lease till next tick

Signed-off-by: linning <linningde25@gmail.com>

* refactor

Signed-off-by: linning <linningde25@gmail.com>

* don't use inspect_lease

Signed-off-by: linning <linningde25@gmail.com>

* fix ci

Signed-off-by: linning <linningde25@gmail.com>

* add test

Signed-off-by: linning <linningde25@gmail.com>

* remove unneed comment

Signed-off-by: linning <linningde25@gmail.com>

* close #5388

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check pending_read_count before hibernating

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* increase the range to check lease expired

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* renew leader lease after not hibernate

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix hibernate states metrics

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check writes and pre_read_index before proposing read index

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* use a individual tick to renewing lease

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix condition

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* stablize test_node_renew_lease

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* cleanup unused changes

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add config item for interval of tick

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* rename tick

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* disable the tick for test_inconsistent_configuration

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* change deafault value of check_leader_lease_interval

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* disable the tick for test_read_hibernated_region

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* count renew lease read index into raft metrics

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* disable the tick for test_renew_lease

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix lint error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add a 100ms buffer for renew_bound

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* make clippy happy

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: linning <linningde25@gmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* cloud: Add retry for azure blob server busy error (#11813)

* add retry for azure blob server busy error

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* close#11812

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* use to_string

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* tikv_util: make cgroup parsing more robust (#11786)

* make cgroup parsing more robust

Signed-off-by: tabokie <xy.tao@outlook.com>

* fix error message

Signed-off-by: tabokie <xy.tao@outlook.com>

* parse cpu quota in floats

Signed-off-by: tabokie <xy.tao@outlook.com>

* *: make it compile on macos (#11825)

* *: make it compile on macos

close #11823

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* remove extra comment

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* resolved_ts: add more metrics (#11809)

* add metrics

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add slow log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* remove duplicate metric

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add resolved ts metrics for leader peers

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* close #11767

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* init endpoint with store meta

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* rsmeter: fix perf regression caused by arc swap  (#11833)

* rsmeter: fix perf regression caused by arc swap

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* use atomic cell instead

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* debug assert

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Wenxuan <breezewish@pingcap.com>

* load base split: more accurate sample for large key ranges batch (#11039)

* more accurate sample for large key ranges

Signed-off-by: lhy1024 <admin@liudos.us>

* update

Signed-off-by: lhy1024 <admin@liudos.us>

* ref #11521

Signed-off-by: lhy1024 <admin@liudos.us>

* address comment

Signed-off-by: lhy1024 <admin@liudos.us>

* fix test

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftclient: enable delay only in high load (#11772)

and improve CPU efficiency under high load. But it turns out
this can lead to significant regression in normal load. So
this PR adds CPU stats and only enabling delay in high load.

This is also the same strategy used in v4.x.

Close #11769.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* tikv-ctl: fix tikv-ctl's output is incomplete on calling process::exit (#11231)

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>
Co-authored-by: Connor <zbk602423539@gmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: skip deleting snapshot files in peer pending_remove is true (#11782)

* raftstore: skip deleting snapshot files in peer when the peer is pending removal and the snapshot is being applied and canceled -- close #11746

This is to avoid the potential panic when the snapshot files are deleted, but the peer's status (Tombstone) is not persisted in disk due to tikv crash.

Signed-off-by: tonyxuqqi <tonyxuqi@outlook.com>

* address code review feedback  -- close #11746

Signed-off-by: qi.xu <tonxuqi@outlook.com>

* address code review feedback 2  -- close #11746

Signed-off-by: qi.xu <tonxuqi@outlook.com>

* address code review feedback 2  -- close #11746

Signed-off-by: qi.xu <tonxuqi@outlook.com>

* address code review feedback 2  -- close #11746

Signed-off-by: qi.xu <tonxuqi@outlook.com>

* address code review feedback 2  -- close #11746

Signed-off-by: qi.xu <tonxuqi@outlook.com>

Co-authored-by: qi.xu <tonxuqi@outlook.com>

* load_statis: fix compilation on macos (#11851)

close #11772

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* config: override rocksdb config when flow control enabled (#11840)

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* rocksdb: consider env sharing when configuring background threads (#11760)

#9649 increased the default setting of kvdb background flushes (3 for 8c and 4 for 16c).

This PR takes another approach: use smaller concurrency for individual dbs, but configure the global thread pool size with the sum of kvdb and raftdb concurrency. This way we won't over-configure flush workers when raft-engine is enabled.

* consider env sharing when configuring background threads

ref #11119

Signed-off-by: tabokie <xy.tao@outlook.com>

* server/config: keep compatible using option (#11862)

* server/config: keep compatible using option

Using `Option` for `heavy_load_wait_duration` to keep compatible with
versions prior to v5.3.

Close #11861.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* fix format

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* fix clippy

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: zhouqiang <zhouqiang.cl@gmail.com>

* raftstore: fix missing workers' stop on shutdown (#11864)

ref #11159

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: move in-memory pessimistic locks to split regions (#11655)

ref #11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: Remove the entrance of enabling io snooper. (#11394)

* *: Remove the entrance of enabling io snooper. close #10867

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* fix

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* fix test

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* fix test

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: change PeerTicks to enum (#11849)

* raftstore: change PeerTicks to enum

close #11848

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* use bool array for registry

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* rename PeerTicks to PeerTick

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* add associated const for number of PeerTick types

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* change naming

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: check memory limit when inserting in-memory pessimistic locks (#11841)

* *: check memory limit when inserting in-memory pessimistic locks

ref #11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* fix merged upstream

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* fix test

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* fix typo

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* *: fix invalid failpoint caused by typo (#11709)

* fix invalid failpoint caused by typo

close #11734

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* address the comment

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* address the comment

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: Extract significant router (#11750)

* ref#11409 add significant router

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* make format

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* address comment

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* raftstore: move scan delete to raft log gc worker (#11853)

* raftstore: move scan delete to raft log gc worker

When clearing raft metas, raftstore will scan raft logs and delete them
one by one. Seeking can be slow if there are a lot of tombstone keys.
This PR moves the operation to raft log gc worker to reduce the impact.

The final solution should be also moving remaining IO operations to
async write IO threads.

Close #10210.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* address comment

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* speed up destroy

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* fix compile

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* further speed up

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* revert test case configuration

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* address comment

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* address comment

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* doc: update new rules for linking issue and commit message (#11832)

close #11097, close #11831

Signed-off-by: zhangyangyu <angwerzx@126.com>

Co-authored-by: Mini256 <minianter@foxmail.com>

Co-authored-by: Mini256 <minianter@foxmail.com>
Co-authored-by: Xiaoguang Sun <sunxiaoguang@users.noreply.github.com>

* *: update rust toolchain to 2022-01-07 (#11875)

* update rust toolchain to 2022-01-17

Signed-off-by: tabokie <xy.tao@outlook.com>

* address comment: clean up unnecessary `as_ref()`

Signed-off-by: tabokie <xy.tao@outlook.com>

* try fixing the underflow

Signed-off-by: tabokie <xy.tao@outlook.com>

* mute underflow warning for raft metrics

Signed-off-by: tabokie <xy.tao@outlook.com>

* clean up unused data members

Signed-off-by: tabokie <xy.tao@outlook.com>

* format

Signed-off-by: tabokie <xy.tao@outlook.com>

* step back to 2022-01-07

Signed-off-by: tabokie <xy.tao@outlook.com>

* readme: change images based on github theme (#11795)

Signed-off-by: sloorush <aarush.bhatt@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: don't remove other peer's read delegate (#11882)

* don't remove read delegate besides peer_destroy

Signed-off-by: linning <linningde25@gmail.com>

* add test case

Signed-off-by: linning <linningde25@gmail.com>

* make clippy happy

Signed-off-by: linning <linningde25@gmail.com>

* address comment

Signed-off-by: linning <linningde25@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* logger: use eprintln! if the logger is not initialized (#11869)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: fix the calculation of total column size in analyze (#11884)

Signed-off-by: xuyifan <xuyifangreeneyes@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* engine: update raft-engine for data consistency fix (#11885)

Fix https://github.com/tikv/raft-engine/issues/142.

When encountering this bug, TiKV will panic with message "applied index > max(commit index, recorded commit index)" after restart.

* update raft-engine

Signed-off-by: tabokie <xy.tao@outlook.com>

* update one more commit

Signed-off-by: tabokie <xy.tao@outlook.com>

* check docker build during clippy, ref #11312 (#11819)

Signed-off-by: tabokie <xy.tao@outlook.com>

* gc_worker: fix incorrect scheduled_tasks counting (#11904)

* gc_worker: fix incorrect scheduled_tasks counting

close #11903

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* remove check_is_busy totally

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* do not use wildcard match in error handling

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* raftstore: Introduce raft log fetcher (#11900)

* ref#11409 introduce raft log fetcher

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* update kvproto

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* address comment

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* fix test build

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* fix clippy

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* limit capacity

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* update kvproto

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* call stop on worker

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* rename worker

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* status_server: add pprof flamegraph response header (#10951)

* tweak(status_server): add pprof flamegraph response header

Signed-off-by: Suhaha <jklopsdfw@gmail.com>

* test(status_server): add Content-Type asset to test_pprof_profile_service

Signed-off-by: Suhaha <jklopsdfw@gmail.com>

* tweak(status_server): add pprof flamegraph response header

Signed-off-by: Suhaha <jklopsdfw@gmail.com>

* tweak(status_server): add pprof flamegraph response header close #11917

Signed-off-by: Suhaha <jklopsdfw@gmail.com>

* close #11917

Signed-off-by: Suhaha <jklopsdfw@gmail.com>

Co-authored-by: goroutine <ngaut@users.noreply.github.com>
Co-authored-by: Connor <zbk602423539@gmail.com>

* raftstore: renew leader lease in advance when handle read request (#9307)

* renew lease advance

Signed-off-by: linning <linningde25@gmail.com>

* add log

Signed-off-by: linning <linningde25@gmail.com>

* make clippy happy

Signed-off-by: linning <linningde25@gmail.com>

* ref #11579

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* reset raft tick

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* set has ready

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add renew_leader_lease_advance_duration config

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix panic

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* disable renewing for test_lease_unsafe_during_leader_transfers

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: 5kbpers <tangminghua@pingcap.com>

* pprof: ignore cpu profiling on non-x86 arch (#11925)

* Disable cpu profiling on non-x86_64 arch

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Fix warns

Signed-off-by: mornyx <mornyx.z@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* github: add new pr requirement for linking issue and commit message (#11887)

Signed-off-by: zhangyangyu <angwerzx@126.com>

Co-authored-by: Xiaoguang Sun <sunxiaoguang@users.noreply.github.com>

* engine: enable raft engine by default (#11928)

* enable raft engine by default and synchronize docs

Signed-off-by: tabokie <xy.tao@outlook.com>

* update raft engine

Signed-off-by: tabokie <xy.tao@outlook.com>

* update raft-engine and fix tests

Signed-off-by: tabokie <xy.tao@outlook.com>

* gc_worker: Limit the key range to scan for GcKeys tasks (#11922)

close tikv/tikv#11752, close tikv/tikv#11902

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: TransferLeader support multiple target peers (#11063)

ref tikv/tikv#822, ref tikv/tikv#4214, close tikv/tikv#10602

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* raftstore: remove the leaders field of `StoreMeta` (#11934)

close tikv/tikv#11933

Remove `StoreMeta.leaders`
```

### Related changes

### Check List <!--REMOVE the items that are not applicable-->

Tests <!-- At least one of them must be included. -->

- Manual test (add detailed scripts or steps below)
`cargo check`

### Release note <!-- bugfixes or new feature need a release note -->

```release-note
None

Signed-off-by: linning <linningde25@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: Fetch raft log in async manner (#11409)

close tikv/tikv#10408, close tikv/tikv#11320

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: add quarter function (#11935)

ref tikv/tikv#5751

Signed-off-by: zhongyong jin <AT-Fieldless@outlook.com>

* Enable full debug info for dev and test (#11949)

ref tikv/tikv#5049, ref tikv/tikv#5572, close tikv/tikv#5572

```

### Related changes

- PR to update `pingcap/docs`/`pingcap/docs-cn`:
- PR to update `pingcap/tidb-ansible`:
- Need to cherry-pick to the release branch
-->

### Check List <!--REMOVE the items that are not applicable-->

Tests <!-- At least one of them must be included. -->

- Manual test (add detailed scripts or steps below)
- No code

Manual test:
Linux box with 8 cores (3:50 -> 4:00):
```
# Before this PR:
$ make clean
cargo clean
rm -rf bin dist
$ time make build
cargo build --no-default-features --features "jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"
   Compiling libc v0.2.106
...
   Compiling server v0.0.1 (tikv/components/server)
    Finished dev [unoptimized + debuginfo] target(s) in 3m 50s

real	3m50,487s
user	38m11,859s
sys	3m9,540s

# After this PR:
$ git diff
diff --git a/Cargo.toml b/Cargo.toml
index 71f5329d3..67cb9d183 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -267,7 +267,7 @@ default-members = ["cmd/tikv-server", "cmd/tikv-ctl"]
 
 [profile.dev]
 opt-level = 0
-debug = 1 # required for line numbers in tests, see tikv #5049
+debug = true
 codegen-units = 4
 lto = false
 incremental = true
@@ -293,7 +293,7 @@ codegen-units = 4
 
 [profile.test]
 opt-level = 0
-debug = 1 # enable line numbers by default for easy test debugging
+debug = true
 codegen-units = 16
 lto = false
 incremental = true
mattias@msig:~/repos/tikv$ time make clean
cargo clean
rm -rf bin dist

real	0m0,975s
user	0m0,148s
sys	0m0,828s
mattias@msig:~/repos/tikv$ time make build
cargo build --no-default-features --features "jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"
   Compiling libc v0.2.106
...
   Compiling server v0.0.1 (tikv/components/server)
    Finished dev [unoptimized + debuginfo] target(s) in 4m 00s

real	4m0,201s
user	39m45,037s
sys	3m16,397s
```

Macbook Air M1:
```
# Before:
% time make build
cargo build --no-default-features --features " jemalloc test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"
   Compiling libc v0.2.106
....
   Compiling server v0.0.1 (tikv/components/server)
    Finished dev [unoptimized + debuginfo] target(s) in 4m 01s
make build  1107.42s user 116.20s system 506% cpu 4:01.46 total

# After:
% time make build
cargo build --no-default-features --features " jemalloc test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"
   Compiling libc v0.2.106
....
   Compiling server v0.0.1 (tikv/components/server)
    Finished dev [unoptimized + debuginfo] target(s) in 4m 10s
make build  1179.39s user 120.74s system 518% cpu 4:10.98 total
```

Side effects

- Performance regression, Only when building, to the benefit of full debug info by default.


### Release note <!-- bugfixes or new feature need a release note -->

```release-note
None

Signed-off-by: Mattias Jonsson <mjonss@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* correct a metric about compaction filter (#11938)

 

correct a metric about compaction filter

Signed-off-by: qupeng <qupeng@pingcap.com>

* raft: Fix possible panic on entries fetched callback (#11961)

close tikv/tikv#11951

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* split_controller: refine the sample function and add some comments (#11952)

 

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* metrics: fix grid position in TiKV Details (#11936)

ref tikv/tikv#11119

Fix grid positions in TiKV Details dashboard.

Signed-off-by: tabokie <xy.tao@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* cdc: capture old value from txn layer dynamically (#11896)

close tikv/tikv#10091

Signed-off-by: hi-rustin <rustin.liu@gmail.com>
Signed-off-by: qupeng <qupeng@pingcap.com>

Co-authored-by: hi-rustin <rustin.liu@gmail.com>

* split_controller: reorganize the structs and add more comments (#11967)

 

Reorganize the structs and add more comments.

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: support Bit column push downBit column (#11968)

close tikv/tikv#11893, ref tikv/tikv#11893, ref pingcap/tidb#31884

Signed-off-by: xiejiandong <xiejiandong@pingcap.com>

* copr: support push down mod, sysdate to TiKV (#11970)

close tikv/tikv#11916, ref tikv/tikv#11916

Signed-off-by: xiejiandong <xiejiandong@pingcap.com>

* raftstore: propose in-memory pessimistic locks before prepare merge (#11758)

ref tikv/tikv#11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* stats_monitor: reformat stats_monitor tick (#11972)

 

Signed-off-by: lhy1024 <admin@liudos.us>

* copr: support extract scalar value from bit (#11980)

close tikv/tikv#11893

Signed-off-by: yisaer <disxiaofei@163.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* diagnosis: support get cpu time of threads for macOS (#11978)

close tikv/tikv#11977

Add the variant for macOS into `components/tikv_util/src/sys/thread.rs`

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Co-authored-by: Jay <BusyJay@users.noreply.github.com>

* rust: update toolchain to fix missing rls (#11954)

close tikv/tikv#11953

Signed-off-by: Connor1996 <zbk602423539@gmail.com>
Signed-off-by: Yilin Chen <sticnarf@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Co-authored-by: Yilin Chen <sticnarf@gmail.com>

* cdc: separate resolved region outliers (#11991)

Separate broadcasing outlier regions and normal regions,
so 1) downstreams know where they should send resolve lock requests,
and 2) resolved ts of normal regions does not fallback.

close pingcap/tiflow#4516
close pingcap/tiflow#4311
ref pingcap/tiflow#4146

Signed-off-by: Neil Shen <overvenus@gmail.com>

* *: collect key ranges for the read request in ResourceMeteringTag (#11995)

close tikv/tikv#11988

*: collect key ranges for the read request in ResourceMeteringTag

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* split_controller: move sample_threshold check out of the split_key iteration (#11986)

 

split_controller: move sample_threshold check out of the split_key iteration

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: support insertutf8 and lowerutf8 (#11987)

ref tikv/tikv#5751

Signed-off-by: yisaer <disxiaofei@163.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: support rest greatest/least functions (#12003)

ref tikv/tikv#5751

Signed-off-by: yisaer <disxiaofei@163.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: force compact with gentleness (#12006)

* force compact with gentleness

Signed-off-by: tabokie <xy.tao@outlook.com>

* address comment

Signed-off-by: tabokie <xy.tao@outlook.com>

* split_controller: refine the LOAD_BASE_SPLIT_EVENT metrics label definitions (#12010)

 

split_controller: refine the LOAD_BASE_SPLIT_EVENT metrics label definitions

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* split_controller: fix the incorrect sampled key ranges number check (#12013)

close tikv/tikv#12012

split_controller: fix the incorrect sampled key ranges number check

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: Support adjusting max_inflight_msgs dynamically (#11866)

close tikv/tikv#11865

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: fix missing flag solvement for bit column in decoding (#12016)

ref tikv/tikv#7, close pingcap/tidb#32506

Signed-off-by: yisaer <disxiaofei@163.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* file_system: overhaul proc-based io tracing utilities (#11873)

ref tikv/tikv#10867, fix tikv/tikv#11775

Signed-off-by: tabokie <xy.tao@outlook.com>

* raftstore: Wait for the apply index equals to the commit index (#11716)

ref tikv/tikv#10483

Signed-off-by: v01dstar <yang.zhang@pingcap.com>

* copr: revert 11968/11980/12016 (#12030)

ref tikv/tikv#11968, ref tikv/tikv#11980, ref tikv/tikv#12016, ref pingcap/tidb#32506

Signed-off-by: yisaer <disxiaofei@163.com>

* build: fix a Mac build issue (#12027)

ref tikv/tikv#10867, ref tikv/tikv#11873

Signed-off-by: tabokie <xy.tao@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: Extra physical table id column (#11931)

close tikv/tikv#11888

Added EXTRA_PHYSICAL_TABLE_ID_COL_ID support, to support TiDB's table partition dynamic prune mode, where a single request includes multiple partitions, but when Pessimistic lock (like `SELECT FOR UPDATE`) or ongoing transaction (having something in the tidb session local transaction buffer) each row needs its partition id / physical table ID to either lock that row or to check against the transaction buffer.

Signed-off-by: Mattias Jonsson <mjonss@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: hotfix panic from tokio-timer (#12004)

close tikv/tikv#11940, ref tikv/tikv#11940

*: hotfix panic from tokio-timer by enlarging the length of the level vector

Signed-off-by: you06 <you1474600@gmail.com>

* build: add arm64 check on SSE in makefile (#12035)

close tikv/tikv#12034

Signed-off-by: Jin Dong <djdongjin95@gmail.com>

Co-authored-by: Yilin Chen <sticnarf@gmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* replication mode: sync state to pd (#11751)

 

Signed-off-by: disksing <i@disksing.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: revert pessimistic locks status when failing to propose PrepareMerge (#11985)

ref tikv/tikv#11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* copr: support push bit column down (#12037)

ref pingcap/tidb#30738

Signed-off-by: yisaer <disxiaofei@163.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* split_controller: add more sample function test cases (#12058)

 

split_controller: add more sample function test cases

Signed-off-by: JmPotato <ghzpotato@gmail.com>

* copr: fix greatest/least time/date args type (#12056)

 

Signed-off-by: yisaer <disxiaofei@163.com>

* cdc: advancing resolved ts correctly for clusters with tiflash (#12050)

close pingcap/tiflow#4461

cdc: advancing resolved ts correctly for clusters with tiflash

Signed-off-by: qupeng <qupeng@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: bump master branch version to v6.0.0-alpha (#12077)

close tikv/tikv#12075

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* raftstore: check uninitialized destroy for merge (#12055)

close tikv/tikv#12048

When a peer is destroyed without being initialized, it will store
itself to peer list and the region epoch is missing. In merge if
such state is detected, it should abort merging instead of panicking.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: Add TruncateManager and TruncateWorker (#11553)

ref tikv/rfcs#81

Signed-off-by: longfangsong <longfangsong@icloud.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* TiKV supports buckets (#11763)

ref tikv/tikv#11759

Signed-off-by: qi.xu <tonxuqi@outlook.com>

Co-authored-by: qi.xu <tonxuqi@outlook.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* api_version: Codec for RawKV key (#12036)

ref tikv/tikv#11965

Key of RawKV is encoded as `user-key + memcomparable-padding + timestamp` in API v2.

Signed-off-by: pingyu <yuping@pingcap.com>

Co-authored-by: Andy Lok <andylokandy@hotmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: fix some typos (#12066)

 

Signed-off-by: cuishuang <imcusg@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: fix stale message cause panic (#12054)

close tikv/tikv#12023

raftstore: fix stale message cause panic

Signed-off-by: linning <linningde25@gmail.com>

* raftstore: ignore async fetch result when the peer is pending removed (#12038)

close tikv/tikv#11973, close tikv/tikv#12026

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: remove part about wechat in doc (#12101)

close tikv/tikv#12100

Signed-off-by: jackwener <jakevingoo@gmail.com>

* engine: upgrade raft engine (#12095)

ref tikv/tikv#165, ref tikv/raft-engine#165, ref tikv/tikv#11119

Signed-off-by: Randy <ztelur@gmail.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>

* *: Fix possible undefined behavior for transmuting vec (#12096)

close tikv/tikv#12070

Signed-off-by: Connor1996 <zbk602423539@gm…
CalvinNeo added a commit to pingcap/tidb-engine-ext that referenced this issue May 30, 2022
* storage: Add API V2 check for RawKV and TxnKV requests (txn part) (#11216)

* wip

Signed-off-by: pingyu <yuping@pingcap.com>

* wip

Signed-off-by: pingyu <yuping@pingcap.com>

* ref #10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref #10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref #10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref #10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref #10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref #10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* fix enable_ttl

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* simplify test

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* ref #10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

Co-authored-by: andylokandy <andylokandy@hotmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* cdc: load uninlined value more effectively (#11615)

Signed-off-by: qupeng <qupeng@pingcap.com>

* cdc: load old value with prefix seek if possible (#11643)

* close #11642

Signed-off-by: qupeng <qupeng@pingcap.com>

* a more comment

Signed-off-by: qupeng <qupeng@pingcap.com>

* ctl: split into multiple modules (#11658)

* ctl: split into multiple modules

ref #10938

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* refactor if let to match

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* Fix build issues on non-linux platforms (#11671)

* close #11676

Signed-off-by: devillove084 <786537003@qq.com>

* all: Migrate from std:: types to primitives. (#11677)

* Deprecation is planned for MIN,MAX,EPSLION,INFINITY,NEG_INFINITY of
  the (i|u|f)(8|16|32|64) types in standard.

ref #11678

```release-note
NONE
```

Signed-off-by: Harold Dost <harolddost@gmail.com>

* metrics alert: fix apply cpu alert (#11672)

Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* resrc_mtr: introduce scanned keys statistics (#11085)

close #11690

Signed-off-by: mornyx <mornyx.z@gmail.com>

* file_system: support collect the io flow by proc (#11667)

* file_system: support collect the read flow by proc

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* fix lint

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* fix clippy

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* Delete unused content & Adjust copyright text

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Add bench_fetch_io_bytes

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* fix lint

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Integrate thread_io

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* record bytes in request

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* fix logical bug

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Update test write_bytes

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Delete unused & Rename func

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Add test for read & Update test for write

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Add buffered thread io

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Replace DashMap with ThreadLocal

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* Rename variables and function & Remove unused call

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Change fetch logic. close #10867

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Add AtomicIOBytes

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Split ThreadIOSentinel

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* add THREAD_IO_TOTAL for metrics bytes fetcher

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Adjust code style & fix

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* improve write_bytes test

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: check epoch and validness of lock table when writing pessimistic locks (#11591)

ref #11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* rsmeter: introduce datasink (#11688)

* add datasink trait

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* RAII removes on_reporter_closing

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fmt

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix build

ref #11691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* datasink -> data_sink

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fmt

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix build

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* clippy: Fixing up clippy errors. (#11193)

* clippy: Fixing up clippy errors.

* Fix crate references
* Fix anonymous lifetimes references.

ref #4301
Signed-off-by: Harold Dost <harolddost@gmail.com>

* Test Revert of Iterations

ref #4301

Signed-off-by: Harold Dost <harolddost@gmail.com>

* clippy: Enable check for rust-2018-idioms

ref #4301

Signed-off-by: Harold Dost <harolddost@gmail.com>

* rsmeter: add centralized place to construct protobuf data (#11696)

ref #11691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* tikv_util: update procfs to 0.12.0 (#11703)

Signed-off-by: linning <linningde25@gmail.com>

* Use generic for api version to reduce runtime branching (#11687)

* Use generic for api version to reduce runtime branching

ref #10938

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* fix test

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* add test

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* fix clippy

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* fmt

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* rename vars

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* improve vars

Signed-off-by: andylokandy <andylokandy@hotmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* sst_importer: check api version when importing (#11664)

* sst_importer: check api version when importing

ref #10938

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Fix bugs

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* fix bug of Iterator::{scan,scan_cf}

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* remove redundant log

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Show the entire key

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Fix lint issue

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* resolve conflict

Signed-off-by: andylokandy <andylokandy@hotmail.com>

Co-authored-by: Andy Lok <andylokandy@hotmail.com>

* backup: pipeline scan and save step (#11528)

* br: sperate io and scan threads

Signed-off-by: Yu Juncen <yu745514916@live.com>

* br/stream: make clippy happy

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: fix tests

Signed-off-by: Yu Juncen <yu745514916@live.com>

* backup: fix a dummy bug

Signed-off-by: Yu Juncen <yu745514916@live.com>

* backup: add config of io-threads

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: ref #11350: some minior change

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: added some metrics

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: better tuning concurrency, and added a metric

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: added some slow log and metrics

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: address comments

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: fix build

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* Revert "backup: fix build"

This reverts commit 74537be56c410d0f91af2f48b7e65356ab53720e.

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* Revert "backup: address comments"

This reverts commit 77d75756028bea3496f878f0ad8bbc562f5d00aa.

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: address comments

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: always set IO type to Export

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup:make clippy happy

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: address comments

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* metrics: fix the metrics cannot be displayed (#11710)

ref #11662

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* backup: return api-version to BR when backup (#11704)

* Update kvproto

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Return api-version for br

ref #10938

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Reformat code

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Update components/external_storage/export/src/export.rs

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

Co-authored-by: Andy Lok <andylokandy@hotmail.com>

* format code

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

Co-authored-by: Andy Lok <andylokandy@hotmail.com>

* file_system: limit thread_io target os (#11715)

* limit thread_io target os

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* limit thread_io target os. typo. close #11698

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

Co-authored-by: Yilin Chen <sticnarf@gmail.com>

* file_system: bypass file allocate implementation in fs2 (#11700)

Close #10688

Patch fs2 to https://github.com/tabokie/fs2-rs/tree/tikv. In which, `posix_fallocate` is replaced with `fallocate` (https://github.com/danburkert/fs2-rs/pull/42).

* bypass file allocate implementation in fs2

Signed-off-by: tabokie <xy.tao@outlook.com>

* metrics: remove min legend of some Grafana panels (#11487)

Minimum legend isn't very useful, and it occupied valuable visual real estate.

Remove some min legends from Grafana metrics, except for the following:
- Level 0 chance
- Number files at each level
- File Count
- Entry Count
- Allocator Stats
- Encrypted files
- Encryption meta files size

Ref #11119

Signed-off-by: tabokie <xy.tao@outlook.com>

* coprocessor: not return rows when there is no input for simple aggregation (#11708)

* not return rows when there is no input for simple aggregation

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

* close #11735, and address comments

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

* backup: reduce the default thread pool size of backup, enable auto tune by default (#11699)

* backup: clamp auto tune values

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* ref #11000: change the default backup threads.

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: run tasks in background

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: run rustfmt

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: set remain threads to 20% of vcpu

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br-stream: fix build

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* Backup: add S3 metrics && add s3_multi_part_size config  (#11666)

* s3: add request metrics for s3 storage

Signed-off-by: 3pointer <luancheng@pingcap.com>

* s3: add grafana json

Signed-off-by: 3pointer <luancheng@pingcap.com>

* br: add config for s3 multi part upload

Signed-off-by: 3pointer <luancheng@pingcap.com>

* update comment

Signed-off-by: 3pointer <luancheng@pingcap.com>

* update comment

Signed-off-by: 3pointer <luancheng@pingcap.com>

* address comment

Signed-off-by: 3pointer <luancheng@pingcap.com>

* hidden the new config

Signed-off-by: 3pointer <luancheng@pingcap.com>

* only hidden config in config-template.toml

Signed-off-by: 3pointer <luancheng@pingcap.com>

* address comment

Signed-off-by: 3pointer <luancheng@pingcap.com>

* close #11727 && format

Signed-off-by: 3pointer <luancheng@pingcap.com>

* fix

Signed-off-by: 3pointer <luancheng@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* rsmeter: support multiple datasinks (#11707)

* rsmeter: support multiple datasinks

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* wrap config notifier

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* add unit tests

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* polish
ref #11691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix test

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix outdated comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* engine: properly estimate store size (#11728)

* properly estimate store size, ref #11119

Signed-off-by: tabokie <xy.tao@outlook.com>

* address comments

Signed-off-by: tabokie <xy.tao@outlook.com>

* implement global config client for tikv (#11685)

* implement global config client

close #11686

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* implement unit test for global config in pd client

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* resolved formatting issues

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* impl pd client asyncly

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* according to reviewer's opinion
removing store method implementation

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* Merge branch 'master' of https://github.com/tikv/tikv into global_conf

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* write test case for watch global config when
grpc server is closed.

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* resove issues

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* clippy

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* watch global config rentry future in test deadlock just removed

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

Co-authored-by: Zhenchi <zhongzc_arch@outlook.com>

* storage: skip scanning lock when using rc (#11701)

* skip scanning lock when using rc, ref #11485

Signed-off-by: you06 <you1474600@gmail.com>

* format code

Signed-off-by: you06 <you1474600@gmail.com>

* add test

Signed-off-by: you06 <you1474600@gmail.com>

* address comment

Signed-off-by: you06 <you1474600@gmail.com>

* add an integration test

Signed-off-by: you06 <you1474600@gmail.com>

* format code

Signed-off-by: you06 <you1474600@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: pushdown substring to tikv (#11494)

* corp: pushdown substring to tikv

Signed-off-by: guo-shaoge <shaoge1994@163.com>

* fix case. close #11495

Signed-off-by: guo-shaoge <shaoge1994@163.com>

* fix comment

Signed-off-by: guo-shaoge <shaoge1994@163.com>

* rsmeter: distinguish between collectors and observers within recorder  (#11712)

* rsmeter: support multiple datasinks

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* wrap config notifier

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* add unit tests

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* polish
ref #11691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* rsmeter: distinguish between collectors and observers within recorder

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fmt & try to resolve #11689

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* add unit tests
ref #11691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* retrigger test

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix test

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix outdated comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* Update components/resource_metering/src/recorder/sub_recorder/mod.rs

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Yexiang Zhang <mornyx.z@gmail.com>

* address comment

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* remove enabled

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Co-authored-by: Wenxuan <breezewish@pingcap.com>
Co-authored-by: Yexiang Zhang <mornyx.z@gmail.com>

* config: validate online configurable thread pools (tikv#11159) (#11714)

ref #11159

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* res_meter: upgrade kvproto (#11749)

* Upgrade kvproto

close #11748

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Fix test compilation

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Fix test compilation

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Fix test compilation

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Use master kvproto

Signed-off-by: mornyx <mornyx.z@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* rsmeter: add pubsub datasink  (#11719)

* rsmeter: add pubsub datasink
close #11691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix build

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* Update components/resource_metering/src/reporter/pubsub.rs

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Wenxuan <hi@breeswish.org>

* fix build

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Wenxuan <hi@breeswish.org>

* status_server: Support online config update for configuration hosting platform (#11693)

close #11692

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* res_meter: optimize top k (#11753)

* Keep top k on recorder

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Modify comments

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Fix pd.rs

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Refactor

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Remove import

Signed-off-by: mornyx <mornyx.z@gmail.com>

* max_resource_groups 200 -> 100

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Fix ut

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Use iterator to avoid clone

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Optimize

close #11754

Signed-off-by: mornyx <mornyx.z@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftclient: delay flush (#11705)

* raftclient: delay flush

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* remove raftstore wait

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* Ref #11309.

tikv#11310 introduces delay in raftstore thread to reduce the flush rate.
It can only reduce flush rate from one thread. If there are many
raftstore threads or IO threads, messages can still be flush frequently.
This PR reduces the flush rate by introducing delay in raft client. So
delay will work at connection level and achieve the maximun batch
affect.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* remove useless field

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* address comment

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* fix unstable raw get case

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Liqi Geng <gengliqiii@gmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* cloud: support azure blob storage as external storage backend (#11732)

* support azure storage blob for external storage

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* modify the priority of the azure login methods

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* fix feature match branch

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* implement token update retriable

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* fix some problems

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* close #11731

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* rustfmt

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* make clippy success

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* simplify uploader

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* commit some suggestions

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* coprocesser: update encoding lib (#11506)

Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* coprocesser: implement upper and lower function on GBK charset (#11756)

Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* logger: support archiving and rotation (#11651) (#11657)

* logger: support archiving and rotation (#11651)

close #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: add testcases to test dt_from_file_name (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: simplify DateTime format (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: do a little refactoring (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: do a little refactoring (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: do a little refactoring (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: do a little refactoring (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: do a little refactoring (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* diagnostics: refactor  (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: run clippy for integration tests (#11768)

* run clippy --fix with nightly-2021-12-17

Signed-off-by: tabokie <xy.tao@outlook.com>

* manual clippy

Signed-off-by: tabokie <xy.tao@outlook.com>

* format and clippy

Signed-off-by: tabokie <xy.tao@outlook.com>

* enable default features when running clippy, ref #11119

Signed-off-by: tabokie <xy.tao@outlook.com>

* fix build

Signed-off-by: tabokie <xy.tao@outlook.com>

* revert box_collection lint

Signed-off-by: tabokie <xy.tao@outlook.com>

* Update tests/integrations/raftstore/test_bootstrap.rs

Signed-off-by: Xinye Tao <xy.tao@outlook.com>

Co-authored-by: Lei Zhao <zlwgx1023@gmail.com>

Co-authored-by: Lei Zhao <zlwgx1023@gmail.com>

* logger: fix config-template's log unit (#11651) (#11777)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: improve testcase (#11651) (#11778)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* raftstore: propose in-memory pessimistic locks before leader transfer (#11577)

* *: check epoch and validness of lock table when writing pessimistic locks

ref #11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* raftstore: propose in-memory pessimistic locks before leader transfer

ref #11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* mark delete in scheduler and delete after apply

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* Add tests for two different orders between proposing locks and write command

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* do not clear pessimistic locks before transfer leader

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* delete memory locks in the apply thread and add more term and version check

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* remove potentially incorrect assert

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* check term and version when reading the lock table

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* revert a log change

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* change term when becoming follower

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* fix lint

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* do not check version when reading locks (temporarily)

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* fix another test

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* only treat MsgTransferLeader with context with valid msg

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* fix a tiny comment error

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* make clippy happy

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* res_meter: ignore the read keys test temporarily (#11779)

ref #11765

Signed-off-by: mornyx <mornyx.z@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* logger: change critical to fatal (#11651) (#11780)

* logger: change critical to fatal (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: fix test_parse_log_level (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* update CHANGELOG.md (#11790)

* update from tikv-server v5.0.1(20210423)

from https://github.com/tikv/tikv/releases

Signed-off-by: xiejiandong <xiejiandong@pingcap.com>

* update CHANGELOG.md close #11167

Signed-off-by: xiejiandong <xiejiandong@pingcap.com>

* config: relax the bounds of online configurable worker number (#11651) (#11798)

close #11776

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: invoke logger_compatible_adjust before run_tikv (#11651) (#11792)

close #11789

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: unify thread related native APIs (#11785)

* *: unify thread related native APIs

Now there are duplicated codes calling syscalls to fetch thread
informations. This PR provides unified APIs and remove duplicated codes.

Close #11784.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* address comment

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* declare platform compatability

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* Fix method on_tick will not update metric when enable_auto_tune is false (#11791)

* Fix method on_tick will not update metric when enable_auto_tune is false
ref #11787
Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* resolve format problem in discussion

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* resolve format problem in discussion again

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: add a new tick to renew lease in advance (#6427)

* renew lease in raft tick

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check lease before sending read index

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* renew lease on raft tick to cover lease till next tick

Signed-off-by: linning <linningde25@gmail.com>

* refactor

Signed-off-by: linning <linningde25@gmail.com>

* don't use inspect_lease

Signed-off-by: linning <linningde25@gmail.com>

* fix ci

Signed-off-by: linning <linningde25@gmail.com>

* add test

Signed-off-by: linning <linningde25@gmail.com>

* remove unneed comment

Signed-off-by: linning <linningde25@gmail.com>

* close #5388

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check pending_read_count before hibernating

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* increase the range to check lease expired

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* renew leader lease after not hibernate

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix hibernate states metrics

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check writes and pre_read_index before proposing read index

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* use a individual tick to renewing lease

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix condition

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* stablize test_node_renew_lease

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* cleanup unused changes

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add config item for interval of tick

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* rename tick

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* disable the tick for test_inconsistent_configuration

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* change deafault value of check_leader_lease_interval

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* disable the tick for test_read_hibernated_region

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* count renew lease read index into raft metrics

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* disable the tick for test_renew_lease

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix lint error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add a 100ms buffer for renew_bound

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* make clippy happy

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: linning <linningde25@gmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* cloud: Add retry for azure blob server busy error (#11813)

* add retry for azure blob server busy error

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* close#11812

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* use to_string

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* tikv_util: make cgroup parsing more robust (#11786)

* make cgroup parsing more robust

Signed-off-by: tabokie <xy.tao@outlook.com>

* fix error message

Signed-off-by: tabokie <xy.tao@outlook.com>

* parse cpu quota in floats

Signed-off-by: tabokie <xy.tao@outlook.com>

* *: make it compile on macos (#11825)

* *: make it compile on macos

close #11823

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* remove extra comment

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* resolved_ts: add more metrics (#11809)

* add metrics

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add slow log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* remove duplicate metric

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add resolved ts metrics for leader peers

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* close #11767

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* init endpoint with store meta

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* rsmeter: fix perf regression caused by arc swap  (#11833)

* rsmeter: fix perf regression caused by arc swap

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* use atomic cell instead

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* debug assert

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Wenxuan <breezewish@pingcap.com>

* load base split: more accurate sample for large key ranges batch (#11039)

* more accurate sample for large key ranges

Signed-off-by: lhy1024 <admin@liudos.us>

* update

Signed-off-by: lhy1024 <admin@liudos.us>

* ref #11521

Signed-off-by: lhy1024 <admin@liudos.us>

* address comment

Signed-off-by: lhy1024 <admin@liudos.us>

* fix test

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftclient: enable delay only in high load (#11772)

and improve CPU efficiency under high load. But it turns out
this can lead to significant regression in normal load. So
this PR adds CPU stats and only enabling delay in high load.

This is also the same strategy used in v4.x.

Close #11769.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* tikv-ctl: fix tikv-ctl's output is incomplete on calling process::exit (#11231)

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>
Co-authored-by: Connor <zbk602423539@gmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: skip deleting snapshot files in peer pending_remove is true (#11782)

* raftstore: skip deleting snapshot files in peer when the peer is pending removal and the snapshot is being applied and canceled -- close #11746

This is to avoid the potential panic when the snapshot files are deleted, but the peer's status (Tombstone) is not persisted in disk due to tikv crash.

Signed-off-by: tonyxuqqi <tonyxuqi@outlook.com>

* address code review feedback  -- close #11746

Signed-off-by: qi.xu <tonxuqi@outlook.com>

* address code review feedback 2  -- close #11746

Signed-off-by: qi.xu <tonxuqi@outlook.com>

* address code review feedback 2  -- close #11746

Signed-off-by: qi.xu <tonxuqi@outlook.com>

* address code review feedback 2  -- close #11746

Signed-off-by: qi.xu <tonxuqi@outlook.com>

* address code review feedback 2  -- close #11746

Signed-off-by: qi.xu <tonxuqi@outlook.com>

Co-authored-by: qi.xu <tonxuqi@outlook.com>

* load_statis: fix compilation on macos (#11851)

close #11772

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* config: override rocksdb config when flow control enabled (#11840)

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* rocksdb: consider env sharing when configuring background threads (#11760)

#9649 increased the default setting of kvdb background flushes (3 for 8c and 4 for 16c).

This PR takes another approach: use smaller concurrency for individual dbs, but configure the global thread pool size with the sum of kvdb and raftdb concurrency. This way we won't over-configure flush workers when raft-engine is enabled.

* consider env sharing when configuring background threads

ref #11119

Signed-off-by: tabokie <xy.tao@outlook.com>

* server/config: keep compatible using option (#11862)

* server/config: keep compatible using option

Using `Option` for `heavy_load_wait_duration` to keep compatible with
versions prior to v5.3.

Close #11861.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* fix format

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* fix clippy

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: zhouqiang <zhouqiang.cl@gmail.com>

* raftstore: fix missing workers' stop on shutdown (#11864)

ref #11159

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: move in-memory pessimistic locks to split regions (#11655)

ref #11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: Remove the entrance of enabling io snooper. (#11394)

* *: Remove the entrance of enabling io snooper. close #10867

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* fix

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* fix test

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* fix test

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: change PeerTicks to enum (#11849)

* raftstore: change PeerTicks to enum

close #11848

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* use bool array for registry

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* rename PeerTicks to PeerTick

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* add associated const for number of PeerTick types

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* change naming

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: check memory limit when inserting in-memory pessimistic locks (#11841)

* *: check memory limit when inserting in-memory pessimistic locks

ref #11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* fix merged upstream

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* fix test

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* fix typo

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* *: fix invalid failpoint caused by typo (#11709)

* fix invalid failpoint caused by typo

close #11734

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* address the comment

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* address the comment

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: Extract significant router (#11750)

* ref#11409 add significant router

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* make format

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* address comment

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* raftstore: move scan delete to raft log gc worker (#11853)

* raftstore: move scan delete to raft log gc worker

When clearing raft metas, raftstore will scan raft logs and delete them
one by one. Seeking can be slow if there are a lot of tombstone keys.
This PR moves the operation to raft log gc worker to reduce the impact.

The final solution should be also moving remaining IO operations to
async write IO threads.

Close #10210.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* address comment

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* speed up destroy

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* fix compile

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* further speed up

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* revert test case configuration

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* address comment

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* address comment

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* doc: update new rules for linking issue and commit message (#11832)

close #11097, close #11831

Signed-off-by: zhangyangyu <angwerzx@126.com>

Co-authored-by: Mini256 <minianter@foxmail.com>

Co-authored-by: Mini256 <minianter@foxmail.com>
Co-authored-by: Xiaoguang Sun <sunxiaoguang@users.noreply.github.com>

* *: update rust toolchain to 2022-01-07 (#11875)

* update rust toolchain to 2022-01-17

Signed-off-by: tabokie <xy.tao@outlook.com>

* address comment: clean up unnecessary `as_ref()`

Signed-off-by: tabokie <xy.tao@outlook.com>

* try fixing the underflow

Signed-off-by: tabokie <xy.tao@outlook.com>

* mute underflow warning for raft metrics

Signed-off-by: tabokie <xy.tao@outlook.com>

* clean up unused data members

Signed-off-by: tabokie <xy.tao@outlook.com>

* format

Signed-off-by: tabokie <xy.tao@outlook.com>

* step back to 2022-01-07

Signed-off-by: tabokie <xy.tao@outlook.com>

* readme: change images based on github theme (#11795)

Signed-off-by: sloorush <aarush.bhatt@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: don't remove other peer's read delegate (#11882)

* don't remove read delegate besides peer_destroy

Signed-off-by: linning <linningde25@gmail.com>

* add test case

Signed-off-by: linning <linningde25@gmail.com>

* make clippy happy

Signed-off-by: linning <linningde25@gmail.com>

* address comment

Signed-off-by: linning <linningde25@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* logger: use eprintln! if the logger is not initialized (#11869)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: fix the calculation of total column size in analyze (#11884)

Signed-off-by: xuyifan <xuyifangreeneyes@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* engine: update raft-engine for data consistency fix (#11885)

Fix https://github.com/tikv/raft-engine/issues/142.

When encountering this bug, TiKV will panic with message "applied index > max(commit index, recorded commit index)" after restart.

* update raft-engine

Signed-off-by: tabokie <xy.tao@outlook.com>

* update one more commit

Signed-off-by: tabokie <xy.tao@outlook.com>

* check docker build during clippy, ref #11312 (#11819)

Signed-off-by: tabokie <xy.tao@outlook.com>

* gc_worker: fix incorrect scheduled_tasks counting (#11904)

* gc_worker: fix incorrect scheduled_tasks counting

close #11903

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* remove check_is_busy totally

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* do not use wildcard match in error handling

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* raftstore: Introduce raft log fetcher (#11900)

* ref#11409 introduce raft log fetcher

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* update kvproto

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* address comment

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* fix test build

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* fix clippy

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* limit capacity

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* update kvproto

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* call stop on worker

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* rename worker

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* status_server: add pprof flamegraph response header (#10951)

* tweak(status_server): add pprof flamegraph response header

Signed-off-by: Suhaha <jklopsdfw@gmail.com>

* test(status_server): add Content-Type asset to test_pprof_profile_service

Signed-off-by: Suhaha <jklopsdfw@gmail.com>

* tweak(status_server): add pprof flamegraph response header

Signed-off-by: Suhaha <jklopsdfw@gmail.com>

* tweak(status_server): add pprof flamegraph response header close #11917

Signed-off-by: Suhaha <jklopsdfw@gmail.com>

* close #11917

Signed-off-by: Suhaha <jklopsdfw@gmail.com>

Co-authored-by: goroutine <ngaut@users.noreply.github.com>
Co-authored-by: Connor <zbk602423539@gmail.com>

* raftstore: renew leader lease in advance when handle read request (#9307)

* renew lease advance

Signed-off-by: linning <linningde25@gmail.com>

* add log

Signed-off-by: linning <linningde25@gmail.com>

* make clippy happy

Signed-off-by: linning <linningde25@gmail.com>

* ref #11579

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* reset raft tick

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* set has ready

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add renew_leader_lease_advance_duration config

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix panic

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* disable renewing for test_lease_unsafe_during_leader_transfers

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: 5kbpers <tangminghua@pingcap.com>

* pprof: ignore cpu profiling on non-x86 arch (#11925)

* Disable cpu profiling on non-x86_64 arch

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Fix warns

Signed-off-by: mornyx <mornyx.z@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* github: add new pr requirement for linking issue and commit message (#11887)

Signed-off-by: zhangyangyu <angwerzx@126.com>

Co-authored-by: Xiaoguang Sun <sunxiaoguang@users.noreply.github.com>

* engine: enable raft engine by default (#11928)

* enable raft engine by default and synchronize docs

Signed-off-by: tabokie <xy.tao@outlook.com>

* update raft engine

Signed-off-by: tabokie <xy.tao@outlook.com>

* update raft-engine and fix tests

Signed-off-by: tabokie <xy.tao@outlook.com>

* gc_worker: Limit the key range to scan for GcKeys tasks (#11922)

close tikv/tikv#11752, close tikv/tikv#11902

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: TransferLeader support multiple target peers (#11063)

ref tikv/tikv#822, ref tikv/tikv#4214, close tikv/tikv#10602

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* raftstore: remove the leaders field of `StoreMeta` (#11934)

close tikv/tikv#11933

Remove `StoreMeta.leaders`
```

### Related changes

### Check List <!--REMOVE the items that are not applicable-->

Tests <!-- At least one of them must be included. -->

- Manual test (add detailed scripts or steps below)
`cargo check`

### Release note <!-- bugfixes or new feature need a release note -->

```release-note
None

Signed-off-by: linning <linningde25@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: Fetch raft log in async manner (#11409)

close tikv/tikv#10408, close tikv/tikv#11320

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: add quarter function (#11935)

ref tikv/tikv#5751

Signed-off-by: zhongyong jin <AT-Fieldless@outlook.com>

* Enable full debug info for dev and test (#11949)

ref tikv/tikv#5049, ref tikv/tikv#5572, close tikv/tikv#5572

```

### Related changes

- PR to update `pingcap/docs`/`pingcap/docs-cn`:
- PR to update `pingcap/tidb-ansible`:
- Need to cherry-pick to the release branch
-->

### Check List <!--REMOVE the items that are not applicable-->

Tests <!-- At least one of them must be included. -->

- Manual test (add detailed scripts or steps below)
- No code

Manual test:
Linux box with 8 cores (3:50 -> 4:00):
```
# Before this PR:
$ make clean
cargo clean
rm -rf bin dist
$ time make build
cargo build --no-default-features --features "jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"
   Compiling libc v0.2.106
...
   Compiling server v0.0.1 (tikv/components/server)
    Finished dev [unoptimized + debuginfo] target(s) in 3m 50s

real	3m50,487s
user	38m11,859s
sys	3m9,540s

# After this PR:
$ git diff
diff --git a/Cargo.toml b/Cargo.toml
index 71f5329d3..67cb9d183 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -267,7 +267,7 @@ default-members = ["cmd/tikv-server", "cmd/tikv-ctl"]
 
 [profile.dev]
 opt-level = 0
-debug = 1 # required for line numbers in tests, see tikv #5049
+debug = true
 codegen-units = 4
 lto = false
 incremental = true
@@ -293,7 +293,7 @@ codegen-units = 4
 
 [profile.test]
 opt-level = 0
-debug = 1 # enable line numbers by default for easy test debugging
+debug = true
 codegen-units = 16
 lto = false
 incremental = true
mattias@msig:~/repos/tikv$ time make clean
cargo clean
rm -rf bin dist

real	0m0,975s
user	0m0,148s
sys	0m0,828s
mattias@msig:~/repos/tikv$ time make build
cargo build --no-default-features --features "jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"
   Compiling libc v0.2.106
...
   Compiling server v0.0.1 (tikv/components/server)
    Finished dev [unoptimized + debuginfo] target(s) in 4m 00s

real	4m0,201s
user	39m45,037s
sys	3m16,397s
```

Macbook Air M1:
```
# Before:
% time make build
cargo build --no-default-features --features " jemalloc test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"
   Compiling libc v0.2.106
....
   Compiling server v0.0.1 (tikv/components/server)
    Finished dev [unoptimized + debuginfo] target(s) in 4m 01s
make build  1107.42s user 116.20s system 506% cpu 4:01.46 total

# After:
% time make build
cargo build --no-default-features --features " jemalloc test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"
   Compiling libc v0.2.106
....
   Compiling server v0.0.1 (tikv/components/server)
    Finished dev [unoptimized + debuginfo] target(s) in 4m 10s
make build  1179.39s user 120.74s system 518% cpu 4:10.98 total
```

Side effects

- Performance regression, Only when building, to the benefit of full debug info by default.


### Release note <!-- bugfixes or new feature need a release note -->

```release-note
None

Signed-off-by: Mattias Jonsson <mjonss@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* correct a metric about compaction filter (#11938)

 

correct a metric about compaction filter

Signed-off-by: qupeng <qupeng@pingcap.com>

* raft: Fix possible panic on entries fetched callback (#11961)

close tikv/tikv#11951

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* split_controller: refine the sample function and add some comments (#11952)

 

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* metrics: fix grid position in TiKV Details (#11936)

ref tikv/tikv#11119

Fix grid positions in TiKV Details dashboard.

Signed-off-by: tabokie <xy.tao@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* cdc: capture old value from txn layer dynamically (#11896)

close tikv/tikv#10091

Signed-off-by: hi-rustin <rustin.liu@gmail.com>
Signed-off-by: qupeng <qupeng@pingcap.com>

Co-authored-by: hi-rustin <rustin.liu@gmail.com>

* split_controller: reorganize the structs and add more comments (#11967)

 

Reorganize the structs and add more comments.

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: support Bit column push downBit column (#11968)

close tikv/tikv#11893, ref tikv/tikv#11893, ref pingcap/tidb#31884

Signed-off-by: xiejiandong <xiejiandong@pingcap.com>

* copr: support push down mod, sysdate to TiKV (#11970)

close tikv/tikv#11916, ref tikv/tikv#11916

Signed-off-by: xiejiandong <xiejiandong@pingcap.com>

* raftstore: propose in-memory pessimistic locks before prepare merge (#11758)

ref tikv/tikv#11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* stats_monitor: reformat stats_monitor tick (#11972)

 

Signed-off-by: lhy1024 <admin@liudos.us>

* copr: support extract scalar value from bit (#11980)

close tikv/tikv#11893

Signed-off-by: yisaer <disxiaofei@163.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* diagnosis: support get cpu time of threads for macOS (#11978)

close tikv/tikv#11977

Add the variant for macOS into `components/tikv_util/src/sys/thread.rs`

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Co-authored-by: Jay <BusyJay@users.noreply.github.com>

* rust: update toolchain to fix missing rls (#11954)

close tikv/tikv#11953

Signed-off-by: Connor1996 <zbk602423539@gmail.com>
Signed-off-by: Yilin Chen <sticnarf@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Co-authored-by: Yilin Chen <sticnarf@gmail.com>

* cdc: separate resolved region outliers (#11991)

Separate broadcasing outlier regions and normal regions,
so 1) downstreams know where they should send resolve lock requests,
and 2) resolved ts of normal regions does not fallback.

close pingcap/tiflow#4516
close pingcap/tiflow#4311
ref pingcap/tiflow#4146

Signed-off-by: Neil Shen <overvenus@gmail.com>

* *: collect key ranges for the read request in ResourceMeteringTag (#11995)

close tikv/tikv#11988

*: collect key ranges for the read request in ResourceMeteringTag

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* split_controller: move sample_threshold check out of the split_key iteration (#11986)

 

split_controller: move sample_threshold check out of the split_key iteration

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: support insertutf8 and lowerutf8 (#11987)

ref tikv/tikv#5751

Signed-off-by: yisaer <disxiaofei@163.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: support rest greatest/least functions (#12003)

ref tikv/tikv#5751

Signed-off-by: yisaer <disxiaofei@163.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: force compact with gentleness (#12006)

* force compact with gentleness

Signed-off-by: tabokie <xy.tao@outlook.com>

* address comment

Signed-off-by: tabokie <xy.tao@outlook.com>

* split_controller: refine the LOAD_BASE_SPLIT_EVENT metrics label definitions (#12010)

 

split_controller: refine the LOAD_BASE_SPLIT_EVENT metrics label definitions

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* split_controller: fix the incorrect sampled key ranges number check (#12013)

close tikv/tikv#12012

split_controller: fix the incorrect sampled key ranges number check

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: Support adjusting max_inflight_msgs dynamically (#11866)

close tikv/tikv#11865

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: fix missing flag solvement for bit column in decoding (#12016)

ref tikv/tikv#7, close pingcap/tidb#32506

Signed-off-by: yisaer <disxiaofei@163.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* file_system: overhaul proc-based io tracing utilities (#11873)

ref tikv/tikv#10867, fix tikv/tikv#11775

Signed-off-by: tabokie <xy.tao@outlook.com>

* raftstore: Wait for the apply index equals to the commit index (#11716)

ref tikv/tikv#10483

Signed-off-by: v01dstar <yang.zhang@pingcap.com>

* copr: revert 11968/11980/12016 (#12030)

ref tikv/tikv#11968, ref tikv/tikv#11980, ref tikv/tikv#12016, ref pingcap/tidb#32506

Signed-off-by: yisaer <disxiaofei@163.com>

* build: fix a Mac build issue (#12027)

ref tikv/tikv#10867, ref tikv/tikv#11873

Signed-off-by: tabokie <xy.tao@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: Extra physical table id column (#11931)

close tikv/tikv#11888

Added EXTRA_PHYSICAL_TABLE_ID_COL_ID support, to support TiDB's table partition dynamic prune mode, where a single request includes multiple partitions, but when Pessimistic lock (like `SELECT FOR UPDATE`) or ongoing transaction (having something in the tidb session local transaction buffer) each row needs its partition id / physical table ID to either lock that row or to check against the transaction buffer.

Signed-off-by: Mattias Jonsson <mjonss@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: hotfix panic from tokio-timer (#12004)

close tikv/tikv#11940, ref tikv/tikv#11940

*: hotfix panic from tokio-timer by enlarging the length of the level vector

Signed-off-by: you06 <you1474600@gmail.com>

* build: add arm64 check on SSE in makefile (#12035)

close tikv/tikv#12034

Signed-off-by: Jin Dong <djdongjin95@gmail.com>

Co-authored-by: Yilin Chen <sticnarf@gmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* replication mode: sync state to pd (#11751)

 

Signed-off-by: disksing <i@disksing.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: revert pessimistic locks status when failing to propose PrepareMerge (#11985)

ref tikv/tikv#11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* copr: support push bit column down (#12037)

ref pingcap/tidb#30738

Signed-off-by: yisaer <disxiaofei@163.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* split_controller: add more sample function test cases (#12058)

 

split_controller: add more sample function test cases

Signed-off-by: JmPotato <ghzpotato@gmail.com>

* copr: fix greatest/least time/date args type (#12056)

 

Signed-off-by: yisaer <disxiaofei@163.com>

* cdc: advancing resolved ts correctly for clusters with tiflash (#12050)

close pingcap/tiflow#4461

cdc: advancing resolved ts correctly for clusters with tiflash

Signed-off-by: qupeng <qupeng@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: bump master branch version to v6.0.0-alpha (#12077)

close tikv/tikv#12075

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* raftstore: check uninitialized destroy for merge (#12055)

close tikv/tikv#12048

When a peer is destroyed without being initialized, it will store
itself to peer list and the region epoch is missing. In merge if
such state is detected, it should abort merging instead of panicking.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: Add TruncateManager and TruncateWorker (#11553)

ref tikv/rfcs#81

Signed-off-by: longfangsong <longfangsong@icloud.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* TiKV supports buckets (#11763)

ref tikv/tikv#11759

Signed-off-by: qi.xu <tonxuqi@outlook.com>

Co-authored-by: qi.xu <tonxuqi@outlook.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* api_version: Codec for RawKV key (#12036)

ref tikv/tikv#11965

Key of RawKV is encoded as `user-key + memcomparable-padding + timestamp` in API v2.

Signed-off-by: pingyu <yuping@pingcap.com>

Co-authored-by: Andy Lok <andylokandy@hotmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: fix some typos (#12066)

 

Signed-off-by: cuishuang <imcusg@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: fix stale message cause panic (#12054)

close tikv/tikv#12023

raftstore: fix stale message cause panic

Signed-off-by: linning <linningde25@gmail.com>

* raftstore: ignore async fetch result when the peer is pending removed (#12038)

close tikv/tikv#11973, close tikv/tikv#12026

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: remove part about wechat in doc (#12101)

close tikv/tikv#12100

Signed-off-by: jackwener <jakevingoo@gmail.com>

* engine: upgrade raft engine (#12095)

ref tikv/tikv#165, ref tikv/raft-engine#165, ref tikv/tikv#11119

Signed-off-by: Randy <ztelur@gmail.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>

* *: Fix possible undefined behavior for transmuting vec (#12096)

close tikv/tikv#12070

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* github: Remove the pingcap/tidb-ansible from the PR template (#12102)

close tikv/tikv#12103

Signed-off-by: hi-rustin <rustin.liu@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: extract reschedule to inject a fail point (#12108)

 

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* raftstore: reactivate in-memory pessimistic locking when leader transfer fails (#11883)

ref tikv/tikv#11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: check memory locks for replica read only on the leader (#12115)

close tikv/tikv#12109

Consider a reader sends read index to the leader it supposed to be, but when
the leader receives the read index message, it has stepped down to a
follower. Without this commit, a peer will check the locks and re-fill the
read index context with the result no matter what role it is. So, when the
read index request is redirected to the new leader again, it no longer carries
the context of lock checking.

This commmit changes to only do the check when the peer is a leader. Then, the
read index request will remain unchanged before being redirected to the leader.

If the lease of the leader expires, it is still safe to check on the
uncertained leader. If the heartbeat check passes and no new leader is elected,
then the check works fine. If there is a new leader, when the old leader
becomes a follower, it will clear its pending read index. Then, the reader has
to resend the read index again to the correct leader. So, generally it is safe
as long as we guarantee the check only happens on the leader.

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* apiv2 rawkv: logical deletion flag (#12090)

ref tikv/tikv#11965

Signed-off-by: haojinming <jinming.hao@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: add projection (#10689)

close tikv/tikv#12114

Signed-off-by: ichn-hu <zfhu16@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Co-authored-by: Alex Chi <iskyzh@gmail.com>

* raftstore: adjust raft max msg size dynamically (#12018)

close tikv/tikv#12017

Signed-off-by: glorv <glorvs@163.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: update deps (#12098)

- Upgrade grpcio to 0.10
- Upgrade openssl to 1.1.1m
- Remove dependency on prost

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* tests: make raftstore tests about transactions more stable (#12122)

close tikv/tikv#12120

A new fail point "after_propose" is added to insert callback from the test.
Then we can ensure we release other key fail points after they take effect.

The other change is to retry request automatically to avoid stale command
errors.

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* metrics: support multi k8s in grafana dashboards (#12008)

close tikv/tikv#12104

Signed-off-by: just1900 <legendj228@gmail.com>

* storage: set in-memory pessimistic lock feature gate to 6.0.0 (#12078)

ref tikv/tikv#11452

The in-memory pessimistic lock feature should be only enabled after
all TiKV instances have been upgraded to the new version. Otherwise,
transferring leader may be blocked because of the new protocol and
procedure when in-memory pessimistic locks exist.

So, we need to use a feature gate to make sure this feature is only
enabled after all TiKVs have been upgraded to the new version.

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore,cdc: pass leader transferee to cdc observer (#12124)

Signed-off-by: qupeng <qupeng@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* config: simplify handling of nullable configurations (#12133)

Support deriving `O…
CalvinNeo added a commit to pingcap/tidb-engine-ext that referenced this issue May 30, 2022
…roxy (#72)

* ctl: validate column family name when scanning data using ctl (#11269)

* ctl: should check column name validation when scan data using ctl

close #11268
Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* fix format problem

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>

* txn: Do assertions on data for mutations to reduce potential data inconsistency (#10977)

* Add mutation checking logic

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix build

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix usages of mutation

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Update kvproto and renaming

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add a simple test

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix incorrect assertion

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Support check existance during pessimistic lock

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add assertion level support

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add tests

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add more test cases

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add test for check_existence of pessimistic_lock request

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix build

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* ref #11542

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Add a metric for counting different way of checking assertion, for reference about the performance impact

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* metrics: fix memory metrics (#11656)

Signed-off-by: Shuaipeng Yu <jackysp@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* coprocesser: refactor encoding and add gbk_chinese_ci map (#11661)

* close #11660

Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com>

* close #11660

Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* storage: Add API V2 check for RawKV and TxnKV requests (txn part) (#11216)

* wip

Signed-off-by: pingyu <yuping@pingcap.com>

* wip

Signed-off-by: pingyu <yuping@pingcap.com>

* ref #10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref #10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref #10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref #10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref #10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref #10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

* fix enable_ttl

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* simplify test

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* ref #10974: Add API V2 check for RawKV and TxnKV requests (txn)

Signed-off-by: pingyu <yuping@pingcap.com>

Co-authored-by: andylokandy <andylokandy@hotmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* cdc: load uninlined value more effectively (#11615)

Signed-off-by: qupeng <qupeng@pingcap.com>

* cdc: load old value with prefix seek if possible (#11643)

* close #11642

Signed-off-by: qupeng <qupeng@pingcap.com>

* a more comment

Signed-off-by: qupeng <qupeng@pingcap.com>

* ctl: split into multiple modules (#11658)

* ctl: split into multiple modules

ref #10938

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* refactor if let to match

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* Fix build issues on non-linux platforms (#11671)

* close #11676

Signed-off-by: devillove084 <786537003@qq.com>

* all: Migrate from std:: types to primitives. (#11677)

* Deprecation is planned for MIN,MAX,EPSLION,INFINITY,NEG_INFINITY of
  the (i|u|f)(8|16|32|64) types in standard.

ref #11678

```release-note
NONE
```

Signed-off-by: Harold Dost <harolddost@gmail.com>

* metrics alert: fix apply cpu alert (#11672)

Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* resrc_mtr: introduce scanned keys statistics (#11085)

close #11690

Signed-off-by: mornyx <mornyx.z@gmail.com>

* file_system: support collect the io flow by proc (#11667)

* file_system: support collect the read flow by proc

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* fix lint

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* fix clippy

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* Delete unused content & Adjust copyright text

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Add bench_fetch_io_bytes

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* fix lint

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Integrate thread_io

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* record bytes in request

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* fix logical bug

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Update test write_bytes

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Delete unused & Rename func

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Add test for read & Update test for write

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Add buffered thread io

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Replace DashMap with ThreadLocal

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* Rename variables and function & Remove unused call

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Change fetch logic. close #10867

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Add AtomicIOBytes

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Split ThreadIOSentinel

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* add THREAD_IO_TOTAL for metrics bytes fetcher

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* Adjust code style & fix

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* improve write_bytes test

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: check epoch and validness of lock table when writing pessimistic locks (#11591)

ref #11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* rsmeter: introduce datasink (#11688)

* add datasink trait

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* RAII removes on_reporter_closing

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fmt

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix build

ref #11691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* datasink -> data_sink

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fmt

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix build

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* clippy: Fixing up clippy errors. (#11193)

* clippy: Fixing up clippy errors.

* Fix crate references
* Fix anonymous lifetimes references.

ref #4301
Signed-off-by: Harold Dost <harolddost@gmail.com>

* Test Revert of Iterations

ref #4301

Signed-off-by: Harold Dost <harolddost@gmail.com>

* clippy: Enable check for rust-2018-idioms

ref #4301

Signed-off-by: Harold Dost <harolddost@gmail.com>

* rsmeter: add centralized place to construct protobuf data (#11696)

ref #11691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* tikv_util: update procfs to 0.12.0 (#11703)

Signed-off-by: linning <linningde25@gmail.com>

* Use generic for api version to reduce runtime branching (#11687)

* Use generic for api version to reduce runtime branching

ref #10938

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* fix test

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* add test

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* fix clippy

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* fmt

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* address comment

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* rename vars

Signed-off-by: andylokandy <andylokandy@hotmail.com>

* improve vars

Signed-off-by: andylokandy <andylokandy@hotmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* sst_importer: check api version when importing (#11664)

* sst_importer: check api version when importing

ref #10938

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Fix bugs

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* fix bug of Iterator::{scan,scan_cf}

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* remove redundant log

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Show the entire key

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Fix lint issue

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* resolve conflict

Signed-off-by: andylokandy <andylokandy@hotmail.com>

Co-authored-by: Andy Lok <andylokandy@hotmail.com>

* backup: pipeline scan and save step (#11528)

* br: sperate io and scan threads

Signed-off-by: Yu Juncen <yu745514916@live.com>

* br/stream: make clippy happy

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br/stream: fix tests

Signed-off-by: Yu Juncen <yu745514916@live.com>

* backup: fix a dummy bug

Signed-off-by: Yu Juncen <yu745514916@live.com>

* backup: add config of io-threads

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: ref #11350: some minior change

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: added some metrics

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: better tuning concurrency, and added a metric

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: added some slow log and metrics

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: address comments

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: fix build

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* Revert "backup: fix build"

This reverts commit 74537be56c410d0f91af2f48b7e65356ab53720e.

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* Revert "backup: address comments"

This reverts commit 77d75756028bea3496f878f0ad8bbc562f5d00aa.

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: address comments

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: always set IO type to Export

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup:make clippy happy

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: address comments

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* metrics: fix the metrics cannot be displayed (#11710)

ref #11662

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* backup: return api-version to BR when backup (#11704)

* Update kvproto

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Return api-version for br

ref #10938

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Reformat code

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

* Update components/external_storage/export/src/export.rs

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

Co-authored-by: Andy Lok <andylokandy@hotmail.com>

* format code

Signed-off-by: Peng Guanwen <pg999w@outlook.com>

Co-authored-by: Andy Lok <andylokandy@hotmail.com>

* file_system: limit thread_io target os (#11715)

* limit thread_io target os

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* limit thread_io target os. typo. close #11698

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

Co-authored-by: Yilin Chen <sticnarf@gmail.com>

* file_system: bypass file allocate implementation in fs2 (#11700)

Close #10688

Patch fs2 to https://github.com/tabokie/fs2-rs/tree/tikv. In which, `posix_fallocate` is replaced with `fallocate` (https://github.com/danburkert/fs2-rs/pull/42).

* bypass file allocate implementation in fs2

Signed-off-by: tabokie <xy.tao@outlook.com>

* metrics: remove min legend of some Grafana panels (#11487)

Minimum legend isn't very useful, and it occupied valuable visual real estate.

Remove some min legends from Grafana metrics, except for the following:
- Level 0 chance
- Number files at each level
- File Count
- Entry Count
- Allocator Stats
- Encrypted files
- Encryption meta files size

Ref #11119

Signed-off-by: tabokie <xy.tao@outlook.com>

* coprocessor: not return rows when there is no input for simple aggregation (#11708)

* not return rows when there is no input for simple aggregation

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

* close #11735, and address comments

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

* backup: reduce the default thread pool size of backup, enable auto tune by default (#11699)

* backup: clamp auto tune values

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* ref #11000: change the default backup threads.

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: run tasks in background

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: run rustfmt

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: set remain threads to 20% of vcpu

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* br-stream: fix build

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* Backup: add S3 metrics && add s3_multi_part_size config  (#11666)

* s3: add request metrics for s3 storage

Signed-off-by: 3pointer <luancheng@pingcap.com>

* s3: add grafana json

Signed-off-by: 3pointer <luancheng@pingcap.com>

* br: add config for s3 multi part upload

Signed-off-by: 3pointer <luancheng@pingcap.com>

* update comment

Signed-off-by: 3pointer <luancheng@pingcap.com>

* update comment

Signed-off-by: 3pointer <luancheng@pingcap.com>

* address comment

Signed-off-by: 3pointer <luancheng@pingcap.com>

* hidden the new config

Signed-off-by: 3pointer <luancheng@pingcap.com>

* only hidden config in config-template.toml

Signed-off-by: 3pointer <luancheng@pingcap.com>

* address comment

Signed-off-by: 3pointer <luancheng@pingcap.com>

* close #11727 && format

Signed-off-by: 3pointer <luancheng@pingcap.com>

* fix

Signed-off-by: 3pointer <luancheng@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* rsmeter: support multiple datasinks (#11707)

* rsmeter: support multiple datasinks

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* wrap config notifier

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* add unit tests

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* polish
ref #11691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix test

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix outdated comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* engine: properly estimate store size (#11728)

* properly estimate store size, ref #11119

Signed-off-by: tabokie <xy.tao@outlook.com>

* address comments

Signed-off-by: tabokie <xy.tao@outlook.com>

* implement global config client for tikv (#11685)

* implement global config client

close #11686

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* implement unit test for global config in pd client

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* resolved formatting issues

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* impl pd client asyncly

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* according to reviewer's opinion
removing store method implementation

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* Merge branch 'master' of https://github.com/tikv/tikv into global_conf

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* write test case for watch global config when
grpc server is closed.

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* resove issues

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* clippy

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

* watch global config rentry future in test deadlock just removed

Signed-off-by: lemonhx <lemonhx@lemonhx.tech>

Co-authored-by: Zhenchi <zhongzc_arch@outlook.com>

* storage: skip scanning lock when using rc (#11701)

* skip scanning lock when using rc, ref #11485

Signed-off-by: you06 <you1474600@gmail.com>

* format code

Signed-off-by: you06 <you1474600@gmail.com>

* add test

Signed-off-by: you06 <you1474600@gmail.com>

* address comment

Signed-off-by: you06 <you1474600@gmail.com>

* add an integration test

Signed-off-by: you06 <you1474600@gmail.com>

* format code

Signed-off-by: you06 <you1474600@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: pushdown substring to tikv (#11494)

* corp: pushdown substring to tikv

Signed-off-by: guo-shaoge <shaoge1994@163.com>

* fix case. close #11495

Signed-off-by: guo-shaoge <shaoge1994@163.com>

* fix comment

Signed-off-by: guo-shaoge <shaoge1994@163.com>

* rsmeter: distinguish between collectors and observers within recorder  (#11712)

* rsmeter: support multiple datasinks

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* wrap config notifier

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* add unit tests

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* polish
ref #11691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* rsmeter: distinguish between collectors and observers within recorder

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fmt & try to resolve #11689

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* add unit tests
ref #11691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* retrigger test

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* address comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix test

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix outdated comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* Update components/resource_metering/src/recorder/sub_recorder/mod.rs

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Yexiang Zhang <mornyx.z@gmail.com>

* address comment

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* remove enabled

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Co-authored-by: Wenxuan <breezewish@pingcap.com>
Co-authored-by: Yexiang Zhang <mornyx.z@gmail.com>

* config: validate online configurable thread pools (tikv#11159) (#11714)

ref #11159

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* res_meter: upgrade kvproto (#11749)

* Upgrade kvproto

close #11748

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Fix test compilation

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Fix test compilation

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Fix test compilation

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Use master kvproto

Signed-off-by: mornyx <mornyx.z@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* rsmeter: add pubsub datasink  (#11719)

* rsmeter: add pubsub datasink
close #11691

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix build

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* Update components/resource_metering/src/reporter/pubsub.rs

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Wenxuan <hi@breeswish.org>

* fix build

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Wenxuan <hi@breeswish.org>

* status_server: Support online config update for configuration hosting platform (#11693)

close #11692

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* res_meter: optimize top k (#11753)

* Keep top k on recorder

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Modify comments

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Fix pd.rs

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Refactor

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Remove import

Signed-off-by: mornyx <mornyx.z@gmail.com>

* max_resource_groups 200 -> 100

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Fix ut

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Use iterator to avoid clone

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Optimize

close #11754

Signed-off-by: mornyx <mornyx.z@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftclient: delay flush (#11705)

* raftclient: delay flush

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* remove raftstore wait

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* Ref #11309.

tikv#11310 introduces delay in raftstore thread to reduce the flush rate.
It can only reduce flush rate from one thread. If there are many
raftstore threads or IO threads, messages can still be flush frequently.
This PR reduces the flush rate by introducing delay in raft client. So
delay will work at connection level and achieve the maximun batch
affect.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* remove useless field

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* address comment

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* fix unstable raw get case

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Liqi Geng <gengliqiii@gmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* cloud: support azure blob storage as external storage backend (#11732)

* support azure storage blob for external storage

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* modify the priority of the azure login methods

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* fix feature match branch

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* implement token update retriable

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* fix some problems

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* close #11731

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* rustfmt

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* make clippy success

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* simplify uploader

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* commit some suggestions

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* coprocesser: update encoding lib (#11506)

Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* coprocesser: implement upper and lower function on GBK charset (#11756)

Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* logger: support archiving and rotation (#11651) (#11657)

* logger: support archiving and rotation (#11651)

close #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: add testcases to test dt_from_file_name (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: simplify DateTime format (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: do a little refactoring (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: do a little refactoring (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: do a little refactoring (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: do a little refactoring (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: do a little refactoring (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* diagnostics: refactor  (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: run clippy for integration tests (#11768)

* run clippy --fix with nightly-2021-12-17

Signed-off-by: tabokie <xy.tao@outlook.com>

* manual clippy

Signed-off-by: tabokie <xy.tao@outlook.com>

* format and clippy

Signed-off-by: tabokie <xy.tao@outlook.com>

* enable default features when running clippy, ref #11119

Signed-off-by: tabokie <xy.tao@outlook.com>

* fix build

Signed-off-by: tabokie <xy.tao@outlook.com>

* revert box_collection lint

Signed-off-by: tabokie <xy.tao@outlook.com>

* Update tests/integrations/raftstore/test_bootstrap.rs

Signed-off-by: Xinye Tao <xy.tao@outlook.com>

Co-authored-by: Lei Zhao <zlwgx1023@gmail.com>

Co-authored-by: Lei Zhao <zlwgx1023@gmail.com>

* logger: fix config-template's log unit (#11651) (#11777)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: improve testcase (#11651) (#11778)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* raftstore: propose in-memory pessimistic locks before leader transfer (#11577)

* *: check epoch and validness of lock table when writing pessimistic locks

ref #11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* raftstore: propose in-memory pessimistic locks before leader transfer

ref #11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* mark delete in scheduler and delete after apply

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* Add tests for two different orders between proposing locks and write command

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* do not clear pessimistic locks before transfer leader

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* delete memory locks in the apply thread and add more term and version check

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* remove potentially incorrect assert

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* check term and version when reading the lock table

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* revert a log change

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* change term when becoming follower

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* fix lint

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* do not check version when reading locks (temporarily)

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* fix another test

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* only treat MsgTransferLeader with context with valid msg

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* fix a tiny comment error

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* make clippy happy

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* res_meter: ignore the read keys test temporarily (#11779)

ref #11765

Signed-off-by: mornyx <mornyx.z@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* logger: change critical to fatal (#11651) (#11780)

* logger: change critical to fatal (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: fix test_parse_log_level (#11651)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* update CHANGELOG.md (#11790)

* update from tikv-server v5.0.1(20210423)

from https://github.com/tikv/tikv/releases

Signed-off-by: xiejiandong <xiejiandong@pingcap.com>

* update CHANGELOG.md close #11167

Signed-off-by: xiejiandong <xiejiandong@pingcap.com>

* config: relax the bounds of online configurable worker number (#11651) (#11798)

close #11776

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

* logger: invoke logger_compatible_adjust before run_tikv (#11651) (#11792)

close #11789

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: unify thread related native APIs (#11785)

* *: unify thread related native APIs

Now there are duplicated codes calling syscalls to fetch thread
informations. This PR provides unified APIs and remove duplicated codes.

Close #11784.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* address comment

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* declare platform compatability

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* Fix method on_tick will not update metric when enable_auto_tune is false (#11791)

* Fix method on_tick will not update metric when enable_auto_tune is false
ref #11787
Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* resolve format problem in discussion

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* resolve format problem in discussion again

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: add a new tick to renew lease in advance (#6427)

* renew lease in raft tick

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check lease before sending read index

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* renew lease on raft tick to cover lease till next tick

Signed-off-by: linning <linningde25@gmail.com>

* refactor

Signed-off-by: linning <linningde25@gmail.com>

* don't use inspect_lease

Signed-off-by: linning <linningde25@gmail.com>

* fix ci

Signed-off-by: linning <linningde25@gmail.com>

* add test

Signed-off-by: linning <linningde25@gmail.com>

* remove unneed comment

Signed-off-by: linning <linningde25@gmail.com>

* close #5388

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check pending_read_count before hibernating

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* increase the range to check lease expired

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* renew leader lease after not hibernate

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix hibernate states metrics

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check writes and pre_read_index before proposing read index

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* use a individual tick to renewing lease

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix condition

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* stablize test_node_renew_lease

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* cleanup unused changes

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add config item for interval of tick

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* rename tick

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* disable the tick for test_inconsistent_configuration

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* change deafault value of check_leader_lease_interval

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* disable the tick for test_read_hibernated_region

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* count renew lease read index into raft metrics

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* disable the tick for test_renew_lease

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix lint error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add a 100ms buffer for renew_bound

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* make clippy happy

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: linning <linningde25@gmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* cloud: Add retry for azure blob server busy error (#11813)

* add retry for azure blob server busy error

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* close#11812

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

* use to_string

Signed-off-by: Leavrth <jianjun.liao@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* tikv_util: make cgroup parsing more robust (#11786)

* make cgroup parsing more robust

Signed-off-by: tabokie <xy.tao@outlook.com>

* fix error message

Signed-off-by: tabokie <xy.tao@outlook.com>

* parse cpu quota in floats

Signed-off-by: tabokie <xy.tao@outlook.com>

* *: make it compile on macos (#11825)

* *: make it compile on macos

close #11823

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* remove extra comment

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* resolved_ts: add more metrics (#11809)

* add metrics

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add slow log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* remove duplicate metric

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add resolved ts metrics for leader peers

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* close #11767

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* init endpoint with store meta

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* rsmeter: fix perf regression caused by arc swap  (#11833)

* rsmeter: fix perf regression caused by arc swap

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* use atomic cell instead

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* debug assert

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Wenxuan <breezewish@pingcap.com>

* load base split: more accurate sample for large key ranges batch (#11039)

* more accurate sample for large key ranges

Signed-off-by: lhy1024 <admin@liudos.us>

* update

Signed-off-by: lhy1024 <admin@liudos.us>

* ref #11521

Signed-off-by: lhy1024 <admin@liudos.us>

* address comment

Signed-off-by: lhy1024 <admin@liudos.us>

* fix test

Signed-off-by: lhy1024 <admin@liudos.us>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftclient: enable delay only in high load (#11772)

and improve CPU efficiency under high load. But it turns out
this can lead to significant regression in normal load. So
this PR adds CPU stats and only enabling delay in high load.

This is also the same strategy used in v4.x.

Close #11769.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* tikv-ctl: fix tikv-ctl's output is incomplete on calling process::exit (#11231)

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>
Co-authored-by: Connor <zbk602423539@gmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: skip deleting snapshot files in peer pending_remove is true (#11782)

* raftstore: skip deleting snapshot files in peer when the peer is pending removal and the snapshot is being applied and canceled -- close #11746

This is to avoid the potential panic when the snapshot files are deleted, but the peer's status (Tombstone) is not persisted in disk due to tikv crash.

Signed-off-by: tonyxuqqi <tonyxuqi@outlook.com>

* address code review feedback  -- close #11746

Signed-off-by: qi.xu <tonxuqi@outlook.com>

* address code review feedback 2  -- close #11746

Signed-off-by: qi.xu <tonxuqi@outlook.com>

* address code review feedback 2  -- close #11746

Signed-off-by: qi.xu <tonxuqi@outlook.com>

* address code review feedback 2  -- close #11746

Signed-off-by: qi.xu <tonxuqi@outlook.com>

* address code review feedback 2  -- close #11746

Signed-off-by: qi.xu <tonxuqi@outlook.com>

Co-authored-by: qi.xu <tonxuqi@outlook.com>

* load_statis: fix compilation on macos (#11851)

close #11772

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* config: override rocksdb config when flow control enabled (#11840)

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* rocksdb: consider env sharing when configuring background threads (#11760)

#9649 increased the default setting of kvdb background flushes (3 for 8c and 4 for 16c).

This PR takes another approach: use smaller concurrency for individual dbs, but configure the global thread pool size with the sum of kvdb and raftdb concurrency. This way we won't over-configure flush workers when raft-engine is enabled.

* consider env sharing when configuring background threads

ref #11119

Signed-off-by: tabokie <xy.tao@outlook.com>

* server/config: keep compatible using option (#11862)

* server/config: keep compatible using option

Using `Option` for `heavy_load_wait_duration` to keep compatible with
versions prior to v5.3.

Close #11861.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* fix format

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* fix clippy

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: zhouqiang <zhouqiang.cl@gmail.com>

* raftstore: fix missing workers' stop on shutdown (#11864)

ref #11159

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: move in-memory pessimistic locks to split regions (#11655)

ref #11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: Remove the entrance of enabling io snooper. (#11394)

* *: Remove the entrance of enabling io snooper. close #10867

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* fix

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

* fix test

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* fix test

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: change PeerTicks to enum (#11849)

* raftstore: change PeerTicks to enum

close #11848

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* use bool array for registry

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* rename PeerTicks to PeerTick

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* add associated const for number of PeerTick types

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* change naming

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: check memory limit when inserting in-memory pessimistic locks (#11841)

* *: check memory limit when inserting in-memory pessimistic locks

ref #11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* fix merged upstream

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* fix test

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* fix typo

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* *: fix invalid failpoint caused by typo (#11709)

* fix invalid failpoint caused by typo

close #11734

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* address the comment

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* address the comment

Signed-off-by: Ryan Leung <rleungx@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: Extract significant router (#11750)

* ref#11409 add significant router

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* make format

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* address comment

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* raftstore: move scan delete to raft log gc worker (#11853)

* raftstore: move scan delete to raft log gc worker

When clearing raft metas, raftstore will scan raft logs and delete them
one by one. Seeking can be slow if there are a lot of tombstone keys.
This PR moves the operation to raft log gc worker to reduce the impact.

The final solution should be also moving remaining IO operations to
async write IO threads.

Close #10210.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* address comment

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* speed up destroy

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* fix compile

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* further speed up

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* revert test case configuration

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* address comment

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* address comment

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* doc: update new rules for linking issue and commit message (#11832)

close #11097, close #11831

Signed-off-by: zhangyangyu <angwerzx@126.com>

Co-authored-by: Mini256 <minianter@foxmail.com>

Co-authored-by: Mini256 <minianter@foxmail.com>
Co-authored-by: Xiaoguang Sun <sunxiaoguang@users.noreply.github.com>

* *: update rust toolchain to 2022-01-07 (#11875)

* update rust toolchain to 2022-01-17

Signed-off-by: tabokie <xy.tao@outlook.com>

* address comment: clean up unnecessary `as_ref()`

Signed-off-by: tabokie <xy.tao@outlook.com>

* try fixing the underflow

Signed-off-by: tabokie <xy.tao@outlook.com>

* mute underflow warning for raft metrics

Signed-off-by: tabokie <xy.tao@outlook.com>

* clean up unused data members

Signed-off-by: tabokie <xy.tao@outlook.com>

* format

Signed-off-by: tabokie <xy.tao@outlook.com>

* step back to 2022-01-07

Signed-off-by: tabokie <xy.tao@outlook.com>

* readme: change images based on github theme (#11795)

Signed-off-by: sloorush <aarush.bhatt@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: don't remove other peer's read delegate (#11882)

* don't remove read delegate besides peer_destroy

Signed-off-by: linning <linningde25@gmail.com>

* add test case

Signed-off-by: linning <linningde25@gmail.com>

* make clippy happy

Signed-off-by: linning <linningde25@gmail.com>

* address comment

Signed-off-by: linning <linningde25@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* logger: use eprintln! if the logger is not initialized (#11869)

ref #11651

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: fix the calculation of total column size in analyze (#11884)

Signed-off-by: xuyifan <xuyifangreeneyes@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* engine: update raft-engine for data consistency fix (#11885)

Fix https://github.com/tikv/raft-engine/issues/142.

When encountering this bug, TiKV will panic with message "applied index > max(commit index, recorded commit index)" after restart.

* update raft-engine

Signed-off-by: tabokie <xy.tao@outlook.com>

* update one more commit

Signed-off-by: tabokie <xy.tao@outlook.com>

* check docker build during clippy, ref #11312 (#11819)

Signed-off-by: tabokie <xy.tao@outlook.com>

* gc_worker: fix incorrect scheduled_tasks counting (#11904)

* gc_worker: fix incorrect scheduled_tasks counting

close #11903

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* remove check_is_busy totally

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* do not use wildcard match in error handling

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* raftstore: Introduce raft log fetcher (#11900)

* ref#11409 introduce raft log fetcher

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* update kvproto

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* address comment

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* fix test build

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* fix clippy

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* limit capacity

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* update kvproto

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* call stop on worker

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* rename worker

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* status_server: add pprof flamegraph response header (#10951)

* tweak(status_server): add pprof flamegraph response header

Signed-off-by: Suhaha <jklopsdfw@gmail.com>

* test(status_server): add Content-Type asset to test_pprof_profile_service

Signed-off-by: Suhaha <jklopsdfw@gmail.com>

* tweak(status_server): add pprof flamegraph response header

Signed-off-by: Suhaha <jklopsdfw@gmail.com>

* tweak(status_server): add pprof flamegraph response header close #11917

Signed-off-by: Suhaha <jklopsdfw@gmail.com>

* close #11917

Signed-off-by: Suhaha <jklopsdfw@gmail.com>

Co-authored-by: goroutine <ngaut@users.noreply.github.com>
Co-authored-by: Connor <zbk602423539@gmail.com>

* raftstore: renew leader lease in advance when handle read request (#9307)

* renew lease advance

Signed-off-by: linning <linningde25@gmail.com>

* add log

Signed-off-by: linning <linningde25@gmail.com>

* make clippy happy

Signed-off-by: linning <linningde25@gmail.com>

* ref #11579

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* reset raft tick

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* set has ready

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add renew_leader_lease_advance_duration config

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix panic

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* disable renewing for test_lease_unsafe_during_leader_transfers

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: 5kbpers <tangminghua@pingcap.com>

* pprof: ignore cpu profiling on non-x86 arch (#11925)

* Disable cpu profiling on non-x86_64 arch

Signed-off-by: mornyx <mornyx.z@gmail.com>

* Fix warns

Signed-off-by: mornyx <mornyx.z@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* github: add new pr requirement for linking issue and commit message (#11887)

Signed-off-by: zhangyangyu <angwerzx@126.com>

Co-authored-by: Xiaoguang Sun <sunxiaoguang@users.noreply.github.com>

* engine: enable raft engine by default (#11928)

* enable raft engine by default and synchronize docs

Signed-off-by: tabokie <xy.tao@outlook.com>

* update raft engine

Signed-off-by: tabokie <xy.tao@outlook.com>

* update raft-engine and fix tests

Signed-off-by: tabokie <xy.tao@outlook.com>

* gc_worker: Limit the key range to scan for GcKeys tasks (#11922)

close tikv/tikv#11752, close tikv/tikv#11902

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: TransferLeader support multiple target peers (#11063)

ref tikv/tikv#822, ref tikv/tikv#4214, close tikv/tikv#10602

Signed-off-by: MrCroxx <mrcroxx@outlook.com>

* raftstore: remove the leaders field of `StoreMeta` (#11934)

close tikv/tikv#11933

Remove `StoreMeta.leaders`
```

### Related changes

### Check List <!--REMOVE the items that are not applicable-->

Tests <!-- At least one of them must be included. -->

- Manual test (add detailed scripts or steps below)
`cargo check`

### Release note <!-- bugfixes or new feature need a release note -->

```release-note
None

Signed-off-by: linning <linningde25@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: Fetch raft log in async manner (#11409)

close tikv/tikv#10408, close tikv/tikv#11320

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: add quarter function (#11935)

ref tikv/tikv#5751

Signed-off-by: zhongyong jin <AT-Fieldless@outlook.com>

* Enable full debug info for dev and test (#11949)

ref tikv/tikv#5049, ref tikv/tikv#5572, close tikv/tikv#5572

```

### Related changes

- PR to update `pingcap/docs`/`pingcap/docs-cn`:
- PR to update `pingcap/tidb-ansible`:
- Need to cherry-pick to the release branch
-->

### Check List <!--REMOVE the items that are not applicable-->

Tests <!-- At least one of them must be included. -->

- Manual test (add detailed scripts or steps below)
- No code

Manual test:
Linux box with 8 cores (3:50 -> 4:00):
```
# Before this PR:
$ make clean
cargo clean
rm -rf bin dist
$ time make build
cargo build --no-default-features --features "jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"
   Compiling libc v0.2.106
...
   Compiling server v0.0.1 (tikv/components/server)
    Finished dev [unoptimized + debuginfo] target(s) in 3m 50s

real	3m50,487s
user	38m11,859s
sys	3m9,540s

# After this PR:
$ git diff
diff --git a/Cargo.toml b/Cargo.toml
index 71f5329d3..67cb9d183 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -267,7 +267,7 @@ default-members = ["cmd/tikv-server", "cmd/tikv-ctl"]
 
 [profile.dev]
 opt-level = 0
-debug = 1 # required for line numbers in tests, see tikv #5049
+debug = true
 codegen-units = 4
 lto = false
 incremental = true
@@ -293,7 +293,7 @@ codegen-units = 4
 
 [profile.test]
 opt-level = 0
-debug = 1 # enable line numbers by default for easy test debugging
+debug = true
 codegen-units = 16
 lto = false
 incremental = true
mattias@msig:~/repos/tikv$ time make clean
cargo clean
rm -rf bin dist

real	0m0,975s
user	0m0,148s
sys	0m0,828s
mattias@msig:~/repos/tikv$ time make build
cargo build --no-default-features --features "jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"
   Compiling libc v0.2.106
...
   Compiling server v0.0.1 (tikv/components/server)
    Finished dev [unoptimized + debuginfo] target(s) in 4m 00s

real	4m0,201s
user	39m45,037s
sys	3m16,397s
```

Macbook Air M1:
```
# Before:
% time make build
cargo build --no-default-features --features " jemalloc test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"
   Compiling libc v0.2.106
....
   Compiling server v0.0.1 (tikv/components/server)
    Finished dev [unoptimized + debuginfo] target(s) in 4m 01s
make build  1107.42s user 116.20s system 506% cpu 4:01.46 total

# After:
% time make build
cargo build --no-default-features --features " jemalloc test-engines-rocksdb cloud-aws cloud-gcp cloud-azure"
   Compiling libc v0.2.106
....
   Compiling server v0.0.1 (tikv/components/server)
    Finished dev [unoptimized + debuginfo] target(s) in 4m 10s
make build  1179.39s user 120.74s system 518% cpu 4:10.98 total
```

Side effects

- Performance regression, Only when building, to the benefit of full debug info by default.


### Release note <!-- bugfixes or new feature need a release note -->

```release-note
None

Signed-off-by: Mattias Jonsson <mjonss@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* correct a metric about compaction filter (#11938)

 

correct a metric about compaction filter

Signed-off-by: qupeng <qupeng@pingcap.com>

* raft: Fix possible panic on entries fetched callback (#11961)

close tikv/tikv#11951

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* split_controller: refine the sample function and add some comments (#11952)

 

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* metrics: fix grid position in TiKV Details (#11936)

ref tikv/tikv#11119

Fix grid positions in TiKV Details dashboard.

Signed-off-by: tabokie <xy.tao@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* cdc: capture old value from txn layer dynamically (#11896)

close tikv/tikv#10091

Signed-off-by: hi-rustin <rustin.liu@gmail.com>
Signed-off-by: qupeng <qupeng@pingcap.com>

Co-authored-by: hi-rustin <rustin.liu@gmail.com>

* split_controller: reorganize the structs and add more comments (#11967)

 

Reorganize the structs and add more comments.

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: support Bit column push downBit column (#11968)

close tikv/tikv#11893, ref tikv/tikv#11893, ref pingcap/tidb#31884

Signed-off-by: xiejiandong <xiejiandong@pingcap.com>

* copr: support push down mod, sysdate to TiKV (#11970)

close tikv/tikv#11916, ref tikv/tikv#11916

Signed-off-by: xiejiandong <xiejiandong@pingcap.com>

* raftstore: propose in-memory pessimistic locks before prepare merge (#11758)

ref tikv/tikv#11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* stats_monitor: reformat stats_monitor tick (#11972)

 

Signed-off-by: lhy1024 <admin@liudos.us>

* copr: support extract scalar value from bit (#11980)

close tikv/tikv#11893

Signed-off-by: yisaer <disxiaofei@163.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* diagnosis: support get cpu time of threads for macOS (#11978)

close tikv/tikv#11977

Add the variant for macOS into `components/tikv_util/src/sys/thread.rs`

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Co-authored-by: Jay <BusyJay@users.noreply.github.com>

* rust: update toolchain to fix missing rls (#11954)

close tikv/tikv#11953

Signed-off-by: Connor1996 <zbk602423539@gmail.com>
Signed-off-by: Yilin Chen <sticnarf@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Co-authored-by: Yilin Chen <sticnarf@gmail.com>

* cdc: separate resolved region outliers (#11991)

Separate broadcasing outlier regions and normal regions,
so 1) downstreams know where they should send resolve lock requests,
and 2) resolved ts of normal regions does not fallback.

close pingcap/tiflow#4516
close pingcap/tiflow#4311
ref pingcap/tiflow#4146

Signed-off-by: Neil Shen <overvenus@gmail.com>

* *: collect key ranges for the read request in ResourceMeteringTag (#11995)

close tikv/tikv#11988

*: collect key ranges for the read request in ResourceMeteringTag

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* split_controller: move sample_threshold check out of the split_key iteration (#11986)

 

split_controller: move sample_threshold check out of the split_key iteration

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: support insertutf8 and lowerutf8 (#11987)

ref tikv/tikv#5751

Signed-off-by: yisaer <disxiaofei@163.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: support rest greatest/least functions (#12003)

ref tikv/tikv#5751

Signed-off-by: yisaer <disxiaofei@163.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: force compact with gentleness (#12006)

* force compact with gentleness

Signed-off-by: tabokie <xy.tao@outlook.com>

* address comment

Signed-off-by: tabokie <xy.tao@outlook.com>

* split_controller: refine the LOAD_BASE_SPLIT_EVENT metrics label definitions (#12010)

 

split_controller: refine the LOAD_BASE_SPLIT_EVENT metrics label definitions

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* split_controller: fix the incorrect sampled key ranges number check (#12013)

close tikv/tikv#12012

split_controller: fix the incorrect sampled key ranges number check

Signed-off-by: JmPotato <ghzpotato@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: Support adjusting max_inflight_msgs dynamically (#11866)

close tikv/tikv#11865

Signed-off-by: Wenbo Zhang <ethercflow@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: fix missing flag solvement for bit column in decoding (#12016)

ref tikv/tikv#7, close pingcap/tidb#32506

Signed-off-by: yisaer <disxiaofei@163.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* file_system: overhaul proc-based io tracing utilities (#11873)

ref tikv/tikv#10867, fix tikv/tikv#11775

Signed-off-by: tabokie <xy.tao@outlook.com>

* raftstore: Wait for the apply index equals to the commit index (#11716)

ref tikv/tikv#10483

Signed-off-by: v01dstar <yang.zhang@pingcap.com>

* copr: revert 11968/11980/12016 (#12030)

ref tikv/tikv#11968, ref tikv/tikv#11980, ref tikv/tikv#12016, ref pingcap/tidb#32506

Signed-off-by: yisaer <disxiaofei@163.com>

* build: fix a Mac build issue (#12027)

ref tikv/tikv#10867, ref tikv/tikv#11873

Signed-off-by: tabokie <xy.tao@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: Extra physical table id column (#11931)

close tikv/tikv#11888

Added EXTRA_PHYSICAL_TABLE_ID_COL_ID support, to support TiDB's table partition dynamic prune mode, where a single request includes multiple partitions, but when Pessimistic lock (like `SELECT FOR UPDATE`) or ongoing transaction (having something in the tidb session local transaction buffer) each row needs its partition id / physical table ID to either lock that row or to check against the transaction buffer.

Signed-off-by: Mattias Jonsson <mjonss@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: hotfix panic from tokio-timer (#12004)

close tikv/tikv#11940, ref tikv/tikv#11940

*: hotfix panic from tokio-timer by enlarging the length of the level vector

Signed-off-by: you06 <you1474600@gmail.com>

* build: add arm64 check on SSE in makefile (#12035)

close tikv/tikv#12034

Signed-off-by: Jin Dong <djdongjin95@gmail.com>

Co-authored-by: Yilin Chen <sticnarf@gmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* replication mode: sync state to pd (#11751)

 

Signed-off-by: disksing <i@disksing.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: revert pessimistic locks status when failing to propose PrepareMerge (#11985)

ref tikv/tikv#11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* copr: support push bit column down (#12037)

ref pingcap/tidb#30738

Signed-off-by: yisaer <disxiaofei@163.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* split_controller: add more sample function test cases (#12058)

 

split_controller: add more sample function test cases

Signed-off-by: JmPotato <ghzpotato@gmail.com>

* copr: fix greatest/least time/date args type (#12056)

 

Signed-off-by: yisaer <disxiaofei@163.com>

* cdc: advancing resolved ts correctly for clusters with tiflash (#12050)

close pingcap/tiflow#4461

cdc: advancing resolved ts correctly for clusters with tiflash

Signed-off-by: qupeng <qupeng@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: bump master branch version to v6.0.0-alpha (#12077)

close tikv/tikv#12075

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

* raftstore: check uninitialized destroy for merge (#12055)

close tikv/tikv#12048

When a peer is destroyed without being initialized, it will store
itself to peer list and the region epoch is missing. In merge if
such state is detected, it should abort merging instead of panicking.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: Add TruncateManager and TruncateWorker (#11553)

ref tikv/rfcs#81

Signed-off-by: longfangsong <longfangsong@icloud.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* TiKV supports buckets (#11763)

ref tikv/tikv#11759

Signed-off-by: qi.xu <tonxuqi@outlook.com>

Co-authored-by: qi.xu <tonxuqi@outlook.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* api_version: Codec for RawKV key (#12036)

ref tikv/tikv#11965

Key of RawKV is encoded as `user-key + memcomparable-padding + timestamp` in API v2.

Signed-off-by: pingyu <yuping@pingcap.com>

Co-authored-by: Andy Lok <andylokandy@hotmail.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: fix some typos (#12066)

 

Signed-off-by: cuishuang <imcusg@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* raftstore: fix stale message cause panic (#12054)

close tikv/tikv#12023

raftstore: fix stale message cause panic

Signed-off-by: linning <linningde25@gmail.com>

* raftstore: ignore async fetch result when the peer is pending removed (#12038)

close tikv/tikv#11973, close tikv/tikv#12026

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: remove part about wechat in doc (#12101)

close tikv/tikv#12100

Signed-off-by: jackwener <jakevingoo@gmail.com>

* engine: upgrade raft engine (#12095)

ref tikv/tikv#165, ref tikv/raft-engine#165, ref tikv/tikv#11119

Signed-off-by: Randy <ztelur@gmail.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>

* *: Fix possible undefined behavior for transmuting vec (#12096)

close tikv/tikv#12070

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* github: Remove the pingcap/tidb-ansible from the PR template (#12102)

close tikv/tikv#12103

Signed-off-by: hi-rustin <rustin.liu@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* copr: extract reschedule to inject a fail point (#12108)

 

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* raftstore: reactivate in-memory pessimistic locking when leader transfer fails (#11883)

ref tikv/tikv#11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

* *: check memory locks for replica read only on the leader (#12115)

close tikv/tikv#12109

Consider a reader sends read index to the leader it supposed to be, but when
the leader receives the read index message, it has stepped down to a
follower. Without this commit, a peer will check the locks and re-fill the
read index context with the result no matter what role it is. So, when the
read index request is redirected to the new leader again, it no longer carries
the context of lock checking.

This commmit changes to only do the check when the peer is a leader. Then, the
read index request will remain unchanged before being redirected to the leader.

If the lease of the leader expires, it is still safe to check on the
uncertained leader. If the heartbeat check passes and no new leader is elected,
then the check works fine. If there is a new leader, when the old leader
becomes a follower, it will clear its pending read index. Then, the reader has
to resend th…
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants