Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support API V2 encode in sst_importer #11041

Closed
Tracked by #10938
andylokandy opened this issue Oct 12, 2021 · 0 comments · Fixed by #11139
Closed
Tracked by #10938

Support API V2 encode in sst_importer #11041

andylokandy opened this issue Oct 12, 2021 · 0 comments · Fixed by #11139

Comments

@andylokandy
Copy link
Contributor

No description provided.

@andylokandy andylokandy mentioned this issue Oct 12, 2021
29 tasks
@andylokandy andylokandy changed the title Add API V2 key-value encode support in sst_importer Support API V2 encode in sst_importer Oct 15, 2021
andylokandy added a commit to andylokandy/tikv that referenced this issue Oct 25, 2021
ref tikv#10938 close tikv#11041
Signed-off-by: Andy Lok <andylokandy@hotmail.com>
ti-chi-bot added a commit that referenced this issue Nov 26, 2021
* implement api v2 raw value

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* storage: support raw value encode for API V2

ref #10938 close #11041
Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix clippy

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* improve

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix clippy

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* remove println

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* improve test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* add test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* disable test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test again

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test again

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test again

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix overflow

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix overflow

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* improve

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* improve comment

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* add case

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* improve comment

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix clippy

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* improve

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
mornyx added a commit to mornyx/tikv that referenced this issue Dec 7, 2021
# This is the 1st commit message:

Add latency stats for kv get and kv batch get

Signed-off-by: mornyx <mornyx.z@gmail.com>

# This is the commit message #2:

tikv_util: upgrade `gag` from 0.1 to 1.0 (tikv#11277)

ref tikv#11276

Signed-off-by: kennytm <kennytm@gmail.com>
# This is the commit message tikv#3:

codec: add compile time assert to guarantee safety (tikv#11357)

* codec: add compile time assert to guarantee safety

close tikv#11356.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* add more explanation.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#4:

rafstore: fix commit state check in apply thread (tikv#11401)

* fix apply check & close tikv#11396

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* make format

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* format

Signed-off-by: gengliqi <gengliqiii@gmail.com>
# This is the commit message tikv#5:

backup: create external storage per-backup reqeust. (tikv#11113)

* backup: create external storage per-backup reqeust.

fixed tikv#11112

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: make clippy happy

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: fix build

Signed-off-by: yujuncen <yujuncen@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#6:

Remove duplicate enum and macro about numberic_enum_mod (tikv#11347)

* remove duplicated and useless `remove duplicated and useless`

close tikv#11297

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* remove duplicate macro rocksdb_enum_mod

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* fix format problem

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* resolve discussion

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* fix test error

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* move macro `numeric_enum_serializing_mod` to a proper location

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* move `perf_level_serde` of mod config in engine trait to mod perf_context

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* Add test about `rate-limiter-mode` and fix compile error in integration test

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#7:

ctl: detect raft db correctly (tikv#11395)

Close tikv#11393.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#8:

 Support to iterate table properties for KvEngine  (tikv#11388)

* support to iterate table properties for KvEngine
* close tikv#11387

Signed-off-by: qupeng <qupeng@pingcap.com>
# This is the commit message tikv#9:

gc: Fix GC scan effectiveness to avoid OOM (tikv#11416)

* close tikv#11410 limit the pending task for gc worker

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* avoid using prefix seek

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* rename test

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* address comment

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* fix build

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#10:

fix a panic about parsing memory.limit_in_bytes (tikv#11354)

* handle int overflow for memory.limit_in_bytes

clsoe tikv#11353.
Signed-off-by: qupeng <qupeng@pingcap.com>

* close tikv#11353

Signed-off-by: qupeng <qupeng@pingcap.com>

* address comments

Signed-off-by: qupeng <qupeng@pingcap.com>
# This is the commit message tikv#11:

build: fix make docker on Linux (tikv#11349)

* fix make docker

fix tikv#11312

Signed-off-by: tabokie <xy.tao@outlook.com>

* fix cargo

Signed-off-by: tabokie <xy.tao@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#12:

engine_rocks: ignore NoSpace error during flush or compaction (tikv#11225)

* ignore NoSpace error during flush or compaction

fix tikv#11224

Signed-off-by: tabokie <xy.tao@outlook.com>

* use constant

Signed-off-by: tabokie <xy.tao@outlook.com>

* fix typo

Signed-off-by: tabokie <xy.tao@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#13:

gc: set pending capacity for gc worker (tikv#11429)

Signed-off-by: Connor1996 <zbk602423539@gmail.com>
# This is the commit message tikv#14:

raftstore: support synchronous write (tikv#11418)

* support sync write & ref tikv#11165

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* add tests

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* change default store-io-pool-size to 0

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* remove unnecessary test

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* merge raftstore: separate read write ready(tikv#10592)

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* address comments

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* fix test

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* make clippy happy

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* address comment

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* remove offset in NormalFsm

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* make format

Signed-off-by: gengliqi <gengliqiii@gmail.com>
# This is the commit message tikv#15:

metrics: add Raft Engine panel (tikv#11415)

Signed-off-by: tabokie <xy.tao@outlook.com>
# This is the commit message tikv#16:

resource-metering: add some metrics (tikv#11414)

* resource-metering: add some metrics

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* more log

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* polish description, ref tikv#11436

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#17:

metrics: add asynchronous write io panels (tikv#11408)

* update metrics name

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* update tikv-details & close tikv#10540

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* rename Raft GC to Raft Log GC

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* address comment

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* fix datasource

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* add more metrics

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* update metrics

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* update

Signed-off-by: gengliqi <gengliqiii@gmail.com>
# This is the commit message tikv#18:

storage: fix reverse scan check memory locks (tikv#11447)

close tikv#11440

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>
# This is the commit message tikv#19:

test: make test_sending_fail_with_net_error more statable (tikv#11366)

* sending count should deregister on time

Signed-off-by: bufferflies <1045931706@qq.com>

* make test statable

Signed-off-by: bufferflies <1045931706@qq.com>

* remove unused change

Signed-off-by: bufferflies <1045931706@qq.com>

* remove unused change

Signed-off-by: bufferflies <1045931706@qq.com>

* close tikv#11365

Signed-off-by: bufferflies <1045931706@qq.com>

* reformat annotation

Signed-off-by: bufferflies <1045931706@qq.com>

* add send once filter

Signed-off-by: bufferflies <1045931706@qq.com>

* lint pass

Signed-off-by: bufferflies <1045931706@qq.com>

* lint pass

Signed-off-by: bufferflies <1045931706@qq.com>

* replace sendOnceMsgFilter

Signed-off-by: bufferflies <1045931706@qq.com>

* remove unnecessary change

Signed-off-by: bufferflies <1045931706@qq.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#20:

raftstore: increase batch for raftlog-gc-worker (tikv#11405)

* increase batch

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

* do not change seek

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

* fix fmt

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

* fix test

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

* fix number overflow

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

* increase batch

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

* revert time

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

* set last_compacted_idx for split region

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

* log large range gc task

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

* address comment

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

* close tikv#11404

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

Co-authored-by: 5kbpers <tangminghua@pingcap.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#21:

Make tikv-server fully generic over kv engines (tikv#10945)

* server: Allow flow_info_receiver to be None

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Move init_flow_receiver call into init_raw_engines

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Extract debug service creation to CreateKvEngine::start_debug_service

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Extract backup service creation to CreateKvEngine::start_backup_service

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Extract rocks cfg controller registration to CreateKvEngine::register_kv_config

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Extract create_raftstore_compaction_listener to free function

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Extract create_kv_engine to CreateKvEngine trait

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Parameterize EngineResourceInfo over EK

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Parameterize TikvServer over EK

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Document CreateKvEngine trait

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* clippy

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Extract raft engine creation to CreateRaftEngine::create_raft_engine

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Extract CreateRaftEngine::register_raft_config method

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Extract CreateRaftEngine::create_engine_infos method

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Unspecialize init_raw_engines method

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Extract setup code to generic run_server function

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* clippy

ref tikv#6402

Signed-off-by: Brian Anderson <andersrb@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#22:

Revert "Make tikv-server fully generic over kv engines (tikv#10945)" (tikv#11462)

This reverts commit df2e921.

Ref tikv#11461.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>
# This is the commit message tikv#23:

raftstore: flush waterfall metrics when using synchronous write (tikv#11451)

Signed-off-by: gengliqi <gengliqiii@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#24:

fix one disk full unstable test case on assert condition. (tikv#11391)

close tikv#11389

Signed-off-by: tier-cap <zhengxiaojin@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#25:

resolved_ts: return early after regions already had quorum (tikv#11352)

* close tikv#11351

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: Zixiong Liu <liuzixiong@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* make clippy happy

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* close tikv#11400

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: qupeng <qupeng@pingcap.com>
Co-authored-by: Zixiong Liu <liuzixiong@pingcap.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#26:

support to get approximate keys and size for a given range from UserCollectedProperties (tikv#11465)

Signed-off-by: qupeng <qupeng@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#27:

copr: add paging API support for streaming-like processling (tikv#11300)

* copr response with range

Signed-off-by: you06 <you1474600@gmail.com>

* switch to new kvproto

Signed-off-by: you06 <you1474600@gmail.com>

* fix bugs...

Signed-off-by: you06 <you1474600@gmail.com>

* format

Signed-off-by: you06 <you1474600@gmail.com>

* remove unsed field

Signed-off-by: you06 <you1474600@gmail.com>

* add comments

Signed-off-by: you06 <you1474600@gmail.com>

* update kvproto

Signed-off-by: you06 <you1474600@gmail.com>

* fix bench

Signed-off-by: you06 <you1474600@gmail.com>

* update cargo.lock

Signed-off-by: you06 <you1474600@gmail.com>

* update comment

Signed-off-by: you06 <you1474600@gmail.com>

* add test for copr paging

Signed-off-by: you06 <you1474600@gmail.com>

* close tikv#11448

Signed-off-by: you06 <you1474600@gmail.com>

* Update components/tidb_query_executors/src/runner.rs

Co-authored-by: Lei Zhao <zlwgx1023@gmail.com>
Signed-off-by: you06 <you1474600@gmail.com>

* check range in response

Signed-off-by: you06 <you1474600@gmail.com>

* format code

Signed-off-by: you06 <you1474600@gmail.com>

* add test for reverse scan

Signed-off-by: you06 <you1474600@gmail.com>

* add test for multi ranges

Signed-off-by: you06 <you1474600@gmail.com>

* test drained with multi ranges

Signed-off-by: you06 <you1474600@gmail.com>

* fix clippy

Signed-off-by: you06 <you1474600@gmail.com>

* fix lint

Signed-off-by: you06 <you1474600@gmail.com>

* paging by all record count

Signed-off-by: you06 <you1474600@gmail.com>

* format

Signed-off-by: you06 <you1474600@gmail.com>

Co-authored-by: Lei Zhao <zlwgx1023@gmail.com>
# This is the commit message tikv#28:

encryption: support seek for encryption writer (tikv#10929)

* support seek for encryption writer

Signed-off-by: tabokie <xy.tao@outlook.com>

* minor fix, ref: tikv#11119

Signed-off-by: tabokie <xy.tao@outlook.com>

* fix error from merge

Signed-off-by: tabokie <xy.tao@outlook.com>

* fight format checker, ref tikv#11119

Signed-off-by: tabokie <xy.tao@outlook.com>

* add wrappers

Signed-off-by: tabokie <xy.tao@outlook.com>

* remove pub quantifiers and fix lint

Signed-off-by: tabokie <xy.tao@outlook.com>

* fix name export

Signed-off-by: tabokie <xy.tao@outlook.com>

* revert naming changes

Signed-off-by: tabokie <xy.tao@outlook.com>

* address comment on unit test

Signed-off-by: tabokie <xy.tao@outlook.com>

* fix test and inline wrapper calls

Signed-off-by: tabokie <xy.tao@outlook.com>

* fix comments

Signed-off-by: tabokie <xy.tao@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#29:

server: remove register PD process (tikv#10985)

* remove register PD process

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* resolve conflicts

close tikv#11472

Signed-off-by: Ryan Leung <rleungx@gmail.com>
# This is the commit message tikv#30:

engine: provide customized file builder to raft engine (tikv#10937)

* open raft engine with customized file builder

ref tikv#11119

Signed-off-by: tabokie <xy.tao@outlook.com>

* overhaul configurations

Signed-off-by: tabokie <xy.tao@outlook.com>

* update default config and address comment

Signed-off-by: tabokie <xy.tao@outlook.com>

Co-authored-by: qupeng <qupeng@pingcap.com>
# This is the commit message tikv#31:

*: update some dependencies (tikv#11458)

close tikv#10997, ref tikv#11276

Signed-off-by: tabokie <xy.tao@outlook.com>
# This is the commit message tikv#32:

coprocessor: add `to_binary` and `from_binary` function (tikv#11397)

Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#33:

server: fix IOType configure for recving snapshot (tikv#11122)

* fix IOType for recving snapshot

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* Set io_type before Write::write_all. close tikv#10659

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#34:

raftstore: add pessimistic lock table to peer (tikv#11459)

ref tikv#11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#35:

engine: fix backward compatibility issue of raft-engine.recovery-mode (tikv#11490)

* add test case

Signed-off-by: tabokie <xy.tao@outlook.com>

* update raft-engine, fix tikv#11489

Signed-off-by: tabokie <xy.tao@outlook.com>

* handle timeout error

Signed-off-by: tabokie <xy.tao@outlook.com>

Co-authored-by: qupeng <qupeng@pingcap.com>
# This is the commit message tikv#36:

metrics: add duration composition panel (tikv#11491)

* metrics: add duration composition pannel

Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com>

* close tikv#11492

Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#37:

*: support read-through lock (tikv#11238)

* storage: get/batch_get support read-through lock

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

* storage: scan support read-through lock

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

* coprocessor: support read-through lock

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

* don't check access_locks when checking memory locks

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

* fix clippy
close tikv#11402

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

* update kvproto and add some comments

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

* test desc scan

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#38:

storage: support raw value encode for API V2 (tikv#11139)

* implement api v2 raw value

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* storage: support raw value encode for API V2

ref tikv#10938 close tikv#11041
Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix clippy

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* improve

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix clippy

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* remove println

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* improve test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* add test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* disable test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test again

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test again

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test again

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix overflow

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix overflow

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* improve

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* improve comment

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* add case

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* improve comment

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix clippy

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* improve

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#39:

*: replace snapshot `CbContext` with `SnapshotExt` (tikv#11471)

ref tikv#11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>
# This is the commit message tikv#40:

Add comments

Signed-off-by: mornyx <mornyx.z@gmail.com>

# This is the commit message tikv#41:

Extract one stats struct for KvGet/KvBatchGet

Signed-off-by: mornyx <mornyx.z@gmail.com>

# This is the commit message tikv#42:

raftstore: relax merge result check (tikv#11478)

* add test case

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* raftstore: relax merge result check

Close tikv#11475.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* fix unstable transfer leader case

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Liqi Geng <gengliqiii@gmail.com>
# This is the commit message tikv#43:

engine: move raft engine purge job to a dedicated thread (tikv#11497)

* update raft engine and implement batch_gc, ref tikv#11119

Signed-off-by: tabokie <xy.tao@outlook.com>

* introduce purge-worker

Signed-off-by: tabokie <xy.tao@outlook.com>

* clean up old worker

Signed-off-by: tabokie <xy.tao@outlook.com>

* update raft engine

Signed-off-by: tabokie <xy.tao@outlook.com>
# This is the commit message tikv#44:

Update max_ts for Prewrite and AcquirePessimisticLock with should_not_exist flag. (tikv#11488)

* update max_ts for prewrite with should_not_exist flag

Signed-off-by: dwangxxx <dwang_slam@163.com>

* close tikv#11315

Signed-off-by: dwangxxx <dwang_slam@163.com>

* close tikv#11315

Signed-off-by: dwangxxx <dwang_slam@163.com>

* fix unit test

Signed-off-by: dwangxxx <dwang_slam@163.com>

* fix format

Signed-off-by: dwangxxx <dwang_slam@163.com>

* fix test

Signed-off-by: dwangxxx <dwang_slam@163.com>

* add test.

Signed-off-by: dwangxxx <dwang_slam@163.com>

* fix clippy error

Signed-off-by: dwangxxx <dwang_slam@163.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#45:

Rename KvGetLatencyStats

Signed-off-by: mornyx <mornyx.z@gmail.com>

# This is the commit message tikv#46:

Rename

Signed-off-by: mornyx <mornyx.z@gmail.com>

# This is the commit message tikv#47:

*: simplify pull request template (tikv#11504)

We only needs to explain what's change. Problem description should be
traced as issue.

This PR also polishes contributing guide to explain the new flow.

Close tikv#11503.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: qupeng <qupeng@pingcap.com>
# This is the commit message tikv#48:

cdc: introduce TsFilter into incremental scan (tikv#11385)

* close tikv#11384

Signed-off-by: qupeng <qupeng@pingcap.com>
# This is the commit message tikv#49:

metric: add stale read count metric (tikv#11051)

Signed-off-by: linning <linningde25@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#50:

storage:change configuration storage.max-key-size default value from 4096 to 8192 (tikv#11331)

close tikv#11241

Signed-off-by: TonsnakeLin <lpbgytong@163.com>
# This is the commit message tikv#51:

storage: make it cheaper to clone Scheduler (tikv#11501)

closes tikv#11500

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#52:

raftclient: count term and index in estimated size (tikv#11493)

* add test case

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* count term and index tag

If there are many entries in a message, the estimated size of message
can be way smaller than the actual size. This PR fixes the error by
also counting index and term in estimation.

It also remove the hard limit as the estimation is closed enough.

Close tikv#9714.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#53:

scanner/forward: move to next key if accesses a lock with delete type (tikv#11544)

close tikv#11541

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>
# This is the commit message tikv#54:

*: add configuration for in-memory pessimistic lock (tikv#11481)

ref tikv#11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
# This is the commit message tikv#55:

backup: fix background panic in `endpoint::tests::test_adjust_thread_pool_size` (tikv#11556)

* backup: fix tikv#11548

Signed-off-by: yujuncen <yujuncen@pingcap.com>
# This is the commit message tikv#56:

tikv_util: reimplement poll_future_notify (tikv#11550)

The current implementation is very easy to get deadlock if the future is
woken immediately during polling. This PR resolves the issue by
maintaining states instead.

Note, if the future is implemented in the wrong way that always
notifying the future without checking if it has readiness, the new
implementation can run into dead loop.

Close tikv#11549.

Co-authored-by: qupeng <qupeng@pingcap.com>
# This is the commit message tikv#57:

fix taking rootfs as filesystem by mistake (tikv#11554)

* fix adding rootfs as filesystem by mistake

Signed-off-by: onlyacat <huangzhe8263@gmail.com>

* reformat by lints

Signed-off-by: onlyacat <huangzhe8263@gmail.com>

* close tikv#11552

Signed-off-by: onlyacat <huangzhe8263@gmail.com>
# This is the commit message tikv#58:

raftstore: set term when catch up logs for merge  (tikv#11545)

* add test case

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* raftstore: set term when catch up logs for merge

Prepare merge may includes logs from different terms, which may be even
larger than the term of a follower that is lag behind. So when catching
up logs by commit merge, its term should also be set to get a correct
metadata.

Close tikv#11526.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* add log for becoming follower

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* correct commit log check

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
mornyx added a commit to mornyx/tikv that referenced this issue Dec 7, 2021
Signed-off-by: mornyx <mornyx.z@gmail.com>

close tikv#8942

tikv_util: upgrade `gag` from 0.1 to 1.0 (tikv#11277)

ref tikv#11276

Signed-off-by: kennytm <kennytm@gmail.com>

codec: add compile time assert to guarantee safety (tikv#11357)

* codec: add compile time assert to guarantee safety

close tikv#11356.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* add more explanation.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

rafstore: fix commit state check in apply thread (tikv#11401)

* fix apply check & close tikv#11396

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* make format

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* format

Signed-off-by: gengliqi <gengliqiii@gmail.com>

backup: create external storage per-backup reqeust. (tikv#11113)

* backup: create external storage per-backup reqeust.

fixed tikv#11112

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: make clippy happy

Signed-off-by: yujuncen <yujuncen@pingcap.com>

* backup: fix build

Signed-off-by: yujuncen <yujuncen@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

Remove duplicate enum and macro about numberic_enum_mod (tikv#11347)

* remove duplicated and useless `remove duplicated and useless`

close tikv#11297

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* remove duplicate macro rocksdb_enum_mod

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* fix format problem

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* resolve discussion

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* fix test error

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* move macro `numeric_enum_serializing_mod` to a proper location

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* move `perf_level_serde` of mod config in engine trait to mod perf_context

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

* Add test about `rate-limiter-mode` and fix compile error in integration test

Signed-off-by: yuqi1129 <yuqi4733@gmail.com>

Co-authored-by: Xinye Tao <xy.tao@outlook.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

ctl: detect raft db correctly (tikv#11395)

Close tikv#11393.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

 Support to iterate table properties for KvEngine  (tikv#11388)

* support to iterate table properties for KvEngine
* close tikv#11387

Signed-off-by: qupeng <qupeng@pingcap.com>

gc: Fix GC scan effectiveness to avoid OOM (tikv#11416)

* close tikv#11410 limit the pending task for gc worker

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* avoid using prefix seek

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* rename test

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* address comment

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* fix build

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

fix a panic about parsing memory.limit_in_bytes (tikv#11354)

* handle int overflow for memory.limit_in_bytes

clsoe tikv#11353.
Signed-off-by: qupeng <qupeng@pingcap.com>

* close tikv#11353

Signed-off-by: qupeng <qupeng@pingcap.com>

* address comments

Signed-off-by: qupeng <qupeng@pingcap.com>

build: fix make docker on Linux (tikv#11349)

* fix make docker

fix tikv#11312

Signed-off-by: tabokie <xy.tao@outlook.com>

* fix cargo

Signed-off-by: tabokie <xy.tao@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

engine_rocks: ignore NoSpace error during flush or compaction (tikv#11225)

* ignore NoSpace error during flush or compaction

fix tikv#11224

Signed-off-by: tabokie <xy.tao@outlook.com>

* use constant

Signed-off-by: tabokie <xy.tao@outlook.com>

* fix typo

Signed-off-by: tabokie <xy.tao@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

gc: set pending capacity for gc worker (tikv#11429)

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

raftstore: support synchronous write (tikv#11418)

* support sync write & ref tikv#11165

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* add tests

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* change default store-io-pool-size to 0

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* remove unnecessary test

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* merge raftstore: separate read write ready(tikv#10592)

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* address comments

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* fix test

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* make clippy happy

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* address comment

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* remove offset in NormalFsm

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* make format

Signed-off-by: gengliqi <gengliqiii@gmail.com>

metrics: add Raft Engine panel (tikv#11415)

Signed-off-by: tabokie <xy.tao@outlook.com>

resource-metering: add some metrics (tikv#11414)

* resource-metering: add some metrics

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* more log

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* polish description, ref tikv#11436

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

metrics: add asynchronous write io panels (tikv#11408)

* update metrics name

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* update tikv-details & close tikv#10540

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* rename Raft GC to Raft Log GC

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* address comment

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* fix datasource

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* add more metrics

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* update metrics

Signed-off-by: gengliqi <gengliqiii@gmail.com>

* update

Signed-off-by: gengliqi <gengliqiii@gmail.com>

storage: fix reverse scan check memory locks (tikv#11447)

close tikv#11440

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

test: make test_sending_fail_with_net_error more statable (tikv#11366)

* sending count should deregister on time

Signed-off-by: bufferflies <1045931706@qq.com>

* make test statable

Signed-off-by: bufferflies <1045931706@qq.com>

* remove unused change

Signed-off-by: bufferflies <1045931706@qq.com>

* remove unused change

Signed-off-by: bufferflies <1045931706@qq.com>

* close tikv#11365

Signed-off-by: bufferflies <1045931706@qq.com>

* reformat annotation

Signed-off-by: bufferflies <1045931706@qq.com>

* add send once filter

Signed-off-by: bufferflies <1045931706@qq.com>

* lint pass

Signed-off-by: bufferflies <1045931706@qq.com>

* lint pass

Signed-off-by: bufferflies <1045931706@qq.com>

* replace sendOnceMsgFilter

Signed-off-by: bufferflies <1045931706@qq.com>

* remove unnecessary change

Signed-off-by: bufferflies <1045931706@qq.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

raftstore: increase batch for raftlog-gc-worker (tikv#11405)

* increase batch

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

* do not change seek

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

* fix fmt

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

* fix test

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

* fix number overflow

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

* increase batch

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

* revert time

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

* set last_compacted_idx for split region

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

* log large range gc task

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

* address comment

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

* close tikv#11404

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

Co-authored-by: 5kbpers <tangminghua@pingcap.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

Make tikv-server fully generic over kv engines (tikv#10945)

* server: Allow flow_info_receiver to be None

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Move init_flow_receiver call into init_raw_engines

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Extract debug service creation to CreateKvEngine::start_debug_service

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Extract backup service creation to CreateKvEngine::start_backup_service

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Extract rocks cfg controller registration to CreateKvEngine::register_kv_config

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Extract create_raftstore_compaction_listener to free function

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Extract create_kv_engine to CreateKvEngine trait

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Parameterize EngineResourceInfo over EK

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Parameterize TikvServer over EK

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Document CreateKvEngine trait

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* clippy

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Extract raft engine creation to CreateRaftEngine::create_raft_engine

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Extract CreateRaftEngine::register_raft_config method

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Extract CreateRaftEngine::create_engine_infos method

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Unspecialize init_raw_engines method

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* server: Extract setup code to generic run_server function

Signed-off-by: Brian Anderson <andersrb@gmail.com>

* clippy

ref tikv#6402

Signed-off-by: Brian Anderson <andersrb@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

Revert "Make tikv-server fully generic over kv engines (tikv#10945)" (tikv#11462)

This reverts commit df2e921.

Ref tikv#11461.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

raftstore: flush waterfall metrics when using synchronous write (tikv#11451)

Signed-off-by: gengliqi <gengliqiii@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

fix one disk full unstable test case on assert condition. (tikv#11391)

close tikv#11389

Signed-off-by: tier-cap <zhengxiaojin@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

resolved_ts: return early after regions already had quorum (tikv#11352)

* close tikv#11351

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: Zixiong Liu <liuzixiong@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* make clippy happy

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* close tikv#11400

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: qupeng <qupeng@pingcap.com>
Co-authored-by: Zixiong Liu <liuzixiong@pingcap.com>
Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

support to get approximate keys and size for a given range from UserCollectedProperties (tikv#11465)

Signed-off-by: qupeng <qupeng@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

copr: add paging API support for streaming-like processling (tikv#11300)

* copr response with range

Signed-off-by: you06 <you1474600@gmail.com>

* switch to new kvproto

Signed-off-by: you06 <you1474600@gmail.com>

* fix bugs...

Signed-off-by: you06 <you1474600@gmail.com>

* format

Signed-off-by: you06 <you1474600@gmail.com>

* remove unsed field

Signed-off-by: you06 <you1474600@gmail.com>

* add comments

Signed-off-by: you06 <you1474600@gmail.com>

* update kvproto

Signed-off-by: you06 <you1474600@gmail.com>

* fix bench

Signed-off-by: you06 <you1474600@gmail.com>

* update cargo.lock

Signed-off-by: you06 <you1474600@gmail.com>

* update comment

Signed-off-by: you06 <you1474600@gmail.com>

* add test for copr paging

Signed-off-by: you06 <you1474600@gmail.com>

* close tikv#11448

Signed-off-by: you06 <you1474600@gmail.com>

* Update components/tidb_query_executors/src/runner.rs

Co-authored-by: Lei Zhao <zlwgx1023@gmail.com>
Signed-off-by: you06 <you1474600@gmail.com>

* check range in response

Signed-off-by: you06 <you1474600@gmail.com>

* format code

Signed-off-by: you06 <you1474600@gmail.com>

* add test for reverse scan

Signed-off-by: you06 <you1474600@gmail.com>

* add test for multi ranges

Signed-off-by: you06 <you1474600@gmail.com>

* test drained with multi ranges

Signed-off-by: you06 <you1474600@gmail.com>

* fix clippy

Signed-off-by: you06 <you1474600@gmail.com>

* fix lint

Signed-off-by: you06 <you1474600@gmail.com>

* paging by all record count

Signed-off-by: you06 <you1474600@gmail.com>

* format

Signed-off-by: you06 <you1474600@gmail.com>

Co-authored-by: Lei Zhao <zlwgx1023@gmail.com>

encryption: support seek for encryption writer (tikv#10929)

* support seek for encryption writer

Signed-off-by: tabokie <xy.tao@outlook.com>

* minor fix, ref: tikv#11119

Signed-off-by: tabokie <xy.tao@outlook.com>

* fix error from merge

Signed-off-by: tabokie <xy.tao@outlook.com>

* fight format checker, ref tikv#11119

Signed-off-by: tabokie <xy.tao@outlook.com>

* add wrappers

Signed-off-by: tabokie <xy.tao@outlook.com>

* remove pub quantifiers and fix lint

Signed-off-by: tabokie <xy.tao@outlook.com>

* fix name export

Signed-off-by: tabokie <xy.tao@outlook.com>

* revert naming changes

Signed-off-by: tabokie <xy.tao@outlook.com>

* address comment on unit test

Signed-off-by: tabokie <xy.tao@outlook.com>

* fix test and inline wrapper calls

Signed-off-by: tabokie <xy.tao@outlook.com>

* fix comments

Signed-off-by: tabokie <xy.tao@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

server: remove register PD process (tikv#10985)

* remove register PD process

Signed-off-by: Ryan Leung <rleungx@gmail.com>

* resolve conflicts

close tikv#11472

Signed-off-by: Ryan Leung <rleungx@gmail.com>

engine: provide customized file builder to raft engine (tikv#10937)

* open raft engine with customized file builder

ref tikv#11119

Signed-off-by: tabokie <xy.tao@outlook.com>

* overhaul configurations

Signed-off-by: tabokie <xy.tao@outlook.com>

* update default config and address comment

Signed-off-by: tabokie <xy.tao@outlook.com>

Co-authored-by: qupeng <qupeng@pingcap.com>

*: update some dependencies (tikv#11458)

close tikv#10997, ref tikv#11276

Signed-off-by: tabokie <xy.tao@outlook.com>

coprocessor: add `to_binary` and `from_binary` function (tikv#11397)

Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

server: fix IOType configure for recving snapshot (tikv#11122)

* fix IOType for recving snapshot

Signed-off-by: MuZhou233 <muzhou233@outlook.com>

* Set io_type before Write::write_all. close tikv#10659

Signed-off-by: MuZhou233 <MuZhou233@outlook.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

raftstore: add pessimistic lock table to peer (tikv#11459)

ref tikv#11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

engine: fix backward compatibility issue of raft-engine.recovery-mode (tikv#11490)

* add test case

Signed-off-by: tabokie <xy.tao@outlook.com>

* update raft-engine, fix tikv#11489

Signed-off-by: tabokie <xy.tao@outlook.com>

* handle timeout error

Signed-off-by: tabokie <xy.tao@outlook.com>

Co-authored-by: qupeng <qupeng@pingcap.com>

metrics: add duration composition panel (tikv#11491)

* metrics: add duration composition pannel

Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com>

* close tikv#11492

Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

*: support read-through lock (tikv#11238)

* storage: get/batch_get support read-through lock

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

* storage: scan support read-through lock

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

* coprocessor: support read-through lock

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

* don't check access_locks when checking memory locks

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

* fix clippy
close tikv#11402

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

* update kvproto and add some comments

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

* test desc scan

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

storage: support raw value encode for API V2 (tikv#11139)

* implement api v2 raw value

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* storage: support raw value encode for API V2

ref tikv#10938 close tikv#11041
Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix clippy

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* improve

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix clippy

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* remove println

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* improve test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* add test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* disable test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test again

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test again

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test again

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix overflow

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix overflow

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* improve

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* improve comment

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* add case

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* improve comment

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix clippy

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* improve

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

* fix test

Signed-off-by: Andy Lok <andylokandy@hotmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

*: replace snapshot `CbContext` with `SnapshotExt` (tikv#11471)

ref tikv#11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

Add comments

Signed-off-by: mornyx <mornyx.z@gmail.com>

Extract one stats struct for KvGet/KvBatchGet

Signed-off-by: mornyx <mornyx.z@gmail.com>

raftstore: relax merge result check (tikv#11478)

* add test case

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* raftstore: relax merge result check

Close tikv#11475.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* fix unstable transfer leader case

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Liqi Geng <gengliqiii@gmail.com>

engine: move raft engine purge job to a dedicated thread (tikv#11497)

* update raft engine and implement batch_gc, ref tikv#11119

Signed-off-by: tabokie <xy.tao@outlook.com>

* introduce purge-worker

Signed-off-by: tabokie <xy.tao@outlook.com>

* clean up old worker

Signed-off-by: tabokie <xy.tao@outlook.com>

* update raft engine

Signed-off-by: tabokie <xy.tao@outlook.com>

Update max_ts for Prewrite and AcquirePessimisticLock with should_not_exist flag. (tikv#11488)

* update max_ts for prewrite with should_not_exist flag

Signed-off-by: dwangxxx <dwang_slam@163.com>

* close tikv#11315

Signed-off-by: dwangxxx <dwang_slam@163.com>

* close tikv#11315

Signed-off-by: dwangxxx <dwang_slam@163.com>

* fix unit test

Signed-off-by: dwangxxx <dwang_slam@163.com>

* fix format

Signed-off-by: dwangxxx <dwang_slam@163.com>

* fix test

Signed-off-by: dwangxxx <dwang_slam@163.com>

* add test.

Signed-off-by: dwangxxx <dwang_slam@163.com>

* fix clippy error

Signed-off-by: dwangxxx <dwang_slam@163.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

Rename KvGetLatencyStats

Signed-off-by: mornyx <mornyx.z@gmail.com>

Rename

Signed-off-by: mornyx <mornyx.z@gmail.com>

*: simplify pull request template (tikv#11504)

We only needs to explain what's change. Problem description should be
traced as issue.

This PR also polishes contributing guide to explain the new flow.

Close tikv#11503.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: qupeng <qupeng@pingcap.com>

cdc: introduce TsFilter into incremental scan (tikv#11385)

* close tikv#11384

Signed-off-by: qupeng <qupeng@pingcap.com>

metric: add stale read count metric (tikv#11051)

Signed-off-by: linning <linningde25@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

storage:change configuration storage.max-key-size default value from 4096 to 8192 (tikv#11331)

close tikv#11241

Signed-off-by: TonsnakeLin <lpbgytong@163.com>

storage: make it cheaper to clone Scheduler (tikv#11501)

closes tikv#11500

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

raftclient: count term and index in estimated size (tikv#11493)

* add test case

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* count term and index tag

If there are many entries in a message, the estimated size of message
can be way smaller than the actual size. This PR fixes the error by
also counting index and term in estimation.

It also remove the hard limit as the estimation is closed enough.

Close tikv#9714.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

scanner/forward: move to next key if accesses a lock with delete type (tikv#11544)

close tikv#11541

Signed-off-by: youjiali1995 <zlwgx1023@gmail.com>

*: add configuration for in-memory pessimistic lock (tikv#11481)

ref tikv#11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

backup: fix background panic in `endpoint::tests::test_adjust_thread_pool_size` (tikv#11556)

* backup: fix tikv#11548

Signed-off-by: yujuncen <yujuncen@pingcap.com>

tikv_util: reimplement poll_future_notify (tikv#11550)

The current implementation is very easy to get deadlock if the future is
woken immediately during polling. This PR resolves the issue by
maintaining states instead.

Note, if the future is implemented in the wrong way that always
notifying the future without checking if it has readiness, the new
implementation can run into dead loop.

Close tikv#11549.

Co-authored-by: qupeng <qupeng@pingcap.com>

fix taking rootfs as filesystem by mistake (tikv#11554)

* fix adding rootfs as filesystem by mistake

Signed-off-by: onlyacat <huangzhe8263@gmail.com>

* reformat by lints

Signed-off-by: onlyacat <huangzhe8263@gmail.com>

* close tikv#11552

Signed-off-by: onlyacat <huangzhe8263@gmail.com>

raftstore: set term when catch up logs for merge  (tikv#11545)

* add test case

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* raftstore: set term when catch up logs for merge

Prepare merge may includes logs from different terms, which may be even
larger than the term of a follower that is lag behind. So when catching
up logs by commit merge, its term should also be set to get a correct
metadata.

Close tikv#11526.

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* add log for becoming follower

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

* correct commit log check

Signed-off-by: Jay Lee <BusyJayLee@gmail.com>

Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>

storage: Add API V2 check for RawKV and TxnKV requests (part 2) (tikv#11228)

* wip

Signed-off-by: pingyu <yuping@pingcap.com>

* wip

Signed-off-by: pingyu <yuping@pingcap.com>

* wip

Signed-off-by: pingyu <yuping@pingcap.com>

* wip

Signed-off-by: pingyu <yuping@pingcap.com>

* wip

Signed-off-by: pingyu <yuping@pingcap.com>

* wip

Signed-off-by: pingyu <yuping@pingcap.com>

* ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2)

Signed-off-by: pingyu <yuping@pingcap.com>

* resolve conflict

Signed-off-by: pingyu <yuping@pingcap.com>

* add raw_key_maybe_unbounded_into_option

Signed-off-by: pingyu <yuping@pingcap.com>

* ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref tikv#10974: address comments

Signed-off-by: pingyu <yuping@pingcap.com>

* ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2)

Signed-off-by: pingyu <yuping@pingcap.com>

* ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2)

Signed-off-by: pingyu <yuping@pingcap.com>

* remove make_invalid_key_prefix_err

Signed-off-by: andylokandy <andylokandy@hotmail.com>

Co-authored-by: andylokandy <andylokandy@hotmail.com>

storage: read and write in-memory pessimistic locks in the scheduler  (tikv#11529)

ref tikv#11452

Signed-off-by: Yilin Chen <sticnarf@gmail.com>

tolerate unmounted cgroup controller (tikv#11570)

* tolerate unmounted cgroup controller, fix tikv#11569

Signed-off-by: tabokie <xy.tao@outlook.com>

* add test

Signed-off-by: tabokie <xy.tao@outlook.com>

* address comment

Signed-off-by: tabokie <xy.tao@outlook.com>

cdc: stabilize  some cdc cases (tikv#11586)

* close tikv#11560

Signed-off-by: qupeng <qupeng@pingcap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant