raftstore: support asynchronous write io #10540
Assignees
Labels
affects-5.3
This bug affects 5.3.x versions.
sig/raft
Component: Raft, RaftStore, etc.
type/enhancement
Type: Issue - Enhancement
Projects
Comments
gengliqi
changed the title
gengliqi
changed the title
Not yet. We have a design doc(in chinese) about analyzing the benefits of asynchronous io for raftstore. But it lacks of some implementation details. For now, I am busy with merging PRs in master and doing tests. I will translate the design doc to english after getting these things done. |
gengliqi
added a commit
to gengliqi/tikv
that referenced
this issue
Oct 19, 2021
Signed-off-by: gengliqi <gengliqiii@gmail.com>
ti-chi-bot
added a commit
that referenced
this issue
Oct 20, 2021
* use asynchronous io(squash commits due to sign-off) Signed-off-by: gengliqi <gengliqiii@gmail.com> * update hash Signed-off-by: gengliqi <gengliqiii@gmail.com> * add one async io test Signed-off-by: gengliqi <gengliqiii@gmail.com> * update async io test Signed-off-by: gengliqi <gengliqiii@gmail.com> * move async_io file Signed-off-by: gengliqi <gengliqiii@gmail.com> * introduce async io Signed-off-by: gengliqi <gengliqiii@gmail.com> * make clippy happy Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * update comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * add one more test Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * make clippy happy Signed-off-by: gengliqi <gengliqiii@gmail.com> * reset raft-engine to master Signed-off-by: gengliqi <gengliqiii@gmail.com> * add config-template Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix compile error Signed-off-by: gengliqi <gengliqiii@gmail.com> * move code Signed-off-by: gengliqi <gengliqiii@gmail.com> * remove useless phantom data Signed-off-by: gengliqi <gengliqiii@gmail.com> * remove RegionNotifier Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comment Signed-off-by: gengliqi <gengliqiii@gmail.com> * make write public Signed-off-by: gengliqi <gengliqiii@gmail.com> * update raft crate Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix Signed-off-by: gengliqi <gengliqiii@gmail.com> * demo apply unpersisted log Signed-off-by: gengliqi <gengliqiii@gmail.com> * add msg flush counter Signed-off-by: gengliqi <gengliqiii@gmail.com> * add syncer thread Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix syncer bug Signed-off-by: gengliqi <gengliqiii@gmail.com> * change back to channel Signed-off-by: gengliqi <gengliqiii@gmail.com> * random writer id Signed-off-by: gengliqi <gengliqiii@gmail.com> * support reschedule io Signed-off-by: gengliqi <gengliqiii@gmail.com> * revert remove committed entries memory control Signed-off-by: gengliqi <gengliqiii@gmail.com> * polish code & change writer channel to bounded channel & add unit test for sync worker Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix compile error Signed-off-by: gengliqi <gengliqiii@gmail.com> * change some default value Signed-off-by: gengliqi <gengliqiii@gmail.com> * update Signed-off-by: gengliqi <gengliqiii@gmail.com> * update metric description Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix compile error Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix compile error Signed-off-by: gengliqi <gengliqiii@gmail.com> * make clippy happy Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix compile error Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix wrong batch_wait metric Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix wrong batch_wait metric Signed-off-by: gengliqi <gengliqiii@gmail.com> * add write msg block wait metric Signed-off-by: gengliqi <gengliqiii@gmail.com> * change waterfall metric granularity to request time Signed-off-by: gengliqi <gengliqiii@gmail.com> * change waterfall metric granularity to request time Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * make some config can be updated online Signed-off-by: gengliqi <gengliqiii@gmail.com> * make some config can be updated online Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * remove the wrong sanity check Signed-off-by: gengliqi <gengliqiii@gmail.com> * remove the wrong sanity check Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix compile error Signed-off-by: gengliqi <gengliqiii@gmail.com> * remove sync worker Signed-off-by: gengliqi <gengliqiii@gmail.com> * remove sync worker Signed-off-by: gengliqi <gengliqiii@gmail.com> * add comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * flush msg if any Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix compile error Signed-off-by: gengliqi <gengliqiii@gmail.com> * guarantee the entry must be exist in entry cache before persisting Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix peer_storage unit test Signed-off-by: gengliqi <gengliqiii@gmail.com> * change back is_applying_snapshot function name Signed-off-by: gengliqi <gengliqiii@gmail.com> * handle msg metrics Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * add more comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * add write router Signed-off-by: gengliqi <gengliqiii@gmail.com> * add write router implementation Signed-off-by: gengliqi <gengliqiii@gmail.com> * add log Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix wrong usage of write router Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix test Signed-off-by: gengliqi <gengliqiii@gmail.com> * merge introduce-write-router Signed-off-by: gengliqi <gengliqiii@gmail.com> * make format Signed-off-by: gengliqi <gengliqiii@gmail.com> * update log format Signed-off-by: gengliqi <gengliqiii@gmail.com> * use fetch_update Signed-off-by: gengliqi <gengliqiii@gmail.com> * add raft client delay flush Signed-off-by: gengliqi <gengliqiii@gmail.com> * record next writer id Signed-off-by: gengliqi <gengliqiii@gmail.com> * make format Signed-off-by: gengliqi <gengliqiii@gmail.com> * always enable entry cache Signed-off-by: gengliqi <gengliqiii@gmail.com> * add test Signed-off-by: gengliqi <gengliqiii@gmail.com> * add more comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix failed test_read_after_peer_destroyed Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comment Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix compact_to Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix compact_to Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix waterfall metric Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * update unit test Signed-off-by: gengliqi <gengliqiii@gmail.com> * make clippy happy Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix compile error Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix test_storage_cache_update Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comment Signed-off-by: gengliqi <gengliqiii@gmail.com> * remove delay flush Signed-off-by: gengliqi <gengliqiii@gmail.com> * io tasks are written in sync mode by default Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix tests Signed-off-by: gengliqi <gengliqiii@gmail.com> * update RaftLogBatch Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix test failed Signed-off-by: gengliqi <gengliqiii@gmail.com> * change waterfall metrics name Signed-off-by: gengliqi <gengliqiii@gmail.com> * add some tests Signed-off-by: gengliqi <gengliqiii@gmail.com> * remove InvokeContext Signed-off-by: gengliqi <gengliqiii@gmail.com> * add test_async_io Signed-off-by: gengliqi <gengliqiii@gmail.com> * make clippy happy Signed-off-by: gengliqi <gengliqiii@gmail.com> * change test default config Signed-off-by: gengliqi <gengliqiii@gmail.com> * tiny rename Signed-off-by: gengliqi <gengliqiii@gmail.com> * add more comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * tiny change Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comment Signed-off-by: gengliqi <gengliqiii@gmail.com> * remove persisted msg wait Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * change some assert to panic Signed-off-by: gengliqi <gengliqiii@gmail.com> * add concurrent_ready_max_count Signed-off-by: gengliqi <gengliqiii@gmail.com> * add cmd-batch-concurrent-ready-max-count and remove raft-ready-size-limit Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * make clippy happy Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix test Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * delay build raft msg Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * add pos for handle_normal Signed-off-by: gengliqi <gengliqiii@gmail.com> * add test_async_io_cannot_destroy_when_persist_snapshot test Signed-off-by: gengliqi <gengliqiii@gmail.com> * remove sync io Signed-off-by: gengliqi <gengliqiii@gmail.com> * remove unnecessary pub Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix compile error Signed-off-by: gengliqi <gengliqiii@gmail.com> * add debug log Signed-off-by: gengliqi <gengliqiii@gmail.com> * add test_async_io_cannot_handle_ready_when_persist_snapshot test Signed-off-by: gengliqi <gengliqiii@gmail.com> * make format & tiny change Signed-off-by: gengliqi <gengliqiii@gmail.com> * update comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * tiny fix & ref #10540 Signed-off-by: gengliqi <gengliqiii@gmail.com> * revert change in write Signed-off-by: gengliqi <gengliqiii@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Co-authored-by: qupeng <qupeng@pingcap.com>
gengliqi
added a commit
to gengliqi/tikv
that referenced
this issue
Oct 26, 2021
Signed-off-by: gengliqi <gengliqiii@gmail.com>
gengliqi
added a commit
to gengliqi/tikv
that referenced
this issue
Oct 26, 2021
Signed-off-by: gengliqi <gengliqiii@gmail.com>
MrCroxx
added a commit
to MrCroxx/tikv
that referenced
this issue
Nov 1, 2021
eliminate follower raft gc tick will reduce some cpu usage for ref tikv#10540 Signed-off-by: MrCroxx <mrcroxx@outlook.com>
ti-chi-bot
pushed a commit
that referenced
this issue
Nov 7, 2021
* optimize store and grpc cpu Signed-off-by: gengliqi <gengliqiii@gmail.com> * optimize proposed cb Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix compile error ref #10540 Signed-off-by: gengliqi <gengliqiii@gmail.com> * close #11121 Signed-off-by: gengliqi <gengliqiii@gmail.com> * make clippy happy Signed-off-by: gengliqi <gengliqiii@gmail.com> * make clippy happy Signed-off-by: gengliqi <gengliqiii@gmail.com> * make clippy happy Signed-off-by: gengliqi <gengliqiii@gmail.com> * add debug log Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix transfer leader Signed-off-by: gengliqi <gengliqiii@gmail.com> * propose cmd before transfer leader & split & merge only when header check is done Signed-off-by: gengliqi <gengliqiii@gmail.com> * add more tests Signed-off-by: gengliqi <gengliqiii@gmail.com> * polish code Signed-off-by: gengliqi <gengliqiii@gmail.com> * make clippy happy Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix Signed-off-by: gengliqi <gengliqiii@gmail.com> * tiny fix Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix Signed-off-by: gengliqi <gengliqiii@gmail.com> * make clippy happy Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * make CmdEpochChecker private Signed-off-by: gengliqi <gengliqiii@gmail.com>
ti-chi-bot
pushed a commit
that referenced
this issue
Nov 7, 2021
* Remove `skip_gc_raft_log_ticks` and shorten tick interval. Previously (before #8668), trigger gc will force WAL sync in apply thread, and use `skip_gc_raft_log_ticks` to reduce syncing. After [#8668](#8668), syncing is moved to background worker thread, and `CompactLog` cmds are batched. So there is no need to use `skip_gc_raft_log_ticks` anymore. Signed-off-by: MrCroxx <mrcroxx@outlook.com> * restore raft log gc tick interval and fix typo restore raft log gc tick interval, it's better to seperate default config modification from this pr. Signed-off-by: MrCroxx <mrcroxx@outlook.com> * remove raft log gc tick register whthin threshold. Signed-off-by: MrCroxx <mrcroxx@outlook.com> * remove tests for discarded config. Signed-off-by: MrCroxx <mrcroxx@outlook.com> * Hide `raft-log-gc-threshold` and set default value to 1. This config is not needed after #8668, but some tests rely on it to prevent log compaction. Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix typo Co-authored-by: Xinye Tao <xy.tao@outlook.com> Signed-off-by: MrCroxx <mrcroxx@outlook.com> * Use optimized `raft_log_gc_tick_interval` config. Signed-off-by: MrCroxx <mrcroxx@outlook.com> * remove stale config `raft-log-reserve-max-ticks` Signed-off-by: MrCroxx <mrcroxx@outlook.com> * stop ticking raft log gc if role becomes follower Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix bug eliminate follower raft gc tick will reduce some cpu usage for ref #10540 Signed-off-by: MrCroxx <mrcroxx@outlook.com> * update config items for async-io - Restore config `raft-log-gc-threshold`. - Make `raft-log-gc-batch-size` configurable. Signed-off-by: MrCroxx <mrcroxx@outlook.com> * remove raft gc tick batch, make compact interval configurable Signed-off-by: MrCroxx <mrcroxx@outlook.com> * restore raft-log-reserve-max-ticks Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix metrics Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix bug Signed-off-by: MrCroxx <mrcroxx@outlook.com> * restore register tick, remove flush per recv task Signed-off-by: MrCroxx <mrcroxx@outlook.com> * rename to raft-log-comapct-sync-interval Signed-off-by: MrCroxx <mrcroxx@outlook.com> * update optimized config Signed-off-by: MrCroxx <mrcroxx@outlook.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com>
MrCroxx
added a commit
to ti-srebot/tikv
that referenced
this issue
Nov 8, 2021
Signed-off-by: MrCroxx <mrcroxx@outlook.com>
ti-chi-bot
added a commit
that referenced
this issue
Nov 8, 2021
* cherry pick #10599 to release-5.3 Signed-off-by: ti-srebot <ti-srebot@pingcap.com> * skip sync when no raft log gc task ref #10599 Signed-off-by: MrCroxx <mrcroxx@outlook.com> * ref #10540 Signed-off-by: MrCroxx <mrcroxx@outlook.com> Co-authored-by: Croxx <mrcroxx@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
gengliqi
added a commit
to gengliqi/tikv
that referenced
this issue
Nov 17, 2021
Signed-off-by: gengliqi <gengliqiii@gmail.com>
gengliqi
added a commit
to gengliqi/tikv
that referenced
this issue
Nov 17, 2021
Signed-off-by: gengliqi <gengliqiii@gmail.com>
gengliqi
added a commit
that referenced
this issue
Nov 22, 2021
* update metrics name Signed-off-by: gengliqi <gengliqiii@gmail.com> * update tikv-details & close #10540 Signed-off-by: gengliqi <gengliqiii@gmail.com> * rename Raft GC to Raft Log GC Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comment Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix datasource Signed-off-by: gengliqi <gengliqiii@gmail.com> * add more metrics Signed-off-by: gengliqi <gengliqiii@gmail.com> * update metrics Signed-off-by: gengliqi <gengliqiii@gmail.com> * update Signed-off-by: gengliqi <gengliqiii@gmail.com>
gengliqi
added a commit
to ti-srebot/tikv
that referenced
this issue
Nov 23, 2021
Signed-off-by: gengliqi <gengliqiii@gmail.com>
you06
pushed a commit
to you06/tikv
that referenced
this issue
Nov 23, 2021
* update metrics name Signed-off-by: gengliqi <gengliqiii@gmail.com> * update tikv-details & close tikv#10540 Signed-off-by: gengliqi <gengliqiii@gmail.com> * rename Raft GC to Raft Log GC Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comment Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix datasource Signed-off-by: gengliqi <gengliqiii@gmail.com> * add more metrics Signed-off-by: gengliqi <gengliqiii@gmail.com> * update metrics Signed-off-by: gengliqi <gengliqiii@gmail.com> * update Signed-off-by: gengliqi <gengliqiii@gmail.com>
mornyx
added a commit
to mornyx/tikv
that referenced
this issue
Dec 7, 2021
# This is the 1st commit message: Add latency stats for kv get and kv batch get Signed-off-by: mornyx <mornyx.z@gmail.com> # This is the commit message #2: tikv_util: upgrade `gag` from 0.1 to 1.0 (tikv#11277) ref tikv#11276 Signed-off-by: kennytm <kennytm@gmail.com> # This is the commit message tikv#3: codec: add compile time assert to guarantee safety (tikv#11357) * codec: add compile time assert to guarantee safety close tikv#11356. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * add more explanation. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#4: rafstore: fix commit state check in apply thread (tikv#11401) * fix apply check & close tikv#11396 Signed-off-by: gengliqi <gengliqiii@gmail.com> * make format Signed-off-by: gengliqi <gengliqiii@gmail.com> * format Signed-off-by: gengliqi <gengliqiii@gmail.com> # This is the commit message tikv#5: backup: create external storage per-backup reqeust. (tikv#11113) * backup: create external storage per-backup reqeust. fixed tikv#11112 Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: make clippy happy Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: fix build Signed-off-by: yujuncen <yujuncen@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#6: Remove duplicate enum and macro about numberic_enum_mod (tikv#11347) * remove duplicated and useless `remove duplicated and useless` close tikv#11297 Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * remove duplicate macro rocksdb_enum_mod Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * fix format problem Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * resolve discussion Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * fix test error Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * move macro `numeric_enum_serializing_mod` to a proper location Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * move `perf_level_serde` of mod config in engine trait to mod perf_context Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * Add test about `rate-limiter-mode` and fix compile error in integration test Signed-off-by: yuqi1129 <yuqi4733@gmail.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#7: ctl: detect raft db correctly (tikv#11395) Close tikv#11393. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#8: Support to iterate table properties for KvEngine (tikv#11388) * support to iterate table properties for KvEngine * close tikv#11387 Signed-off-by: qupeng <qupeng@pingcap.com> # This is the commit message tikv#9: gc: Fix GC scan effectiveness to avoid OOM (tikv#11416) * close tikv#11410 limit the pending task for gc worker Signed-off-by: Connor1996 <zbk602423539@gmail.com> * avoid using prefix seek Signed-off-by: Connor1996 <zbk602423539@gmail.com> * rename test Signed-off-by: Connor1996 <zbk602423539@gmail.com> * address comment Signed-off-by: Connor1996 <zbk602423539@gmail.com> * fix build Signed-off-by: Connor1996 <zbk602423539@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#10: fix a panic about parsing memory.limit_in_bytes (tikv#11354) * handle int overflow for memory.limit_in_bytes clsoe tikv#11353. Signed-off-by: qupeng <qupeng@pingcap.com> * close tikv#11353 Signed-off-by: qupeng <qupeng@pingcap.com> * address comments Signed-off-by: qupeng <qupeng@pingcap.com> # This is the commit message tikv#11: build: fix make docker on Linux (tikv#11349) * fix make docker fix tikv#11312 Signed-off-by: tabokie <xy.tao@outlook.com> * fix cargo Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#12: engine_rocks: ignore NoSpace error during flush or compaction (tikv#11225) * ignore NoSpace error during flush or compaction fix tikv#11224 Signed-off-by: tabokie <xy.tao@outlook.com> * use constant Signed-off-by: tabokie <xy.tao@outlook.com> * fix typo Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#13: gc: set pending capacity for gc worker (tikv#11429) Signed-off-by: Connor1996 <zbk602423539@gmail.com> # This is the commit message tikv#14: raftstore: support synchronous write (tikv#11418) * support sync write & ref tikv#11165 Signed-off-by: gengliqi <gengliqiii@gmail.com> * add tests Signed-off-by: gengliqi <gengliqiii@gmail.com> * change default store-io-pool-size to 0 Signed-off-by: gengliqi <gengliqiii@gmail.com> * remove unnecessary test Signed-off-by: gengliqi <gengliqiii@gmail.com> * merge raftstore: separate read write ready(tikv#10592) Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix test Signed-off-by: gengliqi <gengliqiii@gmail.com> * make clippy happy Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comment Signed-off-by: gengliqi <gengliqiii@gmail.com> * remove offset in NormalFsm Signed-off-by: gengliqi <gengliqiii@gmail.com> * make format Signed-off-by: gengliqi <gengliqiii@gmail.com> # This is the commit message tikv#15: metrics: add Raft Engine panel (tikv#11415) Signed-off-by: tabokie <xy.tao@outlook.com> # This is the commit message tikv#16: resource-metering: add some metrics (tikv#11414) * resource-metering: add some metrics Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * more log Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * polish description, ref tikv#11436 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#17: metrics: add asynchronous write io panels (tikv#11408) * update metrics name Signed-off-by: gengliqi <gengliqiii@gmail.com> * update tikv-details & close tikv#10540 Signed-off-by: gengliqi <gengliqiii@gmail.com> * rename Raft GC to Raft Log GC Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comment Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix datasource Signed-off-by: gengliqi <gengliqiii@gmail.com> * add more metrics Signed-off-by: gengliqi <gengliqiii@gmail.com> * update metrics Signed-off-by: gengliqi <gengliqiii@gmail.com> * update Signed-off-by: gengliqi <gengliqiii@gmail.com> # This is the commit message tikv#18: storage: fix reverse scan check memory locks (tikv#11447) close tikv#11440 Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> # This is the commit message tikv#19: test: make test_sending_fail_with_net_error more statable (tikv#11366) * sending count should deregister on time Signed-off-by: bufferflies <1045931706@qq.com> * make test statable Signed-off-by: bufferflies <1045931706@qq.com> * remove unused change Signed-off-by: bufferflies <1045931706@qq.com> * remove unused change Signed-off-by: bufferflies <1045931706@qq.com> * close tikv#11365 Signed-off-by: bufferflies <1045931706@qq.com> * reformat annotation Signed-off-by: bufferflies <1045931706@qq.com> * add send once filter Signed-off-by: bufferflies <1045931706@qq.com> * lint pass Signed-off-by: bufferflies <1045931706@qq.com> * lint pass Signed-off-by: bufferflies <1045931706@qq.com> * replace sendOnceMsgFilter Signed-off-by: bufferflies <1045931706@qq.com> * remove unnecessary change Signed-off-by: bufferflies <1045931706@qq.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#20: raftstore: increase batch for raftlog-gc-worker (tikv#11405) * increase batch Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * do not change seek Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix fmt Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix test Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix number overflow Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * increase batch Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * revert time Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * set last_compacted_idx for split region Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * log large range gc task Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * address comment Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * close tikv#11404 Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> Co-authored-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#21: Make tikv-server fully generic over kv engines (tikv#10945) * server: Allow flow_info_receiver to be None Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Move init_flow_receiver call into init_raw_engines Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract debug service creation to CreateKvEngine::start_debug_service Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract backup service creation to CreateKvEngine::start_backup_service Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract rocks cfg controller registration to CreateKvEngine::register_kv_config Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract create_raftstore_compaction_listener to free function Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract create_kv_engine to CreateKvEngine trait Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Parameterize EngineResourceInfo over EK Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Parameterize TikvServer over EK Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Document CreateKvEngine trait Signed-off-by: Brian Anderson <andersrb@gmail.com> * clippy Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract raft engine creation to CreateRaftEngine::create_raft_engine Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract CreateRaftEngine::register_raft_config method Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract CreateRaftEngine::create_engine_infos method Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Unspecialize init_raw_engines method Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract setup code to generic run_server function Signed-off-by: Brian Anderson <andersrb@gmail.com> * clippy ref tikv#6402 Signed-off-by: Brian Anderson <andersrb@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#22: Revert "Make tikv-server fully generic over kv engines (tikv#10945)" (tikv#11462) This reverts commit df2e921. Ref tikv#11461. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> # This is the commit message tikv#23: raftstore: flush waterfall metrics when using synchronous write (tikv#11451) Signed-off-by: gengliqi <gengliqiii@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#24: fix one disk full unstable test case on assert condition. (tikv#11391) close tikv#11389 Signed-off-by: tier-cap <zhengxiaojin@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#25: resolved_ts: return early after regions already had quorum (tikv#11352) * close tikv#11351 Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Zixiong Liu <liuzixiong@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * make clippy happy Signed-off-by: 5kbpers <tangminghua@pingcap.com> * close tikv#11400 Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: qupeng <qupeng@pingcap.com> Co-authored-by: Zixiong Liu <liuzixiong@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#26: support to get approximate keys and size for a given range from UserCollectedProperties (tikv#11465) Signed-off-by: qupeng <qupeng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#27: copr: add paging API support for streaming-like processling (tikv#11300) * copr response with range Signed-off-by: you06 <you1474600@gmail.com> * switch to new kvproto Signed-off-by: you06 <you1474600@gmail.com> * fix bugs... Signed-off-by: you06 <you1474600@gmail.com> * format Signed-off-by: you06 <you1474600@gmail.com> * remove unsed field Signed-off-by: you06 <you1474600@gmail.com> * add comments Signed-off-by: you06 <you1474600@gmail.com> * update kvproto Signed-off-by: you06 <you1474600@gmail.com> * fix bench Signed-off-by: you06 <you1474600@gmail.com> * update cargo.lock Signed-off-by: you06 <you1474600@gmail.com> * update comment Signed-off-by: you06 <you1474600@gmail.com> * add test for copr paging Signed-off-by: you06 <you1474600@gmail.com> * close tikv#11448 Signed-off-by: you06 <you1474600@gmail.com> * Update components/tidb_query_executors/src/runner.rs Co-authored-by: Lei Zhao <zlwgx1023@gmail.com> Signed-off-by: you06 <you1474600@gmail.com> * check range in response Signed-off-by: you06 <you1474600@gmail.com> * format code Signed-off-by: you06 <you1474600@gmail.com> * add test for reverse scan Signed-off-by: you06 <you1474600@gmail.com> * add test for multi ranges Signed-off-by: you06 <you1474600@gmail.com> * test drained with multi ranges Signed-off-by: you06 <you1474600@gmail.com> * fix clippy Signed-off-by: you06 <you1474600@gmail.com> * fix lint Signed-off-by: you06 <you1474600@gmail.com> * paging by all record count Signed-off-by: you06 <you1474600@gmail.com> * format Signed-off-by: you06 <you1474600@gmail.com> Co-authored-by: Lei Zhao <zlwgx1023@gmail.com> # This is the commit message tikv#28: encryption: support seek for encryption writer (tikv#10929) * support seek for encryption writer Signed-off-by: tabokie <xy.tao@outlook.com> * minor fix, ref: tikv#11119 Signed-off-by: tabokie <xy.tao@outlook.com> * fix error from merge Signed-off-by: tabokie <xy.tao@outlook.com> * fight format checker, ref tikv#11119 Signed-off-by: tabokie <xy.tao@outlook.com> * add wrappers Signed-off-by: tabokie <xy.tao@outlook.com> * remove pub quantifiers and fix lint Signed-off-by: tabokie <xy.tao@outlook.com> * fix name export Signed-off-by: tabokie <xy.tao@outlook.com> * revert naming changes Signed-off-by: tabokie <xy.tao@outlook.com> * address comment on unit test Signed-off-by: tabokie <xy.tao@outlook.com> * fix test and inline wrapper calls Signed-off-by: tabokie <xy.tao@outlook.com> * fix comments Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#29: server: remove register PD process (tikv#10985) * remove register PD process Signed-off-by: Ryan Leung <rleungx@gmail.com> * resolve conflicts close tikv#11472 Signed-off-by: Ryan Leung <rleungx@gmail.com> # This is the commit message tikv#30: engine: provide customized file builder to raft engine (tikv#10937) * open raft engine with customized file builder ref tikv#11119 Signed-off-by: tabokie <xy.tao@outlook.com> * overhaul configurations Signed-off-by: tabokie <xy.tao@outlook.com> * update default config and address comment Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: qupeng <qupeng@pingcap.com> # This is the commit message tikv#31: *: update some dependencies (tikv#11458) close tikv#10997, ref tikv#11276 Signed-off-by: tabokie <xy.tao@outlook.com> # This is the commit message tikv#32: coprocessor: add `to_binary` and `from_binary` function (tikv#11397) Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#33: server: fix IOType configure for recving snapshot (tikv#11122) * fix IOType for recving snapshot Signed-off-by: MuZhou233 <muzhou233@outlook.com> * Set io_type before Write::write_all. close tikv#10659 Signed-off-by: MuZhou233 <MuZhou233@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#34: raftstore: add pessimistic lock table to peer (tikv#11459) ref tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#35: engine: fix backward compatibility issue of raft-engine.recovery-mode (tikv#11490) * add test case Signed-off-by: tabokie <xy.tao@outlook.com> * update raft-engine, fix tikv#11489 Signed-off-by: tabokie <xy.tao@outlook.com> * handle timeout error Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: qupeng <qupeng@pingcap.com> # This is the commit message tikv#36: metrics: add duration composition panel (tikv#11491) * metrics: add duration composition pannel Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> * close tikv#11492 Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#37: *: support read-through lock (tikv#11238) * storage: get/batch_get support read-through lock Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * storage: scan support read-through lock Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * coprocessor: support read-through lock Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * don't check access_locks when checking memory locks Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * fix clippy close tikv#11402 Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * update kvproto and add some comments Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * test desc scan Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#38: storage: support raw value encode for API V2 (tikv#11139) * implement api v2 raw value Signed-off-by: Andy Lok <andylokandy@hotmail.com> * storage: support raw value encode for API V2 ref tikv#10938 close tikv#11041 Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix clippy Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix clippy Signed-off-by: Andy Lok <andylokandy@hotmail.com> * remove println Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * add test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * disable test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test again Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test again Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test again Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix overflow Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix overflow Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve comment Signed-off-by: Andy Lok <andylokandy@hotmail.com> * add case Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve comment Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix clippy Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#39: *: replace snapshot `CbContext` with `SnapshotExt` (tikv#11471) ref tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> # This is the commit message tikv#40: Add comments Signed-off-by: mornyx <mornyx.z@gmail.com> # This is the commit message tikv#41: Extract one stats struct for KvGet/KvBatchGet Signed-off-by: mornyx <mornyx.z@gmail.com> # This is the commit message tikv#42: raftstore: relax merge result check (tikv#11478) * add test case Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * raftstore: relax merge result check Close tikv#11475. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * fix unstable transfer leader case Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Liqi Geng <gengliqiii@gmail.com> # This is the commit message tikv#43: engine: move raft engine purge job to a dedicated thread (tikv#11497) * update raft engine and implement batch_gc, ref tikv#11119 Signed-off-by: tabokie <xy.tao@outlook.com> * introduce purge-worker Signed-off-by: tabokie <xy.tao@outlook.com> * clean up old worker Signed-off-by: tabokie <xy.tao@outlook.com> * update raft engine Signed-off-by: tabokie <xy.tao@outlook.com> # This is the commit message tikv#44: Update max_ts for Prewrite and AcquirePessimisticLock with should_not_exist flag. (tikv#11488) * update max_ts for prewrite with should_not_exist flag Signed-off-by: dwangxxx <dwang_slam@163.com> * close tikv#11315 Signed-off-by: dwangxxx <dwang_slam@163.com> * close tikv#11315 Signed-off-by: dwangxxx <dwang_slam@163.com> * fix unit test Signed-off-by: dwangxxx <dwang_slam@163.com> * fix format Signed-off-by: dwangxxx <dwang_slam@163.com> * fix test Signed-off-by: dwangxxx <dwang_slam@163.com> * add test. Signed-off-by: dwangxxx <dwang_slam@163.com> * fix clippy error Signed-off-by: dwangxxx <dwang_slam@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#45: Rename KvGetLatencyStats Signed-off-by: mornyx <mornyx.z@gmail.com> # This is the commit message tikv#46: Rename Signed-off-by: mornyx <mornyx.z@gmail.com> # This is the commit message tikv#47: *: simplify pull request template (tikv#11504) We only needs to explain what's change. Problem description should be traced as issue. This PR also polishes contributing guide to explain the new flow. Close tikv#11503. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: qupeng <qupeng@pingcap.com> # This is the commit message tikv#48: cdc: introduce TsFilter into incremental scan (tikv#11385) * close tikv#11384 Signed-off-by: qupeng <qupeng@pingcap.com> # This is the commit message tikv#49: metric: add stale read count metric (tikv#11051) Signed-off-by: linning <linningde25@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#50: storage:change configuration storage.max-key-size default value from 4096 to 8192 (tikv#11331) close tikv#11241 Signed-off-by: TonsnakeLin <lpbgytong@163.com> # This is the commit message tikv#51: storage: make it cheaper to clone Scheduler (tikv#11501) closes tikv#11500 Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#52: raftclient: count term and index in estimated size (tikv#11493) * add test case Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * count term and index tag If there are many entries in a message, the estimated size of message can be way smaller than the actual size. This PR fixes the error by also counting index and term in estimation. It also remove the hard limit as the estimation is closed enough. Close tikv#9714. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#53: scanner/forward: move to next key if accesses a lock with delete type (tikv#11544) close tikv#11541 Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> # This is the commit message tikv#54: *: add configuration for in-memory pessimistic lock (tikv#11481) ref tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> # This is the commit message tikv#55: backup: fix background panic in `endpoint::tests::test_adjust_thread_pool_size` (tikv#11556) * backup: fix tikv#11548 Signed-off-by: yujuncen <yujuncen@pingcap.com> # This is the commit message tikv#56: tikv_util: reimplement poll_future_notify (tikv#11550) The current implementation is very easy to get deadlock if the future is woken immediately during polling. This PR resolves the issue by maintaining states instead. Note, if the future is implemented in the wrong way that always notifying the future without checking if it has readiness, the new implementation can run into dead loop. Close tikv#11549. Co-authored-by: qupeng <qupeng@pingcap.com> # This is the commit message tikv#57: fix taking rootfs as filesystem by mistake (tikv#11554) * fix adding rootfs as filesystem by mistake Signed-off-by: onlyacat <huangzhe8263@gmail.com> * reformat by lints Signed-off-by: onlyacat <huangzhe8263@gmail.com> * close tikv#11552 Signed-off-by: onlyacat <huangzhe8263@gmail.com> # This is the commit message tikv#58: raftstore: set term when catch up logs for merge (tikv#11545) * add test case Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * raftstore: set term when catch up logs for merge Prepare merge may includes logs from different terms, which may be even larger than the term of a follower that is lag behind. So when catching up logs by commit merge, its term should also be set to get a correct metadata. Close tikv#11526. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * add log for becoming follower Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * correct commit log check Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
mornyx
added a commit
to mornyx/tikv
that referenced
this issue
Dec 7, 2021
Signed-off-by: mornyx <mornyx.z@gmail.com> close tikv#8942 tikv_util: upgrade `gag` from 0.1 to 1.0 (tikv#11277) ref tikv#11276 Signed-off-by: kennytm <kennytm@gmail.com> codec: add compile time assert to guarantee safety (tikv#11357) * codec: add compile time assert to guarantee safety close tikv#11356. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * add more explanation. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> rafstore: fix commit state check in apply thread (tikv#11401) * fix apply check & close tikv#11396 Signed-off-by: gengliqi <gengliqiii@gmail.com> * make format Signed-off-by: gengliqi <gengliqiii@gmail.com> * format Signed-off-by: gengliqi <gengliqiii@gmail.com> backup: create external storage per-backup reqeust. (tikv#11113) * backup: create external storage per-backup reqeust. fixed tikv#11112 Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: make clippy happy Signed-off-by: yujuncen <yujuncen@pingcap.com> * backup: fix build Signed-off-by: yujuncen <yujuncen@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Remove duplicate enum and macro about numberic_enum_mod (tikv#11347) * remove duplicated and useless `remove duplicated and useless` close tikv#11297 Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * remove duplicate macro rocksdb_enum_mod Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * fix format problem Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * resolve discussion Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * fix test error Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * move macro `numeric_enum_serializing_mod` to a proper location Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * move `perf_level_serde` of mod config in engine trait to mod perf_context Signed-off-by: yuqi1129 <yuqi4733@gmail.com> * Add test about `rate-limiter-mode` and fix compile error in integration test Signed-off-by: yuqi1129 <yuqi4733@gmail.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> ctl: detect raft db correctly (tikv#11395) Close tikv#11393. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Support to iterate table properties for KvEngine (tikv#11388) * support to iterate table properties for KvEngine * close tikv#11387 Signed-off-by: qupeng <qupeng@pingcap.com> gc: Fix GC scan effectiveness to avoid OOM (tikv#11416) * close tikv#11410 limit the pending task for gc worker Signed-off-by: Connor1996 <zbk602423539@gmail.com> * avoid using prefix seek Signed-off-by: Connor1996 <zbk602423539@gmail.com> * rename test Signed-off-by: Connor1996 <zbk602423539@gmail.com> * address comment Signed-off-by: Connor1996 <zbk602423539@gmail.com> * fix build Signed-off-by: Connor1996 <zbk602423539@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> fix a panic about parsing memory.limit_in_bytes (tikv#11354) * handle int overflow for memory.limit_in_bytes clsoe tikv#11353. Signed-off-by: qupeng <qupeng@pingcap.com> * close tikv#11353 Signed-off-by: qupeng <qupeng@pingcap.com> * address comments Signed-off-by: qupeng <qupeng@pingcap.com> build: fix make docker on Linux (tikv#11349) * fix make docker fix tikv#11312 Signed-off-by: tabokie <xy.tao@outlook.com> * fix cargo Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> engine_rocks: ignore NoSpace error during flush or compaction (tikv#11225) * ignore NoSpace error during flush or compaction fix tikv#11224 Signed-off-by: tabokie <xy.tao@outlook.com> * use constant Signed-off-by: tabokie <xy.tao@outlook.com> * fix typo Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> gc: set pending capacity for gc worker (tikv#11429) Signed-off-by: Connor1996 <zbk602423539@gmail.com> raftstore: support synchronous write (tikv#11418) * support sync write & ref tikv#11165 Signed-off-by: gengliqi <gengliqiii@gmail.com> * add tests Signed-off-by: gengliqi <gengliqiii@gmail.com> * change default store-io-pool-size to 0 Signed-off-by: gengliqi <gengliqiii@gmail.com> * remove unnecessary test Signed-off-by: gengliqi <gengliqiii@gmail.com> * merge raftstore: separate read write ready(tikv#10592) Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comments Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix test Signed-off-by: gengliqi <gengliqiii@gmail.com> * make clippy happy Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comment Signed-off-by: gengliqi <gengliqiii@gmail.com> * remove offset in NormalFsm Signed-off-by: gengliqi <gengliqiii@gmail.com> * make format Signed-off-by: gengliqi <gengliqiii@gmail.com> metrics: add Raft Engine panel (tikv#11415) Signed-off-by: tabokie <xy.tao@outlook.com> resource-metering: add some metrics (tikv#11414) * resource-metering: add some metrics Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * more log Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> * polish description, ref tikv#11436 Signed-off-by: Zhenchi <zhongzc_arch@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> metrics: add asynchronous write io panels (tikv#11408) * update metrics name Signed-off-by: gengliqi <gengliqiii@gmail.com> * update tikv-details & close tikv#10540 Signed-off-by: gengliqi <gengliqiii@gmail.com> * rename Raft GC to Raft Log GC Signed-off-by: gengliqi <gengliqiii@gmail.com> * address comment Signed-off-by: gengliqi <gengliqiii@gmail.com> * fix datasource Signed-off-by: gengliqi <gengliqiii@gmail.com> * add more metrics Signed-off-by: gengliqi <gengliqiii@gmail.com> * update metrics Signed-off-by: gengliqi <gengliqiii@gmail.com> * update Signed-off-by: gengliqi <gengliqiii@gmail.com> storage: fix reverse scan check memory locks (tikv#11447) close tikv#11440 Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> test: make test_sending_fail_with_net_error more statable (tikv#11366) * sending count should deregister on time Signed-off-by: bufferflies <1045931706@qq.com> * make test statable Signed-off-by: bufferflies <1045931706@qq.com> * remove unused change Signed-off-by: bufferflies <1045931706@qq.com> * remove unused change Signed-off-by: bufferflies <1045931706@qq.com> * close tikv#11365 Signed-off-by: bufferflies <1045931706@qq.com> * reformat annotation Signed-off-by: bufferflies <1045931706@qq.com> * add send once filter Signed-off-by: bufferflies <1045931706@qq.com> * lint pass Signed-off-by: bufferflies <1045931706@qq.com> * lint pass Signed-off-by: bufferflies <1045931706@qq.com> * replace sendOnceMsgFilter Signed-off-by: bufferflies <1045931706@qq.com> * remove unnecessary change Signed-off-by: bufferflies <1045931706@qq.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> raftstore: increase batch for raftlog-gc-worker (tikv#11405) * increase batch Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * do not change seek Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix fmt Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix test Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * fix number overflow Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * increase batch Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * revert time Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * set last_compacted_idx for split region Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * log large range gc task Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * address comment Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> * close tikv#11404 Signed-off-by: Little-Wallace <bupt2013211450@gmail.com> Co-authored-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Make tikv-server fully generic over kv engines (tikv#10945) * server: Allow flow_info_receiver to be None Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Move init_flow_receiver call into init_raw_engines Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract debug service creation to CreateKvEngine::start_debug_service Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract backup service creation to CreateKvEngine::start_backup_service Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract rocks cfg controller registration to CreateKvEngine::register_kv_config Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract create_raftstore_compaction_listener to free function Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract create_kv_engine to CreateKvEngine trait Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Parameterize EngineResourceInfo over EK Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Parameterize TikvServer over EK Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Document CreateKvEngine trait Signed-off-by: Brian Anderson <andersrb@gmail.com> * clippy Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract raft engine creation to CreateRaftEngine::create_raft_engine Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract CreateRaftEngine::register_raft_config method Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract CreateRaftEngine::create_engine_infos method Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Unspecialize init_raw_engines method Signed-off-by: Brian Anderson <andersrb@gmail.com> * server: Extract setup code to generic run_server function Signed-off-by: Brian Anderson <andersrb@gmail.com> * clippy ref tikv#6402 Signed-off-by: Brian Anderson <andersrb@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Revert "Make tikv-server fully generic over kv engines (tikv#10945)" (tikv#11462) This reverts commit df2e921. Ref tikv#11461. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> raftstore: flush waterfall metrics when using synchronous write (tikv#11451) Signed-off-by: gengliqi <gengliqiii@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> fix one disk full unstable test case on assert condition. (tikv#11391) close tikv#11389 Signed-off-by: tier-cap <zhengxiaojin@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> resolved_ts: return early after regions already had quorum (tikv#11352) * close tikv#11351 Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * fix tests Signed-off-by: 5kbpers <tangminghua@pingcap.com> * Apply suggestions from code review Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: Zixiong Liu <liuzixiong@pingcap.com> * address comments Signed-off-by: 5kbpers <tangminghua@pingcap.com> * make clippy happy Signed-off-by: 5kbpers <tangminghua@pingcap.com> * close tikv#11400 Signed-off-by: 5kbpers <tangminghua@pingcap.com> Co-authored-by: qupeng <qupeng@pingcap.com> Co-authored-by: Zixiong Liu <liuzixiong@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> support to get approximate keys and size for a given range from UserCollectedProperties (tikv#11465) Signed-off-by: qupeng <qupeng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> copr: add paging API support for streaming-like processling (tikv#11300) * copr response with range Signed-off-by: you06 <you1474600@gmail.com> * switch to new kvproto Signed-off-by: you06 <you1474600@gmail.com> * fix bugs... Signed-off-by: you06 <you1474600@gmail.com> * format Signed-off-by: you06 <you1474600@gmail.com> * remove unsed field Signed-off-by: you06 <you1474600@gmail.com> * add comments Signed-off-by: you06 <you1474600@gmail.com> * update kvproto Signed-off-by: you06 <you1474600@gmail.com> * fix bench Signed-off-by: you06 <you1474600@gmail.com> * update cargo.lock Signed-off-by: you06 <you1474600@gmail.com> * update comment Signed-off-by: you06 <you1474600@gmail.com> * add test for copr paging Signed-off-by: you06 <you1474600@gmail.com> * close tikv#11448 Signed-off-by: you06 <you1474600@gmail.com> * Update components/tidb_query_executors/src/runner.rs Co-authored-by: Lei Zhao <zlwgx1023@gmail.com> Signed-off-by: you06 <you1474600@gmail.com> * check range in response Signed-off-by: you06 <you1474600@gmail.com> * format code Signed-off-by: you06 <you1474600@gmail.com> * add test for reverse scan Signed-off-by: you06 <you1474600@gmail.com> * add test for multi ranges Signed-off-by: you06 <you1474600@gmail.com> * test drained with multi ranges Signed-off-by: you06 <you1474600@gmail.com> * fix clippy Signed-off-by: you06 <you1474600@gmail.com> * fix lint Signed-off-by: you06 <you1474600@gmail.com> * paging by all record count Signed-off-by: you06 <you1474600@gmail.com> * format Signed-off-by: you06 <you1474600@gmail.com> Co-authored-by: Lei Zhao <zlwgx1023@gmail.com> encryption: support seek for encryption writer (tikv#10929) * support seek for encryption writer Signed-off-by: tabokie <xy.tao@outlook.com> * minor fix, ref: tikv#11119 Signed-off-by: tabokie <xy.tao@outlook.com> * fix error from merge Signed-off-by: tabokie <xy.tao@outlook.com> * fight format checker, ref tikv#11119 Signed-off-by: tabokie <xy.tao@outlook.com> * add wrappers Signed-off-by: tabokie <xy.tao@outlook.com> * remove pub quantifiers and fix lint Signed-off-by: tabokie <xy.tao@outlook.com> * fix name export Signed-off-by: tabokie <xy.tao@outlook.com> * revert naming changes Signed-off-by: tabokie <xy.tao@outlook.com> * address comment on unit test Signed-off-by: tabokie <xy.tao@outlook.com> * fix test and inline wrapper calls Signed-off-by: tabokie <xy.tao@outlook.com> * fix comments Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> server: remove register PD process (tikv#10985) * remove register PD process Signed-off-by: Ryan Leung <rleungx@gmail.com> * resolve conflicts close tikv#11472 Signed-off-by: Ryan Leung <rleungx@gmail.com> engine: provide customized file builder to raft engine (tikv#10937) * open raft engine with customized file builder ref tikv#11119 Signed-off-by: tabokie <xy.tao@outlook.com> * overhaul configurations Signed-off-by: tabokie <xy.tao@outlook.com> * update default config and address comment Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: qupeng <qupeng@pingcap.com> *: update some dependencies (tikv#11458) close tikv#10997, ref tikv#11276 Signed-off-by: tabokie <xy.tao@outlook.com> coprocessor: add `to_binary` and `from_binary` function (tikv#11397) Signed-off-by: xiongjiwei <xiongjiwei1996@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> server: fix IOType configure for recving snapshot (tikv#11122) * fix IOType for recving snapshot Signed-off-by: MuZhou233 <muzhou233@outlook.com> * Set io_type before Write::write_all. close tikv#10659 Signed-off-by: MuZhou233 <MuZhou233@outlook.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> raftstore: add pessimistic lock table to peer (tikv#11459) ref tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> engine: fix backward compatibility issue of raft-engine.recovery-mode (tikv#11490) * add test case Signed-off-by: tabokie <xy.tao@outlook.com> * update raft-engine, fix tikv#11489 Signed-off-by: tabokie <xy.tao@outlook.com> * handle timeout error Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: qupeng <qupeng@pingcap.com> metrics: add duration composition panel (tikv#11491) * metrics: add duration composition pannel Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> * close tikv#11492 Signed-off-by: zhangjinpeng1987 <zhangjinpeng@pingcap.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> *: support read-through lock (tikv#11238) * storage: get/batch_get support read-through lock Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * storage: scan support read-through lock Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * coprocessor: support read-through lock Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * don't check access_locks when checking memory locks Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * fix clippy close tikv#11402 Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * update kvproto and add some comments Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> * test desc scan Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> storage: support raw value encode for API V2 (tikv#11139) * implement api v2 raw value Signed-off-by: Andy Lok <andylokandy@hotmail.com> * storage: support raw value encode for API V2 ref tikv#10938 close tikv#11041 Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix clippy Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix clippy Signed-off-by: Andy Lok <andylokandy@hotmail.com> * remove println Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * add test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * disable test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test again Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test again Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test again Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix overflow Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix overflow Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve comment Signed-off-by: Andy Lok <andylokandy@hotmail.com> * add case Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve comment Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix clippy Signed-off-by: Andy Lok <andylokandy@hotmail.com> * improve Signed-off-by: Andy Lok <andylokandy@hotmail.com> * fix test Signed-off-by: Andy Lok <andylokandy@hotmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> *: replace snapshot `CbContext` with `SnapshotExt` (tikv#11471) ref tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> Add comments Signed-off-by: mornyx <mornyx.z@gmail.com> Extract one stats struct for KvGet/KvBatchGet Signed-off-by: mornyx <mornyx.z@gmail.com> raftstore: relax merge result check (tikv#11478) * add test case Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * raftstore: relax merge result check Close tikv#11475. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * fix unstable transfer leader case Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Liqi Geng <gengliqiii@gmail.com> engine: move raft engine purge job to a dedicated thread (tikv#11497) * update raft engine and implement batch_gc, ref tikv#11119 Signed-off-by: tabokie <xy.tao@outlook.com> * introduce purge-worker Signed-off-by: tabokie <xy.tao@outlook.com> * clean up old worker Signed-off-by: tabokie <xy.tao@outlook.com> * update raft engine Signed-off-by: tabokie <xy.tao@outlook.com> Update max_ts for Prewrite and AcquirePessimisticLock with should_not_exist flag. (tikv#11488) * update max_ts for prewrite with should_not_exist flag Signed-off-by: dwangxxx <dwang_slam@163.com> * close tikv#11315 Signed-off-by: dwangxxx <dwang_slam@163.com> * close tikv#11315 Signed-off-by: dwangxxx <dwang_slam@163.com> * fix unit test Signed-off-by: dwangxxx <dwang_slam@163.com> * fix format Signed-off-by: dwangxxx <dwang_slam@163.com> * fix test Signed-off-by: dwangxxx <dwang_slam@163.com> * add test. Signed-off-by: dwangxxx <dwang_slam@163.com> * fix clippy error Signed-off-by: dwangxxx <dwang_slam@163.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Rename KvGetLatencyStats Signed-off-by: mornyx <mornyx.z@gmail.com> Rename Signed-off-by: mornyx <mornyx.z@gmail.com> *: simplify pull request template (tikv#11504) We only needs to explain what's change. Problem description should be traced as issue. This PR also polishes contributing guide to explain the new flow. Close tikv#11503. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: qupeng <qupeng@pingcap.com> cdc: introduce TsFilter into incremental scan (tikv#11385) * close tikv#11384 Signed-off-by: qupeng <qupeng@pingcap.com> metric: add stale read count metric (tikv#11051) Signed-off-by: linning <linningde25@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> storage:change configuration storage.max-key-size default value from 4096 to 8192 (tikv#11331) close tikv#11241 Signed-off-by: TonsnakeLin <lpbgytong@163.com> storage: make it cheaper to clone Scheduler (tikv#11501) closes tikv#11500 Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> raftclient: count term and index in estimated size (tikv#11493) * add test case Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * count term and index tag If there are many entries in a message, the estimated size of message can be way smaller than the actual size. This PR fixes the error by also counting index and term in estimation. It also remove the hard limit as the estimation is closed enough. Close tikv#9714. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> scanner/forward: move to next key if accesses a lock with delete type (tikv#11544) close tikv#11541 Signed-off-by: youjiali1995 <zlwgx1023@gmail.com> *: add configuration for in-memory pessimistic lock (tikv#11481) ref tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> backup: fix background panic in `endpoint::tests::test_adjust_thread_pool_size` (tikv#11556) * backup: fix tikv#11548 Signed-off-by: yujuncen <yujuncen@pingcap.com> tikv_util: reimplement poll_future_notify (tikv#11550) The current implementation is very easy to get deadlock if the future is woken immediately during polling. This PR resolves the issue by maintaining states instead. Note, if the future is implemented in the wrong way that always notifying the future without checking if it has readiness, the new implementation can run into dead loop. Close tikv#11549. Co-authored-by: qupeng <qupeng@pingcap.com> fix taking rootfs as filesystem by mistake (tikv#11554) * fix adding rootfs as filesystem by mistake Signed-off-by: onlyacat <huangzhe8263@gmail.com> * reformat by lints Signed-off-by: onlyacat <huangzhe8263@gmail.com> * close tikv#11552 Signed-off-by: onlyacat <huangzhe8263@gmail.com> raftstore: set term when catch up logs for merge (tikv#11545) * add test case Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * raftstore: set term when catch up logs for merge Prepare merge may includes logs from different terms, which may be even larger than the term of a follower that is lag behind. So when catching up logs by commit merge, its term should also be set to get a correct metadata. Close tikv#11526. Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * add log for becoming follower Signed-off-by: Jay Lee <BusyJayLee@gmail.com> * correct commit log check Signed-off-by: Jay Lee <BusyJayLee@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> storage: Add API V2 check for RawKV and TxnKV requests (part 2) (tikv#11228) * wip Signed-off-by: pingyu <yuping@pingcap.com> * wip Signed-off-by: pingyu <yuping@pingcap.com> * wip Signed-off-by: pingyu <yuping@pingcap.com> * wip Signed-off-by: pingyu <yuping@pingcap.com> * wip Signed-off-by: pingyu <yuping@pingcap.com> * wip Signed-off-by: pingyu <yuping@pingcap.com> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <yuping@pingcap.com> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <yuping@pingcap.com> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <yuping@pingcap.com> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <yuping@pingcap.com> * resolve conflict Signed-off-by: pingyu <yuping@pingcap.com> * add raw_key_maybe_unbounded_into_option Signed-off-by: pingyu <yuping@pingcap.com> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <yuping@pingcap.com> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <yuping@pingcap.com> * ref tikv#10974: address comments Signed-off-by: pingyu <yuping@pingcap.com> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <yuping@pingcap.com> * ref tikv#10974: Add API V2 check for RawKV and TxnKV requests (part 2) Signed-off-by: pingyu <yuping@pingcap.com> * remove make_invalid_key_prefix_err Signed-off-by: andylokandy <andylokandy@hotmail.com> Co-authored-by: andylokandy <andylokandy@hotmail.com> storage: read and write in-memory pessimistic locks in the scheduler (tikv#11529) ref tikv#11452 Signed-off-by: Yilin Chen <sticnarf@gmail.com> tolerate unmounted cgroup controller (tikv#11570) * tolerate unmounted cgroup controller, fix tikv#11569 Signed-off-by: tabokie <xy.tao@outlook.com> * add test Signed-off-by: tabokie <xy.tao@outlook.com> * address comment Signed-off-by: tabokie <xy.tao@outlook.com> cdc: stabilize some cdc cases (tikv#11586) * close tikv#11560 Signed-off-by: qupeng <qupeng@pingcap.com>
5kbpers
pushed a commit
to 5kbpers/tikv
that referenced
this issue
Apr 14, 2022
* Remove `skip_gc_raft_log_ticks` and shorten tick interval. Previously (before tikv#8668), trigger gc will force WAL sync in apply thread, and use `skip_gc_raft_log_ticks` to reduce syncing. After [tikv#8668](tikv#8668), syncing is moved to background worker thread, and `CompactLog` cmds are batched. So there is no need to use `skip_gc_raft_log_ticks` anymore. Signed-off-by: MrCroxx <mrcroxx@outlook.com> * restore raft log gc tick interval and fix typo restore raft log gc tick interval, it's better to seperate default config modification from this pr. Signed-off-by: MrCroxx <mrcroxx@outlook.com> * remove raft log gc tick register whthin threshold. Signed-off-by: MrCroxx <mrcroxx@outlook.com> * remove tests for discarded config. Signed-off-by: MrCroxx <mrcroxx@outlook.com> * Hide `raft-log-gc-threshold` and set default value to 1. This config is not needed after tikv#8668, but some tests rely on it to prevent log compaction. Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix typo Co-authored-by: Xinye Tao <xy.tao@outlook.com> Signed-off-by: MrCroxx <mrcroxx@outlook.com> * Use optimized `raft_log_gc_tick_interval` config. Signed-off-by: MrCroxx <mrcroxx@outlook.com> * remove stale config `raft-log-reserve-max-ticks` Signed-off-by: MrCroxx <mrcroxx@outlook.com> * stop ticking raft log gc if role becomes follower Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix bug eliminate follower raft gc tick will reduce some cpu usage for ref tikv#10540 Signed-off-by: MrCroxx <mrcroxx@outlook.com> * update config items for async-io - Restore config `raft-log-gc-threshold`. - Make `raft-log-gc-batch-size` configurable. Signed-off-by: MrCroxx <mrcroxx@outlook.com> * remove raft gc tick batch, make compact interval configurable Signed-off-by: MrCroxx <mrcroxx@outlook.com> * restore raft-log-reserve-max-ticks Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix metrics Signed-off-by: MrCroxx <mrcroxx@outlook.com> * fix bug Signed-off-by: MrCroxx <mrcroxx@outlook.com> * restore register tick, remove flush per recv task Signed-off-by: MrCroxx <mrcroxx@outlook.com> * rename to raft-log-comapct-sync-interval Signed-off-by: MrCroxx <mrcroxx@outlook.com> * update optimized config Signed-off-by: MrCroxx <mrcroxx@outlook.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Development Task
The sync write io in store thread introduces much extra latency. After detailed analysis, we found that it increases commit duration by about one IO+fsync time. After tikv/raft-rs#403, we can use async write io in store thread.
Task list(updated 2021/9/8)
It is split into several PRs for review easily.
raftstore: introduce asynchronous write io #10326
raftstore: add write router for async write io #10667
raftstore: always enable entry cache for async write io #10672
The text was updated successfully, but these errors were encountered: