Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: rollup of in-memory engine commits #17152

Merged

Conversation

overvenus
Copy link
Member

What is changed and how it works?

What's Changed:

Rollup of in-memory engine commits for release-8.1 branch.

Check List

Tests

  • Unit test
  • Integration test

Side effects

Release note

None

ti-chi-bot and others added 30 commits May 24, 2024 08:44
 

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
…17068)

ref tikv#16818

Refactor logs for the unexpected path, print both the exsiting row and input key/value.

Signed-off-by: cfzjywxk <lsswxrxr@163.com>

Co-authored-by: cfzjywxk <lsswxrxr@163.com>
) (tikv#17108)

close tikv#17004

Signed-off-by: wuhuizuo <wuhuizuo@126.com>

Co-authored-by: wuhuizuo <wuhuizuo@126.com>
…tikv#17120)

 

Signed-off-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
Signed-off-by: Jinpeng Zhang <zzzhangjinpeng@gmail.com>

Co-authored-by: Jinpeng Zhang <zzzhangjinpeng@gmail.com>
Co-authored-by: wuhuizuo <wuhuizuo@126.com>
…arte file (tikv#16792)

ref tikv#16141

strip off read related code from engine.rs to a separte file

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
ref tikv#16141

collector iteration related metrics

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
ref tikv#16141

expose range cache engine configuration

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
…ange (tikv#16794)

ref tikv#16141

do not load a range overlapped with an unresolved range

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
close tikv#16782

Signed-off-by: zhangjinpeng87 <zzzhangjinpeng@gmail.com>
Signed-off-by: Jinpeng Zhang <zzzhangjinpeng@gmail.com>

Co-authored-by: lucasliang <nkcs_lykx@hotmail.com>
ref tikv#16141

add metrics for memory usage

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>

Co-authored-by: Jinpeng Zhang <zzzhangjinpeng@gmail.com>
…nd write duration (tikv#16831)

ref tikv#16141

add metrics for gc duration, range load duration, and write duration

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
…ne (tikv#16852)

ref tikv#16141

delete lock physically rather than writting tombstone

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
…kv#16850)

ref tikv#16141

fix missing memory record when snapshot loading

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
…ikv#16816)

ref tikv#16141

In-Memory Engine: add a tag into Range for diagnostics

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
Signed-off-by: Qi Xu <tonyxuqqi@outlook.com>

Co-authored-by: SpadeA-Tang <u6748471@anu.edu.au>
Co-authored-by: Qi Xu <tonyxuqqi@outlook.com>
ref tikv#16141

remove out of date pending ranges

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
…r memory-engine (tikv#16897)

ref tikv#16141

compile the in-memory-engine by default and support dynamic config update for memory-engine.
The enabled flag would support enabled:true -> enabled:false and soft/hard limit also supports online change. 
But if enabled:false is set initially, the config change to make enabled:true needs the TiKV restart.

Signed-off-by: tonyxuqqi <tonyxuqi@outlook.com>
…gc (tikv#16944)

ref tikv#16141

skiplist tombstone should be handled separetely in gc

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
…id the consideration of tag (tikv#16959)

ref tikv#16141

manually implement partial_eq for CacheRange to avoid the consideration of tag

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>

Co-authored-by: tonyxuqqi <tonyxuqi@outlook.com>
…atistics (tikv#16945)

ref tikv#16141

implement core local array and collect read flow statistics

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
ref tikv#16141

fix seek does not set self.valid properly

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
…ions (tikv#16997)

ref tikv#16141

add metrics for different types of iteration operations

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
ref tikv#16141

add metrics for range count

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
Signed-off-by: Spade  A <71589810+SpadeA-Tang@users.noreply.github.com>

Co-authored-by: lucasliang <nkcs_lykx@hotmail.com>
ref tikv#16141

fix gc worker not used

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
…tart-time (tikv#17069)

close tikv#17066

fix online config change panic for periodic-full-compact-start-time

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
close tikv#17080

* Test both formats +HHMM offsets and as well as +HH:MM for ReadableSchedule.

Signed-off-by: Alex Feinberg <alex@strlen.net>
…y be handled once in GC (tikv#17083)

ref tikv#16141, close tikv#17060

mvcc keys with different sequen number should only be handled once in GC

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
Signed-off-by: Spade  A <71589810+SpadeA-Tang@users.noreply.github.com>

Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: lucasliang <nkcs_lykx@hotmail.com>
ref tikv#16141, close tikv#17086

enable compaction filter for HybridEngine

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
…ikv#17122)

ref tikv#16141, close tikv#17114

sequence number should be increased for each key

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
…7129)

ref tikv#16141, close tikv#17079

support reverse direction when iterating

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
…v#17128)

ref tikv#16141, close tikv#17127

clean lock cf tombstone in a background worker

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
ref tikv#16141, close tikv#17131

consider range overlaps when eviction

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
Copy link
Contributor

ti-chi-bot bot commented Jun 17, 2024

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

SpadeA-Tang and others added 14 commits June 18, 2024 17:21
ref tikv#16141

support delete range

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
close tikv#17153

evict range should consider loading range

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ref tikv#16141, close tikv#17103

clear write batch when range load failed

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
…e to memory (tikv#17149)

close tikv#17104

fix concurrency issue between delete range and write to memory

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ref tikv#16141

add seek duration metrics

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
)

ref tikv#16141

add metrics for prepare for write duration

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
ref tikv#16141

set disk engine when start

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ref tikv#16141

avoid using snapshot cache

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
ref tikv#16141

evict range when becomeing follower, merge, and deleting data ranges.

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
…tikv#17171)

ref tikv#16141

filter plus one when any key is removed during GC

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
Signed-off-by: Spade  A <71589810+SpadeA-Tang@users.noreply.github.com>
Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
@overvenus overvenus changed the base branch from release-8.1 to release-8.1-20240621-v8.1.0 June 21, 2024 11:52
@overvenus overvenus changed the title [DNM] *: rollup of in-memory engine commits *: rollup of in-memory engine commits Jun 21, 2024
Signed-off-by: Neil Shen <overvenus@gmail.com>
Signed-off-by: Neil Shen <overvenus@gmail.com>
Copy link
Contributor

ti-chi-bot bot commented Jun 21, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@overvenus overvenus marked this pull request as ready for review June 21, 2024 12:10
@overvenus overvenus merged commit c57aef1 into tikv:release-8.1-20240621-v8.1.0 Jun 21, 2024
2 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants