Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to near 7.6(a0e8a7a163302bc9a7be5) #362

Merged
merged 204 commits into from
Jan 2, 2024

Conversation

CalvinNeo
Copy link
Member

@CalvinNeo CalvinNeo commented Dec 22, 2023

What problem does this PR solve?

Issue Number: close #xxx

Problem Summary:
Origin 24c3802

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

Connor1996 and others added 30 commits August 24, 2023 04:09
close tikv#11161

Add back heap profile HTTP API and make it secure. The API is removed by tikv#11162 due to a 
secure issue that can visit arbitrary files on the server. This PR makes it only show the file 
name instead of the absolute path, and adds a paranoid check to make sure the passed file
name is in the set of heap profiles.

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
… tablet (tikv#15332)

ref tikv#12842

- Fix a bug of compact range that causes a dirty tablet being reported as clean.
- Added an additional check to ensure trim's correctness.
- Fix a bug that some tablets are not destroyed and block peer destroy progress.

Signed-off-by: tabokie <xy.tao@outlook.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
…shutting dowm (tikv#15426)

ref tikv#15202

not panic in the case of unexepected dropped channel when shutting dowm

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
…ikv#15427)

close tikv#15282

disable duplicated mvcc key check compaction by default

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
close tikv#15357

Correct the raft_router/apply_router's alive and leak metrics.

Signed-off-by: tonyxuqqi <tonyxuqi@outlook.com>
…ikv#15440)

close tikv#15438

fix unwrap panic of region_compact_redundant_rows_percent

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
close tikv#15430

Use concurrent hashmap to avoid router cache occupying too much memory

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
close tikv#13311

Fix the possible meta inconsistency issue.

Signed-off-by: cfzjywxk <lsswxrxr@163.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ref tikv#14864

This is the first PR to fix OOM caused by Resolver tracking large txns.
Resolver checks memory quota before tracking a lock, and returns false
if it exceeds memory quota.

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
…tikv#15425)

close tikv#15424

Signed-off-by: glorv <glorvs@163.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
…5421)

ref tikv#15409

Signed-off-by: bufferflies <1045931706@qq.com>

Co-authored-by: Spade  A <71589810+SpadeA-Tang@users.noreply.github.com>
close tikv#14864

Fix resolved ts OOM caused by Resolver tracking large txns. `ObserveRegion` is
deregistered if it exceeds memory quota. It may cause higher CPU usage because
of scanning locks, but it's better than OOM.

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
…15453)

ref tikv#12842

support column family based write buffer manager

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
ref tikv/pd#6556, close tikv#15428

pc_client: add store-level backoff for the reconnect retries

Signed-off-by: nolouch <nolouch@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
close tikv#15405

Signed-off-by: bufferflies <1045931706@qq.com>

Co-authored-by: Spade  A <71589810+SpadeA-Tang@users.noreply.github.com>
ref tikv#12842

- Initialize `persisted_apply_index` on startup.

Signed-off-by: tabokie <xy.tao@outlook.com>
…for mvcc scan (tikv#15455)

ref tikv#14654

consider unmatch between region range and tablet range for mvcc scan
close tikv#12304

Add logs for assertion failure

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
close tikv#15403

1. split config support to update dynamic. In past, the `optimize_for` function will set the config immutable.

Signed-off-by: bufferflies <1045931706@qq.com>
ref tikv#15409

reuse failpoint tests in async_io_test

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
close tikv#15490

avoid duplicated Instant:now

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
close tikv#15458

Resolver owns a hash map to tracking locks and unlock events, and so
for calculating resolved ts. However, it does not shrink map even after
all lock are removed, this may result OOM if there are transactions
that modify many rows across many regions. The total memory usage is
proportional to the number of modified rows.

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
close tikv#15468

Return `RegionNotFound` while cannot find peer in the current store.

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ref tikv#8235

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
…15504)

close tikv#15503

fix panic of dynamic changing write-buffer-limit

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
close tikv#15487

Signed-off-by: qupeng <qupeng@pingcap.com>
ref tikv#15409

reuse failpoint tests in test_early_apply

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
…15456)

close tikv#15457

there are three triggers will split the regions:
1. load split include sizekeys, load etc. In this cases, the new region should contains the data  after split.
2. tidb split tables or partition table, such like `create table test.t1(id int,b int) shard_row_id_bits=4 partition by hash(id) partitions 2000`. 
In this cases , the new region shouldn't contains any data  after split.

Signed-off-by: bufferflies <1045931706@qq.com>
ref tikv#15461

limit the flush times during server stop

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
ref tikv#14864

* Fix resolved ts OOM caused by adding large txns locks to `ResolverStatus`.
* Add initial scan backoff duration metrics.

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Co-authored-by: Connor <zbk602423539@gmail.com>
cfzjywxk and others added 6 commits December 20, 2023 07:52
…kv#16180)

ref tikv#15066

Change in-memory pessimsitic locks from hash map to btree map, support collecting pessimistic locks for scan lock command.
Then:
1. GC could collect expired pessimistic locks.
2. Pessimistic rollback could use read scan first and then clean up expired pessimistic locks at one time.

Signed-off-by: cfzjywxk <lsswxrxr@163.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
close tikv#16197

This commit add histograms for status server APIs, so that we can inspect
the duration for each API request.
It helps us to identify any correlations between specific API requests and
potential incidents.

Signed-off-by: Neil Shen <overvenus@gmail.com>
ref tikv#16141

implement read flow of in-memory engine -- hybrid engine part

Signed-off-by: SpadeA-Tang <u6748471@anu.edu.au>
close tikv#16191

change metrics label name for resource-group-name from "name" to "resource_group".

To be backward compatible with old grafana panel, we add a new label name and keep the old one. We are going to deprecate the old label in v8.0.

Signed-off-by: glorv <glorvs@163.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ref tikv#15066

Add scan lock read duration panel to check read lock holding time.

Signed-off-by: cfzjywxk <lsswxrxr@163.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
close tikv#16015

Revert "*: make unified-pool use FuturePool (tikv#15925)"
- revert due to performance regression

Signed-off-by: nolouch <nolouch@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@CLAassistant
Copy link

CLAassistant commented Dec 22, 2023

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 4 committers have signed the CLA.

✅ glorv
✅ CalvinNeo
❌ cfzjywxk
❌ nolouch
You have signed the CLA already but the status is still pending? Let us recheck it.

@CalvinNeo CalvinNeo changed the title Upgrade to near 7.6 Upgrade to near 7.6(a0e8a7a163302bc9a7be5) Dec 22, 2023
@CalvinNeo CalvinNeo force-pushed the upgrade-to-near-7.6 branch 7 times, most recently from 3cf236e to e898f3e Compare December 22, 2023 06:17
@CalvinNeo
Copy link
Member Author

/run-all-tests

@CalvinNeo CalvinNeo force-pushed the upgrade-to-near-7.6 branch 9 times, most recently from 2f8d1cc to 38a171b Compare December 24, 2023 06:00
@ti-chi-bot ti-chi-bot bot added the size/XXL label Jan 2, 2024
@CalvinNeo CalvinNeo merged commit b7500ab into pingcap:raftstore-proxy Jan 2, 2024
2 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.