pitr checkpoint ts lag reached more than 8h after inject network partition between one of tikv and pd leader

[Lily2025]

Bug Report

What version of TiKV are you using?

./tikv-server -V
TiKV
Release Version: 8.0.0-alpha
Edition: Community
Git Commit Hash: 43d0e06
Git Commit Branch: heads/refs/tags/v8.0.0-alpha
UTC Build Time: 2024-01-26 11:47:08
Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27)
Enable Features: pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure trace-async-tasks openssl-vendored
Profile: dist_release

What operating system and CPU are you using?

8c/32g

Steps to reproduce

1、start pitr
2、run workload
go-tpc tpcc run -D tpcc20000 --host tc-tidb.endless-ha-test-oltp-pitr-tps-6570032-1-702 -P4000 --warehouses 20000 -T 32 --ignore-error '2013,1213,1105,1205,8022,8028,9004,9007,1062' --user root --password '' --interval '10s'
3、inject network partition between one of tikv and pd leader

What did you expect?

pitr checkpoint ts lag less than 10mins after fault recover

What did happened?

pitr checkpoint ts lag reached more than 8h after inject network partition between one of tikv and pd leader

[Lily2025]

/severity critical

[Lily2025]

/assign BornChanger

[ti-chi-bot]

@Lily2025: GitHub didn't allow me to assign the following users: BornChanger.

Note that only tikv members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign BornChanger

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[BornChanger]

/component br

[ti-chi-bot]

@BornChanger: The label(s) component/br cannot be applied, because the repository doesn't have them.

In response to this:

/component br

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[BornChanger]

/component backup-restore

[Lily2025]

/assign YuJuncen

[YuJuncen]

This should be a mistake in #16008, which added a stale checking to every Start command.

The time line is:

Region R become leader -> StartObserve(R) sent -> Region epoch of R changed -> StartObserve(R) received but dropped

Then, the RefreshObserver will be dropped because there isn't any subscription record before.

The solution is to always add a phantom record in the subscription tracer if there isn't one while we are starting.

[YuJuncen]

This will only affect master because #16008 haven't been brought to any release version. I think once we are going to pick that PR, we can (and we should) also pick this.