br: advancer owner failed to update global checkpoint

[BornChanger]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Replace a TiKV instance by file copy from one EC2 to another EC2

2. What did you expect to see? (Required)

Log backup continue running

3. What did you see instead (Required)

Log backup stuck with error of

"Alert: RPO of log backup is too high - critical
 Description: The log data is not persisted to the storage for more than 30 minutes. This alert often indicates anomalies. You can check the TiKV logs to find the cause."

And TiDB log of the advancer owner has log of:

 [2024/01/15 16:55:52.168 +00:00] [WARN] [advancer.go:315] ["failed to update global checkpoint: stale"] [old=447044033015185426] [new=0]
[2024/01/15 16:55:52.172 +00:00] [WARN] [owner_daemon.go:63] ["failed on tick"] [error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 10.29.45.253:20160: i/o timeout\""] 

4. What is your TiDB version? (Required)

[BornChanger]

/component br

[BornChanger]

/severity major

[BornChanger]

This file copy way to replace tikv reuses the old store ID. but advancer owner has cache the old store id with old ip. so the connection is unavaiable.
To workaround, we can 1. restart tidb(advance owner ), 2. rebuild the pitr task.

[seiya-annie]

/found customer