pitr restore fail but report successfully

[seiya-annie]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. start pitr job
2. do full backup
3. prepare 5 * 350000 * 100k data
4. run update for 20 minutes
5. do flashback to after prepare before update
6. pitr restore data to new cluster
7. check data is consistent with upstream

2. What did you expect to see? (Required)

data in consistent

3. What did you see instead (Required)

MySQL [pitr]> admin check table user_data1;
ERROR 8223 (HY000): data inconsistency in table: user_data1, index: k_1, handle: 131674, index-values:"handle: 131674, values: [KindNull <nil> KindInt64 131674]" != record-values:""
MySQL [pitr]> admin check table user_data2;
ERROR 1105 (HY000): other error: default not found: key:7480000000000001FFDE5F728000000000FF0286070000000000FAF9EBD093D5FBFFF5, maybe read truncated/dropped table data?
MySQL [pitr]> admin check table user_data3;
ERROR 8223 (HY000): data inconsistency in table: user_data3, index: k_1, handle: 222917, index-values:"handle: 222917, values: [KindNull <nil> KindInt64 222917]" != record-values:""
MySQL [pitr]> admin check table user_data4;
ERROR 1105 (HY000): tikv aborts txn: Error(Txn(Error(Mvcc(Error(DefaultNotFound { key: [116, 128, 0, 0, 0, 0, 0, 1, 255, 220, 95, 114, 128, 0, 0, 0, 0, 255, 2, 144, 65, 0, 0, 0, 0, 0, 250, 249, 235, 208, 149, 177, 199, 255, 248] })))))
MySQL [pitr]> admin check table user_data5;
ERROR 8223 (HY000): data inconsistency in table: user_data5, index: k_1, handle: 128606, index-values:"handle: 128606, values: [KindNull <nil> KindInt64 128606]" != record-values:""
MySQL [pitr]>
### 4. What is your TiDB version? (Required)

[root@node1 ~]# tiup br:nightly -V
Starting component `br`: /root/.tiup/components/br/v6.5.0-alpha-nightly-20221213/br -V
Release Version: v6.5.0-alpha
Git Commit Hash: 827d8ff2d22ac4c93ae1b841b79d468211e1d393
Git Branch: heads/refs/tags/v6.5.0-alpha
Go Version: go1.19.3
UTC Build Time: 2022-12-13 14:25:00
Race Enabled: false

[root@node1 ~]# //home/pitr3/tidb-deploy/tikv-13560/bin/tikv-server -V
TiKV 
Release Version:   6.5.0-alpha
Edition:           Community
Git Commit Hash:   931cf7fd75c12900332a3a458f54ef7ef496c68b
Git Commit Branch: heads/refs/tags/v6.5.0-alpha
UTC Build Time:    2022-12-13 14:40:55
Rust Version:      rustc 1.67.0-nightly (96ddd32c4 2022-11-14)
Enable Features:   pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure
Profile:           dist_release

[seiya-annie]

data restore success:
Restore KV Files <---------------------------------------------------------------------------------> 100.00%[2022/12/14 14:34:45.392 +08:00] [INFO] [collector.go:73] ["restore log success summary"] [total-take=37m36.298644169s] [restore-from=438027052646400000] [restore-to=438027839078400000] [restore-from="2022-12-13 22:10:00 +0800"] [restore-to="2022-12-13 23:00:00 +0800"] [total-kv-count=24971899] [total-size=761.8GB] [average-speed=337.6MB/s]

[seiya-annie]

[2022/12/14 17:39:02.786 +08:00] [FATAL] [[lib.rs:495](http://lib.rs:495/)] ["called `Result::unwrap()` on an `Err` value: Other(\"[components/raft_log_engine/src/[engine.rs:622](http://engine.rs:622/)]: Full\")"] [backtrace="   0: tikv_util::set_panic_hook::{{closure}}\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/tikv_util/src/lib.rs:494:18\n](http://github.com/pingcap/tikv/components/tikv_util/src/lib.rs:494:18/n)   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/[boxed.rs:2032](http://boxed.rs:2032/):9\n      std::panicking::rust_panic_with_hook\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/[panicking.rs:692](http://panicking.rs:692/):13\n   2: std::panicking::begin_panic_handler::{{closure}}\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/[panicking.rs:579](http://panicking.rs:579/):13\n   3: std::sys_common::backtrace::__rust_end_short_backtrace\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/[backtrace.rs:137](http://backtrace.rs:137/):18\n   4: rust_begin_unwind\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/[panicking.rs:575](http://panicking.rs:575/):5\n   5: core::panicking::panic_fmt\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/[panicking.rs:65](http://panicking.rs:65/):14\n   6: core::result::unwrap_failed\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/[result.rs:1791](http://result.rs:1791/):5\n   7: core::result::Result<T,E>::unwrap\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/[result.rs:1113](http://result.rs:1113/):23\n      raftstore::store::async_io::write::WriteTaskBatch<EK,ER>::add_write_task\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/raftstore/src/store/async_io/write.rs:447:9\n](http://github.com/pingcap/tikv/components/raftstore/src/store/async_io/write.rs:447:9/n)      raftstore::store::async_io::write::Worker<EK,ER,N,T>::handle_write_task\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/raftstore/src/store/async_io/write.rs:697:9\n](http://github.com/pingcap/tikv/components/raftstore/src/store/async_io/write.rs:697:9/n)   8: raftstore::store::peer::Peer<EK,ER>::handle_raft_ready_append\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/raftstore/src/store/peer.rs:2787:21\n](http://github.com/pingcap/tikv/components/raftstore/src/store/peer.rs:2787:21/n)   9: raftstore::store::fsm::peer::PeerFsmDelegate<EK,ER,T>::collect_ready\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/raftstore/src/store/fsm/peer.rs:1979:19\n](http://github.com/pingcap/tikv/components/raftstore/src/store/fsm/peer.rs:1979:19/n)      <raftstore::store::fsm::store::RaftPoller<EK,ER,T> as batch_system::batch::PollHandler<raftstore::store::fsm::peer::PeerFsm<EK,ER>,raftstore::store::fsm::store::StoreFsm<EK>>>::handle_normal\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/raftstore/src/store/fsm/store.rs:945:13\n](http://github.com/pingcap/tikv/components/raftstore/src/store/fsm/store.rs:945:13/n)  10: batch_system::batch::Poller<N,C,Handler>::poll\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/batch-system/src/batch.rs:472:27\n](http://github.com/pingcap/tikv/components/batch-system/src/batch.rs:472:27/n)  11: batch_system::batch::BatchSystem<N,C>::start_poller::{{closure}}\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/batch-system/src/batch.rs:600:17\n](http://github.com/pingcap/tikv/components/batch-system/src/batch.rs:600:17/n)      <std::thread::Builder as tikv_util::sys::thread::StdThreadBuildWrapper>::spawn_wrapper::{{closure}}\n             at /home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/tikv_util/src/sys/thread.rs:415:23\n](http://github.com/pingcap/tikv/components/tikv_util/src/sys/thread.rs:415:23/n)      std::sys_common::backtrace::__rust_begin_short_backtrace\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/[backtrace.rs:121](http://backtrace.rs:121/):18\n  12: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/[mod.rs:551](http://mod.rs:551/):17\n      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/panic/[unwind_safe.rs:271](http://unwind_safe.rs:271/):9\n      std::panicking::try::do_call\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/[panicking.rs:483](http://panicking.rs:483/):40\n      std::panicking::try\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/[panicking.rs:447](http://panicking.rs:447/):19\n      std::panic::catch_unwind\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/[panic.rs:137](http://panic.rs:137/):14\n      std::thread::Builder::spawn_unchecked_::{{closure}}\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/[mod.rs:550](http://mod.rs:550/):30\n      core::ops::function::FnOnce::call_once{{vtable.shim}}\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/[function.rs:513](http://function.rs:513/):5\n  13: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/[boxed.rs:2000](http://boxed.rs:2000/):9\n      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/[boxed.rs:2000](http://boxed.rs:2000/):9\n      std::sys::unix::thread::Thread::new::thread_start\n             at /rust/toolchains/nightly-2022-11-15-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys/unix/[thread.rs:108](http://thread.rs:108/):17\n  14: start_thread\n  15: clone\n"] [location=/home/jenkins/agent/workspace/build-common/go/src/[github.com/pingcap/tikv/components/raftstore/src/store/async_io/write.rs:449](http://github.com/pingcap/tikv/components/raftstore/src/store/async_io/write.rs:449)] [thread_name=raftstore-9-0]

[seiya-annie]

error report restore successfully, in fact, it is restored fail.

[seiya-annie]

fix by #39943