pitr restore data fail for "failed: raft entry is too large, region 170, entry size 8616655"

[seiya-annie]

Bug Report

What version of TiKV are you using?

2.28 nightly

What operating system and CPU are you using?

Steps to reproduce

start log backup
do full backup every sometime
run paypay workload
do pitr restore with latest full backup repeatedly (clean target TiDB eachtime)

What did you expect?

pitr backup and restore works normal

What did happened?

failed: raft entry is too large, region 170, entry size 8616655; execute over region id:170 start_key:\"t\\200\\
000\\000\\000\\000\\000\\000\\377e_i\\200\\000\\000\\000\\000\\377\\000\\000\\001\\003\\200\\000\\000\\000\\377\\000
\\031\\014\\336\\000\\000\\000\\000\\373\" end_key:\"t\\200\\000\\000\\000\\000\\000\\000\\377e_i\\200\\000\\000\\00
0\\000\\377\\000\\000\\001\\003\\200\\000\\000\\000\\377\\000\\035O\\237\\000\\000\\000\\000\\373\" region_epoch:<co
nf_ver:1 version:158 > peers:<id:171 store_id:1 >  failed: raft entry is too large, region 170, entry size 8616655;
execute over region id:170 start_key:\"t\\200\\000\\000\\000\\000\\000\\000\\377e_i\\200\\000\\000\\000\\000\\377\\0
00\\000\\001\\003\\200\\000\\000\\000\\377\\000\\031\\014\\336\\000\\000\\000\\000\\373\" end_key:\"t\\200\\000\\000
\\000\\000\\000\\000\\377e_i\\200\\000\\000\\000\\000\\377\\000\\000\\001\\003\\200\\000\\000\\000\\377\\000\\035O\\
237\\000\\000\\000\\000\\373\" region_epoch:<conf_ver:1 version:158 > peers:<id:171 store_id:1 >  failed: raft entry
 is too large, region 170, entry size 8616655; execute over region id:170 start_key:\"t\\200\\000\\000\\000\\000\\00
0\\000\\377e_i\\200\\000\\000\\000\\000\\377\\000\\000\\001\\003\\200\\000\\000\\000\\377\\000\\031\\014\\336\\000\\
000\\000\\000\\373\" end_key:\"t\\200\\000\\000\\000\\000\\000\\000\\377e_i\\200\\000\\000\\000\\000\\377\\000\\000\
\001\\003\\200\\000\\000\\000\\377\\000\\035O\\237\\000\\000\\000\\000\\373\" region_epoch:<conf_ver:1 version:158 >
 peers:<id:171 store_id:1 >  failed: raft entry is too large, region 170, entry size 8616655; execute over region id
:170 start_key:\"t\\200\\000\\000\\000\\000\\000\\000\\377e_i\\200\\000\\000\\000\\000\\377\\000\\000\\001\\003\\200
\\000\\000\\000\\377\\000\\031\\014\\336\\000\\000\\000\\000\\373\" end_key:\"t\\200\\000\\000\\000\\000\\000\\000\\
377e_i\\200\\000\\000\\000\\000\\377\\000\\000\\001\\003\\200\\000\\000\\000\\377\\000\\035O\\237\\000\\000\\000\\00
0\\373\" region_epoch:<conf_ver:1 version:158 > peers:<id:171 store_id:1 >  failed: raft entry is too large, region
170, entry size 8616655\n", "error": "command terminated with exit code 1"} 

[seiya-annie]

tikv.log.gz
br.log.2023-03-01T01.44.39Z.gz

[YuJuncen]

I think this is because we replaced the RaftStoreRouter by the Engine trait at #14294, where we have only calculated the size of the write itself, hence won't affect release-6.5.

[YuJuncen]

This may happen in v6.5.1 in some scenarios. Perhaps due to we haven't computed the wire header for EACH REQUEST.

I think we should batch upto 1/2 of the raft_max_size.