Not expecting massive fetch data request to happen during I/O load

[yamingk]

Still no flip, no restarts on any replica, seeing a lot of fetch data requests happending, which seems massive push data rquests are not received on follower side, this is not expected and needs to be investigated.

This is seen with real drives, not file.

root@test-vol-test-kernel-hdd-001-aj2no:/output# grep -ri "We haven't received data" log
[05/16/24 16:06:41-07:00] [D] [test_raft_repl_dev_replica_2] [2483] [raft_repl_dev.cpp:419] [rd=rdev1] We haven't received data for 1 out 3 in reqs batch, will fetch and wait for 10000 ms, in_resync_mode()=false
[05/16/24 16:06:48-07:00] [D] [test_raft_repl_dev_replica_2] [2483] [raft_repl_dev.cpp:419] [rd=rdev1] We haven't received data for 3 out 12 in reqs batch, will fetch and wait for 10000 ms, in_resync_mode()=false
[05/16/24 16:06:56-07:00] [D] [test_raft_repl_dev_replica_1] [2479] [raft_repl_dev.cpp:419] [rd=rdev1] We haven't received data for 1 out 10 in reqs batch, will fetch and wait for 10000 ms, in_resync_mode()=false
[05/16/24 16:06:58-07:00] [D] [test_raft_repl_dev_replica_1] [2479] [raft_repl_dev.cpp:419] [rd=rdev1] We haven't received data for 61 out 64 in reqs batch, will fetch and wait for 10000 ms, in_resync_mode()=false
[05/16/24 16:07:01-07:00] [D] [test_raft_repl_dev_replica_1] [2479] [raft_repl_dev.cpp:419] [rd=rdev1] We haven't received data for 22 out 64 in reqs batch, will fetch and wait for 10000 ms, in_resync_mode()=false
[05/16/24 16:07:04-07:00] [D] [test_raft_repl_dev_replica_1] [2526] [raft_repl_dev.cpp:419] [rd=rdev1] We haven't received data for 57 out 64 in reqs batch, will fetch and wait for 10000 ms, in_resync_mode()=false
[05/16/24 16:07:07-07:00] [D] [test_raft_repl_dev_replica_1] [2526] [raft_repl_dev.cpp:419] [rd=rdev1] We haven't received data for 60 out 64 in reqs batch, will fetch and wait for 10000 ms, in_resync_mode()=false
[05/16/24 16:07:10-07:00] [D] [test_raft_repl_dev_replica_1] [2526] [raft_repl_dev.cpp:419] [rd=rdev1] We haven't received data for 62 out 64 in reqs batch, will fetch and wait for 10000 ms, in_resync_mode()=false
[05/16/24 16:07:11-07:00] [D] [test_raft_repl_dev_replica_2] [2483] [raft_repl_dev.cpp:419] [rd=rdev1] We haven't received data for 1 out 5 in reqs batch, will fetch and wait for 10000 ms, in_resync_mode()=false
[05/16/24 16:07:13-07:00] [D] [test_raft_repl_dev_replica_1] [2479] [raft_repl_dev.cpp:419] [rd=rdev1] We haven't received data for 62 out 64 in reqs batch, will fetch and wait for 10000 ms, in_resync_mode()=false
[05/16/24 16:07:15-07:00] [D] [test_raft_repl_dev_replica_1] [2479] [raft_repl_dev.cpp:419] [rd=rdev1] We haven't received data for 58 out 64 in reqs batch, will fetch and wait for 10000 ms, in_resync_mode()=false
[05/16/24 16:07:20-07:00] [D] [test_raft_repl_dev_replica_1] [2479] [raft_repl_dev.cpp:419] [rd=rdev1] We haven't received data for 64 out 64 in reqs batch, will fetch and wait for 10000 ms, in_resync_mode()=false
[05/16/24 16:07:20-07:00] [D] [test_raft_repl_dev_replica_2] [2483] [raft_repl_dev.cpp:419] [rd=rdev1] We haven't received data for 1 out 2 in reqs batch, will fetch and wait for 10000 ms, in_resync_mode()=false
[05/16/24 16:07:26-07:00] [D] [test_raft_repl_dev_replica_2] [2483] [raft_repl_dev.cpp:419] [rd=rdev1] We haven't received data for 1 out 1 in reqs batch, will fetch and wait for 10000 ms, in_resync_mode()=false