Leader crashed at BR (reading snapshot)

[Besroy]

Seems that leader is send snapshot to an old member (out_member) and a new member (in_member) and crashed at reading blobs, not sure if it is resolved by #312 or it is a new issue. Need check.

TestEnv:
cluster: 38
namespace: sdsnuobj03-prod
pod: sm-long-running1-5
HomeObjVer = homeobject/2.5.3@oss/main

Details:
more details in Issue61

1. SH starts write 100,000 blobs in one PG, blob_size=512K, replica_count=5
2. sm-long-running3-5 is lagged during write, and leader sm-long-running1-5 is sending snapshot to it
3. SH triggers replace member after write completed, out_member=sm-long-running3-5, in_member=sm-long-running4-5
4. leader sm-long-running1-5 crashed while reading snapshot

Some coredump stack:
#0 0x000055aabecc547d in std::__shared_ptr<sisl::MetricsGroupImpl, (__gnu_cxx::_Lock_policy)2>::get (this=0xc0) at /usr/include/c++/11/bits/shared_ptr_base.h:1296
#1 std::__shared_ptr_access<sisl::MetricsGroupImpl, (__gnu_cxx::_Lock_policy)2, false, false>::_M_get (this=0xc0) at /usr/include/c++/11/bits/shared_ptr_base.h:993
#2 std::__shared_ptr_access<sisl::MetricsGroupImpl, (__gnu_cxx::_Lock_policy)2, false, false>::operator-> (this=0xc0) at /usr/include/c++/11/bits/shared_ptr_base.h:987
#3 operator()<folly::Expected<homeobject::HSHomeObject::PGBlobIterator::blob_read_result, homeobject::BlobError> > (result=..., __closure=0x7f6f15edc770)
at /home/ubuntu/HomeObject/src/lib/homestore_backend/pg_blob_iterator.cpp:322
#4 operator() (__closure=, __closure=) at /home/ubuntu/.conan2/p/b/folly8afde91948858/p/include/folly/futures/Future-inl.h:96
#5 folly::futures::detail::InvokeResultWrapperBase<folly::SemiFuture<folly::Expected<homeobject::HSHomeObject::PGBlobIterator::blob_read_result, homeobject::BlobError> > >::wrapResult<folly::futures::detail::wrapInvoke<folly::Expected<homeobject::HSHomeObject::PGBlobIterator::blob_read_result, homeobject::BlobError>, homeobject::HSHomeObject::PGBlobIterator::prefetch_blobs_snapshot_data()::<lambda(auto:283&&)> >(folly::Try<folly::Expected<homeobject::HSHomeObject::PGBlobIterator::blob_read_result, homeobject::BlobError> >&&, homeobject::HSHomeObject::PGBlobIterator::prefetch_blobs_snapshot_data()::<lambda(auto:283&&)>&&)::<lambda()> > (fn=...) at /home/ubuntu/.conan2/p/b/folly8afde91948858/p/include/folly/futures/Future-inl.h:68
#6 folly::futures::detail::wrapInvoke<folly::Expected<homeobject::HSHomeObject::PGBlobIterator::blob_read_result, homeobject::BlobError>, homeobject::HSHomeObject::PGBlobIterator::prefetch_blobs_snapshot_data()::<lambda(auto:283&&)> > (f=..., t=...) at /home/ubuntu/.conan2/p/b/folly8afde91948858/p/include/folly/futures/Future-inl.h:103
#7 operator() (t=..., __closure=0x7f6f15edc770) at /home/ubuntu/.conan2/p/b/folly8afde91948858/p/include/folly/futures/Future-inl.h:998
#8 folly::futures::detail::CoreCallbackState<folly::Expected<homeobject::HSHomeObject::PGBlobIterator::blob_read_result, homeobject::BlobError>, folly::Future<folly::Expected<homeobject::HSHomeObject::PGBlobIterator::blob_read_result, homeobject::BlobError> >::thenValue<homeobject::HSHomeObject::PGBlobIterator::prefetch_blobs_snapshot_data()::<lambda(auto:283&&)> >(homeobject::HSHomeObject::PGBlobIterator::prefetch_blobs_snapshot_data()::<lambda(auto:283&&)>&&) &&::<lambda(folly::Executor::KeepAlivefolly::Executor&&, folly::Try<folly::Expected<homeobject::HSHomeObject::PGBlobIterator::blob_read_result, homeobject::BlobError> >&&)> >::invoke<folly::Executor::KeepAlivefolly::Executor, folly::Try<folly::Expected<homeobject::HSHomeObject::PGBlobIterator::blob_read_result, homeobject::BlobError> > > (
this=0x7f6f15edc770) at /home/ubuntu/.conan2/p/b/folly8afde91948858/p/include/folly/futures/Future-inl.h:139
#9 operator() (__closure=) at /home/ubuntu/.conan2/p/b/folly8afde91948858/p/include/folly/futures/Future-inl.h:144
#10 folly::makeTryWithNoUnwrap<folly::futures::detail::CoreCallbackState<folly::Expected<homeobject::HSHomeObject::PGBlobIterator::blob_read_result, homeobject::BlobError>, folly::Future<folly::Expected<homeobject::HSHomeObject::PGBlobIterator::blob_read_result, homeobject::BlobError> >::thenValue<homeobject::HSHomeObject::PGBlobIterator::prefetch_blobs_snapshot_data()::<lambda(auto:283&&)> >(homeobject::HSHomeObject::PGBlobIterator::prefetch_blobs_snapshot_data()::<lambda(auto:283&&)>&&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Expected<homeobject::HSHomeObject::PGBlobIterator::blob_read_result, homeobject::BlobError> >&&)> >::tryInvoke<folly::Executor::KeepAlivefolly::Executor, folly::Try<folly::Expected<homeobject::HSHomeObject::PGBlobIterator::blob_read_result, homeobject::BlobError> > >(folly::Executor::KeepAlive<>&&, folly::Try<folly::Expected<homeobject::HSHomeObject::PGBlobIterator::blob_read_result, homeobject::BlobError> >&&)::<lambda()> > (f=..., f=...)
at /home/ubuntu/.conan2/p/b/folly8afde91948858/p/include/folly/Try-inl.h:254
#11 folly::makeTryWith<folly::futures::detail::CoreCallbackState<folly::Expected<homeobject::HSHomeObject::PGBlobIterator::blob_read_result, homeobject::BlobError>, folly::Future<folly::Expected<homeobject::HSHomeObject::PGBlobIterator::blob_read_result, homeobject::BlobError> >::thenValue<homeobject::HSHomeObject::PGBlobIterator::prefetch_blobs_snapshot_data()::<lambda(auto:283&&)> >(homeobject::HSHomeObject::PGBlobIterator::prefetch_blobs_snapshot_data()::<lambda(auto:283&&)>&&) &&::<lambda(folly::Executor::KeepAlive<>&&, folly::Try<folly::Expected<homeobject::HSHomeObject::PGBlobIterator::blob_read_result, homeobject::BlobError> >&&)> >::tryInvoke<folly::Executor::KeepAlivefolly::Executor, folly::Try<folly::Expected<homeobject::HSHomeObject::PGBlobIterator::blob_read_result, homeobject::BlobError> > >(folly::Executor::KeepAlive<>&&, folly::Try<folly::Expected<homeobject::HSHomeObject::PGBlobIterator::blob_read_result, homeobject::BlobError> >&&)::<lambda()> > (f=...)
at /home/ubuntu/.conan2/p/b/folly8afde91948858/p/include/folly/Try-inl.h:276