coredump in sortkey_count

[xihong08]

Bug Report

In our production environment has appeared some times coredump. it Seems happened in sortkey_count process. from replica-server log, coredump occurs after replica's close. so i try reproduce in our test environment，because i cannot make replica down scenario, i use kill-partition to make it. and i reproduce this coredump
My steps

1. use sortkey count in one client such as in pegaus_shell run 'count 2'
2. at the same time use remote-command to kill-partition of replica in the first step data
   coredump stack as follows

#0  0x00000000007c08cb in pegasus::server::capacity_unit_calculator::add_read_cu (this=0x0, read_data_size=1)
    at /incubator-pegasus/src/server/capacity_unit_calculator.cpp:84
#1  0x00000000007c0d18 in pegasus::server::capacity_unit_calculator::add_sortkey_count_cu (this=0x0, status=0)
    at /incubator-pegasus/src/server/capacity_unit_calculator.cpp:165
#2  0x000000000087d5d7 in pegasus::server::pegasus_server_impl::on_sortkey_count (this=0x3264c00, hash_key=..., reply=...)
    at /incubator-pegasus/src/server/pegasus_server_impl.cpp:1181
#3  0x00000000008135fa in dsn::apps::rrdb_service::on_sortkey_count (svc=0x3264c00, args=..., reply=...)
    at /incubator-pegasus/src/include/rrdb/rrdb.server.h:300
#4  0x0000000000814aed in dsn::replication::storage_serverlet<dsn::apps::rrdb_service>::register_async_rpc_handler<dsn::blob, dsn::apps::count_response>(dsn::task_code, char const*, void (*)(dsn::apps::rrdb_service*, dsn::blob const&, dsn::rpc_replier<dsn::apps::count_response>&))::{lambda(dsn::apps::rrdb_service*, dsn::message_ex*)#1}::operator()(dsn::apps::rrdb_service*, dsn::message_ex*) const (this=0x281b6f8, p=0x3264c00,
    r=0x588b9c9a0) at /incubator-pegasus/DSN_ROOT/include/dsn/dist/replication/storage_serverlet.h:29
#5  0x0000000000824390 in std::__invoke_impl<void, dsn::replication::storage_serverlet<dsn::apps::rrdb_service>::register_async_rpc_handler<dsn::blob, dsn::apps::count_response>(dsn::task_code, char const*, void (*)(dsn::apps::rrdb_service*, dsn::blob const&, dsn::rpc_replier<dsn::apps::count_response>&))::{lambda(dsn::apps::rrdb_service*, dsn::message_ex*)#1}&, dsn::apps::rrdb_service*, dsn::message_ex*>(std::__invoke_other, dsn::replication::storage_serverlet<dsn::apps::rrdb_service>::register_async_rpc_handler<dsn::blob, dsn::apps::count_response>(dsn::task_code, char const*, void (*)(dsn::apps::rrdb_service*, dsn::blob const&, dsn::rpc_replier<dsn::apps::count_response>&))::{lambda(dsn::apps::rrdb_service*, dsn::message_ex*)#1}&, dsn::apps::rrdb_service*&&, dsn::message_ex*&&) (__f=...) at /opt/gcc10/include/c++/10.2.0/bits/invoke.h:60
#6  0x000000000081f200 in std::__invoke_r<void, dsn::replication::storage_serverlet<dsn::apps::rrdb_service>::register_async_rpc_handler<dsn::blob, dsn::apps::count_response>(dsn::task_code, char const*, void (*)(dsn::apps::rrdb_service*, dsn::blob const&, dsn::rpc_replier<dsn::apps::count_response>&))::{lambda(dsn::apps::rrdb_service*, dsn::message_ex*)#1}&, dsn::apps::rrdb_service*, dsn::message_ex*>(dsn::replication::storage_serverlet<dsn::apps::rrdb_service>::register_async_rpc_handler<dsn::blob, dsn::apps::count_response>(dsn::task_code, char const*, void (*)(dsn::apps::rrdb_service*, dsn::blob const&, dsn::rpc_replier<dsn::apps::count_response>&))::{lambda(dsn::apps::rrdb_service*, dsn::message_ex*)#1}&, (std::__is_invocable&&)...) (__fn=...) at /opt/gcc10/include/c++/10.2.0/bits/invoke.h:153
#7  0x000000000081a219 in std::_Function_handler<void (dsn::apps::rrdb_service*, dsn::message_ex*), dsn::replication::storage_serverlet<dsn::apps::rrdb_service>::register_async_rpc_handler<dsn::blob, dsn::apps::count_response>(dsn::task_code, char const*, void (*)(dsn::apps::rrdb_service*, dsn::blob const&, dsn::rpc_replier<dsn::apps::count_response>&))::{lambda(dsn::apps::rrdb_service*, dsn::message_ex*)#1}>::_M_invoke(std::_Any_data const&, dsn::apps::rrdb_service*&&, dsn::message_ex*&&) (__functor=..., __args#0=@0x7ff78d7f7bf0: 0x3264c00

[acelyc111]

Good find! It's a bug indeed, could you please help to fix it?