Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DocDB] Tserver process rpc_tp_TabletSe crashes in rocksdb::Block::NewIterator() #13109

Open
sonalsagarwal opened this issue Jun 30, 2022 · 2 comments
Open

[DocDB] Tserver process rpc_tp_TabletSe crashes in rocksdb::Block::NewIterator() #13109

sonalsagarwal opened this issue Jun 30, 2022 · 2 comments
Assignees
@ttyusupov
Labels
area/docdb YugabyteDB core features kind/bug This issue is a bug priority/high High Priority

Comments

@sonalsagarwal
Copy link

sonalsagarwal commented Jun 30, 2022 •
edited by yugabyte-ci

Jira Link: DB-2817

Description

YB version: 2.15.1.0-b91
Test run : TPCC 2700 WH load phase
3 node RF3 single zone

The tserver process was restarted when the rcp_tp_TabletSe process crashed.

From tserver err log :

*** Aborted at 1656537840 (unix time) try "date -d @1656537840" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0x0) received by PID 19961 (TID 0x7f91a6d40700) from PID 0; stack trace: ***
    @          0x3364449 rocksdb::Block::NewIterator()
    @          0x3387a79 rocksdb::BlockBasedTable::NewDataBlockIterator()
    @          0x33af087 rocksdb::(anonymous namespace)::TwoLevelIterator::InitDataBlock()
    @          0x33ae9d4 rocksdb::(anonymous namespace)::TwoLevelIterator::Seek()
    @          0x339bbf3 rocksdb::MergingIterator::Seek()
    @          0x32ffc5c rocksdb::DBIter::Seek()
    @          0x2fb33c2 yb::docdb::IntentsWriter::Apply()
    @          0x3351d37 rocksdb::WriteBatch::Iterate()
    @          0x33565b1 rocksdb::WriteBatchInternal::InsertInto()
    @          0x32ca1f2 rocksdb::DBImpl::WriteImpl()
    @          0x32bb5c1 rocksdb::DBImpl::Write()
    @          0x356c425 yb::tablet::Tablet::WriteToRocksDB()
    @          0x3568b18 yb::tablet::Tablet::ApplyIntents()
    @          0x3569722 yb::tablet::Tablet::ApplyIntents()
    @          0x360c612 yb::tablet::TransactionParticipant::Impl::ProcessReplicated()
    @          0x354f9c3 yb::tablet::UpdateTxnOperation::DoReplicated()
    @          0x35450af yb::tablet::Operation::Replicated()
    @          0x35463d5 yb::tablet::OperationDriver::ReplicationFinished()
    @          0x2e29175 yb::consensus::ConsensusRound::NotifyReplicationFinished()
    @          0x2e75a45 yb::consensus::ReplicaState::ApplyPendingOperationsUnlocked()
    @          0x2e74d6d yb::consensus::ReplicaState::AdvanceCommittedOpIdUnlocked()
    @          0x2e5ef8f yb::consensus::RaftConsensus::UpdateReplica()
    @          0x2e44c4f yb::consensus::RaftConsensus::Update()
    @          0x374c045 yb::tserver::ConsensusServiceImpl::UpdateConsensus()
    @          0x2eabdf2 std::__1::__function::__func<>::operator()()
    @          0x2eaca66 yb::consensus::ConsensusServiceIf::Handle()
    @          0x34c0c9f yb::rpc::ServicePoolImpl::Handle()
    @          0x34126e5 yb::rpc::InboundCall::InboundCallTask::Run()
    @          0x34cfbbd yb::rpc::(anonymous namespace)::Worker::Execute()
    @          0x3a96514 yb::Thread::SuperviseThread()
    @     0x7f94af9c1694 start_thread
    @     0x7f94afec341d __clone

Stack from Core dump :

warning: File "/home/yugabyte/yb-software/yugabyte-2.15.1.0-b91-centos-x86_64/linuxbrew/lib/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".
To enable execution of this file add
	add-auto-load-safe-path /home/yugabyte/yb-software/yugabyte-2.15.1.0-b91-centos-x86_64/linuxbrew/lib/libthread_db.so.1
line to your configuration file "/root/.gdbinit".
To completely disable this security protection add
	set auto-load safe-path /
line to your configuration file "/root/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
	info "(gdb)Auto-loading safe path"

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

warning: File "/home/yugabyte/yb-software/yugabyte-2.15.1.0-b91-centos-x86_64/linuxbrew/lib/libthread_db.so.1" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".

warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by `/home/yugabyte/tserver/bin/yb-tserver --flagfile /home/yugabyte/tserver/conf/se'.
Program terminated with signal 11, Segmentation fault.
Dwarf Error: Cannot find DIE at 0xd5fb referenced from DIE at 0x1f3e92 [in module /home/yugabyte/yb-software/yugabyte-2.15.1.0-b91-centos-x86_64/bin/yb-tserver]
#-1 DecodeFixed32 (ptr=<optimized out>) at ../../src/yb/rocksdb/util/coding.h:110
110	../../src/yb/rocksdb/util/coding.h: No such file or directory.
(gdb) bt
#-1 DecodeFixed32 (ptr=<optimized out>) at ../../src/yb/rocksdb/util/coding.h:110
#0  DecodeFixed32 (ptr=<optimized out>) at ../../src/yb/rocksdb/util/coding.h:110
#1  NumRestarts (this=0x44cd08390) at ../../src/yb/rocksdb/table/block.cc:512
#2  rocksdb::Block::NewIterator(rocksdb::Comparator const*, rocksdb::KeyValueEncodingFormat, rocksdb::BlockIter*, bool) const (this=0x44cd08390, cmp=0xbf6fa98, key_value_encoding_format=1, iter=0x0, total_order_seek=true)
    at ../../src/yb/rocksdb/table/block.cc:543
#3  0x0000000003387a79 in rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::ReadOptions const&, yb::Slice const&, rocksdb::BlockType, rocksdb::BlockIter*) (this=0x2999780d0, ro=..., index_value=..., block_type=kData, input_iter=0x0)
    at ../../src/yb/rocksdb/table/block_based_table_reader.cc:1260
#4  0x00000000033af087 in rocksdb::(anonymous namespace)::TwoLevelIterator::InitDataBlock() (this=0x3047dee8) at ../../src/yb/rocksdb/table/two_level_iterator.cc:207
#5  0x00000000033ae9d4 in rocksdb::(anonymous namespace)::TwoLevelIterator::Seek(yb::Slice const&) (this=0x3047dee8, target=...) at ../../src/yb/rocksdb/table/two_level_iterator.cc:117
#6  0x000000000339bbf3 in Seek (this=0x3047dd20, k=...) at ../../src/yb/rocksdb/table/iterator_wrapper.h:126
#7  rocksdb::MergingIterator::Seek(yb::Slice const&) (this=0x3047dca0, target=...) at ../../src/yb/rocksdb/table/merger.cc:154
#8  0x00000000032ffc5c in rocksdb::DBIter::Seek(yb::Slice const&) (this=0x3047db38, target=...) at ../../src/yb/rocksdb/db/db_iter.cc:759
#9  0x0000000002fb33c2 in yb::docdb::IntentsWriter::Apply(rocksdb::DirectWriteHandler*) (this=0x7f91a6d36fc0, handler=0x7f91a6d36518) at ../../src/yb/docdb/rocksdb_writer.cc:386
#10 0x0000000003351d37 in DirectInsert (handler=0x7f91a6d36a28, writer=0x7f91a6d36fc0) at ../../src/yb/rocksdb/db/write_batch.cc:983
#11 rocksdb::WriteBatch::Iterate(rocksdb::WriteBatch::Handler*) const (this=<optimized out>, handler=0x7f91a6d36a28) at ../../src/yb/rocksdb/db/write_batch.cc:341
#12 0x00000000033565b1 in rocksdb::WriteBatchInternal::InsertInto(rocksdb::autovector<rocksdb::WriteThread::Writer*, 8ul> const&, unsigned long, rocksdb::ColumnFamilyMemTables*, rocksdb::FlushScheduler*, bool, unsigned long, rocksdb::DB*, yb::EnumBitSet<rocksdb::InsertFlag>) (writers=..., sequence=<optimized out>, memtables=<optimized out>, flush_scheduler=<optimized out>, ignore_missing_column_families=<optimized out>, log_number=0, db=0x6ee1c000, insert_flags=...)
    at ../../src/yb/rocksdb/db/write_batch.cc:927
#13 0x00000000032ca1f2 in rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*) (this=<optimized out>, write_options=..., my_batch=<optimized out>, callback=<optimized out>)
    at ../../src/yb/rocksdb/db/db_impl.cc:5291
#14 0x00000000032bb5c1 in rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*) (this=0xbf6fa98, write_options=..., my_batch=0x0) at ../../src/yb/rocksdb/db/db_impl.cc:4924
#15 0x000000000356c425 in yb::tablet::Tablet::WriteToRocksDB(rocksdb::UserFrontiers const*, rocksdb::WriteBatch*, yb::docdb::StorageDbType) (this=0x3469d0020, frontiers=<optimized out>, write_batch=0x7f91a6d37088, storage_db_type=0)
    at ../../src/yb/tablet/tablet.cc:1335
#16 0x0000000003568b18 in yb::tablet::Tablet::ApplyIntents(yb::tablet::TransactionApplyData const&) (this=0x3469d0020, data=...) at ../../src/yb/tablet/tablet.cc:1776
#17 0x0000000003569722 in non-virtual thunk to yb::tablet::Tablet::ApplyIntents(yb::tablet::TransactionApplyData const&) () at ../../src/yb/util/status.cc:632
#18 0x000000000360c612 in ProcessApply (this=0x25bb0700, data=...) at ../../src/yb/tablet/transaction_participant.cc:561
#19 ReplicatedApplying (this=<optimized out>, id=..., data=...) at ../../src/yb/tablet/transaction_participant.cc:1420
#20 yb::tablet::TransactionParticipant::Impl::ProcessReplicated(yb::tablet::TransactionParticipant::ReplicatedData const&) (this=<optimized out>, data=...) at ../../src/yb/tablet/transaction_participant.cc:492
#21 0x000000000354f9c3 in ProcessReplicated (this=0x2ee058ba0, data=...) at ../../src/yb/tablet/transaction_participant.cc:1695
#22 yb::tablet::UpdateTxnOperation::DoReplicated(long, yb::Status*) (this=0x931c8c0, leader_term=<optimized out>, complete_status=<optimized out>) at ../../src/yb/tablet/operations/update_txn_operation.cc:65
#23 0x00000000035450af in yb::tablet::Operation::Replicated(long, yb::StronglyTypedBool<yb::tablet::WasPending_Tag>) (this=0x931c8c0, leader_term=1, was_pending=...) at ../../src/yb/tablet/operations/operation.cc:70
#24 0x00000000035463d5 in ApplyTask (this=0x241f4c480, leader_term=-1, applied_op_ids=0x7f91a6d377c0) at ../../src/yb/tablet/operations/operation_driver.cc:395
#25 yb::tablet::OperationDriver::ReplicationFinished(yb::Status const&, long, std::__1::vector<yb::OpId, std::__1::allocator<yb::OpId> >*) (this=<optimized out>, status=..., leader_term=1, applied_op_ids=0x7f91a6d377c0)
    at ../../src/yb/tablet/operations/operation_driver.cc:353
#26 0x0000000002e29175 in yb::consensus::ConsensusRound::NotifyReplicationFinished(yb::Status const&, long, std::__1::vector<yb::OpId, std::__1::allocator<yb::OpId> >*) (this=0xc0010780, status=..., leader_term=-1, applied_op_ids=0x7f91a6d377c0)
    at ../../src/yb/consensus/consensus_round.cc:47
#27 0x0000000002e75a45 in NotifyReplicationFinishedUnlocked (this=0x171ae700, round=..., status=..., leader_term=-1, applied_op_ids=0x7f91a6d377c0) at ../../src/yb/consensus/replica_state.cc:1378
#28 yb::consensus::ReplicaState::ApplyPendingOperationsUnlocked(yb::OpId const&, yb::StronglyTypedBool<yb::consensus::CouldStop_Tag>) (this=0x171ae700, committed_op_id=..., could_stop=...) at ../../src/yb/consensus/replica_state.cc:956
#29 0x0000000002e74d6d in yb::consensus::ReplicaState::AdvanceCommittedOpIdUnlocked(yb::OpId const&, yb::StronglyTypedBool<yb::consensus::CouldStop_Tag>) (this=0x171ae700, committed_op_id=..., could_stop=...)
    at ../../src/yb/consensus/replica_state.cc:869
#30 0x0000000002e5ef8f in EarlyCommitUnlocked (this=0x1084cfb98, request=..., deduped_req=...) at ../../src/yb/consensus/raft_consensus.cc:2033
#31 yb::consensus::RaftConsensus::UpdateReplica(yb::consensus::ConsensusRequestPB*, yb::consensus::ConsensusResponsePB*) (this=<optimized out>, request=0x377113e20, response=0x377113ea0) at ../../src/yb/consensus/raft_consensus.cc:1967
#32 0x0000000002e44c4f in yb::consensus::RaftConsensus::Update(yb::consensus::ConsensusRequestPB*, yb::consensus::ConsensusResponsePB*, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > >) (this=0x1084cfb98, request=0x377113e20, response=0x377113ea0, deadline=...) at ../../src/yb/consensus/raft_consensus.cc:1530
#33 0x000000000374c045 in yb::tserver::ConsensusServiceImpl::UpdateConsensus(yb::consensus::ConsensusRequestPB const*, yb::consensus::ConsensusResponsePB*, yb::rpc::RpcContext) (this=<optimized out>, req=0x377113e20, resp=0x377113ea0, context=...)
    at ../../src/yb/tserver/tablet_service.cc:1867
#34 0x0000000002eabdf2 in operator() (this=<optimized out>, req=0x377113e20, resp=0x377113ea0, rpc_context=...) at src/yb/consensus/consensus.service.cc:288
#35 HandleCall<yb::rpc::RpcCallPBParamsImpl<yb::consensus::ConsensusRequestPB, yb::consensus::ConsensusResponsePB>, (lambda at src/yb/consensus/consensus.service.cc:287:28)> (call=..., f=...) at ../../src/yb/rpc/local_call.h:124
#36 operator() (this=<optimized out>, call=...) at src/yb/consensus/consensus.service.cc:286
#37 __invoke<(lambda at src/yb/consensus/consensus.service.cc:284:16) &, std::shared_ptr<yb::rpc::InboundCall> > (__f=..., __args=<optimized out>)
    at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220531054925-62ed7bc00f-almalinux8-x86_64-clang12-linuxbrew-full-lto/installed/uninstrumented/libcxx/include/c++/v1/type_traits:3694
#38 __call<(lambda at src/yb/consensus/consensus.service.cc:284:16) &, std::shared_ptr<yb::rpc::InboundCall> > (__args=<optimized out>, __args=<optimized out>)
    at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220531054925-62ed7bc00f-almalinux8-x86_64-clang12-linuxbrew-full-lto/installed/uninstrumented/libcxx/include/c++/v1/__functional_base:348
#39 operator() (this=<optimized out>, __arg=<optimized out>) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220531054925-62ed7bc00f-almalinux8-x86_64-clang12-linuxbrew-full-lto/installed/uninstrumented/libcxx/include/c++/v1/functional:1558
#40 std::__1::__function::__func<yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0, std::__1::allocator<yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0>, void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>&&) (this=<optimized out>, __arg=<optimized out>)
    at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220531054925-62ed7bc00f-almalinux8-x86_64-clang12-linuxbrew-full-lto/installed/uninstrumented/libcxx/include/c++/v1/functional:1732
#41 0x0000000002eaca66 in operator() (this=<optimized out>, __args=<unknown type in /home/yugabyte/yb-software/yugabyte-2.15.1.0-b91-centos-x86_64/bin/yb-tserver, CU 0x15649dc, DIE 0x15b547f>)
    at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220531054925-62ed7bc00f-almalinux8-x86_64-clang12-linuxbrew-full-lto/installed/uninstrumented/libcxx/include/c++/v1/functional:1885
#42 operator() (this=<optimized out>, __arg=...) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220531054925-62ed7bc00f-almalinux8-x86_64-clang12-linuxbrew-full-lto/installed/uninstrumented/libcxx/include/c++/v1/functional:2560
#43 yb::consensus::ConsensusServiceIf::Handle(std::__1::shared_ptr<yb::rpc::InboundCall>) (this=<optimized out>, call=...) at src/yb/consensus/consensus.service.cc:271
#44 0x00000000034c0c9f in yb::rpc::ServicePoolImpl::Handle(std::__1::shared_ptr<yb::rpc::InboundCall>) (this=0x6bdd680, incoming=...) at ../../src/yb/rpc/service_pool.cc:270
#45 0x00000000034126e5 in yb::rpc::InboundCall::InboundCallTask::Run() (this=<optimized out>) at ../../src/yb/rpc/inbound_call.cc:237
#46 0x00000000034cfbbd in yb::rpc::(anonymous namespace)::Worker::Execute() (this=<optimized out>) at ../../src/yb/rpc/thread_pool.cc:104
#47 0x0000000003a96514 in operator() (this=0xbb4c4170) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220531054925-62ed7bc00f-almalinux8-x86_64-clang12-linuxbrew-full-lto/installed/uninstrumented/libcxx/include/c++/v1/functional:1885
#48 operator() (this=0xbb4c4170) at /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220531054925-62ed7bc00f-almalinux8-x86_64-clang12-linuxbrew-full-lto/installed/uninstrumented/libcxx/include/c++/v1/functional:2560
#49 yb::Thread::SuperviseThread(void*) (arg=0xbb4c4120) at ../../src/yb/util/thread.cc:774
#50 0x00007f94af9c1694 in start_thread (arg=0x7f91a6d40700) at pthread_create.c:333
#51 0x00007f94afec341d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
@sonalsagarwal sonalsagarwal added area/docdb YugabyteDB core features status/awaiting-triage Issue awaiting triage labels Jun 30, 2022
@yugabyte-ci yugabyte-ci added kind/bug This issue is a bug priority/medium Medium priority issue labels Jun 30, 2022
@sonalsagarwal
Copy link
Author

Universe Link : http://10.9.131.126/universes/9567c618-f6f3-4399-b24d-b0d18293f0df

@sonalsagarwal
Copy link
Author

Similar to issue : #12681

@sonalsagarwal sonalsagarwal mentioned this issue Jun 30, 2022
Open
@yugabyte-ci yugabyte-ci removed the status/awaiting-triage Issue awaiting triage label Jul 5, 2022
@yugabyte-ci yugabyte-ci added priority/critical Critical issue and removed priority/medium Medium priority issue labels Jul 20, 2022
@yugabyte-ci yugabyte-ci assigned ttyusupov and unassigned rthallamko3 Jul 29, 2022
@yugabyte-ci yugabyte-ci added priority/high High Priority and removed priority/critical Critical issue labels Feb 22, 2023
This was referenced Jun 22, 2023
Open
Open
@arybochkin arybochkin mentioned this issue Jan 30, 2024
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ttyusupov ttyusupov

Labels
area/docdb YugabyteDB core features kind/bug This issue is a bug priority/high High Priority
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ttyusupov @sonalsagarwal @rthallamko3 @yugabyte-ci