Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DocDB] Potential deadlock in rocksdb when applying intents #11687

Closed
bmatican opened this issue Mar 8, 2022 · 0 comments
Closed

[DocDB] Potential deadlock in rocksdb when applying intents #11687

bmatican opened this issue Mar 8, 2022 · 0 comments
Assignees
@spolitov @ttyusupov @jmeehan16 @rthallamko3
Labels
area/docdb YugabyteDB core features priority/high High Priority

Comments

@bmatican
Copy link
Contributor

bmatican commented Mar 8, 2022 •
edited

Description

    @     0x7faf175a9b39  __lll_lock_wait
    @     0x7faf175a46e2  __GI___pthread_mutex_lock
    @          0x33b8370  rocksdb::InstrumentedMutex::Lock()
    @          0x32aae5f  rocksdb::ColumnFamilyData::GetThreadLocalSuperVersion()
    @          0x32d3de7  rocksdb::DBImpl::NewIterator()
    @          0x30c68ca  yb::docdb::BoundedRocksDbIterator::BoundedRocksDbIterator()
    @          0x3120018  yb::docdb::CreateRocksDBIterator()
    @          0x315ff68  yb::docdb::IntentsWriter::Apply()
    @          0x3362adc  rocksdb::WriteBatch::Iterate()
    @          0x32e3e1f  rocksdb::DBImpl::WriteImpl()
    @          0x32d2700  rocksdb::DBImpl::Write()
    @          0x34c0624  yb::tablet::Tablet::WriteToRocksDB()
    @          0x34bd9fe  yb::tablet::Tablet::ApplyIntents()
    @          0x34be1b1  yb::tablet::Tablet::ApplyIntents()
    @          0x354d83d  yb::tablet::TransactionParticipant::Impl::ProcessReplicated()
    @          0x34a7e37  yb::tablet::UpdateTxnOperation::DoReplicated()

Total number of threads: 1

Slack link for more context + dump of all threads: https://yugabyte.slack.com/archives/C01CB38CZHU/p1646669491969579?thread_ts=1646378629.443139&cid=C01CB38CZHU

version 2.13.0.0 build 21 revision 3156825 build_type RELEASE built at 01 Mar 2022 02:16:00 UTC
@bmatican bmatican added area/docdb YugabyteDB core features priority/high High Priority labels Mar 8, 2022
spolitov added a commit that referenced this issue Mar 9, 2022
@spolitov
[#11687] DocDB: Fix deadlock while applying intents
4432bd3 
Summary:
Intents DB iterator are being created while apply intents.
After 6f9c6a4/D15582 it is done in callback as a part of write to regular db.
But we have to lock intents db mutex while creating iterator.

Also we have logic to flush regular db before flushing intents db.
It is performed while holding intents db mutex.
But flush waits until running write is complete.

So there is a deadlock between intents db mutex and write to regular db.

This diff fixes deadlock by creating iterator outside of write callback.

Test Plan: Jenkins

Reviewers: bogdan, timur

Reviewed By: timur

Subscribers: ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D15879
spolitov added a commit that referenced this issue Mar 10, 2022
@spolitov
[Backport 2.13.0] [#11687] DocDB: Fix deadlock while applying intents
c1f3fb1 
Summary:
Intents DB iterator are being created while apply intents.
After 6f9c6a4/D15582 it is done in callback as a part of write to regular db.
But we have to lock intents db mutex while creating iterator.

Also we have logic to flush regular db before flushing intents db.
It is performed while holding intents db mutex.
But flush waits until running write is complete.

So there is a deadlock between intents db mutex and write to regular db.

This diff fixes deadlock by creating iterator outside of write callback.

Original commit: 4432bd3/D15879

Test Plan: Jenkins

Reviewers: bogdan, timur

Reviewed By: timur

Subscribers: ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D15907
pkj415 pushed a commit to pkj415/yugabyte-db that referenced this issue Mar 30, 2022
@spolitov @pkj415
[yugabyte#11687] DocDB: Fix deadlock while applying intents
e98c871 
Summary:
Intents DB iterator are being created while apply intents.
After 6f9c6a4/D15582 it is done in callback as a part of write to regular db.
But we have to lock intents db mutex while creating iterator.

Also we have logic to flush regular db before flushing intents db.
It is performed while holding intents db mutex.
But flush waits until running write is complete.

So there is a deadlock between intents db mutex and write to regular db.

This diff fixes deadlock by creating iterator outside of write callback.

Test Plan: Jenkins

Reviewers: bogdan, timur

Reviewed By: timur

Subscribers: ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D15879
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@spolitov spolitov

@ttyusupov ttyusupov

@jmeehan16 jmeehan16

@rthallamko3 rthallamko3

Labels
area/docdb YugabyteDB core features priority/high High Priority
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@bmatican @spolitov @ttyusupov @jmeehan16 @rthallamko3