[Master] Move away from simple_spinlock usage in catalog_manager and friends #1540
Assignees
Labels
area/docdb
YugabyteDB core features
kind/enhancement
This is an enhancement of an existing feature
priority/medium
Medium priority issue
Projects
Comments
bmatican
added
the
kind/enhancement
yugabyte-ci
pushed a commit
that referenced
this issue
Jun 12, 2019
…1540 Summary: We have some concurrency issues on the master side, but we have yet to do a good enough analysis into what and where. I ended up porting and adapting 2 diffs from Kudu: port1: apache/kudu@c74e678 port2: apache/kudu@717349b Furthermore, I added a script to analyze and sort pprof-style contention stacks so we can trace which is the worst offender and try to optimize that. Using that, I changed both the TSDescriptor locking from `simple_spinlock` to `rw_spinlock`, from the Kudu port, but I also dug into the `CatalogManager` cow object, `TableInfo`, to do the same. Test Plan: ` YB_ALLOW_SLOW_TESTS=1 ./build/latest/tests-integration-tests/create-table-stress-test --gtest_filter=\*Bench\* --max_create_tablets_per_ts 100 --benchmark_num_tablets 300 --benchmark_num_threads 30 --benchmark_runtime_secs=10` Sample histogram info at the end: ``` I0530 16:46:49.703567 5846 create-table-stress-test.cc:247] BENCHMARK HISTOGRAM: I0530 16:46:49.703572 5846 create-table-stress-test.cc:248] Count: 2538 Mean: 27109.5 Percentiles: 0% (min) = 2042 25% = 3408 50% (med) = 4032 75% = 30848 95% = 129536 99% = 231424 99.9% = 301056 99.99% = 329728 100% (max) = 330648 ``` Sample stack info at the end (much like the web pprof contention output): ``` 46273152 2 @ 00007f36f4bd8687 00007f36fec7a057 00007f36fecae863 00007f36f9e59143 00007f36f7bc96e8 00007f36f7b77134 00007f36f7bd4dd8 00007f36f4bec819 00007f36fc27c694 00007f36f06be41d 0000000000000000 @ 0x7f36f4bd8686 yb::(anonymous namespace)::SubmitSpinLockProfileData() @ 0x7f36fec7a056 yb::master::ScopedLeaderSharedLock::ScopedLeaderSharedLock() @ 0x7f36fecae862 yb::master::MasterServiceImpl::TSHeartbeat() @ 0x7f36f9e59142 yb::master::MasterServiceIf::Handle() @ 0x7f36f7bc96e7 yb::rpc::ServicePoolImpl::Handle() @ 0x7f36f7b77133 yb::rpc::InboundCall::InboundCallTask::Run() @ 0x7f36f7bd4dd7 yb::rpc::(anonymous namespace)::Worker::Execute() @ 0x7f36f4bec818 yb::Thread::SuperviseThread() @ 0x7f36fc27c693 start_thread @ 0x7f36f06be41c __clone @ 0xffffffffffffffff (unknown) ----------- ``` Reviewers: timur, mikhail, raju, sergei Reviewed By: sergei Subscribers: kannan, ybase Differential Revision: https://phabricator.dev.yugabyte.com/D6685
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-1473
Using
simple_spinlockacross catalog_manager and friends (like table/tablet info) is becoming a bottleneck in clusters with:At the very least we can replace these with a
rw_spinlock...The text was updated successfully, but these errors were encountered: