Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the stop watch usage during compactions. #4626

Closed
psudheer21 opened this issue May 29, 2020 · 0 comments
Closed

Remove the stop watch usage during compactions. #4626

psudheer21 opened this issue May 29, 2020 · 0 comments
Assignees
Labels
area/docdb YugabyteDB core features

Comments

@psudheer21
Copy link
Contributor

While investigating a performance bottleneck we noticed that almost all the worker threads were performing clock_gettime() with the following stack trace:

 @     0x7fc84f95411e  (unknown)
    @     0x7ffff07c67c1  ([vdso]+0xffffffffff7007c1)
    @     0x7fc84fa13fe4  __GI___clock_gettime
    @     0x7fc85979e9f4  rocksdb::(anonymous namespace)::PosixEnv::NowNanos()
    @     0x7fc859632eeb  rocksdb::StopWatchNano::StopWatchNano()
    @     0x7fc85964a408  rocksdb::CompactionIterator::NextFromInput()
    @     0x7fc85964ac68  rocksdb::CompactionIterator::Next()
    @     0x7fc859650083  rocksdb::CompactionJob::ProcessKeyValueCompaction()
    @     0x7fc8596516cd  rocksdb::CompactionJob::Run()
    @     0x7fc85968e4ef  rocksdb::DBImpl::BackgroundCompaction()
    @     0x7fc8596900fd  rocksdb::DBImpl::BackgroundCallCompaction()
    @     0x7fc8596a1be6  rocksdb::DBImpl::CompactionTask::DoRun()
    @     0x7fc859696607  rocksdb::DBImpl::ThreadPoolTask::Run()
    @     0x7fc855a690a6  yb::(anonymous namespace)::PriorityThreadPoolWorker::Run()
    @     0x7fc855a9430d  yb::Thread::SuperviseThread()
    @     0x7fc8502ca692  start_thread

This usage of the stop watch is done just for stats and is something we can remove.

@kmuthukk kmuthukk added the area/docdb YugabyteDB core features label May 29, 2020
@bmatican bmatican added this to To Do in YBase features via automation Jun 3, 2020
@bmatican bmatican assigned mbautin and unassigned bmatican Jun 3, 2020
@mbautin mbautin removed their assignment Jun 6, 2020
rajukumaryb added a commit that referenced this issue Jun 11, 2020
Summary:
Remove StopWatch from compaction iterator to avoid two time queries per compaction filter call.

  Subset of changes in https://phabricator.dev.yugabyte.com/D8604, authored by @mbautin

  Benchmark Results:

    ./yb_build.sh release --cxx-test docdb_randomized_docdb-test --gtest_filter bool/RandomizedDocDBTest.Snapshots/0 -n 10 -- -p 1 -k
    Before: 14656.4 ms, After: 14423.5 ms, Gain: 1.02x

    ./yb_build.sh release --cxx-test docdb_randomized_docdb-test --gtest_filter bool/RandomizedDocDBTest.Snapshots/1 -n 10 -- -p 1 -k
    Before: 5798.1 ms, After: 5715.8 ms, Gain: 1.01x

    ------------------------

    ./yb_build.sh release --cxx-test docdb_randomized_docdb-test --gtest_filter bool/RandomizedDocDBTest.SnapshotsWithHistoryCleanup/0 -n 10 -- -p 1 -k
    Before: 9929.9 ms, After: 9841.2 ms, Gain: 1.01x

    ./yb_build.sh release --cxx-test docdb_randomized_docdb-test --gtest_filter bool/RandomizedDocDBTest.SnapshotsWithHistoryCleanup/1 -n 10 -- -p 1 -k
    Before: 5314.5 ms, After: 5127.1 ms, Gain: 1.04x

    ------------------------

    ./yb_build.sh release --cxx-test pg_mini-test --gtest_filter PgMiniTest.BigRead -n 10 -- -p 1 -k
    Before: 93300.3 ms, After: 92834.1 ms, Gain: 1.0x

    ./yb_build.sh release --cxx-test pg_mini-test --gtest_filter PgMiniTest.BigReadWithCompaction -n 10 -- -p 1 -k
    Before: 97169.5 ms, After: 96333.1 ms, Gain: 1.01x

    ./yb_build.sh release --cxx-test pg_mini-test --gtest_filter PgMiniTest.SmallRead -n 10 -- -p 1 -k
    Before: 7108 ms, After: 6782.3 ms, Gain: 1.05x

    ------------------------

	  Setup: Ubuntu 18.04, GCP n1-standard-4

Test Plan: jenkins

Reviewers: bogdan, mikhail

Reviewed By: mikhail

Subscribers: ybase, kannan, sergei

Differential Revision: https://phabricator.dev.yugabyte.com/D8649
@rajukumaryb rajukumaryb self-assigned this Jun 11, 2020
YBase features automation moved this from To Do to Done Jun 11, 2020
deeps1991 pushed a commit to deeps1991/yugabyte-db that referenced this issue Jul 22, 2020
Summary:
Remove StopWatch from compaction iterator to avoid two time queries per compaction filter call.

  Subset of changes in https://phabricator.dev.yugabyte.com/D8604, authored by @mbautin

  Benchmark Results:

    ./yb_build.sh release --cxx-test docdb_randomized_docdb-test --gtest_filter bool/RandomizedDocDBTest.Snapshots/0 -n 10 -- -p 1 -k
    Before: 14656.4 ms, After: 14423.5 ms, Gain: 1.02x

    ./yb_build.sh release --cxx-test docdb_randomized_docdb-test --gtest_filter bool/RandomizedDocDBTest.Snapshots/1 -n 10 -- -p 1 -k
    Before: 5798.1 ms, After: 5715.8 ms, Gain: 1.01x

    ------------------------

    ./yb_build.sh release --cxx-test docdb_randomized_docdb-test --gtest_filter bool/RandomizedDocDBTest.SnapshotsWithHistoryCleanup/0 -n 10 -- -p 1 -k
    Before: 9929.9 ms, After: 9841.2 ms, Gain: 1.01x

    ./yb_build.sh release --cxx-test docdb_randomized_docdb-test --gtest_filter bool/RandomizedDocDBTest.SnapshotsWithHistoryCleanup/1 -n 10 -- -p 1 -k
    Before: 5314.5 ms, After: 5127.1 ms, Gain: 1.04x

    ------------------------

    ./yb_build.sh release --cxx-test pg_mini-test --gtest_filter PgMiniTest.BigRead -n 10 -- -p 1 -k
    Before: 93300.3 ms, After: 92834.1 ms, Gain: 1.0x

    ./yb_build.sh release --cxx-test pg_mini-test --gtest_filter PgMiniTest.BigReadWithCompaction -n 10 -- -p 1 -k
    Before: 97169.5 ms, After: 96333.1 ms, Gain: 1.01x

    ./yb_build.sh release --cxx-test pg_mini-test --gtest_filter PgMiniTest.SmallRead -n 10 -- -p 1 -k
    Before: 7108 ms, After: 6782.3 ms, Gain: 1.05x

    ------------------------

	  Setup: Ubuntu 18.04, GCP n1-standard-4

Test Plan: jenkins

Reviewers: bogdan, mikhail

Reviewed By: mikhail

Subscribers: ybase, kannan, sergei

Differential Revision: https://phabricator.dev.yugabyte.com/D8649
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/docdb YugabyteDB core features
Projects
Development

No branches or pull requests

5 participants