Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[docdb] Multiple test failures on shutdown path due to ~MemTracker #3750

Closed
bmatican opened this issue Feb 27, 2020 · 0 comments
Closed

[docdb] Multiple test failures on shutdown path due to ~MemTracker #3750

bmatican opened this issue Feb 27, 2020 · 0 comments
Assignees
@rajukumaryb
Labels
area/docdb YugabyteDB core features kind/failing-test Tests and testing infra
Projects
YBase features

Comments

@bmatican
Copy link
Contributor

Example (thanks @jj-kim ) https://detective-gcp.dev.yugabyte.com/job/github-yugabyte-db-centos-master-gcc-debug%2F658%2Fartifact%2Fbuild%2Fdebug-gcc-dynamic-ninja%2Fyb-test-logs%2Ftests-pgwrapper__pg_mini-test%2FPgMiniTest_DropDBMarkDeleted.log?max_lines=3000&start_line=12001

#23 std::vector<std::shared_ptr<yb::MemTracker>, std::allocator<std::shared_ptr<yb::MemTracker> > >::~vector (this=0x7ff513e47f30 <yb::server::(anonymous namespace)::common_mem_trackers>, __in_chrg=<optimized out>) at /opt/yb-build/brew/linuxbrew-20181203T161736v2-3ba4c2ed9b0587040949a4a9a95b576f520bae/Cellar/gcc/5.5.0_4/include/c++/5.5.0/bits/stl_vector.h:424

I haven't done a bisect, but I did grep through the jenkins logs, just for gcc release and could see this going back 100 builds:

/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/606/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-integration-tests__twodc-test/BatchSize__TwoDCTest_TestWalRetentionSet__0.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/606/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-integration-tests_twodc-test__BatchSize_TwoDCTest.TestWalRetentionSet_0__error.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/608/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-integration-tests__encryption-test/EncryptionTest_RollingMasterRestart.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/608/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-integration-tests_encryption-test__EncryptionTest.RollingMasterRestart__error.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/609/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-client__snapshot-txn-test/SnapshotTxnTest_BankAccountsWithTimeStrobe.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/609/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-client_snapshot-txn-test__SnapshotTxnTest.BankAccountsWithTimeStrobe__error.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/611/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-client__ql-transaction-test/QLTransactionTest_SimpleWriteConflict.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/611/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-client_ql-transaction-test__QLTransactionTest.SimpleWriteConflict__error.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/621/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-client__snapshot-txn-test/SnapshotTxnTest_BankAccountsWithTimeStrobe.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/621/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-client_snapshot-txn-test__SnapshotTxnTest.BankAccountsWithTimeStrobe__error.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/630/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-client__client-test/ClientTest_TestReplicatedMultiTabletTableFailover.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/630/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-client_client-test__ClientTest.TestReplicatedMultiTabletTableFailover__error.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/635/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_libpq-test/PgLibPqTest_ConcurrentIndexInsert.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/635/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper_pg_libpq-test__PgLibPqTest.ConcurrentIndexInsert__error.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/638/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper__pg_libpq-test/PgLibPqTest_SecondaryIndexInsertSelect.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/638/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-pgwrapper_pg_libpq-test__PgLibPqTest.SecondaryIndexInsertSelect__error.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/640/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-client__serializable-txn-test/SerializableTxnTest_ReadWriteConflict.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/640/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-client_serializable-txn-test__SerializableTxnTest.ReadWriteConflict__error.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/656/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-client__snapshot-txn-test/SnapshotTxnTest_BankAccountsWithTimeStrobe.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/656/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-client_snapshot-txn-test__SnapshotTxnTest.BankAccountsWithTimeStrobe__error.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/666/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-integration-tests__twodc-test/BatchSize__TwoDCTest_TestInsertDeleteWorkloadWithRestart__0.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/666/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-integration-tests_twodc-test__BatchSize_TwoDCTest.TestInsertDeleteWorkloadWithRestart_0__error.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/671/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-client__ql-transaction-test/QLTransactionTest_ReadRestartWithIntents.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/671/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-client__ql-transaction-test/QLTransactionTest_SimpleWriteConflict.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/671/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-client_ql-transaction-test__QLTransactionTest.SimpleWriteConflict__error.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/671/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-client_ql-transaction-test__QLTransactionTest.ReadRestartWithIntents__error.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/673/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-client__ql-transaction-test/QLTransactionTest_WriteConflicts.log
/var/lib/jenkins/jobs/github-yugabyte-db-centos-master-gcc-release/builds/673/archive/build/release-gcc-dynamic-ninja/yb-test-logs/tests-client_ql-transaction-test__QLTransactionTest.WriteConflicts__error.log

I suspect this traces back to 317ef8e

Since this is failing across tests and build types, it would be great to fix soon.

cc @spolitov
@bmatican bmatican added kind/failing-test Tests and testing infra area/docdb YugabyteDB core features labels Feb 27, 2020
@bmatican bmatican added this to To Do in YBase features via automation Feb 27, 2020
rajukumaryb added a commit that referenced this issue Mar 14, 2020
@rajukumaryb
#3750: [docdb] Deregister callback for common mem trackers from root …
d6fa3f9 
…mem tracker in destructor of vector

Summary:
Tree of `MemTrackers` is organized so that a node maintains pointers to all nodes along the path to root. Hence (almost) all traversal is in the upward direction. But common `MemTracker`s are accessed from the root - i.e., in the         downward direction. In the destructor of a common `MemTracker`, an upward traversal to root is followed by a downward traversal back into vector of common `MemTracker`s. This vector should not be accessed by root mem tracker once any     common mem tracker in it is destroyed.

Prevent vector access from root mem tracker by setting the lambda for downward traversal to null in the destructor of vector.

Test Plan: Jenkins

Reviewers: amitanand, bogdan, sergei

Reviewed By: sergei

Subscribers: ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D8087
YBase features automation moved this from To Do to Done Mar 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rajukumaryb rajukumaryb

Labels
area/docdb YugabyteDB core features kind/failing-test Tests and testing infra
Projects
YBase features
  
Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bmatican @rajukumaryb