Skip to content

2.27.0.0-b277

2.27.0.0-b277: [#27810] DocDB: Fix potential high memory consumption issue for pg_locks
@basavaraj29 basavaraj29 tagged this 27 Jun 16:18 
Summary:
In one of the customer envs, we saw `pg_locks` controbuting to high memory with the following stack traces.
```
Estimated bytes: 22817013760, estimated count: 9, sampled bytes: 22817013760, sampled count: 9, stack:
;;tcmalloc::tcmalloc_internal::SampleifyAllocation<>()
;;tcmalloc::tcmalloc_internal::slow_alloc_large<>()
;;yb::tablet::Tablet::GetLockStatus()
;;yb::tserver::TabletServiceImpl::GetLockStatus()

Estimated bytes: 1421082624, estimated count: 43368, sampled bytes: 43581440, sampled count: 1330, stack:
;;tcmalloc::tcmalloc_internal::SampleifyAllocation<>()
;;tcmalloc::tcmalloc_internal::slow_alloc_small<>()
;;rocksdb::UncompressBlockContents()
;;rocksdb::ReadBlockContents()
;;rocksdb::block_based_table::ReadBlockFromFile()
;;rocksdb::BlockBasedTable::ReadBlockFromFileAndMaybePutToCache()
;;rocksdb::BlockBasedTable::BlockEntryIteratorState::NewSecondaryIterator()
;;rocksdb::(anonymous namespace)::TwoLevelIterator::InitDataBlock()
;;rocksdb::(anonymous namespace)::TwoLevelIterator::Seek()
;;rocksdb::MergingIteratorBase<>::Seek()
;;rocksdb::DBIter::Seek()
;;yb::tablet::Tablet::GetLockStatus()
;;yb::tserver::TabletServiceImpl::GetLockStatus()

Estimated bytes: 1112997888, estimated count: 33966, sampled bytes: 34177024, sampled count: 1043, stack:
;;tcmalloc::tcmalloc_internal::SampleifyAllocation<>()
;;tcmalloc::tcmalloc_internal::slow_alloc_small<>()
;;rocksdb::UncompressBlockContents()
;;rocksdb::ReadBlockContents()
;;rocksdb::block_based_table::ReadBlockFromFile()
;;rocksdb::BlockBasedTable::ReadBlockFromFileAndMaybePutToCache()
;;rocksdb::BlockBasedTable::BlockEntryIteratorState::NewSecondaryIterator()
;;rocksdb::(anonymous namespace)::TwoLevelIterator::InitDataBlock()
;;rocksdb::(anonymous namespace)::TwoLevelIterator::DoSkipEmptyDa
taBlocksForward();;rocksdb::MergingIteratorBase<>::Next()
;;rocksdb::DBIter::Next();;yb::tablet::Tablet::GetLockStatus()
;;yb::tserver::TabletServiceImpl::GetLockStatus()
```

Stacks 2 & 3 can be avoided by explicitly setting iterator read options to not fill the cache. Cause for stack 1 is still not clear. It could have been from the local vector - `txn_intent_keys` -  where we store the value section of the interested reverse index keys from intentsdb. Note that this part doesn't respect `max_txn_locks_per_tablet` set in the request and reads all intents of the txn. So it is better we limit reading of intents to `max_txn_locks_per_tablet` per txn as well, since we anyways don't populate records corresponding to the additional ones in the response.

This revision addresses the issue by:
1. Explicitly setting the interator read options to not cache the contents in block cache
2. Limiting the number of intents records we read per txn
Jira: DB-17405

Test Plan: ./yb_build.sh --cxx-test='TEST_F(PgGetLockStatusTest, TestGetLockStatusLimitNumTxnLocks) {'

Reviewers: yyan, rthallam

Reviewed By: yyan

Subscribers: ybase

Differential Revision: https://phorge.dev.yugabyte.com/D45028
Assets 2
Loading