Skip to content

2.27.0.0-b211

@amitanandaiyer amitanandaiyer tagged this 11 Jun 00:26
Summary:
If there is a long running operations may create lot of entries for the
traces causing memory bloat.

This diff adds gflags to limit the number of entries/child-traces that
may be added to any given trace.

Due to recent QA testing, we have identified 2 potential places where we are creating too many traces. This is caused due to the interaction with threadpool, where the threadpool task inherits the trace from the thread which submits the job. This has caused the following tasks/threads to hold onto an rpc trace, and keep appending to the trace, without destroying it, even after the rpc is done.
   - Raft Log appenders, and
   - shared memory exchange threads

To reduce the odds of this, we have made the following changes :
 - use trace level 3 for denoting when a wait state is pulled, (2c496914857c0a7c4e3f2c55d770d3f23308a340) and
 - create a gflag that controls whether or not a thread pool task should inherit a trace (from an incoming rpc task) a0e6d550eb268818f9189d79d3066a61cba10be4 ;

The current diff is meant to cap the memory growth due to traces, in case there are still any unknown areas of memory growth due to traces.
Jira: DB-16843

Test Plan:
yb_build.sh fastdebug --cxx-test trace-test
stress test

Reviewers: rthallam, bkolagani, sergei

Reviewed By: bkolagani

Subscribers: ybase

Differential Revision: https://phorge.dev.yugabyte.com/D44256
Assets 2
Loading