Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DocDB] Add per-tablet memory overhead to the MemTracker #16196

Closed
Tracked by #16787
bmatican opened this issue Feb 22, 2023 · 2 comments
Closed
Tracked by #16787

[DocDB] Add per-tablet memory overhead to the MemTracker #16196

bmatican opened this issue Feb 22, 2023 · 2 comments
Assignees
@yusong-yan
Labels
area/docdb YugabyteDB core features kind/enhancement This is an enhancement of an existing feature priority/medium Medium priority issue

Comments

@bmatican
Copy link
Contributor

bmatican commented Feb 22, 2023 •
edited by jira bot

Jira Link: DB-5627

Description

As a followup to #7998, we should go further and add this memory overhead explicitly into the MemTracker handling.

For metrics, we need to be careful about per-table vs per-tablet metrics, as @amitanandaiyer did some refactoring of those, to better aggregate. So we'd need to consider the tablet lifecycle on these.

For the rocksdb arena, we likely need to consider the lifecycle of the memstores and/or rocksdbs themselves (eg: truncate / drop resetting rocksdbs).
@bmatican bmatican added area/docdb YugabyteDB core features status/awaiting-triage Issue awaiting triage labels Feb 22, 2023
@yugabyte-ci yugabyte-ci added kind/bug This issue is a bug priority/medium Medium priority issue kind/enhancement This is an enhancement of an existing feature and removed kind/bug This issue is a bug status/awaiting-triage Issue awaiting triage labels Feb 22, 2023
@rthallamko3 rthallamko3 mentioned this issue Apr 10, 2023
Open
35 tasks
yusong-yan added a commit that referenced this issue Jul 26, 2023
@yusong-yan
[#16196] docdb : Add per-tablet's metrics memory usage to MemTracker
c965b08 
Summary:
Created a new metric MemTracker under per-tablet's root Memtracker. This new one tracks tablet's counter, gauge, lag, and histogram. And it will be removed when tablet's metric entity's
Jira: DB-5627

Test Plan: Checked the memory breakdown page on local dev server

Reviewers: rthallam, esheng

Reviewed By: esheng

Subscribers: yql, bogdan, ybase

Differential Revision: https://phorge.dev.yugabyte.com/D26876
spolitov added a commit that referenced this issue Jul 27, 2023
@spolitov
[#16196] DocDB: Fix ASAN build
2e30a8e 
Summary:
Fix regression introduced in D26876/c965b08a84527b09591e86b03daa5ebd1a6e40af.

In this diff we got the following circular dependency:
tablet mem tracker retains tablet metric entity
tablet metric entity retains metric mem tracker
metric mem tracker retains tablet mem tracker

Fixed by moving metric mem tracker ownership to the tablets and storing mem tracker in tablet metric entity as weak pointer.
Jira: DB-5627

Test Plan: Jenkins

Reviewers: yyan, xCluster

Reviewed By: yyan

Subscribers: yql, ybase

Differential Revision: https://phorge.dev.yugabyte.com/D27335
@rthallamko3 rthallamko3 mentioned this issue Aug 16, 2023
Closed
@yusong-yan
Copy link
Contributor

I will create a separate Github issue for adding Rocksdb arena overhead.

@rthallamko3
Copy link
Contributor

Given that the per tablet overhead is fixed, we decided to not backport the fix to stable branches.

@rthallamko3 rthallamko3 mentioned this issue Sep 11, 2023
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yusong-yan yusong-yan

Labels
area/docdb YugabyteDB core features kind/enhancement This is an enhancement of an existing feature priority/medium Medium priority issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@bmatican @rthallamko3 @yugabyte-ci @yusong-yan