Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add jemalloc metrics #5448

Merged
merged 6 commits into from Sep 19, 2019

Conversation

@Jing118
Copy link
Contributor

Jing118 commented Sep 11, 2019

What have you changed?

Add 2 jemalloc metrics:

  • dirty: Memory usage of dirty pages , which are defined as previously having been potentially written to by the application, and therefore consuming physical memory, yet having no current use.
  • fragmentation: Calculated as resident - allocated - metadata - dirty

What is the type of the changes?

Improvement.

How is the PR tested?

Manual test, Grafana:

截屏2019-09-1113 57 04

Does this PR affect documentation (docs) or should it be mentioned in the release notes?

No.

Does this PR affect tidb-ansible?

No.

Refer to a related PR or issue link (optional)

Benchmark result if necessary (optional)

Any examples? (optional)

@zhangjinpeng1987

This comment has been minimized.

Copy link
Member

zhangjinpeng1987 commented Sep 11, 2019

The sign-off is missing.

Signed-off-by: Lingjing You <clylj135@gmail.com>
@Jing118 Jing118 force-pushed the Jing118:jemalloc-stats branch from bd79cf0 to 075f329 Sep 11, 2019
Copy link
Member

zhangjinpeng1987 left a comment

LGTM

@zhangjinpeng1987 zhangjinpeng1987 requested a review from youjiali1995 Sep 11, 2019
(
"fragmentation",
stats::resident()?
- (stats::allocated()? + stats::metadata()? + dirty_page_cnt as usize * 4096),

This comment has been minimized.

Copy link
@youjiali1995

youjiali1995 Sep 11, 2019

Contributor

stats.active minus stats.allocated is the fragmentation:

stats.allocated (size_t) r- [--enable-stats]
Total number of bytes allocated by the application.

stats.active (size_t) r- [--enable-stats]
Total number of bytes in active pages allocated by the application. This is a multiple of the page size, and greater than or equal to stats.allocated. This does not include stats.arenas.<i>.pdirty, stats.arenas.<i>.pmuzzy, nor pages entirely devoted to allocator metadata.

Ok(Some(vec![
("allocated", stats::allocated()?),
("active", stats::active()?),
("metadata", stats::metadata()?),
("resident", stats::resident()?),
("mapped", stats::mapped()?),
("retained", stats::retained()?),
("dirty", dirty_page_cnt as usize * 4096),

This comment has been minimized.

Copy link
@youjiali1995

youjiali1995 Sep 11, 2019

Contributor

I think you can calculate dirty pages by resident - active - metadata.

Unused pages in jemalloc have a special state -- muzzy. I'm not sure whether muzzy pages occupy RSS since they're created by madvie(MADV_FREE).

@youjiali1995

This comment has been minimized.

Copy link
Contributor

youjiali1995 commented Sep 11, 2019

Redis has many memory related metrics and it uses jemalloc too.

You can take a look at it https://github.com/antirez/redis/blob/unstable/src/server.c#L3960.

Jing118 added 2 commits Sep 12, 2019
Signed-off-by: Lingjing You <clylj135@gmail.com>
… jemalloc-stats
@sre-bot

This comment has been minimized.

Copy link
Collaborator

sre-bot commented Sep 16, 2019

Your auto merge job has been accepted, waiting for 5401, 5465

@sre-bot

This comment has been minimized.

Copy link
Collaborator

sre-bot commented Sep 16, 2019

/run-all-tests

@sre-bot

This comment has been minimized.

Copy link
Collaborator

sre-bot commented Sep 16, 2019

@Jing118 merge failed.

@Hoverbear

This comment has been minimized.

Copy link
Member

Hoverbear commented Sep 19, 2019

/merge

@sre-bot

This comment has been minimized.

Copy link
Collaborator

sre-bot commented Sep 19, 2019

/run-all-tests

@sre-bot sre-bot merged commit 30963e4 into tikv:master Sep 19, 2019
6 checks passed
6 checks passed
DCO All commits are signed off!
Details
idc-jenkins-ci-tikv/integration-common-test Jenkins job succeeded.
Details
idc-jenkins-ci-tikv/integration-compatibility-test Jenkins job succeeded.
Details
idc-jenkins-ci-tikv/integration-cop-push-down-test Jenkins job succeeded.
Details
idc-jenkins-ci-tikv/integration-ddl-test Jenkins job succeeded.
Details
idc-jenkins-ci/test Jenkins job succeeded.
Details
@Hoverbear

This comment has been minimized.

Copy link
Member

Hoverbear commented Sep 19, 2019

Connor1996 added a commit to Connor1996/tikv that referenced this pull request Sep 24, 2019
Signed-off-by: Lingjing You <clylj135@gmail.com>
sticnarf pushed a commit to sticnarf/tikv that referenced this pull request Oct 27, 2019
Signed-off-by: Lingjing You <clylj135@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.