Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed Docker CPU Percentage's unusual spike when restarting containers #1100

Merged
merged 2 commits into from Aug 24, 2023

Conversation

Arthur1
Copy link
Member

@Arthur1 Arthur1 commented Aug 22, 2023

docker.cpuacct_percentage.*.{user,system} metrics observed after restarting a docker container appear to be an unusually high spike.

スクリーンショット 2023-08-17 22 00 32

This problem is due to the fact that the comparison of two uint values is made by subtraction.

userUsage := float64(currentUserUsage.(uint64) - uint64(prevUserUsage.(float64)))
if userUsage >= 0 {
(*stats)["docker.cpuacct_percentage."+name+".user"] = userUsage / hostUsage * 100.0 * float64(cpuNumsInt)
}

This pull request enables to correctly detect that counters have been reset and also set metrics to moderate values in that case.

operational test

before

You can see that docker.cpuacct_percentage.hoge_2e3173.{user,system} metrics have huge values after executing docker restart command.

$ ./build/darwin/arm64/mackerel-plugin-docker
docker.memory.hoge_2e3173.cache	11956224	1692718764
docker.memory.hoge_2e3173.rss	3805184	1692718764
$ ./build/darwin/arm64/mackerel-plugin-docker
docker.cpuacct_percentage.hoge_2e3173.user	2.154874	1692718768
docker.cpuacct_percentage.hoge_2e3173.system	1.628945	1692718768
docker.memory.hoge_2e3173.cache	11956224	1692718768
docker.memory.hoge_2e3173.rss	3805184	1692718768
$ docker restart 2e31732280a7
2e31732280a7
$ ./build/darwin/arm64/mackerel-plugin-docker
docker.cpuacct_percentage.hoge_2e3173.user	246614225174.769592	1692718776
docker.cpuacct_percentage.hoge_2e3173.system	246614225285.428925	1692718776
docker.memory.hoge_2e3173.cache	11923456	1692718776
docker.memory.hoge_2e3173.rss	3735552	1692718776

after

$ ./build/darwin/arm64/mackerel-plugin-docker
docker.memory.hoge_2e3173.cache	11956224	1692717626
docker.memory.hoge_2e3173.rss	3829760	1692717626
$ ./build/darwin/arm64/mackerel-plugin-docker
docker.cpuacct_percentage.hoge_2e3173.user	3.126744	1692717634
docker.cpuacct_percentage.hoge_2e3173.system	2.455736	1692717634
docker.memory.hoge_2e3173.cache	11956224	1692717634
docker.memory.hoge_2e3173.rss	3829760	1692717634
$ docker restart 2e31732280a7
2e31732280a7
$ ./build/darwin/arm64/mackerel-plugin-docker
docker.memory.hoge_2e3173.cache	11923456	1692717644
docker.memory.hoge_2e3173.rss	3710976	1692717644
docker.cpuacct_percentage.hoge_2e3173.user	1.112812	1692717644
docker.cpuacct_percentage.hoge_2e3173.system	1.229388	1692717644

@Arthur1 Arthur1 changed the title Fix docker cpu counter reset Fixed Docker CPU Percentage's unusual spike when restarting containers Aug 22, 2023
@Arthur1 Arthur1 self-assigned this Aug 22, 2023
Copy link
Member

@ne-sachirou ne-sachirou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🙆‍♀️

@Arthur1 Arthur1 merged commit 78d4ac7 into master Aug 24, 2023
10 of 14 checks passed
@Arthur1 Arthur1 deleted the fix-docker-cpu-counter-reset branch August 29, 2023 20:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants