Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[memory_monitoring] Enhance monitoring the memory usage of containers #10008

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

yozhao101
Copy link
Contributor

@yozhao101 yozhao101 commented Feb 17, 2022

Signed-off-by: Yong Zhao yozhao@microsoft.com

Why I did it

Currently we have a PR (#10017) which will put a hard limitation on memory usage of telemetry container.

During testing the above PR, we found that the commands docker stats, docker start, docker stop and docker exec will be no response if memory usage of telemetry container reaches or exceeds the configured hard limitation. This behavior will fail the background script memory_checker since they depend on the command output of docker stats.

How I did it

Instead of depending on the output of docker stats, the background script memory_checker will calculate the memory usage of a container based on values reading from the cgroup subsystem files /sys/fs/cgroup/memory/docker/<container_name>/memory.usage_in_bytes and /sys/fs/cgroup/memory/docker/<container_name>/memory.stats.

I refer to this Docker official document (https://docs.docker.com/engine/reference/commandline/stats/#extended-description) to make sure the memory usage of a specific container reading from command output of docker stats is equal to the value subtracting cache usage from the total memory usage.

How to verify it

I manually tested this implementation on DuT str-msn2700-20.

Which release branch to backport (provide reason below if selected)

  • 201811
  • 201911
  • 202006
  • [x ] 202012
  • 202106
  • 202111

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

of containers.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>
@yozhao101 yozhao101 changed the title [memory_monitoring] Enhance the feature of monitoring the memory usage of containers [memory_monitoring] Enhance monitoring the memory usage of containers Feb 17, 2022
`contianerd-shim` process.

Signed-off-by: Yong Zhao <yozhao@microsoft.com>
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
@yozhao101
Copy link
Contributor Author

/azp run

@azure-pipelines
Copy link

You have several pipelines (over 10) configured to build pull requests in this repository. Specify which pipelines you would like to run by using /azp run [pipelines] command. You can specify multiple pipelines using a comma separated list.

@yozhao101
Copy link
Contributor Author

/azp run sonic-buildimage

@azure-pipelines
Copy link

No pipelines are associated with this pull request.

Copy link
Contributor

@FengPan-Frank FengPan-Frank left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's conflicts exist on the PR, I update it into #19179 instead, will reply and address comments there as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants