Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inconsistent memory reporting for node containers #1471

Closed
pipex opened this issue Sep 25, 2020 · 7 comments
Closed

Inconsistent memory reporting for node containers #1471

pipex opened this issue Sep 25, 2020 · 7 comments
Assignees
@pipex
Labels
Needs more investigation More investigation is needed to find the cause or fix

Comments

@pipex
Copy link
Contributor

pipex commented Sep 25, 2020

When running node containers, dashboard RAM usage shows a different value that what is reported by the
top and balena stats commands.

In a reported case for a rpi3, the dashboard and the free command show 700MB in use, while top and balena stats show 200MB usage.

This requires further investigation to find the source of truth of memory usage and report the correct value to the user.
@pipex pipex added the Needs more investigation More investigation is needed to find the cause or fix label Sep 25, 2020
@pipex pipex self-assigned this Sep 25, 2020
@jellyfish-bot
Copy link

[shaunmulligan] This issue has attached support thread https://jel.ly.fish/7e2ec3cd-f10e-4b88-9d66-39e8a0bd8b89

@jellyfish-bot
Copy link

[shaunmulligan] This issue has attached support thread https://jel.ly.fish/c57bcf60-6609-4daa-86d1-91bf7fe1c56f

@jellyfish-bot
Copy link

[xginn8] This issue has attached support thread https://jel.ly.fish/f789d0eb-2d55-42bb-ba94-0b03bc112401

@20k-ultra
Copy link
Contributor

20k-ultra commented Sep 30, 2020 •
edited

A user has provided a description of what seems to be the current cause of this issue.

The new added dashboard widget is giving wrong information, the memory usage doesn't take into account memory used for cache, giving a false impression that the device is running out of memory when in reality linux tries to use as much memory as possible for cache that can be take into account as free memory, ie. you can check how htop displays the memory usage. Then the disk usage one, at least in one device, is showing 2.8G of 2.8G used and if we look at df in the device, we can see that 1.6G is free.

@xginn8 looked into how we are retrieving the memory usage of the device and had the following to say:

In terms of memory usage, the supervisor uses a module called systeminformation which in turn calculates used memory as totalmem - freemem from /proc/meminfo (seen here: https://github.com/sebhildebrandt/systeminformation/blob/master/lib/memory.js#L154).

Therefore, we should see if this module is calculating with the cached memory. Personally, unless this module is adding a more robust way of gathering system information that we are using in other places I'm in favour of us creating our own implementation to remove this 3rd party dependency but I haven't seen how much we use it.

@xginn8
Copy link
Contributor

xginn8 commented Sep 30, 2020

This is pretty well outlined in the kernel manpage: https://man7.org/linux/man-pages/man5/proc.5.html

pipex added a commit that referenced this issue Oct 12, 2020
@pipex
Improve calculation for used system memory
b2adaa5 
The memory information reported by the supervisor currently
estimates the value of used memory as `MemTotal - MemFree`.
However, linux systems will try to cache and buffer as much
memory as possible, which will affect the output of `MemFree`
(from /proc/meminfo) and in consequence the memory usage seen
by the user on the dashboard, which will appear much greater than
it is.

The correct calculation should be `MemTotal - MemFree - Buffers - Cached`,
which the calculation performed by `htop` and the `free` commands.

Change-type: patch
Signed-off-by: Felipe Lalanne <felipe@balena.io>
Connects-to: #1471
@pipex pipex mentioned this issue Oct 12, 2020
Merged
pipex added a commit that referenced this issue Oct 12, 2020
@pipex
Improve calculation for used system memory
7565fe8 
The memory information reported by the supervisor currently
estimates the value of used memory as `MemTotal - MemFree`.
However, linux systems will try to cache and buffer as much
memory as possible, which will affect the output of `MemFree`
(from /proc/meminfo) and in consequence the memory usage seen
by the user on the dashboard, which will appear much greater than
it is.

The correct calculation should be `MemTotal - MemFree - Buffers - Cached`,
which the calculation performed by `htop` and the `free` commands.

Change-type: patch
Signed-off-by: Felipe Lalanne <felipe@balena.io>
Connects-to: #1471
pipex added a commit that referenced this issue Oct 14, 2020
@pipex
Improve calculation for used system memory
dd5f622 
The memory information reported by the supervisor currently
estimates the value of used memory as `MemTotal - MemFree`.
However, linux systems will try to cache and buffer as much
memory as possible, which will affect the output of `MemFree`
(from /proc/meminfo) and in consequence the memory usage seen
by the user on the dashboard, which will appear much greater than
it is.

The correct calculation should be `MemTotal - MemFree - Buffers - Cached`,
which the calculation performed by `htop` and the `free` commands.

Change-type: patch
Signed-off-by: Felipe Lalanne <felipe@balena.io>
Connects-to: #1471
@pipex pipex closed this as completed Oct 19, 2020
@jellyfish-bot
Copy link

[shaunmulligan] This issue has attached support thread https://jel.ly.fish/c30f67dc-d242-465b-a999-070f83552a36

@jellyfish-bot
Copy link

[tmigone] This issue has attached support thread https://jel.ly.fish/073ce7d0-232c-401b-8438-d55c366880e5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pipex pipex

Labels
Needs more investigation More investigation is needed to find the cause or fix
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@pipex @20k-ultra @xginn8 @jellyfish-bot