New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
The memory statistics in 4.2.0 beta1 are wrong #7591
Comments
For me, on Ubuntu 16.04, with repo.powerdns.com master builds, the Metronome number is equal to /proc/PID/status's VmData plus VmStk numbers. |
This is related to #7502 - the precise memory measurements were too slow. We are very much open to suggestions how to get a better number that is not too slow to calculate. Note that many kernels are not slow to gather this number.. but some are. And it caused 750 millisecond slowdowns. |
The new counting includes r/o pages and clean r/w pages. Those pages are assigned to the process. But they can be unassigned without cost. So you can count them either way. statm includes them. |
Wouldn't On dnsdist 1.4.0-alpha1 the special/non-special numbers are now quite different.
|
The problem of "how much mem does a program use" in a demand paged virtual memory system having features like shared pages is essentially unanswerable in a value of 1 dimension. "Resident" does not count pages in swap. "data" includes virtual pages allocated but not touched. The new measurement shows a different number, yes, but not wrong. |
We should export all 5 memory stats available from |
Well, it turns out it is wrong some of the time. See #10161 |
No, that is not a good thing. A few numbers here are either useless (always zero) or wrong. See |
Sorry for reopening, but I believe this change (removal of drs from stats) is not the right thing to do. First, the referenced kernel documentation is outdated - description of statm table mentions ancient pre-3.0 kernels thus unlikely it was ever changed since, and even then "broken" is clarifying that it also includes text (code), not only data+stack. In any case, there is no evidence that "drs" is broken or misbehaving (i.e. shows wrong data) in recent kernels (or even in 3.10 which is used by CentOS 7 - which is history by now anyway) - in my experience, it was always correct - shows requested (allocated) data+stack size. Second, RSS by itself is meaningless - it shows only amount of RAM which is resident - in particular this means that if memory is aggressively swapped out and stays in swap (for instance, leaked pages which are never reference after initial use) RSS will never expose the problem. VmData aka drs, on the other hand, shows requested (allocated) memory usage - and while it counts also not (yet) touched pages this is exactly what is needed in most cases - if memory is requested then it is likely will be used eventually - so we know the "aim" of the application. There are exceptions of course (apps using sparse memory) but I doubt that pdns is one of them. Taking all this into account we could review the original problem - if pdns did request more RAM that available in the system this would only lead to issues a bit later - once it would start to use this memory, which would lead to heavy swapping or even OOM - thus the indication was quite useful. With all this in mind I believe that both drs and rss should be available in stats. |
Some comments:
I agree that the mem stats should be revisited and extended. |
Short description
There is a huge jump in memory usage between alpha1 and beta1 in metronome
Total memory usage on the system is lower than the reported PowerDNS usage.
Environment
The text was updated successfully, but these errors were encountered: