statistics(memory, _) unexpectedly reports very small numbers

[triska]

Please consider the following sample program:

:- use_module(library(clpfd)).

run(N0) :-
        assertz(pred(mem)),
        N #= N0 + 1,
        (   N mod 100 #= 0 ->
            statistics(memory, [M|_]),
            portray_clause(num_mem(N, M))
        ;   true
        ),
        run(N).

On Debian 9.3, I get with SWI-Prolog 7.7.9:

?- run(0).
num_mem(100, 6144).
num_mem(200, 7928).
num_mem(300, 9712).
num_mem(400, 11496).
num_mem(500, 13280).
num_mem(600, 2352).
num_mem(700, 4136).
num_mem(800, 5920).
num_mem(900, 7704).
num_mem(1000, 9488).
num_mem(1100, 11272).
num_mem(1200, 13056).
num_mem(1300, 2544).
...
num_mem(2482000, 64176).
...
num_mem(2482400, 9552).
...
num_mem(2492200, 60608).
...
num_mem(2492700, 7768).
etc.

It seems that no matter how long this runs, very small values of unshared memory are periodically reported by statistics/2 . However, I see from htop that memory usage of this program grows more or less constantly and without bounds.

A (much) more accurate measure of the memory that is actually used is obviously highly desirable in many situations, for example to assess how much space particular terms take (example: SSL contexts).

Is memory intended to be a suitable measure for such cases? Is there a parameter that lets us reliably obtain the used memory?

For comparison, statistics/2 behaves as I would expect in the following case:

bigger([_|Ls], N0) :-
        N #= N0 + 1,
        (   N mod 100 #= 0 ->
            statistics(memory, [M|_]),
            portray_clause(num_mem(N, M))
        ;   true
        ),
        bigger(Ls, N).

Sample query:

?- bigger(Ls, 0).
num_mem(100, 4964).
num_mem(200, 6272).
num_mem(300, 7580).
num_mem(400, 8888).
num_mem(500, 10196).
num_mem(600, 11504).
...
num_mem(110000, 1442456).
num_mem(110100, 1443764).
num_mem(110200, 1445072).
num_mem(110300, 1446380).
num_mem(110400, 1447688).
etc.

Further, I note the following unexpected result:

?- statistics(memory, X).
X = [10056, -10057].

[JanWielemaker]

On 02-03-18 09:22, Markus Triska wrote: Please consider the following sample program: :- use_module(library(clpfd)). run(N0) :- assertz(pred(mem)), N #= N0 + 1, ( N mod 100 #= 0 -> statistics(memory, [M|_]), portray_clause(num_mem(N, M)) ; true ), run(N). On Debian 9.3, I get with SWI-Prolog 7.7.9: *?- run(0).* num_mem(100, 6144). num_mem(200, 7928). num_mem(300, 9712). num_mem(400, 11496). num_mem(500, 13280). *num_mem(600, 2352).* num_mem(700, 4136). num_mem(800, 5920). num_mem(900, 7704). num_mem(1000, 9488). num_mem(1100, 11272). num_mem(1200, 13056). *num_mem(1300, 2544).* ... num_mem(2482000, 64176). ... *num_mem(2482400, 9552).* ... num_mem(2492200, 60608). ... *num_mem(2492700, 7768).* etc. It seems that no matter how long this runs, very small values of unshared memory are periodically reported by |statistics/2| . However, I see from |htop| that memory usage of this program *grows more or less constantly* and without bounds.

I uses this code: #if defined(HAVE_GETRUSAGE) && defined(HAVE_RU_IDRSS) struct rusage usage; if ( getrusage(RUSAGE_SELF, &usage) == 0 && usage.ru_idrss ) { return usage.ru_idrss; /* total unshared data */ } #endif See UsedMemory() in pl-os.c

A (much) more accurate measure of the memory that is actually used is obviously highly desirable in many situations, for example to assess how much space particular terms take (example: SSL contexts). Is |memory| intended to be a suitable measure for such cases? Is there a parameter that lets us reliably obtain the used memory?

Not really. It is a Quintus compatibility key, but in modern OSes it isn't that easy to get anything usable in a portable manner. If you have glibc (Linux), there is library(mallocinfo) that provides much more accurate and detailed statistics. I recently found the heaptrack program. That is really neat.

For comparison, |statistics/2| behaves as I would expect in the following case:

This suggests it didn't configure getrusage() right on your system. I guess you can figure that out yourself :) Cheers --- Jan

…

bigger([_|Ls], N0) :- N #= N0 + 1, ( N mod 100 #= 0 -> statistics(memory, [M|_]), portray_clause(num_mem(N, M)) ; true ), bigger(Ls, N). Sample query: ?- bigger(Ls, 0). num_mem(100, 4964). num_mem(200, 6272). num_mem(300, 7580). num_mem(400, 8888). num_mem(500, 10196). num_mem(600, 11504). ... num_mem(110000, 1442456). num_mem(110100, 1443764). num_mem(110200, 1445072). num_mem(110300, 1446380). num_mem(110400, 1447688). etc. Further, I note the following unexpected result: ?- statistics(memory, X). X = [10056,*-10057*]. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#290>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AC7cqnBBvBw4n4ajdOCtcPHKL5hiEfz2ks5taX-cgaJpZM4SaOYC>.

[erlanger]

@JanWielemaker
ru_idrss is not maintained in Linux, please see the manpage snippet below. Perhaps the C code below is not the one used in Linux? (since the man page says that the kernel sets the unmaintained values to 0).

#if defined(HAVE_GETRUSAGE) && defined(HAVE_RU_IDRSS) 
   struct rusage usage; 
      if ( getrusage(RUSAGE_SELF, &usage) == 0 && usage.ru_idrss ) { 
         return usage.ru_idrss; /* total unshared data */ 
      } 
#endif

If we run man 2 getrusage we get the following snippet:

    Not  all  fields  are completed; unmaintained fields are set to zero by the kernel.  (The unmaintained fields are provided for compatibility with other systems, and
       because they may one day be supported on Linux.)  The fields are interpreted as follows:

       ru_utime
              This is the total amount of time spent executing in user mode, expressed in a timeval structure (seconds plus microseconds).

       ru_stime
              This is the total amount of time spent executing in kernel mode, expressed in a timeval structure (seconds plus microseconds).

       ru_maxrss (since Linux 2.6.32)
              This is the maximum resident set size used (in kilobytes).  For RUSAGE_CHILDREN, this is the resident set size of the largest child, not the maximum resident
              set size of the process tree.

       ru_ixrss (unmaintained)
              This field is currently unused on Linux.

       ru_idrss (unmaintained)    <----------------------------------    HERE
              This field is currently unused on Linux.

       ru_isrss (unmaintained)
              This field is currently unused on Linux.

Notice ru_idrss is unmaintained. Perhaps it would be useful to find out how top/htop get rss and just copy that code.

[erlanger]

What does htop use?

I looked through htop, and on linux it uses the /proc/*/statm file to obtain the memory usage. For our case it would be /proc/self/statm. You can see the htop code here.

What fields from statm to use?

Mr. Jan mentions that it is difficult to decide what 'memory usage means'. This is so true. Now, statm offers the following fields:

/proc/[pid]/statm
              Provides information about memory usage, measured in pages.
              The columns are:

                  size       (1) total program size
                             (same as VmSize in /proc/[pid]/status)
                  resident   (2) resident set size
                             (same as VmRSS in /proc/[pid]/status)
                  shared     (3) number of resident shared pages (i.e., backed by a file)
                             (same as RssFile+RssShmem in /proc/[pid]/status)
                  text       (4) text (code)
                  lib        (5) library (unused since Linux 2.6; always 0)
                  data       (6) data + stack
                  dt         (7) dirty pages (unused since Linux 2.6; always 0)

Maybe resident?

At first glance, it may seem that we want field resident - (2), but this does not seem right because it includes the size of memory mapped files. As we see in the man page:

   * VmRSS: Resident set size.  Note that the value here is the
                sum of RssAnon, RssFile, and RssShmem.

              * RssAnon: Size of resident anonymous memory.  (since Linux
                4.5).

              * RssFile: Size of resident file mappings.  (since Linux 4.5).

              * RssShmem: Size of resident shared memory (includes System V
                shared memory, mappings from tmpfs(5), and shared anonymous
                mappings).  (since Linux 4.5).

data + text seems better

What I would suggest we need (compatible semantically with the existing code) is the addition of text(4) + data(6). This would mean the size of code+data+stack using the field description from the man page snipped above.

What do you guys think?

Extra information

By the way, cat /proc/self/status has a human readable form of all the process information and it is documented in the man page above.

UPDATE: the resolution of /proc/self/stat is memory pages (usually 4096 bytes), so this might not be useful to measure anything below that resolution.

[JanWielemaker]

The code for UsedMemory() indeed ignores getrusage() as this returns 0 and returns the combined size of the stacks. The statistics(memory, [Used,Free]) is an old Quintus compatibility flag. It is pretty vague what used and free memory means over different modern operating systems. It is also not that clear why you would like to know as a Prolog programmer. I guess the fact that Linux doesn't maintain these more or less standard fields illustrates this.

I don't think it is worth the trouble trying to maintain many platform specific implementations trying to provide an as good as possible approximation of used and free memory for which it is still hard to assess whether that figure provides what the user wants to know and that all means something slightly different. Note that the clib package provides library(mallocinfo) which exports malinfo/1 for glibc based systems. In SWISH you find lib/procps that gets a lot of info from the /proc file system.

[erlanger]

Hi Jan,

Thanks!
Yeah, I think it is better to let it return the swi computed memory statistics rather than trying to provide platform specific code.

Why would the swi computation return that strangely low number? Perhaps there is some interaction
with the garbage collector.

num_mem(400, 11496). 
num_mem(500, 13280). 
num_mem(600, 2352).     % <----- HERE
num_mem(700, 4136).

I have seen this before, but I can't reproduce it now.

[JanWielemaker]

It has the strange habit to report the combined stack size, which are typically low in the toplevel. I'll change this to simply return 0. That at least is less misleading.

[erlanger]

Thanks!

[JanWielemaker]

Pushed 696a260 to resolve this.

[erlanger]

Thanks! I just found term_size/2 gives you the precise size of a term in cells (8 bytes for 64bit; 4 bytes for 32bit). Perhaps @triska and others reading this thread will find this useful for use cases in which you want to know the precise size of a term.