dwidenoise: Progressively increasing memory usage

[Lestropie]

As reported on forum. valgrind is happy at completion, but top reports persistently increasing RAM usage. I tried pre-allocating XtX, eig and s, but it made no difference. Can only guess that some of the Eigen calls are requesting memory that's not being immediately freed; e.g. .adjoint, .eigenvalues(), .asDiagonal().

[jdtournier]

OK, I just had a look into this, and I can't see anything particularly alarming here - at least on my system, running 3.0.2. I'm not sure which fields you're looking at with top, but hopefully it'll match what I'm going to describe here, with the conclusion that there's no problem to worry about...

For reference this is what I did:

• Collect stats about memory usage once per second using ps in one terminal:

  while [[ 1 ]]; do ps h -C dwidenoise -o rsz,vsz,sz; sleep 1; done > log.txt
  

• Run dwidenoise (with a reduced thread count to get sufficient data points) in another terminal:

  dwidenoise dwi.mif dwi_denoised.mif -nthread 4
  

• Allow dwidenoise to run to completion, then plot using gnuplot (first need to clean file slightly due to presence of odd control sequence characters produced when dwidenoise is not found by ps):

  gnuplot> set key top right outside
  gnuplot> plot "log.txt" u 1 w l t "resident set size (RSZ)", "log.txt" u 2 w l t "virtual memory size (VSZ)", "log.txt" u 3 w l t "core size (SZ)"
  

Which gives this:

The main thing to note is that the VSZ is not budging at all: the application has allocated (or memory-mapped) all the memory is needs upfront, and that allocation does not change thereafter. The core image size (which includes the stack) doesn't budge either, so there's no leak on that front. The resident set does increase linearly through processing, though - but that's expected.

Understanding what happens requires some understanding of virtual memory management on modern OS's (there's a pretty good post on the topic on this thread, but plenty of information elsewhere too if you look for it). When memory is allocated, but not yet used, this does NOT consume physical memory: the OS keeps a note that the process has allocated memory pages, but only when the process actually tries to access these pages does the OS respond and provide a blank page of physical RAM. This is all transparent to the process, and allows the OS to over-allocate way beyond its physical RAM, and only fail when actual resource use exceeds capacity. This is also how memory-mapping works by the way: the kernel keeps track of the fact that some portion of the process' address space maps to some file, but it's only when the process tries to access these memory regions that the kernel actually load the file's data onto fresh physical pages for the process to use.

In our case, we have a big allocation for the output image (well, technically a memory-mapping, most of the time, but that doesn't really make much difference here). None of the pages from that allocation need to be resident at that point, but they are counted as part of the process' virtual memory. As processing proceeds, the process starts to write to the output file, and the kernel makes fresh physical memory pages available to the process as needed, and these fresh physical pages become part of the resident set. So we see the resident set size (RSS) increase during processing, with no impact on the overall virtual memory size, confirming that the process does not allocate any further memory during processing.

So to cut a long story short: I don't see anything out of the ordinary here... Also, even on a memory-constrained system, and assuming both input and output are accessed via memory-mapping, the OS will be able to free up physical RAM during processing simply by recycling pages that have already been committed to file. So even in that situation, the process should complete successfully (though it may take substantially longer if it needs to write to disk all the time). Furthermore, if the files are not being accessed through memory-mapping, but there is enough swap space, this should still work...

[dchristiaens]

Thanks @Lestropie and @jdtournier for looking into this. It's reassuring to know that all seems fine, but it doesn't explain the reported segfault on the forum. The first thing I can think of is that this may depend on the Eigen version. Jan reported using Eigen 3.3.7 (forum), which I haven't tested yet. @jdtournier, which Eigen version did you use?

[jdtournier]

$ dwidenoise -version
== dwidenoise 3.0.2-4-g994abf74 ==
64 bit release version, built Oct 14 2020, using Eigen 3.3.7
...

[Lestropie]

Yeah, I'm seeing the same thing also on 3.3.7. I was just manually inspecting %MEM on the interactive top output, not committing any further thought to it, particularly as it lined up with the user report. Maybe we need separate "bug?" and "bug!" issue labels.

[dchristiaens]

As @jdtournier found, the real issue reported on the forum was that the selected patch size exceeded the image dimensions. This is now resolved with #2230. I suggest we close this one.