Multithreaded noise reduction leaks memory #4416
Comments
Beep6581
added
the
type: bug
|
Note: even with the number of threads set to 1, enabling the wavelet module will also leak about 200 MB of memory per save as well. |
|
@PkmX Can not reproduce |
|
Confirmed, though the leak varies. I'll try to find something with the sanitizer... |
|
Win10 RT 5.4-RC1 Cannot reproduce after processing 100 raw photos with noise reduction and use of 8 threads. |
|
No leak with ASAN in debug and release build. Rebuilding now to check RSS rather than using |
|
No leak regarding RSS here. Sorry, but there must be something different with your setup, @PkmX, and I was wrong with my first impression. Best, |
|
Hmm, I can reliably reproduce it here and the RSS goes up after each save. For the record, I'm applying noise reduction + wavelet to a 24MP JPEG image. The settings in the performance tab are:
|
|
@PkmX can you upload the JPG and your options file using filebin.net? |
|
On mobile now. I can probably upload later. However I can reproduce the leak with any JPG or RAW, so the image likely doesn't matter. You can download the X-T2 compressed RAF (which I process these days) or Sony a6000's ARW from raw.pixls.us. I tested both and they all have the same leak. |
This can be done in two ways:
or
I don't know how you export, but can you try the other way too? Mabye we just testing differently... |
|
@heckflosse Both methods leak memory from what I tested.
Start: Add to queue 10 times and start processing: Process another 10 from queue again: For comparison, with number of threads = 1: After exporting 10: |
|
@PkmX I tried it with your PP3 and options, but still no luck: As I assume you are familiar with compiling RT, could you give ASAN a try? There are always some small leaks from libfontconfig and the like, but we're interested in the big ones. 😄 Best, |
|
ASan/LSan aren't really useful for this kind of leaks (only shows a few KBs of leak here and there). The memory is likely not leaked (there is a still a reference to it somewhere), but it is being unnecessarily retained thus increasing RSS usage over time. I will see if I can find anything useful with |
|
I can't reproduce the leak when using I modified Rawtherapee to call Notice that the Taking a closer look, glibc allocates 64 arenas for my system (8 threads * 8 arenas per thread) and some of them have some hundred MBs allocated with little in use: It is not clear to me why glibc isn't actively trimming the extra space for these arenas. My speculation is that the particular allocation pattern by Rawtherapee triggers heap fragmentation, and this is amplified by the number of arenas. The size also happens to be smaller than the threshold to make glibc use The workaround is setting Any thoughts? (For the record, I'm using glibc 2.26.) |
|
Might be I'm in here, too. With the difference that 1) it also happens when just opening and closing an unedited RAW file with default settings (no NR, no sharpening, no wavelets, ...) will increase the memory usage: and 2) the workaround with those env variables (here: MALLOC_ARENA_MAX=1) is of limited use: Furthermore it degrades performance quite severely. I have to say that I had similar issues in the past with several applications. tmux collected memory like mad when resizing the xterm it was running in, the ENV vars for glibc didn't help, I had to use a patch for tmux to manually call malloc_trim. gcc used enormous amounts of memory, I had to do a fresh reboot into plain terminal when I wanted to do a Gentoo update. And clang used in QtCreator/YouCompleteMe/... as backend for completion was unusable... But all this magically settled at some point, all behave well. So I currently don't know what happens with RT - I don't think there is a (big) memory leak, would have been detected by others I think. Probably it's just the way it allocates memory that triggers the problem here on my machine. |
|
I'm seeing a quite severe increase in memory usage as well (using the Appimage, both 5.4 stable and -dev, on Ubuntu bionic). Additionally, the oom killer seems not to be able to deal with it, causing the whole system to freeze when it finally runs out of memory, with SysRq-REISUB being the only way out. I'm not sure whether it is related to multi-threaded NR, though. |
|
Preloading jemalloc also solves the glibc arena issue for me as well as tuning the env variable: further suggesting that it is glibc's arenas causing the issue. |
|
@thirtythreeforty I got a segfault after running RawTherapee a while with jemalloc. I'm confused by that because I thought it was supposed to be a drop in replacement for malloc. Trying to see if I can recreate the segfault with the glibc malloc but so far nothing. |
|
Post the backtrace, if you can. Either the jemalloc or the RT devs will probably consider that a bug. |
|
Here it is; debug build of the latest -dev. |
|
@afontenot Is it thread 1 that segfaulted? The preloaded jemalloc hangs on exit here. Could not reproduce a SEGV, yet. |
|
@Floessie I think it was thread 1, but I'm not sure how I could confirm that if the backtrace itself doesn't. I don't do much debugging of multithreaded applications. If there's additional information I can give you by crashing it again, let me know. I can probably reproduce the crash. |
|
@afontenot Normally, when GDB catches a SEGV a
That would be helpful. 👍 |
|
@Floessie I generated the backtrace with Which is what's suggested here: https://rawpedia.rawtherapee.com/How_to_write_useful_bug_reports The backtrace for the thread 1 is the last one in my log. I think that's the one that got SEGV, given that gdb has it marked as the current thread. I can try to generate another crash though. |
|
|
|
@Beep6581 Yep, this is what @afontenot already mentioned. So the @afontenot Another trace could either back or contradict it. I'd be glad if you could provoke another crash under GDB. |
|
I need to further investigate on this issue. This gets so bad that I can't edit a single image in one go, I have to close RT and lose the whole editing history. It's a matter of minutes that RT eats up my 8 GB of RAM. This is with LocalLab branch. There seem to be some "definitely lost" bytes. One thing I just now realised: Most memory seems to be lost when panning around at 100%. I just activated a whole lot of tools. Panning around will have a higher peak usage then but that drops down to about where I am with panning a non-edited file with neutral profile. That memory also won't be freeded up when the editor for that image is closed. // edit: ah, RT with jemalloc doesn't crash when directly opening the editor (pass file via cmdline). This keeps memory usage down. So it could be just the tile management isn't friendly for glibc memory management. |
To reproduce:
Perferences,Performance & Quality, setMax number of threads for Noise Reduction and Wavelet Levelsto0or8.On my system (Arch Linux, i7-8550U, 4 cores, 8 threads), each save makes Rawtherapee use 200~400 MB more memory until it gets killed by OOM. Setting number of threads to 1 produces no (or very little of) leaks.
The text was updated successfully, but these errors were encountered: