Memory leak in PyTurboJPEG?

[trdastidar]

Hi,

There seems to be a case of unmanaged memory in PyTurboJPEG. Memory is allocated during the image decode process, but not deallocated later (at least not immediately), and the memory usage of the process keeps building up.

As an example, please consider the following code:

from turbojpeg import TurboJPEG, TJPF_GRAY, TJSAMP_GRAY
import psutil
import os
import time
jpeg = TurboJPEG("/usr/lib/x86_64-linux-gnu/libturbojpeg.so.0")
import numpy

i = 0
PROCESS = psutil.Process(os.getpid())

buf = None
with open('some_image.jpg', 'rb') as fin:
    buf = fin.read()
for i in range(100):
    print 'iteration', i
    img_array = jpeg.decode(buf)
    print img_array.shape
    print 'memory = ', PROCESS.memory_info().rss // 1024

The output of this program looks like this:

...
iteration 43
(2160, 4096, 3)
memory =  806464
iteration 44
(2160, 4096, 3)
memory =  832576
iteration 45
(2160, 4096, 3)
memory =  858472
iteration 46
(2160, 4096, 3)
memory =  884340
iteration 47
(2160, 4096, 3)
memory =  910188
iteration 48
(2160, 4096, 3)
memory =  936316
iteration 49
(2160, 4096, 3)
memory =  962216
iteration 50
...

The memory usage keeps building up till more than a GB. It sometimes goes back to a low value, but again the buildup happens.

The other popular turbojpeg python wrapper -- jpeg4py -- doesn't have this problem. However, I prefer using PyTurboJPEG because it offers a scaling factor in the decode function which speeds up things considerably for my use case. But the memory build up is causing problems as I have a tight memory budget.

What could be the reason and are there any known fixes/workarounds?

[lilohuang]

I’m on a vacation and traveling internationally, please try to use the latest TurboJPEG to have a try, I will try to reproduce the bug 10 days later. 從我的 iPhone 傳送

…

Tathagato Rai Dastidar ***@***.***> 於 2019年9月17日 13:58 寫道： x86

[trdastidar]

Thanks. I will await your reply.
Both PyTurboJPEG and TurboJPEG are at the latest versions (1.4.2-0ubuntu3.1).

[lilohuang]

IIUC, you’re using TurboJPEG 1.4.2 which is a quite old version, I suggest using 1.5.3 or later, you may download it or build the source code from https://sourceforge.net/projects/libjpeg-turbo/files/2.0.3/ BTW, PyTurboJPEG library has been used in some 24x7 mission-critical services, so far I didn’t see any memory leak issue though. I will still reproduce the issue with the same environment you’re using, please stay tuned. Thanks! 從我的 iPhone 傳送 Tathagato Rai Dastidar <notifications@github.com<mailto:notifications@github.com>> 於 2019年9月17日 14:32 寫道： Thanks. I will await your reply. Both PyTurboJPEG and TurboJPEG are at the latest versions (1.4.2-0ubuntu3.1). — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#12?email_source=notifications&email_token=AB7UOOEV4QZIWSICGM5ZHTDQKBTVHA5CNFSM4IXKUPPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD63K2BA#issuecomment-532065540>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AB7UOOGNFZYJ2ZU2Z7YQJ4TQKBTVHANCNFSM4IXKUPPA>.

[trdastidar]

I don't think the memory leak is at the TurboJPEG level, as the jpeg4py python wrapper works fine without any memory blowup. It is something to do with PyTurboJPEG.

I downloaded and installed 2.0.3 libjpeg-turbo. The problem persists.

[lilohuang]

The memory growing should be fixed. Please let me know if you still have question. Thank you.

[trdastidar]

Yes, the problem has been addressed. Thanks for your support.