freeimage may not release memory

[Al92130]

I try to convert a huge file of raw video data into many multipage TIFF files by using the "freeimage.write_multipage()" function of the freeimage package from the Mahotas library (Python 2.7). Unfortunately, it seems that this "very easy to use" function doesn't release memory when running the script. So, my script works fine for small input raw files (less than 1 GB) but crashes with bigger files (a 3 GB input file crashes with Win XP pro 32 - ram 3.2 GB). My goal is to convert input files up to 1.5 TB. An extract of the script below:

When I run my script, the Windows Task manager shows an increase of the used ram, output file after output file until the crash which release all the used ram :

An extract of the reported error :

[luispedro]

My guess is that this could be solved by specifying keep_cache_in_memory=False to write_multipage. You need to use the following branch.

https://github.com/luispedro/mahotas/compare/freeimage_keep_cache

Let me know if it works and I will merge to master

Tx
Luis

[Al92130]

Thank you for helping me.
As you recommended, I made the following modifications to the freeimage function located in ..\Python27\Lib\Site-packages\mahotas\io :
line 605 : def write_multipage(arrays, filename, flags=0, keep_cache_in_memory=True)
line 617 : # keep_cache_in_memory = True

According this, my script was modified as follows :

image list saved

freeimage.write_multipage(list_of_images, CurrentFileToStore, flags=0, keep_cache_in_memory=False)
liste_images = []

I didn't get successfull improvment, so I tried to modify again the line 605 with :
def write_multipage(arrays, filename, flags=0, keep_cache_in_memory=False)

Result is the same : I get a "Memory allocation failed" and memory monitoring from the Taskmanager remains the same.

Al

[luispedro]

Hmm, sorry, then. I don't think you'll be able to do it with mahotas/freeimage.

[luispedro]

I'm closing this as I don't think it's doable within the current infrastructure. Best of luck, Luis

[Al92130]

If you're able to reopen this issue, I would like to suggess the following evolution inside the write_multipage script (coded inside freeimage.py):
we have at line 628 : "_FI.FreeImage_AppendPage(multibitmap, bitmap)"
we tried to insert in line 629 : "_FI.FreeImage_Unload(bitmap)".
This added line is a copy of the line 681 (definition of the "_array_to_bitmap(array)" function).
This modification has been added to the modified freeimage library (first modification with added argument "keep_cache_in_memory") for the tests.

We didn't made a lot of trials, but raw data used for testing are now processed up to the end of the input file, output data seem to be not corrupted and the Taskmanager shows a constant used level of RAM during the process.
Many thanks to Remy.

Let us know your evaluation about this freeimage evolution.
Regards,
Al

[luispedro]

Oh, I'm always willing to consider patches...

Do you have any documentation that this is the right way to call the freeimage API? I am traveling and had only time for a cursory look through their docs.

[Al92130]

The FreeImage_Unload() function is also used in the write() function in the freeimage.py script, line 602.
The FreeImage documentation (ver 3.16.0) - FreeImage_Unload() description warns the user about the use of bitmap date manipulation:

I keep on testing this: with an input file of 11 GB (raw video where pixels are coded as UINT16), my script runs without any memory crash; the raw input file is transformed into tiff multipage files of 1000 images each. the Task manager gives me something like that :

Al