Question about saving and re-opening an image from a memory array

[aferrero2707]

In my photoflow editor I am internally building a pre-computed pyramid of zoom levels whenever I open an image from disk, to speed-up rendering of scaled-down previews.

Currently I am writing each level into a RAW disk buffer, which I re-open with vips_rawload().
For next version, I would like to limit the disk usage and use memory buffers for not-so-large images, and only revert to disk buffers whenever the image would be too large to be kept into RAM.

The solution seems to be simple: use vips_image_write_to_memory() to save the data, and re-open the memory array with either vips_image_new_from_memory() or vips_image_new_from_memory_copy().

The _copy() version has the advantage to avoid any risk of freeing the memory while it is still in use, however it requires a temporary double-allocation of the pixel array which I would like to avoid.

So I was thinking about a way to use vips_image_new_from_memory() and still let VIPS manage the freeing of the memory buffer when it is not needed anymore. One solution I could figure out is to attach the memory array to the image itself as a metadata blob. In this case, the blob would then be deallocated together with the image, right? Would this be a safe and efficient solution?

Thanks!

[jcupitt]

You could also use the close signal on the image to free the memory, that might be easier.

http://www.vips.ecs.soton.ac.uk/supported/current/doc/html/libvips/VipsObject.html#VipsObject-postclose

Actually, you can do it even more simply: just make a memory image with vips_image_new_memory(), then write to that.

http://www.vips.ecs.soton.ac.uk/supported/current/doc/html/libvips/VipsImage.html#vips-image-new-memory

Something like:

VImage mem = VImage::new_memory();
x.write(mem);

Now mem is a memory buffer containing the contents of x.

Your "disc for large files, mem for small files" code might be something like:

VImage cache;

if (is_a_huge_file(x))
  cache = VImage::new_temp_file("%s.v");
else
  cache = VImage::new_memory();

x.write(cache);

You can use an env var to say where new_temp_file should make the temporary file:

http://www.vips.ecs.soton.ac.uk/supported/current/doc/html/libvips/VipsImage.html#vips-image-new-temp-file

[jcupitt]

Oh I forgot, sorry, .write() used to keep a ref to the source image, so you couldn't really use it like this. 8.4 and later make it only keep a ref if the image you are writing to needs it, which a memory or disc image won't.

Anyway, for 8.3 and earlier, yes, you need to make a memory buffer and free it, as you said. From 8.4, you can just use .write(), it's much neater.

[aferrero2707]

If I understand correctly, you propose the following:

• write a callback to free the memory array:

  void free_mem_array (VipsObject *object, gpointer user_data)
  {
  if( user_data ) free( user_data );
  }

• register the callback:

  VipsImage* img = vips_image_new_from_memory( ... );
  g_signal_connect( img, "postclose", G_CALLBACK(free_mem_array), mem_array );

Is that correct?

[jcupitt]

Yes, exactly. This is how vips_malloc() works:

https://github.com/jcupitt/libvips/blob/master/libvips/iofuncs/memory.c#L130

[aferrero2707]

I did a quick test, and it works like a charm. Thanks again!

[jcupitt]

Good stuff, I'll close.