Open CV Mat to/from libvips VipsImage

[fangqiao]

Hi,
I just started to learn libvips. I think that it has much better API for image format and meta data handling, while opencv's Mat is extremely flexible and powerful for pixel manipulation. It would be greate if the two can work together at memoery leve. I went through libvips documentation and found this one :

vips_image_new_from_buffer (const void *buf,
                            size_t len,
                            const char *option_string,
                            ...);

which seems to do the trick.

But I can't find the documentation for defining option_string to tell libvips how many channels and the sequence of channels as RGBA or ARGB etc. as defined by the Mat object , or if the Mat object should have certain spefic sequence of channels as RGBA.

I am wondering if anyone tried to make libvips work with opencv before and would like to share with me the right way to do it or point me to the right documentaiton. Some sample code would be much appreciated.

Thanks in advance.

[jcupitt]

Hello @fangqiao,

new_from_buffer expects a formatted image, so JPEG or TIFF, something like that. You probably need a simple C-style memory array, so you need vips_image_new_from_memory():

http://jcupitt.github.io/libvips/API/current/VipsImage.html#vips-image-new-from-memory

You give that a pointer to an area of memory, plus width, height, bands and format (uchar, float, etc.). libvips expects images to be band-interleaved, so a 2x2 RGB image like this:

R1G1B1	R2G2B2
R3G3B3	R4G4B4

needs to be in memory like this:

R1G1B1R2G2B2R3G3B3R4G4B4

If Mat does something else, you'll need to rearrange pixels, either in libvips or OpenCV.

Could you give an example of what you plan to use Mat for? Perhaps there's a way to do it in libvips?

[fangqiao]

Hi @jcupitt thanks for the quick following up.

I have some image handling program written with opencv and I hope to use libvips' powerfull APIs in meta data and image output for internet publishing optimization. Sometimes I also need to turn Mat data structure to cairo surface to use Cairo's canvas APIs for drawing and also with pango for complex text painting. Mat is really handy, as it provides a suite of APIs to manipulate Mat data structure for specific layouts required by other image libraries. Maybe libvips data structure can do but I need sometime to find that out.

So new vips_image_new_from_memory is the anwser. What if the Mat object has four channels, R,G,B and A, how should the channels be arranged for libvips?

Thanks

[jcupitt]

Channels should be RGBA for libvips. You can have any number of channels, but alpha is always last.

libvips has vips_text() for pango:

http://jcupitt.github.io/libvips/API/current/libvips-create.html#vips-text

There's an example of painting to a Cairo surface from libvips here:

https://github.com/jcupitt/vipsdisp-tiny

But of course OpenCV is very powerful and useful, I agree that linking it to Mat is a good idea. To go the other way, there's vips_image_write_to_memory(), you probably found it.

[fangqiao]

@jcupitt I have done some experiments with libvips. It looks it has covered a majority of my current program based on opencv. But there are two cases that I haven't found a straight forward solution with libvips:

1. dissolve - with opencv, I used some loops to manually calculate overlayed pixel's RGBA values based on dissolve factor, source image and overlay image. I found out from older issues that your approach was to create an alpah channel or update existing alpha chanel, premultiply then use composite to blend with the source image. And I found out you added an vips_addalpha but it seems your c++ wrapper doesn't support it yet.

error: ‘class vips::VImage’ has no member named ‘addalpha’

How to unwrap C++ objec to c so I can use the c function and then wrap it back to c++ object?

Edit:

I have figured out a way to do arbitrary dissolve with your arithmetic apis :

• define dissolve factor, and an linear algorithm of how to calculate pixel bands based on dissolve. could be simple weighted average or other way.
• extract the area for overlay from the target image
• use vips_linear on both the extracted area and the overlay image then use vips_add to get dissolved area
• insert back the dissolved area back.

Also figured out how to wrap and unwrap c++ vimage object, i.e. get_image() and new VImage(VipsImage).

2. direct pixel manipulation - I noticed that your design is leaning toward functional style. VipImages is immutable. Every operation generates a new VipImages. Is there a way to do direct pixel manipulation in place?

Thanks.

[jcupitt]

Hello again,

For dissolve etc. you can just use the C++ operator overloads:

VImage a = VImage::new_from_file("a.jpg");
VImage b = VImage::new_from_file("b.jpg");
double scale = 0.7;

VImage c = scale * a + (1 - scale) * b;

It should be quick, though of course you should test it. The scale factor could also be an image, if you need full RGB blending.

The composite operator implements all the PDF blending modes for stacks of RGBA images, which might be more what you need.

You can add an alpha with:

VImage aa = a.bandjoin(255);

ie. append a constant 255 band.

You can use a.get_image() to get the VipsImage* pointer out of a VImage object, and VImage(v) to wrap a VImage object around a VipsImage* pointer. Have a look at the VImage header, it's got lots of comments and it should be easy to read.

Yes, libvips is functional everywhere by design, you can't modify images at all. It makes the caching and threading system much, much simpler and more efficient.

You can render an image to a memory area, manipulate it in any way you want, then wrap a vips image around it again. Of course that'll need a lot of memory :( You can implement your own libvips operators pretty easily, there are some examples in the manual.

Yes, the C++ wrapper needs hasalpha and addalpha, I've just not got around to it :(

[fangqiao]

Yes the operator overloading works. Thanks for your promote responses. Feel free to close the issue.

[fangqiao]

@jcupitt Sorry for still posting on this closed issue.

I actually am a fan of functional programming. On this image overlay task, and in general image pixel iteration, classic functional pattern of iterator/mapping/zipping/fold etc. can be used if your arithmetic operation plays higher order function and allows custom provided combinators, for example, such a r function like linear( image1, image2, operator), and pixel index, pixel channels and values of both images are passed to operator as paramter and expect operator to return calculated new pixel channels and values.

It would provide much more flexibility to developers.

[jcupitt]

That's not really possible to implement with good performance: you'd get a function call on every pixel. The libvips idea is to implement a complete set of basic vector operations, plus efficient composition. libvips gets quite close to native speed, it's less than a factor of two I think.

You can also add your own loops to the pipeline pretty easily, I'll make you an example.

[jcupitt]

Here's a how to loop over pixels demo:

https://gist.github.com/jcupitt/05b2627e7677bd23a356a71578a83dce

You define a function which can generate an area of output pixels on demand, then attach that to an image.

[fangqiao]

Yes you are right. Higher order function approach is costly performance wise. Thanks for the example.

[fangqiao]

I have tested the conversion between mat and vips. It works. Below is a few lines of codes for your reference.

It is for uchar only.

#include <opencv2/opencv.hpp>
#include <vips/vips8>

Mat* vipsToMat( VImage& in ) {

    int channels = 0;

    switch ( in.bands() ) {
    case 1 :
        channels = CV_8UC1;
        break;
    case 2 :
        channels = CV_8UC2;
        break;

    case 3 :
        channels = CV_8UC3;
        break;

    case 4 :
        channels = CV_8UC4;
        break;

    default :
        return NULL;

    }
    size_t size;

    void* vipsData = in.write_to_memory(&size);

    Mat* m = new Mat( in.height(), in.width(), channels, vipsData);
    return m;

}

VImage matToVips( Mat* mat ) {

    return VImage::new_from_memory( mat->data, mat->elemSize() * mat->cols * mat->rows, mat->cols, mat->rows, mat->channels(), VIPS_FORMAT_UCHAR);

}

[jcupitt]

In vipsToMat you'll need to arrange for vipsData to be freed later, somehow, or you'll have a terrible leak. Is there a finalize hook on Mat you can use?

[fangqiao]

Yes you are right. The code is a just test run, which needs a lot of improvement for production

And I was about to propose an enhancement to your write_to_memory function, to allow writing to a memory preallocated by caller, instead of by vips, which will give caller more flexibilities for meory reuse and preallocation of buffer for performance improvement.