Availability of UMat and "Transparent API" functionality from OpenCV?

[wongjoel]

Does JavaCV currently support the use of the Transparent API by passing the UMat data structure in to algorithms? If so, how is this done correctly? If not, is there a large amount of work that would need to be done to make this possible?

My understanding is that with OpenCV 3.x, the preferred way to do hardware acceleration is via the Transparent API. In theory, this should only require a change in code from using the Mat data structure to the UMat data structure. But when I tried this with JavaCV 1.2, I found that the JavaCV methods expected a Mat, and would not accept a UMat, making it impossible to make use of the Transparent API to suggest hardware acceleration.

If this hasn't been implemented in JavaCV yet, is there anything that can be done to help speed the process of implementing it?

Thanks,

[saudet]

There's no reason why it couldn't work, but I guess we'll need to update things a bit. Adding "UMat" next to "Mat" on this line might do the trick:
https://github.com/bytedeco/javacpp-presets/blob/master/opencv/src/main/java/org/bytedeco/javacpp/presets/opencv_core.java#L199
Let me know if that works! Thanks

[wongjoel]

Thanks. I tried adding UMat in on that line (i.e. I edited the line to be as follows)

.put(new Info("cv::InputArray", "cv::OutputArray", "cv::InputOutputArray", "cv::_InputOutputArray").skip()./*cast().*/pointerTypes("Mat", "UMat"))

But after doing that, javacpp-presets no longer builds successfully - I'm not even quite sure I understand the implication of the error message error: operands to ?: have different types ‘cv::Mat_<double>’ and ‘cv::UMat’.

Here are the last 70 lines of output that I got from running mvn clean install --projects .,opencv,ffmpeg:

[INFO] 
[INFO] --- maven-dependency-plugin:2.10:copy-dependencies (copy-dependencies) @ opencv ---
[INFO] Copying javacpp-1.2.5-SNAPSHOT.jar to /home/celthana/javacpp-presets/target/javacpp.jar
[INFO] 
[INFO] --- maven-resources-plugin:2.7:resources (default-resources) @ opencv ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ opencv ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 23 source files to /home/celthana/javacpp-presets/opencv/target/classes
[INFO] /home/celthana/javacpp-presets/opencv/src/main/java/org/bytedeco/javacpp/helper/opencv_core.java: Some input files use or override a deprecated API.
[INFO] /home/celthana/javacpp-presets/opencv/src/main/java/org/bytedeco/javacpp/helper/opencv_core.java: Recompile with -Xlint:deprecation for details.
[INFO] /home/celthana/javacpp-presets/opencv/src/main/java/org/bytedeco/javacpp/helper/opencv_core.java: Some input files use unchecked or unsafe operations.
[INFO] /home/celthana/javacpp-presets/opencv/src/main/java/org/bytedeco/javacpp/helper/opencv_core.java: Recompile with -Xlint:unchecked for details.
[INFO] 
[INFO] --- javacpp:1.2.5-SNAPSHOT:build (javacpp.compiler) @ opencv ---
[INFO] Detected platform "linux-x86_64"
[INFO] Building for platform "linux-x86_64"
[INFO] Generating /home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jnicvkernels.cpp
[INFO] Compiling /home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/linux-x86_64/libjnicvkernels.so
[INFO] g++ -I/home/celthana/javacpp-presets/opencv/cppbuild/linux-x86_64/include/ -I/usr/lib/jvm/java-8-openjdk-amd64/include -I/usr/lib/jvm/java-8-openjdk-amd64/include/linux /home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jnicvkernels.cpp -march=x86-64 -m64 -O3 -s -msse3 -ffast-math -Wl,-rpath,$ORIGIN/ -Wl,-z,noexecstack -Wl,-Bsymbolic -Wall -fPIC -shared -o libjnicvkernels.so -L/home/celthana/javacpp-presets/opencv/cppbuild/linux-x86_64/lib/ -Wl,-rpath,/home/celthana/javacpp-presets/opencv/cppbuild/linux-x86_64/lib/ -lopencv_imgproc -lopencv_core 
[INFO] Deleting /home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jnicvkernels.cpp
[INFO] Generating /home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jniopencv_bioinspired.cpp
[INFO] Compiling /home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/linux-x86_64/libjniopencv_bioinspired.so
[INFO] g++ -I/home/celthana/javacpp-presets/opencv/cppbuild/linux-x86_64/include/ -I/usr/lib/jvm/java-8-openjdk-amd64/include -I/usr/lib/jvm/java-8-openjdk-amd64/include/linux /home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jniopencv_bioinspired.cpp -march=x86-64 -m64 -O3 -s -Wl,-rpath,$ORIGIN/ -Wl,-z,noexecstack -Wl,-Bsymbolic -Wall -fPIC -shared -o libjniopencv_bioinspired.so -L/home/celthana/javacpp-presets/opencv/cppbuild/linux-x86_64/lib/ -Wl,-rpath,/home/celthana/javacpp-presets/opencv/cppbuild/linux-x86_64/lib/ -lopencv_bioinspired -lopencv_highgui -lopencv_videoio -lopencv_imgcodecs -lopencv_imgproc -lopencv_core 
[INFO] Deleting /home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jniopencv_bioinspired.cpp
[INFO] Copying /home/celthana/javacpp-presets/opencv/cppbuild/linux-x86_64/lib/libopencv_bioinspired.so.3.1
[INFO] Generating /home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jniopencv_calib3d.cpp
[INFO] Compiling /home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/linux-x86_64/libjniopencv_calib3d.so
[INFO] g++ -I/home/celthana/javacpp-presets/opencv/cppbuild/linux-x86_64/include/ -I/usr/lib/jvm/java-8-openjdk-amd64/include -I/usr/lib/jvm/java-8-openjdk-amd64/include/linux /home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jniopencv_calib3d.cpp -march=x86-64 -m64 -O3 -s -Wl,-rpath,$ORIGIN/ -Wl,-z,noexecstack -Wl,-Bsymbolic -Wall -fPIC -shared -o libjniopencv_calib3d.so -L/home/celthana/javacpp-presets/opencv/cppbuild/linux-x86_64/lib/ -Wl,-rpath,/home/celthana/javacpp-presets/opencv/cppbuild/linux-x86_64/lib/ -lopencv_calib3d -lopencv_features2d -lopencv_ml -lopencv_flann -lopencv_highgui -lopencv_videoio -lopencv_imgcodecs -lopencv_imgproc -lopencv_core 
[INFO] Deleting /home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jniopencv_calib3d.cpp
[INFO] Copying /home/celthana/javacpp-presets/opencv/cppbuild/linux-x86_64/lib/libopencv_calib3d.so.3.1
[INFO] Generating /home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jniopencv_core.cpp
[INFO] Compiling /home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/linux-x86_64/libjniopencv_core.so
[INFO] g++ -I/home/celthana/javacpp-presets/opencv/cppbuild/linux-x86_64/include/ -I/usr/lib/jvm/java-8-openjdk-amd64/include -I/usr/lib/jvm/java-8-openjdk-amd64/include/linux /home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jniopencv_core.cpp -march=x86-64 -m64 -O3 -s -Wl,-rpath,$ORIGIN/ -Wl,-z,noexecstack -Wl,-Bsymbolic -Wall -fPIC -shared -o libjniopencv_core.so -L/home/celthana/javacpp-presets/opencv/cppbuild/linux-x86_64/lib/ -Wl,-rpath,/home/celthana/javacpp-presets/opencv/cppbuild/linux-x86_64/lib/ -lopencv_imgproc -lopencv_core 
/home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jniopencv_core.cpp: In function ‘_jobject* Java_org_bytedeco_javacpp_opencv_1core_00024DownhillSolver_create__Lorg_bytedeco_javacpp_opencv_1core_00024MinProblemSolver_00024Function_2Lorg_bytedeco_javacpp_opencv_1core_00024UMat_2Lorg_bytedeco_javacpp_opencv_1core_00024TermCriteria_2(JNIEnv*, jclass, jobject, jobject, jobject)’:
/home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jniopencv_core.cpp:2931:150: error: operands to ?: have different types ‘cv::Mat_<double>’ and ‘cv::UMat’
         PtrAdapter< ::cv::DownhillSolver > radapter(::cv::DownhillSolver::create((cv::Ptr< ::cv::MinProblemSolver::Function >)adapter0, ptr1 == NULL ? cv::Mat_<double>(1,1,0.0) : *ptr1, ptr2 == NULL ? cv
                                                                                                                                                      ^
/home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jniopencv_core.cpp: In function ‘_jfloatArray* Java_org_bytedeco_javacpp_opencv_1core_cvPointTo32f___3I(JNIEnv*, jclass, jintArray)’:
/home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jniopencv_core.cpp:64259:15: warning: unused variable ‘rowner’ [-Wunused-variable]
         void* rowner = (void*)rptr;
               ^
/home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jniopencv_core.cpp:64260:16: warning: unused variable ‘deallocator’ [-Wunused-variable]
         void (*deallocator)(void*) = &JavaCPP__3F_deallocate;
                ^
/home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jniopencv_core.cpp: In function ‘_jobject* Java_org_bytedeco_javacpp_opencv_1core_cvPointTo32f__Ljava_nio_IntBuffer_2(JNIEnv*, jclass, jobject)’:
/home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jniopencv_core.cpp:64292:15: warning: unused variable ‘rowner’ [-Wunused-variable]
         void* rowner = (void*)rptr;
               ^
/home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jniopencv_core.cpp:64293:16: warning: unused variable ‘deallocator’ [-Wunused-variable]
         void (*deallocator)(void*) = &JavaCPP_java_nio_FloatBuffer_deallocate;
                ^
/home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jniopencv_core.cpp: In function ‘_jintArray* Java_org_bytedeco_javacpp_opencv_1core_cvPointFrom32f___3F(JNIEnv*, jclass, jfloatArray)’:
/home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jniopencv_core.cpp:64347:15: warning: unused variable ‘rowner’ [-Wunused-variable]
         void* rowner = (void*)rptr;
               ^
/home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jniopencv_core.cpp:64348:16: warning: unused variable ‘deallocator’ [-Wunused-variable]
         void (*deallocator)(void*) = &JavaCPP__3I_deallocate;
                ^
/home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jniopencv_core.cpp: In function ‘_jobject* Java_org_bytedeco_javacpp_opencv_1core_cvPointFrom32f__Ljava_nio_FloatBuffer_2(JNIEnv*, jclass, jobject)’:
/home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jniopencv_core.cpp:64380:15: warning: unused variable ‘rowner’ [-Wunused-variable]
         void* rowner = (void*)rptr;
               ^
/home/celthana/javacpp-presets/opencv/target/classes/org/bytedeco/javacpp/jniopencv_core.cpp:64381:16: warning: unused variable ‘deallocator’ [-Wunused-variable]
         void (*deallocator)(void*) = &JavaCPP_java_nio_IntBuffer_deallocate;

Any ideas of what is wrong or what to try next?

[saudet]

Looks like there is a problem in the way the default argument for that function gets used. I'll have to think about that...

[saudet]

I've made some changes to JavaCPP and it seems to work fine now. Let me know if you encounter other problems! Thanks

[wongjoel]

Thanks, it seems to work to some extent now, but there are some quirks.

I've found at least one change in method signatures that doesn't seem quite right.
org.bytedeco.javacpp.opencv_highgui.imshow now can be called with:

• imshow(Bytepointer winname, Mat mat)
• imshow(String winname, UMat mat)
• imshow(Bytepointer winname, Texture2D tex)
• imshow(String winname, UMat mat)

But I can no longer call imshow(String winname, Mat mat)

Hopefully this is an isolated problem, but I'm a bit concerned that there might be other cases of this kind of thing happening.

The example from https://www.learnopencv.com/opencv-transparent-api/ doesn't work because org.bytedeco.javacpp.opencv_imgproc.cvtColor doesn't have a signature that matches cvtColor(Mat src, UMat dst, int code). (I chose that one as a quick test since it's a short snippet of code)

I'm planning to convert some of the examples in https://github.com/opencv/opencv/tree/master/samples/tapi to check if the "official" examples work.

[saudet]

You could try to put another "Mat" before the "UMat" in the Info and see what that gives.

Where in the examples are they calling cvtColor() with both a Mat and a UMat? I don't see that line, and I don't think it's possible either.

[wongjoel]

Sorry, I got confused with the cvtColor() thing - I was was using test code that was only half converted to using UMat and thought the conversion process was already complete.

I tried adding a second "Mat" before the "UMat", and things change, but it's still a bit odd.
With the following edit

.put(new Info("cv::InputArray", "cv::OutputArray", "cv::InputOutputArray", "cv::_InputOutputArray").skip()./*cast().*/pointerTypes("Mat", "Mat", "UMat"))

the signatures for imgshow are:

• imshow(Bytepointer winname, Mat mat)
• imshow(String winname, Mat mat)
• imshow(Bytepointer winname, UMat mat)
• imshow(Bytepointer winname, Texture2D tex)
• imshow(String winname, UMat mat)

So then I got the idea of then adding another "UMat" after the first "UMat":
So with the following edit

.put(new Info("cv::InputArray", "cv::OutputArray", "cv::InputOutputArray", "cv::_InputOutputArray").skip()./*cast().*/pointerTypes("Mat", "Mat", "UMat","UMat"))

the signatures for imgshow are then:

• imshow(Bytepointer winname, Mat mat)
• imshow(String winname, Mat mat)
• imshow(Bytepointer winname, UMat mat)
• imshow(String winname, UMat mat)
• imshow(Bytepointer winname, Texture2D tex)
• imshow(String winname, UMat mat)

Which I think are all the expected ones.

I don't know how the presets work (I haven't read up on that part yet), so I have no idea why that worked, but at least it makes some kind of sense.

[wongjoel]

Unfortunately, when using "Mat", "Mat", "UMat","UMat", I get a new problem.

.put(new Info("cv::InputArray", "cv::OutputArray", "cv::InputOutputArray", "cv::_InputOutputArray").skip()./*cast().*/pointerTypes("Mat", "Mat", "UMat","UMat"))

With that edit, I'm no longer able to do use .copyTo() to convert from Mat to UMat (i.e. img_mat.copyTo(img_umat); is no longer allowed, as .copyTo() only accepts a Mat argument and not a UMat argument)

[saudet]

Right, fixed in the commit above. Thanks for reporting!

[saudet]

Support included in version 1.3. Thanks for the feedback!