FrameGrabber.setFormat() doesn't change format leading to bad performance

[AmnonOwed]

I am using JavaCV 1.3.2 on Windows 8 (64-bit) to capture and display in real-time images from my Logitech C922 Pro Stream Webcam. The images are displaying, however the problem is slow framerate at higher resolutions due to the default selected video decoding format.

Some background information: When I use this camera in OBS Studio 18.0.1 I can choose between two video formats: YUY2 and MJPEG. The latter gives much better performance because it is a compressed format. With YUY2 however the performance / framerate is too bad to use higher resolutions, such as 1280 x 720 or 1920 x 1080. So the camera does support framerates up to 60 and 30 fps respectively at those dimensions, however only using MJPEG.

Below is the relevant JavaCV code snippet of my program. While setting the image dimensions works perfectly, setting the format unfortunately does not. Even though I set the format to MJPEG, when I print the format to doublecheck, it still says YUY2.

Long story short, why is the format not changing and/or how can I change this? Of course the end goal being to achieve the actual performance the hardware is capable of.

Thank you for your time and effort!

    grabber = new OpenCVFrameGrabber("");
    grabber.setImageWidth(1280);
    grabber.setImageHeight(720);
    grabber.setFormat("MJPEG");
    grabber.start();
    System.out.println("format: " + grabber.getFormat()); // prints YUY2

[saudet]

OpenCV most likely doesn't support setting that.

Could you instead try to modify VideoInputFrameGrabber and see how we can change the format of your camera? It seems like calling setRequestedMediaSubType(VI_MEDIASUBTYPE_MJPG) is what we need... Thanks!

[AmnonOwed]

I tried using VideoInputFrameGrabber. It gives me an image stream, but based on the framerate and the console feedback below I believe it's still using YUY2:

***** VIDEOINPUT LIBRARY - 0.2000 - TFW2013 *****

SETUP: Setting up device 0
SETUP: C922 Pro Stream Webcam
SETUP: Couldn't find preview pin using SmartTee
SETUP: Default Format is set to 640 by 480
SETUP: trying requested format RGB24 @ 1280 by 720
SETUP: trying format RGB24 @ 1280 by 720
SETUP: trying format RGB32 @ 1280 by 720
SETUP: trying format RGB555 @ 1280 by 720
SETUP: trying format RGB565 @ 1280 by 720
SETUP: trying format YUY2 @ 1280 by 720
SETUP: Capture callback set
SETUP: Device is setup and ready to capture.

Based on the videoInput source, I then tried a modified CustomVideoInputFrameGrabber, which has the following extra call either right after myVideoInput = new videoInput(); or at the end of the start() method:

myVideoInput.setRequestedMediaSubType(18);

Unfortunately there is no improvement in framerate (I do still get an image stream). Not sure if I'm doing this correctly or if there's any other way to test if a setting was actually changed. I seems it's still using YUY2.

[AmnonOwed]

OpenCV most likely doesn't support setting that.

I've researched this and found multiple posts similar to mine with regard to OpenCV defaulting to YUY and people wanting it to use MJPEG. I also found two posts (here and here) with a solution allowing OpenCV to indeed use MJPEG.

In both cases the solution presented was to use FOURCC to set the codec before setting the desired resolution. Perhaps this is something worth exploring.

[saudet]

I see, so we should set CV_CAP_PROP_FOURCC to the specified "format" in start() then. Could you make the necessary modifications?

[AmnonOwed]

Yes, I did. Tried the OpenCV solution with a customized version of OpenCVFrameGrabber with the following line in the start method (to work it has to indeed be before setting the dimensions):

capture.set(CV_CAP_PROP_FOURCC ,CV_FOURCC((byte)'M', (byte)'J', (byte)'P', (byte)'G'));

This seems to get results. First, when using grabber.getFormat() it returns MJPEG instead of YUY2. Second, there is a significant performance increase:

• At 1280x720: from 13 fps (YUY2) to 30 fps (MJPEG)
• At 1920x1080: from 5 fps (YUY2) to 15 fps (MJPEG)

Definitely a big improvement! 😄 Compared with the hardware capability it's still about 50%, because it should be able to reach 60 and 30 fps respectively. Not sure if this is an OpenCV or java bottleneck.

[saudet]

Great, please send a pull request!