Cannot capture images in OpenCV in UVC mode

[nickaein]

Hi,

I have a DFK 42AUC03 camera and I want to grab images using OpenCV in Linux. Using euvccam-fw tool, I've updated the firmware to dmk42uc03_143.euvc and switched camera to UVC mode.

guvcview now gives a grayscale video stream in 1280x960. Here is log of running guvcview with --verbose switch: http://textuploader.com/kleq.

I couldn't find a way to get a color image in guvcview. Changing the Camera output to RGB3 caused guvcview to be closed with this error:

libv4l2: error dequeuing buf: Invalid argument
VIDIOC_DQBUF - Unable to dequeue buffer : Invalid argument

Also, I have a problem in capturing image using OpenCV. This simple code which capture image from the camera:

#include <stdio.h>
#include <opencv2/opencv.hpp>
#include <iostream>
using namespace cv;
using namespace std;

int main(int argc, char** argv )
{
    VideoCapture cap(0);
    Mat img;

    if(cap.isOpened())
        cout << "Device opened." << endl;
    else
        cout << "Device not opened." << endl;

    cap >> img;
    if(!img.empty())
    {
        imwrite("frame.bmp",img);
        cout << "Capture success!" << endl;
    }
    else
    {
        cout << "Capture failed." << endl;
    }

    return 0;
}

Gives the following errors:

libv4l2: error setting pixformat: Invalid argument
HIGHGUI ERROR: libv4l unable to ioctl S_FMT
libv4l2: error setting pixformat: Invalid argument
libv4l1: error setting pixformat: Invalid argument
HIGHGUI ERROR: libv4l unable to ioctl VIDIOCSPICT

Device not opened.
Capture failed.

I wonder if I should set some parameters before using VideoCapture object (I have tried settings resolution which didn't help and I couldn't find a way to set the pixel-format).

[TIS-Edgar]

Hi,

You will not be able to get a color image with guvcview at this point.
The problem is that the uvc driver has no knowledge what so ever about bayer formats and therefor cannot handle our color cameras correctly. That is why you need to use a dmk firmware.

By redeclaring the used format to bayer you can create a correct image as the image data is identical when it comes to gray and bayer. Since guvcview does not know about all of this it will do some weird things that never will be able to succeed and (sadly) crash.

Now to OpenCV:
The VideoCapture class is not worth the trouble. They only handle one format correctly (rgb if I remember correctly) and everything else fails.

I have published an example here:
https://gist.github.com/10f04501f49b6b3bf75e
There you will find working code that directly accesses v4l2 for image retrieval, declares the image buffer as bayer and offers it as IplImage. I did not yet have the chance to clean the code up, which is why it is not yet part of the repository.

[nickaein]

Thanks! That code did help. The only modification I have to do for the DFK 42AUC03 camera was to replace CV_BayerGB2RGB in the cvCvtColor with CV_BayerGR2RGB

[mdxx]

Hello, one colleague of yours gave me your code and i have one basic question. I need from my camera to take pictures, not video... I thought i could change main.cpp and deleting "while()" command so that the snapframe() function be executed only one time. Am i right or i have to do sth else. Also i need to change exposure and gain so i write this part of code:

 //setting exposure
ctrl.id = V4L2_CID_EXPOSURE;
ctrl.value = "an integer for exposure";
ioctl(fd, VIDIOC_S_CTRL, &ctrl);

//setting gain
ctrl.id = V4L2_CID_GAIN;
ctrl.value = "an integer for gain";
ioctl(fd, VIDIOC_S_CTRL, &ctrl);

Is is correct???
And finaly i need to save into a specific folder my picture so i have this:

imwrite( "/home/mdx/Desktop/example/image_name.jpg", pColorCVImage );

is it correct or i have to change sth?

[TIS-Edgar]

Your setting code should work. The only thing you need to change is V4L2_CID_EXPOSURE to V4L2_CID_EXPOSURE_ABSOLUTE as all our cameras use that flag.

From my experience imwrite should work as you expect.

To save a single image it should suffice to call snap_image once.

[mdxx]

Thanks a lot for your reply.

And i think i have a similar problem with guvcview aw i cannot see a color video, but only a greyscale. You said before i have to update drivers by using firmware dmk firmware. I think i have to do this from there:

https://github.com/TheImagingSource/tiscamera/wiki/Getting-Started-with-USB-Cameras---->
Switch Camera Interface Mode and Color Format

But can you tell me what is the correct firmware for DFK 23U274?

sudo ./euvccam-fw -u ../../data/firmware/usb2/dmk72uc02_129.euvc

[mdxx]

With snap_image you mean snapFrame() function am i right?

[TIS-Edgar]

Yeah, I meant snapFrame() sorry for that.

USB-3.0 cameras do not require a firmware change. They work out of the box.
If you only get a mono image you have a kernel prior to version 3.15. All kernel >= 3.15 have a fix I wrote that allows correct color interpretation. For older kernel you will have to reinterpret them manually. Applications like guvcview will not be able to display a color image.

[mdxx]

Hello again, i have one more problem right now.
Even though i tested my code, and i had a picture ( a black picture, but it is still a picture ) now when i am trying to use my cam, from guvcview i am ok... i have this grey video i always had, but when i type ls dev/video0 linux response:ls: cannot access dev/video0: No such file or directory
can you tell me why????
I am using VMWare 10.0

[TIS-Michael]

Hello,

could you send me the outputs of ls /dev, lshw and lsusb?

Thank you in advance

[mdxx]

finally it was sth stupid.... i forgot "/" before dev.....
I run .exe but i didn't have a good picture..... can you say to me what is wrong with that pls?
I have added function set up(int fd,int exp, int gain) in v4ldevice.h and exp and gain are exposure ang gain values. I took the following pic with exp=1000 and gain=180.

I also have this for saving image ( i really don't know if there is an problem or there is sth else that i don't know )
cv::Mat mat_frame(pColorCVImage);

imwrite( "/home/mdx/Desktop/image.jpg", mat_frame);

[TIS-Michael]

It seems that the image is interpreted in a wrong RGB format, but with a compressed image it is hard to tell for sure.
Could you send us a .bmp or a .png?

[mdxx]

so i will have to change imwrite for having .bmp image?

[TIS-Michael]

As far as I understand it, changing the file extension in the path parameter to .bmp should be sufficient.

[mdxx]

i change it from main.cpp (imwrite)

[mdxx]

This is my modified code.
The changes are :
I change opendevice output type from void->int
i add an extra function void set up(ind fd,ind exp,int gain)
i delete loop from main.cpp so i can have only one itme snap image function
i add this for saving image
cv::Mat mat_frame(pColorCVImage);

imwrite( "/home/mdx/Desktop/image.jpg", mat_frame);

Date: Mon, 15 Dec 2014 07:18:02 -0800
From: notifications@github.com
To: tiscamera@noreply.github.com
CC: xiro-drama@hotmail.com
Subject: Re: [tiscamera] Cannot capture images in OpenCV in UVC mode (#11)

As far as I understand it, changing the file extension in the path parameter to .bmp should be sufficient.

—
Reply to this email directly or view it on GitHub.

                  =

[mdxx]

do you see an errors there?

[TIS-Michael]

It looks like the wrong debayering mode is used to convert the picture to RGB.
How do you perform the debayering and which mode do you use?

[mdxx]

aha.... can explain to me a little bit more? Cause i cannot understand :/

[TIS-Michael]

The image data you get from the camera is in a RAW format, in this case black and white with a Bayer Pattern.
In order to get a color image from the RAW data, it has to be converted to an RGB format, using a process called dabayering.
I guess that imwrite() attempts to do that automatically, but fails because it does not use the correct algorithm.
When using a gstreamer pipeline, we achieve this by using the bayer2rgb module. For your code we have to find a different solution.

The documentation for imwrite() suggests Mat::convertTo() and cvtColor() for this, thus I would suggest to read up on the usage of these functions.

[mdxx]

In main there was this line:

cvCvtColor(pOpenCVImage,pColorCVImage,CV_BayerBG2RGB);

[mdxx]

This is the part of code in main that makes everything:

start_capturing();

    ImageBuffer = snapFrame();

    if( ImageBuffer != NULL )

    {

        memcpy( pOpenCVImage->imageData, ImageBuffer, pOpenCVImage->imageSize);

        cvCvtColor(pOpenCVImage,pColorCVImage,CV_BayerBG2RGB); // Create a color image from the raw data          

    }

    else

    {

        printf("No image buffer retrieved.\n");            

    }

stop_capturing();

cv::Mat mat_frame(pcolorCVImage);

imwrite( "/home/mdx/Desktop/image.png", mat_frame);

[mdxx]

can you explain me why there is a double pic when i snap frame?

[mdxx]

I changed image height and width and i finally took this pic, which is a little bit better i think... :P
But it has a pink backround....

[TIS-Michael]

A wrong resolution definitely explains the broken image.
About the color, it looks like there is no white balance. OpenCV seems to offer some white balance functionality, see http://docs.opencv.org/trunk/modules/xphoto/doc/whitebalance.html

[mdxx]

Stefan, can i have an exposure speed up to 10sec with DFK 23U274?

[TIS-Stefan]

Yes, Take care of the snap image time, as I already wrote by email to you

[mdxx]

Hello again, after some time. I want to ask you a critical question. I have DFK 23U274 and i tried it in windows with IC Capture 2.3. There i can snap images with Gain 9Db and Exposure time 2sec and 4,5 sec. In linux i use your C++. I saw from V4L2 API that i can have exposure time up to 30 secs, but in DFK 23U274 specs i read that i can use only up to 1/4sec . So i am wondering why in windows and in IC Capture 2.3 i can have a photo with 9DB and 2secs and in Linux i cannot. Can you help me pls with this final problem before my last tests?

Marios Dimiou
Date: Tue, 16 Dec 2014 02:45:22 -0800
From: notifications@github.com
To: tiscamera@noreply.github.com
CC: xiro-drama@hotmail.com
Subject: Re: [tiscamera] Cannot capture images in OpenCV in UVC mode (#11)

Yes, Take care of the snap image time, as I already wrote by email to you

—
Reply to this email directly or view it on GitHub.

[TIS-Stefan]

Hi

Unfortunately the spec of the DFK 23U274 is somewhat inexact about exposure times. 30 seconds is correct.

Please excuse this.

Stefan

[mdxx]

Goodmorning. In Linux i change exposure time at 7000 for example and i have timestamp. can i correct that somehow?

I saw in V4l2 that exposure value of 10.000 is 1sec

[TIS-Stefan]

Hi

I do not understand your question " In Linux i change exposure time at 7000 for example and i have timestamp. can i correct that somehow?" What is wrong, what should be corrected?

When I query the available properties of the DFK 23U274, then I get:

Property id 0x980913 name Gain min: 176, max: 1023
Property id 0x9a0902 name Exposure (Absolute) min: 1, max: 300000
Property id 0x199e201 name Exposure Time (us) min: 100, max: 30000000
Property id 0x199e204 name Gain (dB/100) min: 0, max: 3032
Property id 0x199e208 name Trigger Mode min: 0, max: 1
Property id 0x199e209 name Software Trigger min: 0, max: 0
Property id 0x199e210 name Trigger Delay min: 0, max: 128
Property id 0x199e211 name Strobe Enable min: 0, max: 1
Property id 0x199e212 name Strobe Polarity min: 0, max: 1
Property id 0x199e213 name Strobe Exposure min: 0, max: 1
Property id 0x199e214 name Strobe Duration min: 0, max: 10000
Property id 0x199e215 name Strobe Delay min: -10000, max: 10000
Property id 0x199e216 name GPOUT min: 0, max: 1
Property id 0x199e217 name GPIN min: 0, max: 1

BR
Stefan

[mdxx]

I have this part of code for exposure:
ctrl.id =V4L2_CID_EXPOSURE_ABSOLUTE;
ctrl.value = 7000;
ioctl(fd, VIDIOC_S_CTRL, &ctrl);

but when i am trying to snap a photo i have timestamp message. and i don't have a photo....
Date: Mon, 16 Feb 2015 02:08:25 -0800
From: notifications@github.com
To: tiscamera@noreply.github.com
CC: xiro-drama@hotmail.com
Subject: Re: [tiscamera] Cannot capture images in OpenCV in UVC mode (#11)

Hi

I do not understand your question " In Linux i change exposure time at 7000 for example and i have timestamp. can i correct that somehow?" What is wrong, what should be corrected?

When I query the available properties of the DFK 23U274, then I get:

Property id 0x980913 name Gain min: 176, max: 1023

Property id 0x9a0902 name Exposure (Absolute) min: 1, max: 300000

Property id 0x199e201 name Exposure Time (us) min: 100, max: 30000000

Property id 0x199e204 name Gain (dB/100) min: 0, max: 3032

Property id 0x199e208 name Trigger Mode min: 0, max: 1

Property id 0x199e209 name Software Trigger min: 0, max: 0

Property id 0x199e210 name Trigger Delay min: 0, max: 128

Property id 0x199e211 name Strobe Enable min: 0, max: 1

Property id 0x199e212 name Strobe Polarity min: 0, max: 1

Property id 0x199e213 name Strobe Exposure min: 0, max: 1

Property id 0x199e214 name Strobe Duration min: 0, max: 10000

Property id 0x199e215 name Strobe Delay min: -10000, max: 10000

Property id 0x199e216 name GPOUT min: 0, max: 1

Property id 0x199e217 name GPIN min: 0, max: 1

BR

Stefan

—
Reply to this email directly or view it on GitHub.

[TIS-Stefan]

Hello

I suppose, your code is based on the V4l2 OpenCV sample, that we provided somewhere. I see a snaimage() call in one of the previous calls. I suppose, you want to set the exposure time to a longer value, so the image gets brighter.
The new exposure time is valid after a frame has been delivered by the camera.
So you should snap an image and discard it. The next should have the correct time,

I do not see a relation to "timestamp" here.

BR
Stefan

[TIS-Stefan]

"Hello, i tried to work with IC Capture today and i had this message

I reconnecting it and it is the same. PLs tell me that my camera isn't broken.... :/"

What is "this message"?

[filesync]

If you only get a mono image you have a kernel prior to version 3.15. All kernel >= 3.15 have a fix >>I wrote that allows correct color interpretation. For older kernel you will have to reinterpret them >>manually.

Hi, I have a 3.03 kernel and the same monochrome issue as described here. Can you please shed some more light on how to "reinterpret" these captures manually. Right now I have a bunch of *.mkv files that are all monochrome. I have tried converting them into RGB, avi using ffmpeg but couldn't get to extracting colored bmp out of those. Any help would be appreciated (good astronomical seeing is rare in my parts!).

[TIS-Stefan]

Mr filesync

could you please give some more context? Or create a new issue? This thread here is quite long

Stefan

[labreu]

@mdxx,
could you share or explain your "set up(ind fd,ind exp,int gain)" function? I really need it =)
what type of variable is ctrl?
This function should be in v4ldevice.cpp, yea?

set up(ind fd,int exp,int gain){
var? ctrl;

//setting exposure
ctrl.id = V4L2_CID_EXPOSURE;
ctrl.value = exp;
ioctl(fd, VIDIOC_S_CTRL, &ctrl);

//setting gain
ctrl.id = V4L2_CID_GAIN;
ctrl.value = gain;
ioctl(fd, VIDIOC_S_CTRL, &ctrl);
}

@TIS-Edgar , @TIS-Stefan , @TIS-Michael , any ideas? I created a struct for "var ctrl" with id and value but it didn't work.

[labreu]

btw, I have a problem before the one above. I am getting images like this attached with this code:
https://gist.github.com/TIS-Edgar/10f04501f49b6b3bf75e

All I changed:

• remove while loop and break
• added a function to save image, imwrite. And the saved image is exactly like the one in the cvShowImage output.
  

the raw file is also "broken", it's the "same image", but in grayscale.

I am using raspberry pi.

problem solved..

[TIS-Stefan]

Answer to your first question:

struct v4l2_control ctrl;
ctrl.id = V4L2_CID_EXPOSURE;
ctrl.value = exposure;

v4l2_ioctl(fd, VIDIOC_S_PARM, &ctrl);

Please see also at http://linuxtv.org/downloads/v4l-dvb-apis/vidioc-g-ctrl.html

The Raspberry PI is somewhat weak to handle the images at high frame rate of your DFK 42. Therefore, you must lower the frame rate.

struct v4l2_streamparm setfps;
setfps.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
setfps.parm.capture.timeperframe.numerator = 1;
setfps.parm.capture.timeperframe.denominator = fps;
v4l2_ioctl(fd, VIDIOC_S_PARM, &setfps);

I did not test this code on my own.

[labreu]

it worked, thanks @TIS-Stefan.

[gonewith]

Hi @TIS-Stefan
I am using DFM 72BUC02-ML in ubuntu 14.04. I find here from google
I have asked technical supporter (maybe he is @TIS-Michael) to update the firmware with dmk72uc02_162_uvc.euvct?
Now I can get color pictures using the code you posted before at here: https://gist.github.com/10f04501f49b6b3bf75e but there is a problem: the color is not correct,for example, orange become blue ... I think Maybe I need change some parameters to adjust it in the code, so It 's not a big problem?

I have another important question: Do you have some C sample code Not c++ to get video stream? if you have not ,Do I need write it from V4L2 for my company embedded device(not need GUI to display the Live image,only get the data to memory )?

Thank you very much for your reply very soon

peter zhang

@TIS-Stefan