Problem displaying images on visual studio (Too dark or only shows Invalid pixels)

[ahmadLyneSystems]

Hello,

I've been trying to setup ifm3d with our O3D311 camera on a windows environment with visual studio 2019. I built ifm3d using the PCL library and followed your windows installation guide on multiple versions of ifm3d but I'm still facing the same issue. I'm mainly concerned with obtaining the distance image but when I run the code below, all I get is either a black image or a white one with the black pixels representing the invalid pixels from the VA software.

System: windows
ifm3d version 0.20
Built using PCL boost library

Our code on C++:

int main(int argc, const char** argv)
{
    auto cam = ifm3d::Camera::MakeShared("10.10.0.85");
    auto fg = std::make_shared<ifm3d::FrameGrabber>(cam, ifm3d::IMG_RDIS | ifm3d::IMG_AMP | ifm3d::IMG_CART | ifm3d::IMG_GRAY);
    auto im = std::make_shared<ifm3d::ImageBuffer>();
  
    auto imTypes = cam->ImagerTypes();
    auto st = cam->ToJSONStr();
       for (std::string type : imTypes)
        std::cout << st << '\n';

    cv::Mat amp, dist, gray;
    cv::Mat xyz;

    while (true)
    {
        if (!fg->WaitForFrame(im.get(), 1000))
        {
            std::cerr << "Timeout waiting for camera!" << std::endl;
            return -1;
        }
     
        amp = im->AmplitudeImage();
        xyz = im->XYZImage();
        dist = im->DistanceImage();
        gray = im->GrayImage();
        
        //dist.convertTo(dist, CV_8U);
        cv::imshow("w", dist);
        cv::waitKey(0);
    }
    return 0;
}

`

outputs:
Distance image:

Amplitude image:

XYZ image:

I could not get the gray image to appear either as it gives an error:

OpenCV(3.4.9) Error: Assertion failed (size.width>0 && size.height>0) in cv::imshow, file C:\build\3_4_winpack-build-win64-vc15\opencv\modules\highgui\src\window.cpp, line 382
And our app dump:

{
"ifm3d": {
"Apps": [
{
"Description": "",
"Id": "1146356873",
"Imager": {
"AutoExposureMaxExposureTime": "10000",
"AutoExposureReferencePointX": "88",
"AutoExposureReferencePointY": "66",
"AutoExposureReferenceROI": "{"ROIs":[{"id":0,"group":0,"type":"Rect","width":130,"height":100,"angle":0,"center_x":88,"center_y":66}]}",
"AutoExposureReferenceType": "1",
"Channel": "0",
"ClippingBottom": "263",
"ClippingCuboid": "{"XMin": -3.402823e+38, "XMax": 3.402823e+38, "YMin": -3.402823e+38, "YMax": 3.402823e+38, "ZMin": -3.402823e+38, "ZMax": 3.402823e+38}",
"ClippingLeft": "0",
"ClippingRight": "351",
"ClippingTop": "0",
"ContinuousAutoExposure": "false",
"ContinuousUserFrameCalibration": "false",
"EnableAmplitudeCorrection": "true",
"EnableFastFrequency": "false",
"EnableFilterAmplitudeImage": "false",
"EnableFilterDistanceImage": "true",
"EnableRectificationAmplitudeImage": "true",
"EnableRectificationDistanceImage": "true",
"ExposureTime": "300",
"ExposureTimeList": "300",
"FlyingPixelEraser": "1",
"FrameRate": "5",
"MaxAllowedLEDFrameRate": "227.2",
"MinimumAmplitude": "42",
"Resolution": "1",
"SpatialFilter": {},
"SpatialFilterType": "0",
"SymmetryThreshold": "0.4",
"TemporalFilter": {
"MedianLength": "1"
},
"TemporalFilterType": "3",
"ThreeFreqMax2FLineDistPercentage": "80",
"ThreeFreqMax3FLineDistPercentage": "80",
"TwoFreqMaxLineDistPercentage": "80",
"Type": "under5m_low",
"UseSimpleBinning": "false"
},
"Index": "1",
"LogicGraph": "{\n "IOMap": {\n "OUT1": "RFT",\n "OUT2": "AQUFIN"\n },\n "blocks": {\n "B00001": {\n "pos": {\n "x": 204,\n "y": 224\n },\n "properties": {\n },\n "type": "PIN_EVENT_IMAGE_ACQUISITION_FINISHED"\n },\n "B00002": {\n "pos": {\n "x": 200,\n "y": 75\n },\n "properties": {\n },\n "type": "PIN_EVENT_READY_FOR_TRIGGER"\n },\n "B00003": {\n "pos": {\n "x": 600,\n "y": 75\n },\n "properties": {\n "pulse_duration": 0\n },\n "type": "DIGITAL_OUT1"\n },\n "B00005": {\n "pos": {\n "x": 598,\n "y": 220\n },\n "properties": {\n "pulse_duration": 0\n },\n "type": "DIGITAL_OUT2"\n }\n },\n "connectors": {\n "C00000": {\n "dst": "B00003",\n "dstEP": 0,\n "src": "B00002",\n "srcEP": 0\n },\n "C00001": {\n "dst": "B00005",\n "dstEP": 0,\n "src": "B00001",\n "srcEP": 0\n }\n }\n}\n",
"Name": "App1",
"PcicEipResultSchema": "{ "layouter": "flexible", "format": { "dataencoding": "binary", "order": "big" }, "elements" : [ { "type": "string", "value": "star", "id": "start_string" }, { "type": "records", "id": "models", "elements": [ { "type": "int16", "id": "boxFound" }, { "type": "int16", "id": "width", "format": { "scale": 1000 } }, { "type": "int16", "id": "height", "format": { "scale": 1000 } }, { "type": "int16", "id": "length", "format": { "scale": 1000 } }, { "type": "int16", "id": "xMidTop", "format": { "scale": 1000 } }, { "type": "int16", "id": "yMidTop", "format": { "scale": 1000 } }, { "type": "int16", "id": "zMidTop", "format": { "scale": 1000 } }, { "type": "int16", "id": "yawAngle" }, { "type": "int16", "id": "qualityLength" }, { "type": "int16", "id": "qualityWidth" }, { "type": "int16", "id": "qualityHeight" } ] }, { "type": "string", "value": "stop", "id": "end_string" } ] }",
"PcicPnioResultSchema": "{"layouter" : "flexible", "format": { "dataencoding": "binary", "order": "big" }, "elements" : [ { "type": "string", "value": "star", "id": "start_string" }, { "type": "records", "id": "models", "elements": [ { "type": "int16", "id": "boxFound" }, { "type": "int16", "id": "width", "format": { "scale": 1000 } }, { "type": "int16", "id": "height", "format": { "scale": 1000 } }, { "type": "int16", "id": "length", "format": { "scale": 1000 } }, { "type": "int16", "id": "xMidTop", "format": { "scale": 1000 } }, { "type": "int16", "id": "yMidTop", "format": { "scale": 1000 } }, { "type": "int16", "id": "zMidTop", "format": { "scale": 1000 } }, { "type": "int16", "id": "yawAngle" }, { "type": "int16", "id": "qualityLength" }, { "type": "int16", "id": "qualityWidth" }, { "type": "int16", "id": "qualityHeight" } ] }, { "type": "string", "value": "stop", "id": "end_string" } ] }",
"PcicTcpResultSchema": "{\n "elements": [\n {\n "id": "distance_image",\n "type": "blob"\n }\n ],\n "format": {\n "dataencoding": "ascii"\n },\n "layouter": "flexible"\n}\n",
"RtspOverlayStyle": "{"ROI": {"default": {"visible": true, "filled": false, "use_symbol": false, "label_alignment": "top", "label_content": "", "label_background": "black", "label_fontsize": 8, "label_failonly": false}, "model_defaults": {}, "specific": {} } }",
"TemplateInfo": "",
"TriggerMode": "1",
"Type": "Camera"
}
],
"Device": {
"ActiveApplication": "1",
"ArticleNumber": "O3D311",
"ArticleStatus": "AE",
"Description": "",
"DeviceType": "1:2",
"EIPConsumingSize": "8",
"EIPProducingSize": "450",
"EnableAcquisitionFinishedPCIC": "false",
"EthernetFieldBus": "0",
"EthernetFieldBusEndianness": "0",
"EvaluationFinishedMinHoldTime": "10",
"ExtrinsicCalibRotX": "0",
"ExtrinsicCalibRotY": "0",
"ExtrinsicCalibRotZ": "0",
"ExtrinsicCalibTransX": "0",
"ExtrinsicCalibTransY": "0",
"ExtrinsicCalibTransZ": "0",
"IODebouncing": "true",
"IOExternApplicationSwitch": "0",
"IOLogicType": "1",
"IPAddressConfig": "0",
"ImageTimestampReference": "946843849",
"Name": "New sensor",
"OperatingMode": "0",
"PNIODeviceName": "",
"PasswordActivated": "false",
"PcicProtocolVersion": "3",
"PcicTcpPort": "50010",
"PcicTcpSchemaAutoUpdate": "false",
"SaveRestoreStatsOnApplSwitch": "true",
"ServiceReportFailedBuffer": "15",
"ServiceReportPassedBuffer": "15",
"SessionTimeout": "30",
"TemperatureFront1": "3276.7",
"TemperatureFront2": "3276.7",
"TemperatureIMX6": "32.5880012512207",
"TemperatureIllu": "36.5",
"UpTime": "157.633055555556"
},
"Net": {
"MACAddress": "00:02:01:41:F9:63",
"NetworkSpeed": "0",
"StaticIPv4Address": "10.10.0.85",
"StaticIPv4Gateway": "192.168.0.201",
"StaticIPv4SubNetMask": "255.255.255.0",
"UseDHCP": "false"
},
"Time": {
"CurrentTime": "946843847",
"MaxNumberOfNTPServers": "5",
"NTPServers": "",
"StartingSynchronization": "false",
"Stats": "",
"SynchronizationActivated": "false",
"Syncing": "false",
"WaitSyncTries": "2"
},
"_": {
"Date": "Tue May 11 10:43:13 2021",
"HWInfo": {
"Connector": "#!02_A300_C41_04880799_008026112",
"Diagnose": "#!03_D322_C34_04873267_008026824",
"Frontend": "#!02_F342_C34_20_00027_008023607",
"Illumination": "#!02_I300_001_04880153_008001175",
"MACAddress": "00:02:01:41:F9:63",
"Mainboard": "#!03_M381_C41_04901813_008023690",
"MiraSerial": "8430-9701-0eeb-0794"
},
"SWVersion": {
"Algorithm_Version": "3.0.5",
"Calibration_Device": "00:02:01:41:f9:63",
"Calibration_Version": "1.0.2",
"Diagnostic_Controller": "O2I_1.20.7428-0-g91c60a74be",
"IFM_Recovery": "O3D3XX 0.6.73",
"IFM_Software": "1.71.7444",
"Linux": "Linux version 3.14.34-rt31-yocto-standard-00048-gc8112a91aeb2-d",
"Main_Application": "O2I_1.20.7428-0-g91c60a74be"
},
"ifm3d_version": 2000
}
}
}

I've also tried to use the openCV image buffer instead but I faced some uninitialized pointer errors on that side of things which I could share if required. Please let me know if need any additional info.

[lola-masson]

Hi @ahmadLyneSystems!
I think this problem is a display problem. If we look at the grey (amplitude) image, we can see that the objects are displayed, the contrast is just very limited. The data straight out of the camera is not normalized for proper display. The vision assistant takes care of processing the amplitude and distance data so that it displays "well" (higher contrast, proper coloring, etc). You are using openCV, which does not apply any processing to the data by default. I would recommend looking at something like a logarithmic transformation of the data before displaying.
Hope this helps.

[lola-masson]

As for the invalid pixels: these, at first glance, look over exposed. Did you try reducing the long exposure time or using multiple exposures?

[ahmadLyneSystems]

Hey lola,

Thanks for the reply. You're right about the amplitude image being normalized which I could fix using a simple image filter. I simply thought the dark contrast was a symptom of anothe problem. However, as I mentioned I'm mainly looking to reproduce the distance image output from the VisionAssistant software.

As for the distance image, I should have mentioned that I only get the white image when I change the openCV data format to CV_8U, otherwise it's just a black image.

The invalid pixels themselves aren't the problem because as you said, I could modify the application settings and exposure to eliminate them. When I run the above code for a distance image, I just get a black image. I tried applying different filters and transforms to it with no success.

Edit: Apologies, I tried some further transformations and I could see some output from the distance image now. However, it's still mostly in black and white. Is there anyway for me to simply replicate the output I see in the VisionAssistant software?

[lola-masson]

I am not sure what exact transformations are performed in the VA, I will need to look into this in more details--let me get back to you asap.
In the meantime, you can have a look at the example simple image, as well as this document explaining how to create an image container and scale the data. This page of OpenCV documentation also details some transformations to perform on an image (scrol down to the very end).

[ahmadLyneSystems]

Thanks for the update. I have been reading the operating instructions for the camera, particularly the chunk types section for each type of output image and I have a better understanding now of what ifm3d is outputting. Perhaps I naively assumed it will simply extract whatever output was there on the VA viewer instead of giving me raw data to work with.

If you could somehow assist me with replicating the transformations made in the VA, it would be great. Otherwise I believe this particular issue is resolved.

[graugans]

Hey @ahmadLyneSystems there is some sample code for scaling the image The amplitude data typically does have more dynamics than a 255Bit grayscale image.

Could you please help me understand what exactly you want to achieve. The data displayed in Vision Assistant is exactly the same as in the Vision assistant.

[ahmadLyneSystems]

Hey @graugans, thanks for linking to the example code. I was actually trying to build a similar RGB scaling function on my own yesterday so this will be a great reference. Initially, I thought I had a problem with my ifm3d build which is why I asked the question. But since it's the expected output, I generally know how to proceed now.

Most of the confusion is due to it being my first time working with a 3D camera instead of regular images so I have a much better idea of what to do now.

Thanks for the assistance @graugans & @lola-masson!