Camera Properties adjustment & Color Correction

[somdats]

@TIS-Stefan
I am using Jetson Nano with nvidia arguscamera lib based on your repo( with gstreamer) to stream live camera (DFM 36CX290-MLBoard camera with FPD-Link III).
The streaming pipeline runs a command with the following settings
wbmode = 1 sensor-mode=0 aelock=false,
resolution= 1948x1096, fps=60

[gst-launch-1.0 nvarguscamerasrc wbmode=1 aelock=false sensor-mode=0 sensor_id=0 ! nvvidconv ! 'video/x-raw(memory:NVMM), width=(int)1948, height=(int)1096, format=(string)I420,framerate=(fraction)60/1' ! omxh264enc bitrate=4000000 ! 'video/x-h264, stream-format=(string)byte-stream ! h264parse ! rtph264pay mtu=1400 ! udpsink host=x.x.x.x port=5002 sync=false async=false ]

The following parameters are adapted to stream in a daylight outdoor scene, however, the color from the images is completely messed up, and appears that the issue is with white balance. The histogram for each channel is left aligned
I tried tweaking the settings as per the available option.
I tried with the auto settings but seems not working
Is it possible to camera-color settings ( maybe with BayerHistogram}
** Even the indoor acquisition is warm-toned**

Here is a sample acquired image, the sky color looks completely weird. It would be great to have your suggestions
Image Daylight { Outdoor]

n
Image [Indoorr]

[TIS-Tim]

You are currently using nvidia software, and the data is processed by the hardware ISP. Did you try setting the wbmode property of nvarguscamerasrc to a different value?

Unfortunately, the inner workings of the ISP are a black box. There is a configuration file at /var/nvidia/nvcam/settings, but its capabilities are undocumented.

However, you can try using our software solution:
To get started, you need our tiscamera, tcamtegrasrc and tcamdutils and/or tcamdutils-cuda packages.
Use tcam-capture to check the image quality.
You can then put together a pipeline using the gstreamer elements we provide to capture the images.

[somdats]

You are currently using nvidia software, and the data is processed by the hardware ISP. Did you try setting the wbmode property of nvarguscamerasrc to a different value?

Unfortunately, the inner workings of the ISP are a black box. There is a configuration file at /var/nvidia/nvcam/settings, but its capabilities are undocumented.

However, you can try using our software solution: To get started, you need our tiscamera, tcamtegrasrc and tcamdutils and/or tcamdutils-cuda packages. Use tcam-capture to check the image quality. You can then put together a pipeline using the gstreamer elements we provide to capture the images.

Thanks for the spontaneous reply!

1. Yes, I tried with white balance settings for daylight condition as per argus lib option
2. Ok, Settings files( I can have a look), So, If the properties are adjusted, the auto-settings of the camera are modified, Is that correct?
3. Are all those you mentioned regarding your software solution, part of a single software package or separate individual soft., Could it be used alongside the already installed jetson packages in order to see If we can achieve better flexibility with the camera parameter settings?

[TIS-Tim]

I think you need to restart the stream in order for the camera_overrides to have an effect. But as I said, it is undocumented, so it might not even be possible to control the behavior you see with that.

We provide the software as separate packages, but they are designed to be used together:
https://www.theimagingsource.com/support/downloads-for-linux/install/tiscameraarm64641804/
https://www.theimagingsource.com/support/downloads-for-linux/install/tcamtegrasrcarm64/
https://www.theimagingsource.com/support/downloads-for-linux/install/tiscameradutilsarm64/
https://www.theimagingsource.com/support/downloads-for-linux/install/tcamdutilscudaarm64/

Only the first of thesse packages is open-source, but if the white balance automatic in one of the libraries is indeed working in unexpected ways, we can look at the reason and figure out how to solve it.

[somdats]

Ok,
Trying out the 1st one may be worthwhile
Does the package have UI-based manipulation of the settings, just to see the changes? Or any example usage in your git repos.
Rest of the soft. packages except the 1st need licenses to be purchased?

[TIS-Tim]

The tiscamera package contains tcam-capture, a small UI tool to check image quality and modify all available settings.

All software linked above is free, no need to purchase licenses. But it only works with The Imaging Source cameras.

There are some tiscamera example programs at
https://github.com/TheImagingSource/tiscamera/tree/master/examples
Additional example programs can be found here:
https://github.com/TheImagingSource/Linux-tiscamera-Programming-Samples

[somdats]

Thanks!
Yes, We currently have 4 Imaging sources cameras, with nvidia- Jetson boards.
It would be good to try out
I have a question [ slightly on a different topic but related to cameras]
The 2 Imaging source cameras connected to another Jetson board display RED LED on the serializer board, It was working all good but certainly, now it's not green anymore[ normal operation]. I read in one of your documents that it may be an issue with the voltage/current, however, we are using the same adapter for connection as earlier

[TIS-Tim]

Can you share the kernel log (dmesg) after booting when the LED stays red?

[somdats]

Can you share the kernel log (dmesg) after booting when the LED stays red?

The Jetson does not boot up.

[somdats]

The tiscamera package contains tcam-capture, a small UI tool to check image quality and modify all available settings.

All software linked above is free, no need to purchase licenses. But it only works with The Imaging Source cameras.

There are some tiscamera example programs at https://github.com/TheImagingSource/tiscamera/tree/master/examples Additional example programs can be found here: https://github.com/TheImagingSource/Linux-tiscamera-Programming-Samples

I installed the deb file, however when I launch tcam-capture, I get a pop message that it is unable to find device( I am actually using imaging-source camera as earlier pointed)

[TIS-Tim]

Did you install the tcamtegrasrc package as well?
It provides access to raw data from mipi cameras via gstreamer.

[somdats]

Did you install the tcamtegrasrc package as well? It provides access to raw data from mipi cameras via gstreamer.

@TIS-Tim
Thank you, I installed it and now it works,
Maybe, I can also install
https://www.theimagingsource.com/support/downloads-for-linux/install/tiscameradutilsarm64/ to perform some processing tasks on the live stream
Do you have an option to do some analysis ( for instance, histogram, etc.)?

[TIS-Tim]

The dutils packages provide optimized debayering and simple image manipulation like gamma, saturation, white balance etc.

If they are not installed, there is an implementation inside tiscamera (called tcamconvert) that does not contain all features.
You can select which implementation is used from the tcam-capture options dialog (maybe need to restart after selecting).

https://www.theimagingsource.com/support/downloads-for-linux/install/tiscameradutilsarm64/
provides an optimized implementation with all features which is usually preferrable.

https://www.theimagingsource.com/support/downloads-for-linux/install/tcamdutilscudaarm64/
provides an implementation doing the same thing, but using CUDA hardware. Depending on the use case and system load, this might be an option.

[somdats]

@TIS-Tim
Thanks a lot again and also for a nice tool, I also installed the util package for better enhancement.
From the preliminary observation, I found your tool much more comprehensive and flexible to adjust the parameters ( including ISP).
what exactly is the exposure auto reference parameter?

I tried to adjust different parameters (exposure settings, white balance, enabling tone-mapping).
Here is an image with the auto setting

and white balance values

Ideally, it would be better to have the auto-settings do the majority of processing followed by slight additional tunings, I still have to adjust them to look visually correct, The overexposure might be the cause for the tree to look warm

As per our requirements

1. so, If everything goes ok, we need to use the tcam tool as a pipeline with GStreamer as we are doing with argus library and therefore a launch parameter with only certain tuning parameters should be better.
2. The streams need to be sent over a network preferably as UDP ( current-status) with encoding, maybe this is good, but we need to add tuning parameters:
   https://github.com/TheImagingSource/Linux-tiscamera-Programming-Samples/tree/master/Stream%20over%20Network%20to%20VLC

On the receiving side, we use opencv-gstreamer video capture to get the feed and then read the frame

It would be good If we could have some tuning parameter setting commands as a gst pipeline ( gst launch)

[TIS-Tim]

You can add the parameters to the pipeline, e.g.

gst-launch-1.0 tcambin tcam-properties=tcam,ExposureAuto=Off,ExposureTime=33333 ! ...

You can get a string of all settings (for copy+paste) from the tcam-capture options dialog (maybe need to un-stringify the json there to get it to work in a console.

The ExposureAutoReference setting is the target brightness for the auto exposure/gain control. You should probably lower this, since avoiding overexposure is important, the best post-processing cannot fix it.

Are those colors correct? Or are the red trees because you intentionally don't have an IR-cut-coated lens?

Instead of using the WDR function, you could also try setting a low Gamma (e.g. ~0.45) and low ExposureAutoReference (e.g. ~50). Maybe increase saturation a bit.

[somdats]

You can add the parameters to the pipeline, e.g.

gst-launch-1.0 tcambin tcam-properties=tcam,ExposureAuto=Off,ExposureTime=33333 ! ...

You can get a string of all settings (for copy+paste) from the tcam-capture options dialog (maybe need to un-stringify the json there to get it to work in a console.

The ExposureAutoReference setting is the target brightness for the auto exposure/gain control. You should probably lower this, since avoiding overexposure is important, the best post-processing cannot fix it.

Are those colors correct? Or are the red trees because you intentionally don't have an IR-cut-coated lens?

Instead of using the WDR function, you could also try setting a low Gamma (e.g. ~0.45) and low ExposureAutoReference (e.g. ~50). Maybe increase saturation a bit.

Thanks
Ok,
The colors are not fully correct, The tree leaves are actually green but look different with the settings that I use/modify. They do have IR Filters now that I see the lens

I will try to adopt your advice and observe the outcomes. However, it would be ideal to have parameter combinations that work in most of the outdoor settings
Do you have an example for the opencv video capture at the receiving side of the network stream or can I use a similar capture element syntax:
VideoCapture cap("udpsrc port=5000 ! application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96 ! rtph264depay ! decodebin ! videoconvert ! appsink drop =1",
CAP_GSTREAMER );

[TIS-Tim]

Are you sure about the lens? If all settings are at their default, and the sky is blue (as I assume it really is) the trees/lawn should be green. If the vegetation is red/pink, it is a usual indicator for the lack of a IR-cut filter.

I'm sorry but I have no idea about the receiving end of that stream. In your original question you used a different streaming mechanism; didn't that work for you? You should be able to connect our capture pipeline to the streaming pipeline you already had.

This https://stackoverflow.com/questions/50549584/gstreamer-udpsink-udpsrc-versus-tcpserversink-tcpclientsrc looks like the counterpart to tcpserversink is tcpclientsrc.

[somdats]

Are you sure about the lens? If all settings are at their default, and the sky is blue (as I assume it really is) the trees/lawn should be green. If the vegetation is red/pink, it is a usual indicator for the lack of a IR-cut filter.

I'm sorry but I have no idea about the receiving end of that stream. In your original question you used a different streaming mechanism; didn't that work for you? You should be able to connect our capture pipeline to the streaming pipeline you already had.

This https://stackoverflow.com/questions/50549584/gstreamer-udpsink-udpsrc-versus-tcpserversink-tcpclientsrc looks like the counterpart to tcpserversink is tcpclientsrc.

I agree with you regarding the color
I would say, I am not totally with IR cut filter, but the lens is definitely IR corrected
Additionally, it has manual focusing

[somdats]

@TIS-Stefan
Thanks for the support from you & colleagues, I am now able to tune the camera/image parameters using tcam pipeline with GStreamer.
I had also mentioned to you previously regarding the color issues, would like to know, If the color issue can still be controlled:

The sky color looks dark( probably due to high contrast/ or similar imaging parameters), the other thing is the green color of the plants is completely OFF and appears reddish, Is it due to the unavailability of an IR cut filter? We are using imaging source camera-lens pair
Any thoughts on it? It would be a great help

[TIS-Stefan]

If you ask me... (which you did), I would say, your lens has no IR cut filter. Look on the back side of your lens, so it reflects light. If there is no red or green reflection, there is no IR cut filter.

[somdats]

@TIS-Stefan , Thanks
Indeed, It is definitely what you point to.

[somdats]

@TIS-Stefan
I am grateful for the suggestions from you and your colleagues. It has helped a lot.
I have a few questions:

1. We have 4 cameras from Imaging source, 2 have autofocus, and the other two with manual adjustment. So, regarding the one with auto, Can we even adjust focus further to sharpen the images or whatever value it picks up is optimal?
2. As already discussed, we miss the IR cut filter, which affects the color component of the grabbed image. In daylight conditions, the colors look warm/intense, (the histogram shows mostly flat with a peak on the higher end) I have kept the white balance "OFF", in terms of contrast. i provide value around "40" with gamma :0.75, exposureAutoreference :125, saturation:116, sharpness:2, along with custom rectangle size as earlier mentioned. Do you have any suggestions to improve the quality of an image by adjusting the available properties through tcam? Here is an example image
   

As our application demands day-light working conditions, we need to maximize the quality in such conditions
4. We want to do a histogram equalization on the grabbed image just to improve the overall contrast If further improvisation is not possible on the cam-property side.

Any suggestions would be vital,
Thanks

[TIS-Stefan]

Hello

the average image brightness is 134 in your image. That is near the value, the automatics try to achieve by default.
Step 1: Enable Hightlight Reduction:

Step 2: Reduce Exposure Auto Reference Value:

Step 3: Limit the maximum Exposure time:

[somdats]

Hello

the average image brightness is 134 in your image. That is near the value, the automatics try to achieve by default. Step 1: Enable Hightlight Reduction:

Step 2: Reduce Exposure Auto Reference Value:

Step 3: Limit the maximum Exposure time:

Hello

the average image brightness is 134 in your image. That is near the value, the automatics try to achieve by default. Step 1: Enable Highlight Reduction:

Step 2: Reduce Exposure Auto Reference Value:

Step 3: Limit the maximum Exposure time:

Thanks, enabling highlight reduction brings down the overall brightness
I am using the tcam-capture tool to experiment and then use those parameters in my gstreamer pipeline:
Here is the UI snapshot to modify the parameters: Do you mean adjusting the exposure auto upper limit value? and what about the exposure auto upper limit auto option?

I am already adjusting the exposure auto reference value depending on the day-light condition. Are there any recommended values?
The exposure max. value is limited by the frame rate, currently, if FPS is 30, then it is: 1/30sec, should it be further reduced?

[TIS-Stefan]

Oh, I see, you use Linux. However, the properties are more or less the same in Windows.

A word to "Auto Reference": This value determines the average brightness of the image, that should be achieved by the automatic algorithms. In your case, I would set it to 80.
Which auto functions rectangle do you use?

We want to do a histogram equalization on the grabbed image just to improve the overall contrast If further improvisation is not possible on the cam-property side.

That is a post processing, in particular, it depends on the image data. For that you must gather the minimum and maximum brightness and then create a LUT with the brightness values and there new substitution. For that, we do not have a camera function. I would do that by post processing on the video frames.

(Now your answer came in, while I am writing).
33333µs Auto Exposure Upperlimit is too much for daylight outside. I would use rather 10000µs or less.
You have Exposure Auto Reference still on 128. Use a value between 50 and 80.

Which "Auto ROI" rectangle do you use?

Which Autofocus cameras do you use? The MIPI cameras do not have Auto Focus.

Stefan

[somdats]

Oh, I see, you use Linux. However, the properties are more or less the same in Windows.

A word to "Auto Reference": This value determines the average brightness of the image, that should be achieved by the automatic algorithms. In your case, I would set it to 80. Which auto functions rectangle do you use?

We want to do a histogram equalization on the grabbed image just to improve the overall contrast If further improvisation is not possible on the cam-property side.

That is a post processing, in particular, it depends on the image data. For that you must gather the minimum and maximum brightness and then create a LUT with the brightness values and there new substitution. For that, we do not have a camera function. I would do that by post processing on the video frames.

(Now your answer came in, while I am writing). 33333µs Auto Exposure Upperlimit is too much for daylight outside. I would use rather 10000µs or less. You have Exposure Auto Reference still on 128. Use a value between 50 and 80.

Which "Auto ROI" rectangle do you use?

Which Autofocus cameras do you use? The MIPI cameras do not have Auto Focus.

Stefan

Which "Auto ROI" rectangle do you use?
-> Custom rectangle : left:480, top: 270, width:970, height:540, It was earlier suggested, the center 50% did not work due to a bug
Which Autofocus cameras do you use? The MIPI cameras do not have Auto Focus.
2 of this:

[TIS-Stefan]

The -I67 cameras are no auto focus cameras. There is no focus mechanic.

Stefan

[somdats]

The -I67 cameras are no auto focus cameras. There is no focus mechanic.

Stefan

Ahh ok, So do we need some extra handling?

[somdats]

@TIS-Stefan
Based on your suggestion, I did an experiment in an outdoor scene, I could not directly place it outdoors, but rather place it on a window completely facing outdoors.
highlight reduction: enabled
max exposure limit: 10000µs
exposure auto reference: 110, further reducing darkens the scene
ROI: custom rectangle ( 480,270,970,540), contrast:40, gamma:0.75, white balance:off.
The image looks dark here:

This is how the scene condition looks with cell phone camera

[TIS-Stefan]

Please do not compare a cell phone camera with an industrial camera. An industrial camera provides the image the sensor sees. A cell phone camera provides an image of computed photography. Which means, more than one image is taken for HDR and tons of image processing is applied. But it is definitely not the image, you get directly from the sensor. You may also check, how long it lasts, until the image is shown, after you pressed the capture button on the cell phone.

Which does not solve your image quality issues.

I suppose, you will drive with the cameras around and capture images or a video file, is that correct? If I am right, you can use post processing after image capture for e.g. histogram spreading.

[somdats]

Please do not compare a cell phone camera with an industrial camera. An industrial camera provides the image the sensor sees. A cell phone camera provides an image of computed photography. Which means, more than one image is taken for HDR and tons of image processing is applied. But it is definitely not the image, you get directly from the sensor. You may also check, how long it lasts, until the image is shown, after you pressed the capture button on the cell phone.

Which does not solve your image quality issues.

I suppose, you will drive with the cameras around and capture images or a video file, is that correct? If I am right, you can use post-processing after image capture for e.g. histogram spreading.

@TIS-Stefan
Well, Unfortunately, you misunderstood me and I am sorry for that. The whole intention of sharing the camera image was to just show how the outdoor condition looks like for someone who is not physically present during the capture but not to compare at all.
So, I am rather interested in improvising /modifying the parameters of the cams so that we can rather be sure that it is more or less optimal. As we are using the tcambin APIs, It is rather wise to understand from your side to handle the camera props.
If you have some additional input? We would rather be happy

We intend to use the cameras in real-time scenarios and not just record, store, and do the post-processing for algorithm development.
Yes, indeed, histogram spreading is on our agenda as I wrote you earlier. However, we wanted to check, if the tuning of camera parameters is ok and how it influence based on your camera-prop APIs.
I hope now you understand

Thanks

[TIS-Stefan]

You can play with gamma too. Also you may set saturation to 130% for more saturated colors.

That is currently all I can suggest.

Also you may use the "WDR/Tonmapping" of tcam-capture. This does histogram spreading on the image, but the values you set may are highly light dependend. But with enabled highlight reduction and a lowered Exposure Auto Reference Value you could create good results.

I am sorry, but this is a lot of try and error.

Stefan

[TIS-Stefan]

The -I67 cameras are no auto focus cameras. There is no focus mechanic.

Stefan

Ahh ok, So do we need some extra handling?

There is no support for a focus mechanic on the MIPI and FPD link cameras at all. The USB cameras have auto focus. For example this one: https://www.theimagingsource.com/products/zoom-autofocus-cameras/usb-3.0-color-autofocus-cameras/dfkafux236m12/
In case you want to use that, we have to check it with your big lens first, because the motor block may blocks the movement of the big wide angle lens.

(I am sorry for the weird sequence of my answer to you questions. I had some email issues today.)

Stefan

[somdats]

You can play with gamma too. Also you may set saturation to 130% for more saturated colors.

That is currently all I can suggest.

Also you may use the "WDR/Tonmapping" of tcam-capture. This does histogram spreading on the image, but the values you set may are highly light dependend. But with enabled highlight reduction and a lowered Exposure Auto Reference Value you could create good results.

I am sorry, but this is a lot of try and error.

Stefan
Thank you very much
Yes, I indeed tried enabling highlight reduction and lowering exposure auto reference, The corresponding capture looks darker as already shared. I did try raising the saturation level but the results were not so different.

[somdats]

The -I67 cameras are no auto focus cameras. There is no focus mechanic.

Stefan

Ahh ok, So do we need some extra handling?

There is no support for a focus mechanic on the MIPI and FPD link cameras at all. The USB cameras have auto focus. For example this one: https://www.theimagingsource.com/products/zoom-autofocus-cameras/usb-3.0-color-autofocus-cameras/dfkafux236m12/ In case you want to use that, we have to check it with your big lens first, because the motor block may blocks the movement of the big wide angle lens.

(I am sorry for the weird sequence of my answer to you questions. I had some email issues today.)

Stefan

We will still have to use MIPI cameras

[TIS-Edgar]

Closed due to inactivity.