From 2482cf18df2a5dbe6942dc861cbd75e02f506223 Mon Sep 17 00:00:00 2001 From: Erol444 Date: Sat, 15 May 2021 01:07:18 +0100 Subject: [PATCH 1/4] corrected stereoDepth limitations --- docs/source/components/nodes/stereo_depth.rst | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/source/components/nodes/stereo_depth.rst b/docs/source/components/nodes/stereo_depth.rst index f7ecdffef..306eaffd8 100644 --- a/docs/source/components/nodes/stereo_depth.rst +++ b/docs/source/components/nodes/stereo_depth.rst @@ -53,26 +53,26 @@ Inputs and Outputs Disparity ######### -When calculating the disparity, each pixel in the disparity map gets assigned a confidence value 0..255 by the stereo matching algorithm, as: -- 0 - maximum confidence that it holds a valid value -- 255 - minimum confidence, so there are chances the value is incorrect +When calculating the disparity, each pixel in the disparity map gets assigned a confidence value :code:`0..255` by the stereo matching algorithm, +as: + +- :code:`0` - maximum confidence that it holds a valid value +- :code:`255` - minimum confidence, so there is more chance that the value is incorrect + (this confidence score is kind-of inverted, if say comparing with NN) For the final disparity map, a filtering is applied based on the confidence threshold value: the pixels that have their confidence score larger than -the threshold get invalidated, i.e. their disparity value is set to zero. +the threshold get invalidated, i.e. their disparity value is set to zero. You can set the confidence threshold with :code:`stereo.setConfidenceThreshold()`. Current limitations ################### -If one or more of the additional depth modes (lrcheck, extended, subpixel) are enabled, then: - -- :code:`depth` output is FP16. -- median filtering is disabled on device. -- with subpixel, either depth or disparity has valid data. +If one or more of the additional depth modes (:code:`lrcheck`, :code:`extended`, :code:`subpixel`) are enabled, then: -Otherwise, depth output is U16 (in milimeters) and median is functional. +- median filtering is disabled on device +- with subpixel, if both :code:`depth` and :code:`disparity` are used, only :code:`depth` will have valid output -Like on Gen1, either :code:`depth` or :code:`disparity` has valid data. +Otherwise, :code:`depth` output is **U16** (in millimeters) and median is functional. Usage ##### From baffd6756c1dc79ba6d269dc034177a592b72d3d Mon Sep 17 00:00:00 2001 From: Erol444 Date: Sat, 15 May 2021 01:07:56 +0100 Subject: [PATCH 2/4] corrected colorCamera IO chart and described isp/encoder --- docs/source/components/nodes/color_camera.rst | 33 ++++++++++++++----- 1 file changed, 24 insertions(+), 9 deletions(-) diff --git a/docs/source/components/nodes/color_camera.rst b/docs/source/components/nodes/color_camera.rst index c995f0bc8..3a92d8ef1 100644 --- a/docs/source/components/nodes/color_camera.rst +++ b/docs/source/components/nodes/color_camera.rst @@ -24,20 +24,30 @@ Inputs and Outputs .. code-block:: - ┌───────────────────┐ still - │ ├───────────► - inputConfig │ │ - ──────────────►│ │ preview - │ ColorCamera ├───────────► - inputControl │ │ - ──────────────►│ │ video - │ ├───────────► - └───────────────────┘ + ColorCamera node + ┌──────────────────────────────┐ + │ ┌──────────────┐ │ + │ │ sensor │ │ raw + │ │ (IMX378) │------------├────────► + │ └─────┬───▲────┘ │ + │ │ │ │ + │ ┌─▼───┴─┐ │ isp + inputControl │ │ │------┬--------├────────► + ──────────────►│------│ ISP │ ┌────▼────┐ │ video + │ │ │ | |---├────────► + │ └───────┘ │ │ │ still + inputConfig │ │ Encoder │---├────────► + ──────────────►│----------------| │ │ preview + │ │ │---├────────► + │ └─────────┘ │ + └──────────────────────────────┘ **Message types** - :code:`inputConfig` - :ref:`ImageManipConfig` - :code:`inputControl` - :ref:`CameraControl` +- :code:`raw` - :ref:`ImgFrame` +- :code:`isp` - :ref:`ImgFrame` - :code:`still` - :ref:`ImgFrame` - :code:`preview` - :ref:`ImgFrame` - :code:`video` - :ref:`ImgFrame` @@ -46,6 +56,11 @@ Inputs and Outputs into :ref:`NeuralNetwork`. :code:`video` and :code:`still` are both NV12, so are suitable for bigger sizes. :code:`still` image gets created when a capture event is sent to the ColorCamera, so it's like taking a photo. +**ISP** (image signal processor) is used for bayer transformation, demosaicing, noise reduction etc. Click +`here `__ for more information. + +**Encoder** converts YUV420 planar frames from **ISP** into :code:`video`/:code:`preview`/:code:`still` frames. + Usage ##### From 170a2820c6bbf789d82b15743620a3160d120ade Mon Sep 17 00:00:00 2001 From: Erol444 Date: Sat, 15 May 2021 21:19:36 +0100 Subject: [PATCH 3/4] Updated colorCam IO chart and fixed message descriptions --- docs/source/components/nodes/color_camera.rst | 49 +++++++++---------- 1 file changed, 23 insertions(+), 26 deletions(-) diff --git a/docs/source/components/nodes/color_camera.rst b/docs/source/components/nodes/color_camera.rst index 3a92d8ef1..68c7e6fc7 100644 --- a/docs/source/components/nodes/color_camera.rst +++ b/docs/source/components/nodes/color_camera.rst @@ -26,40 +26,37 @@ Inputs and Outputs ColorCamera node ┌──────────────────────────────┐ - │ ┌──────────────┐ │ - │ │ sensor │ │ raw - │ │ (IMX378) │------------├────────► - │ └─────┬───▲────┘ │ - │ │ │ │ - │ ┌─▼───┴─┐ │ isp - inputControl │ │ │------┬--------├────────► - ──────────────►│------│ ISP │ ┌────▼────┐ │ video - │ │ │ | |---├────────► - │ └───────┘ │ │ │ still - inputConfig │ │ Encoder │---├────────► - ──────────────►│----------------| │ │ preview - │ │ │---├────────► - │ └─────────┘ │ + │ ┌─────────────┐ │ + │ │ Image │ raw │ raw + │ │ Sensor │---┬--------├────────► + │ └────▲────────┘ | │ + │ │ ┌--------┘ │ + │ ┌─┴───▼─┐ │ isp + inputControl │ │ │-------┬-------├────────► + ──────────────►│------│ ISP │ ┌─────▼────┐ │ video + │ │ │ | |--├────────► + │ └───────┘ │ Image │ │ still + inputConfig │ │ Post- │--├────────► + ──────────────►│----------------|Processing│ │ preview + │ │ │--├────────► + │ └──────────┘ │ └──────────────────────────────┘ **Message types** - :code:`inputConfig` - :ref:`ImageManipConfig` - :code:`inputControl` - :ref:`CameraControl` -- :code:`raw` - :ref:`ImgFrame` -- :code:`isp` - :ref:`ImgFrame` -- :code:`still` - :ref:`ImgFrame` -- :code:`preview` - :ref:`ImgFrame` -- :code:`video` - :ref:`ImgFrame` +- :code:`raw` - :ref:`ImgFrame` - RAW10 bayer data. Demo code for unpacking `here `__ +- :code:`isp` - :ref:`ImgFrame` - YUV420 planar (same as YU12/IYUV/I420) +- :code:`still` - :ref:`ImgFrame` - NV12, suitable for bigger size frames. Image gets created when a capture event is sent to the ColorCamera, so it's like taking a photo +- :code:`preview` - :ref:`ImgFrame` - RGB (or BGR planar/interleaved if configured), mostly suited for small size previews and to feed the image into :ref:`NeuralNetwork` +- :code:`video` - :ref:`ImgFrame` - NV12, suitable for bigger size frames -:code:`Preview` is RGB (or BGR planar/interleaved if configured) and is mostly suited for small size previews and to feed the image -into :ref:`NeuralNetwork`. :code:`video` and :code:`still` are both NV12, so are suitable for bigger sizes. :code:`still` image gets created when -a capture event is sent to the ColorCamera, so it's like taking a photo. +**ISP** (image signal processor) interacts with the 3A algorithms: **auto-focus**, **auto-exposure** and **auto-white-balance**. ISP handles image +sensor adjustments such as exposure time, sensitivity (ISO) and lens position (if image sensor in motorized) at runtime. +Click `here `__ for more information. -**ISP** (image signal processor) is used for bayer transformation, demosaicing, noise reduction etc. Click -`here `__ for more information. - -**Encoder** converts YUV420 planar frames from **ISP** into :code:`video`/:code:`preview`/:code:`still` frames. +**Image Post-Processing** converts YUV420 planar frames from the **ISP** into :code:`video`/:code:`preview`/:code:`still` frames. Usage ##### From 147f29c45a5623666125f6c4779057a5bf812066 Mon Sep 17 00:00:00 2001 From: Erol444 Date: Sun, 16 May 2021 00:27:53 +0100 Subject: [PATCH 4/4] Updated colorCamera ISP description --- docs/source/components/nodes/color_camera.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/source/components/nodes/color_camera.rst b/docs/source/components/nodes/color_camera.rst index 68c7e6fc7..daf4dcd8c 100644 --- a/docs/source/components/nodes/color_camera.rst +++ b/docs/source/components/nodes/color_camera.rst @@ -48,12 +48,13 @@ Inputs and Outputs - :code:`inputControl` - :ref:`CameraControl` - :code:`raw` - :ref:`ImgFrame` - RAW10 bayer data. Demo code for unpacking `here `__ - :code:`isp` - :ref:`ImgFrame` - YUV420 planar (same as YU12/IYUV/I420) -- :code:`still` - :ref:`ImgFrame` - NV12, suitable for bigger size frames. Image gets created when a capture event is sent to the ColorCamera, so it's like taking a photo +- :code:`still` - :ref:`ImgFrame` - NV12, suitable for bigger size frames. The image gets created when a capture event is sent to the ColorCamera, so it's like taking a photo - :code:`preview` - :ref:`ImgFrame` - RGB (or BGR planar/interleaved if configured), mostly suited for small size previews and to feed the image into :ref:`NeuralNetwork` - :code:`video` - :ref:`ImgFrame` - NV12, suitable for bigger size frames -**ISP** (image signal processor) interacts with the 3A algorithms: **auto-focus**, **auto-exposure** and **auto-white-balance**. ISP handles image -sensor adjustments such as exposure time, sensitivity (ISO) and lens position (if image sensor in motorized) at runtime. +**ISP** (image signal processor) is used for bayer transformation, demosaicing, noise reduction, and other image enhancements. +It interacts with the 3A algorithms: **auto-focus**, **auto-exposure**, and **auto-white-balance**, which are handling image sensor +adjustments such as exposure time, sensitivity (ISO), and lens position (if the camera module has a motorized lens) at runtime. Click `here `__ for more information. **Image Post-Processing** converts YUV420 planar frames from the **ISP** into :code:`video`/:code:`preview`/:code:`still` frames.