From e02d78402095aac8c6905a000f1bf975dc966391 Mon Sep 17 00:00:00 2001 From: Erol444 Date: Wed, 27 Oct 2021 21:59:03 +0200 Subject: [PATCH 1/6] Added stereoDepthConfig msg and altered the IO ascii art --- .../messages/stereo_depth_config.rst | 32 +++++++++++++++++++ docs/source/components/nodes/stereo_depth.rst | 21 +++++++++--- 2 files changed, 49 insertions(+), 4 deletions(-) create mode 100644 docs/source/components/messages/stereo_depth_config.rst diff --git a/docs/source/components/messages/stereo_depth_config.rst b/docs/source/components/messages/stereo_depth_config.rst new file mode 100644 index 000000000..c0df070ac --- /dev/null +++ b/docs/source/components/messages/stereo_depth_config.rst @@ -0,0 +1,32 @@ +StereoDepthConfig +================= + +This message is used to configure the :ref:`StereoDepth` node. +With this message you can set filters, confidences, thresholds and mode of the :ref:`StereoDepth` node. + +Examples of functionality +######################### + +- :ref:`Stereo Depth from host` + +Reference +######### + +.. tabs:: + + .. tab:: Python + + .. autoclass:: depthai.StereoDepthConfig + :members: + :inherited-members: + :noindex: + + .. tab:: C++ + + .. doxygenclass:: dai::StereoDepthConfig + :project: depthai-core + :members: + :private-members: + :undoc-members: + +.. include:: ../../includes/footer-short.rst diff --git a/docs/source/components/nodes/stereo_depth.rst b/docs/source/components/nodes/stereo_depth.rst index d36cd1204..2f4a19b26 100644 --- a/docs/source/components/nodes/stereo_depth.rst +++ b/docs/source/components/nodes/stereo_depth.rst @@ -25,24 +25,31 @@ Inputs and Outputs .. code-block:: ┌───────────────────┐ + │ │ confidenceMap + │ ├─────────────► │ │rectifiedLeft │ ├─────────────► left │ │ syncedLeft - ──────────────►│ ├─────────────► + ──────────────►│-------------------├─────────────► │ │ depth │ ├─────────────► │ StereoDepth │ disparity │ ├─────────────► - right │ │rectifiedRight - ──────────────►│ ├─────────────► - │ │ syncedRight + right │ │ syncedRight + ──────────────►│-------------------├─────────────► + │ │rectifiedRight │ ├─────────────► + inputConfig │ | outConfig + ──────────────►│-------------------├─────────────► └───────────────────┘ **Message types** - :code:`left` - :ref:`ImgFrame` from the left :ref:`MonoCamera` - :code:`right` - :ref:`ImgFrame` from the right :ref:`MonoCamera` +- :code:`inputConfig` - :ref:`StereoDepthConfig` + +- :code:`confidenceMap` - :ref:`ImgFrame` - :code:`rectifiedLeft` - :ref:`ImgFrame` - :code:`syncedLeft` - :ref:`ImgFrame` - :code:`depth` - :ref:`ImgFrame` @@ -50,6 +57,12 @@ Inputs and Outputs - :code:`rectifiedRight` - :ref:`ImgFrame` - :code:`syncedRight` - :ref:`ImgFrame` +**Debug outputs** + +- :code:`debugDispLrCheckIt1` - :ref:`ImgFrame` +- :code:`debugDispLrCheckIt2` - :ref:`ImgFrame` +- :code:`debugDispCostDump` - :ref:`ImgFrame` + Internal block diagram of StereoDepth node ########################################## From 61c3a4d4717ebdf1d54bfa32acbb21be9fb673b3 Mon Sep 17 00:00:00 2001 From: Erol444 Date: Sun, 21 Nov 2021 20:02:02 +0100 Subject: [PATCH 2/6] Added tabs for IOs --- docs/source/components/nodes/stereo_depth.rst | 41 ++++++++++++------- 1 file changed, 26 insertions(+), 15 deletions(-) diff --git a/docs/source/components/nodes/stereo_depth.rst b/docs/source/components/nodes/stereo_depth.rst index 2f4a19b26..1d83aa9d8 100644 --- a/docs/source/components/nodes/stereo_depth.rst +++ b/docs/source/components/nodes/stereo_depth.rst @@ -43,25 +43,32 @@ Inputs and Outputs ──────────────►│-------------------├─────────────► └───────────────────┘ -**Message types** +.. tabs:: + + .. tab:: **Inputs** + + - :code:`left` - :ref:`ImgFrame` from the left :ref:`MonoCamera` + - :code:`right` - :ref:`ImgFrame` from the right :ref:`MonoCamera` + - :code:`inputConfig` - :ref:`StereoDepthConfig` -- :code:`left` - :ref:`ImgFrame` from the left :ref:`MonoCamera` -- :code:`right` - :ref:`ImgFrame` from the right :ref:`MonoCamera` -- :code:`inputConfig` - :ref:`StereoDepthConfig` + .. tab:: **Outputs** -- :code:`confidenceMap` - :ref:`ImgFrame` -- :code:`rectifiedLeft` - :ref:`ImgFrame` -- :code:`syncedLeft` - :ref:`ImgFrame` -- :code:`depth` - :ref:`ImgFrame` -- :code:`disparity` - :ref:`ImgFrame` -- :code:`rectifiedRight` - :ref:`ImgFrame` -- :code:`syncedRight` - :ref:`ImgFrame` + - :code:`confidenceMap` - :ref:`ImgFrame` + - :code:`rectifiedLeft` - :ref:`ImgFrame` + - :code:`syncedLeft` - :ref:`ImgFrame` + - :code:`depth` - :ref:`ImgFrame` + - :code:`disparity` - :ref:`ImgFrame` + - :code:`rectifiedRight` - :ref:`ImgFrame` + - :code:`syncedRight` - :ref:`ImgFrame` + - :code:`outConfig` - :ref:`StereoDepthConfig` -**Debug outputs** + .. tab:: **Debug outputs** -- :code:`debugDispLrCheckIt1` - :ref:`ImgFrame` -- :code:`debugDispLrCheckIt2` - :ref:`ImgFrame` -- :code:`debugDispCostDump` - :ref:`ImgFrame` + - :code:`debugDispLrCheckIt1` - :ref:`ImgFrame` + - :code:`debugDispLrCheckIt2` - :ref:`ImgFrame` + - :code:`debugExtDispLrCheckIt1` - :ref:`ImgFrame` + - :code:`debugExtDispLrCheckIt2` - :ref:`ImgFrame` + - :code:`debugDispCostDump` - :ref:`ImgFrame` Internal block diagram of StereoDepth node ########################################## @@ -93,6 +100,8 @@ Currently configurable blocks #. Computes disparity by matching in L->R direction #. Combines results from 1 and 2, running on Shave: each pixel d = disparity_LR(x,y) is compared with disparity_RL(x-d,y). If the difference is above a threshold, the pixel at (x,y) in the final disparity map is invalidated. + You can use :code:`debugDispLrCheckIt1` and :code:`debugDispLrCheckIt2` debug outputs for debugging/fine-tuning purposes. + .. tab:: Extended Disparity The :code:`extended disparity` allows detecting closer distance objects for the given baseline. This increases the maximum disparity search from 96 to 191, meaning the range is now: **[0..190]**. @@ -103,6 +112,8 @@ Currently configurable blocks #. Computes disparity on 2x downscaled images (e.g. 640x360) #. Combines the two level disparities on Shave, effectively covering a total disparity range of 191 pixels (in relation to the original resolution). + You can use :code:`debugExtDispLrCheckIt1` and :code:`debugExtDispLrCheckIt2` debug outputs for debugging/fine-tuning purposes. + .. tab:: Subpixel Disparity Subpixel improves the precision and is especially useful for long range measurements. It also helps for better estimating surface normals. From 3baeb1e7843c794422b7042a5a9d76cfdd13439f Mon Sep 17 00:00:00 2001 From: Erol444 Date: Sun, 21 Nov 2021 20:18:26 +0100 Subject: [PATCH 3/6] Update subpixel as per PR review --- docs/source/components/nodes/stereo_depth.rst | 21 +++++++------------ 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/docs/source/components/nodes/stereo_depth.rst b/docs/source/components/nodes/stereo_depth.rst index 1d83aa9d8..aff5adfaf 100644 --- a/docs/source/components/nodes/stereo_depth.rst +++ b/docs/source/components/nodes/stereo_depth.rst @@ -93,7 +93,7 @@ Currently configurable blocks .. tab:: Left-Right Check - Left-Right Check or LR-Check is used to remove incorrectly calculated disparity pixels due to occlusions at object borders (Left and Right camera views + **Left-Right Check** or LR-Check is used to remove incorrectly calculated disparity pixels due to occlusions at object borders (Left and Right camera views are slightly different). #. Computes disparity by matching in R->L direction @@ -104,7 +104,7 @@ Currently configurable blocks .. tab:: Extended Disparity - The :code:`extended disparity` allows detecting closer distance objects for the given baseline. This increases the maximum disparity search from 96 to 191, meaning the range is now: **[0..190]**. + **Extended disparity mode** allows detecting closer distance objects for the given baseline. This increases the maximum disparity search from 96 to 191, meaning the range is now: **[0..190]**. So this cuts the minimum perceivable distance in half, given that the minimum distance is now :code:`focal_length * base_line_dist / 190` instead of :code:`focal_length * base_line_dist / 95`. @@ -116,13 +116,13 @@ Currently configurable blocks .. tab:: Subpixel Disparity - Subpixel improves the precision and is especially useful for long range measurements. It also helps for better estimating surface normals. + **Subpixel mode** improves the precision and is especially useful for long range measurements. It also helps for better estimating surface normals. Besides the integer disparity output, the Stereo engine is programmed to dump to memory the cost volume, that is 96 levels (disparities) per pixel, - then software interpolation is done on Shave, resulting a final disparity with 5 fractional bits, resulting in significantly more granular depth - steps (32 additional steps between the integer-pixel depth steps), and also theoretically, longer-distance depth viewing - as the maximum depth - is no longer limited by a feature being a full integer pixel-step apart, but rather 1/32 of a pixel. In this mode, stereo cameras perform: :code:`96 depth steps * 32 subpixel depth steps = 3,072 depth steps.` - Note that Subpixel and Extended Disparity are not yet supported simultaneously (which would result in :code:`191 * 32 = 6,112 depth steps`), but should be available in the near future (`Pull Request `__). + then software interpolation is done on Shave, resulting a final disparity with 3 fractional bits, resulting in significantly more granular depth + steps (8 additional steps between the integer-pixel depth steps), and also theoretically, longer-distance depth viewing - as the maximum depth + is no longer limited by a feature being a full integer pixel-step apart, but rather 1/8 of a pixel. In this mode, stereo cameras perform: :code:`95 depth steps * 8 subpixel depth steps + 1(invalid disparity 0) = 761 depth steps` + Note that Subpixel and Extended Disparity are not yet supported simultaneously. For comparison of normal disparity vs. subpixel disparity images, click `here `__. @@ -162,12 +162,7 @@ Currently configurable blocks Current limitations ################### -If one or more of the additional depth modes (:code:`lrcheck`, :code:`extended`, :code:`subpixel`) are enabled, then: - -- median filtering is disabled on device -- with subpixel, if both :code:`depth` and :code:`disparity` are used, only :code:`depth` will have valid output - -Otherwise, :code:`depth` output is **U16** (in millimeters) and median is functional. +When using subpixel mode, if both :code:`depth` and :code:`disparity` are used, only :code:`depth` will have valid output. Stereo depth FPS ################ From 68c4363eac1c5f5d1c41015d9039f8e3fab55aaf Mon Sep 17 00:00:00 2001 From: Erol444 Date: Sun, 21 Nov 2021 22:59:29 +0100 Subject: [PATCH 4/6] Added subpixel + filter limitation --- docs/source/components/nodes/stereo_depth.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/source/components/nodes/stereo_depth.rst b/docs/source/components/nodes/stereo_depth.rst index aff5adfaf..7e0907532 100644 --- a/docs/source/components/nodes/stereo_depth.rst +++ b/docs/source/components/nodes/stereo_depth.rst @@ -162,7 +162,8 @@ Currently configurable blocks Current limitations ################### -When using subpixel mode, if both :code:`depth` and :code:`disparity` are used, only :code:`depth` will have valid output. +- When using subpixel mode, if both :code:`depth` and :code:`disparity` are used, only :code:`depth` will have valid output. +- Median filtering is disabled when subpixel mode is set to 4 or 5 bits. Stereo depth FPS ################ From 1d9dfba6194ee5d6456fd2ce33161cd03973be33 Mon Sep 17 00:00:00 2001 From: Erik Date: Mon, 22 Nov 2021 00:11:59 +0100 Subject: [PATCH 5/6] Update docs/source/components/nodes/stereo_depth.rst Co-authored-by: szabi-luxonis <60790666+szabi-luxonis@users.noreply.github.com> --- docs/source/components/nodes/stereo_depth.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/components/nodes/stereo_depth.rst b/docs/source/components/nodes/stereo_depth.rst index 7e0907532..86f93e265 100644 --- a/docs/source/components/nodes/stereo_depth.rst +++ b/docs/source/components/nodes/stereo_depth.rst @@ -121,7 +121,7 @@ Currently configurable blocks Besides the integer disparity output, the Stereo engine is programmed to dump to memory the cost volume, that is 96 levels (disparities) per pixel, then software interpolation is done on Shave, resulting a final disparity with 3 fractional bits, resulting in significantly more granular depth steps (8 additional steps between the integer-pixel depth steps), and also theoretically, longer-distance depth viewing - as the maximum depth - is no longer limited by a feature being a full integer pixel-step apart, but rather 1/8 of a pixel. In this mode, stereo cameras perform: :code:`95 depth steps * 8 subpixel depth steps + 1(invalid disparity 0) = 761 depth steps` + is no longer limited by a feature being a full integer pixel-step apart, but rather 1/8 of a pixel. In this mode, stereo cameras perform: :code:`94 depth steps * 8 subpixel depth steps + 2 (min/max values) = 754 depth steps` Note that Subpixel and Extended Disparity are not yet supported simultaneously. For comparison of normal disparity vs. subpixel disparity images, click `here `__. From adbad27a6fd6c80222b9c03374118ab7ff6d98b3 Mon Sep 17 00:00:00 2001 From: Erik Date: Mon, 22 Nov 2021 00:13:15 +0100 Subject: [PATCH 6/6] Update docs/source/components/nodes/stereo_depth.rst --- docs/source/components/nodes/stereo_depth.rst | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/source/components/nodes/stereo_depth.rst b/docs/source/components/nodes/stereo_depth.rst index 86f93e265..0da516d51 100644 --- a/docs/source/components/nodes/stereo_depth.rst +++ b/docs/source/components/nodes/stereo_depth.rst @@ -162,7 +162,6 @@ Currently configurable blocks Current limitations ################### -- When using subpixel mode, if both :code:`depth` and :code:`disparity` are used, only :code:`depth` will have valid output. - Median filtering is disabled when subpixel mode is set to 4 or 5 bits. Stereo depth FPS