Fixes and improvements for StereoDepth, ColorCamera #82

alex-luxonis · 2021-03-09T05:34:03Z

StereoDepth: allow streaming both disparity and depth (but still not possible to have both when Subpixel mode is enabled)
Fix some crashes when the Stereo extended modes (LR-check/extended-disparity/subpixel) were used in parallel with other Shaves (NN or RGB postprocessing). May not be fully solved yet.
Fix a crash in Stereo standard mode, with depth output enabled, but median filtering disabled.
Add API to override baseline and FOV config (from EEPROM / defaults), useful for depth calculation with .setEmptyCalibration() and already rectified frames streamed from host
Deprecated setOutputRectified/setOutputDepth, the outputs will be automatically enabled when connected in the pipeline
ColorCamera: add isp and raw outputs, scale config for isp. Currently when the scaling is enabled, will need to disable preview scaling (if the output is used). TODO fix.
Other FW changes picked from PR Wip gen2 detection 3d #74:
- undefined/invalid depth set to 0 instead of infinity (65535 mm)
- focal length for depth calculation taken from calib data (right camera intrinsic), instead of using the H-FOV in board config
- depth output changed to U16 instead of FP16 for all modes

Works with LRcheck or LRcheck+Subpixel for now. The updated FW also fixes some crashes with LRcheck mode enabled

themarpe · 2021-03-15T11:01:12Z

include/depthai/pipeline/node/StereoDepth.hpp

+    /**
+     * @param align Set the disparity/depth alignment to the perspective of one camera
+     */
+    void setDepthAlign(Properties::DepthAlign align);


Should we use CameraBoardSocket to which to align to, instead of new DepthAlign enum?

Currently DepthAlign contains: { AUTO = -1, RIGHT, LEFT, CENTER, RGB }

RIGHT (default) and LEFT would map to the respective 'rectified' streams.

CENTER would just displace the disparity map (each pixel by its disparity_value / 2), while RGB (not yet implemented, defaulting with a warning to CENTER) would need an additional 'warp' step (extrinsics alignment / resolution scaling). For boards like BW1092 where the RGB camera is not centered, the disparity displacement would need to be done by a different amount.

Mapping to the real LEFT and RIGHT (from CameraBoardSocket), we would need to do and inverse homography transform, which should be easily doable with the mechanism for RGB in place. But it incurs the additional warp overhead, and in many cases it's not needed (as could be used directly with the rectified streams).

Should then we extend DepthAlign as: { AUTO = -1, RECTIFIED_RIGHT, RECTIFIED_LEFT, CENTER, RGB, LEFT, RIGHT } ?

What about if we separate this two to:

CameraBoardSocket <- which camera it references (can be up to number of MIPI lanes, not added yet to enum)

DepthAlign <- "mode" to referenced, like: RECTIFIED, ORIGINAL, CENTER, .. (TBD)

I think we'd cover all cases with this, without too much repetition.

Definitely adding a separate config with CameraBoardSocket is good, I will add it.

But one case where combining these two might be confusing is with e.g. CameraBoardSocket::RGB and mode: RECTIFIED.

Or with images from host, or running Stereo on OAK-1: one input from the RGB camera, the other from host (actually forwarded by another OAK-1), as one customer wants to use. Actually I think here we should have a way to pass a minimal StereoCalibration structure to StereoDepth, that won't refer to CameraBoardSocket entries, but simply to the 'left' and 'right' inputs to the node. We can discuss further on luxonis/depthai-shared#19

So I was thinking to reference based on the I/Os of the StereoDepth node. Something like
DepthAlign: RECTIFIED_RIGHT, RECTIFIED_LEFT, RECTIFIED_CENTER, INPUT_LEFT, INPUT_RIGHT, OTHER_CAMERA.

When OTHER_CAMERA is set, will specify it by CameraBoardSocket. And for example running standard stereo on OAK-D, INPUT_RIGHT would be equivalent to OTHER_CAMERA + CameraBoardSocket::RIGHT.

Thoughts on that? Also how to have the API, two separate functions, one with two parameters, or an overloaded function (pass either DepthAlign or CameraBoardSocket, and then remove OTHER_CAMERA)?
CC @saching13

What martin suggested works best if we ignore the concept that left, and right are our two global shutter cameras.

In stereo node we set left camera and right camera using CameraBoardSocket and we have

DepthAlign <- "mode" to referenced, like: RECTIFIED, ORIGINAL, CENTER, .. (TBD)

In DepthAlign mode let's have only RECTIFIED_LEFT, RECTIFIED_RIGHT and CENTER.

And setAlignment(cameraBoardSocket) will make sure to align to any camera's that we have.
And another overload function or something like setExtendedAlignment takes DepthAlign mode. But in this case RECTIFIED_LEFT will be the rectified_left frame of the camera/image passed to the stereo left. and RECTIFIED_RIGHT will be the rectified_right frame of the camera/image passed as the right image to the stereo node.

One such case is where rgb camera can be stereo left and right camera can be stereo right.
And CENTER would not be the center of the device but the center of the stereo baseline.

Thoughts ?

Any resolution here? I think I've understood, and a lot of these are sounding good to me.

Changed as an overloaded function, as the options (DepthAlign and CameraBoardSocket) cannot be combined, and would be confusing to set both.

…egrees', will be refactored when the new calibration structure is integrated

themarpe

Also don't forget to add outputs 'isp' & 'raw' to the list of outputs

std::vector<Node::Output> ColorCamera::getOutputs() {
    return {video, preview, still};
}

themarpe · 2021-03-25T16:36:06Z

src/pipeline/node/ColorCamera.cpp

+void ColorCamera::setIspScale(int numerator, int denominator) {
+    properties.ispScale.horizNumerator = numerator;
+    properties.ispScale.horizDenominator = denominator;
+    properties.ispScale.vertNumerator = numerator;
+    properties.ispScale.vertDenominator = denominator;
+}
+
+void ColorCamera::setIspScaleFull(int horizNum, int horizDenom, int vertNum, int vertDenom) {
+    properties.ispScale.horizNumerator = horizNum;
+    properties.ispScale.horizDenominator = horizDenom;
+    properties.ispScale.vertNumerator = vertNum;
+    properties.ispScale.vertDenominator = vertDenom;
+}
+


Lets overload these two as well.
As well as an tuple overload (plays nicely with initializer lists and tuples/lists in python)

setIspScale(int numerator, int denominator) setIspScale(std::tuple<int, int> scale) setIspScale(int horizNum, int horizDenom, int vertNum, int vertDenom) setIspScale(std::tuple<int, int> horizScale, std::tuple<int, int> vertScale)

Also same for setPreviewSize, setVideoSize and setStillSize

themarpe · 2021-03-25T16:38:09Z

include/depthai/pipeline/node/ColorCamera.hpp

+    Output isp{*this, "isp", Output::Type::MSender, {{DatatypeEnum::ImgFrame, false}}};
+    Output raw{*this, "raw", Output::Type::MSender, {{DatatypeEnum::ImgFrame, false}}};


Missing docstring

TODO depth and subpixel (U16)

- add `isp` and `raw` to ColorCamera list of outputs, add docstrings - overloaded `setIspScale`, with tuple inputs as options - also overloaded `setPreviewSize`, `setVideoSize` and `setStillSize` with tuple inputs

remove non-implemented setNoiseReductionStrength. Updated FW: all initial controls can be applied for Mono/ColorCamera (no longer limited to focus settings for Color)

the chroma planes were swapped

alex-luxonis · 2021-04-01T15:15:04Z

src/opencv/ImgFrame.cpp

@@ -118,7 +118,7 @@ cv::Mat ImgFrame::getCvFrame() {
        } break;

        case Type::YUV420p:
-            cv::cvtColor(frame, output, cv::ColorConversionCodes::COLOR_YUV420p2BGR);
+            cv::cvtColor(frame, output, cv::ColorConversionCodes::COLOR_YUV2BGR_IYUV);


About the change here, it may look counter-intuitive, but the YUV420p frames we receive from device (currently from ColorCamera::isp output that is added on this PR) have the plane order: Y, U, V.

In OpenCV the format YUV420p is defined as an equivalent to YV12:
cv::COLOR_YUV420p2BGR = COLOR_YUV2BGR_YV12,
https://docs.opencv.org/4.5.1/d8/d01/group__imgproc__color__conversions.html

While YV12 has the plane order: Y, V, U:
https://www.fourcc.org/pixel-format/yuv-yv12/

to prevent an out-of-memory situation due to too many threads created

themarpe

I'd only apply the --parallel 8 option to CI: main.workflow.yml and
And leave --parallel elsewhere.

alex-luxonis · 2021-04-02T00:20:30Z

So I added --parallel 8 in the Readme command snippets more as a guideline for the users to limit the number of build jobs, as by default --parallel seems to create as many processes as the underlying build tool (make, etc) needs (I see over 40 build jobs fired at once on my 8 CPU threads machine), and needs several GB of RAM, memory that if isn't free/available, this could easily hang up the system.
But can remove it. Not sure then if we should put a note somewhere else.

(still not optimized)

themarpe · 2021-04-02T08:36:49Z

The problem with --parallel 8 is that it won't solve the issue for eg RPi, etc... and the value is arbitrary.
Should we leave it as --parallel but add a note something like:

ℹ️ Use --parallel [num CPU cores] to reduce memory consumption

Or add this directly into the code block.

On other hand, for script like in depthai-python/docs/docker/update_docs.sh we can do:

cmake --build build --target sphinx --parallel $(nproc)

Also apparently ninja uses num cpu + 2 as default multithreading number of concurrent builds, which is nice:)

don't link depth in pipeline if not used, and other cleanup. Update FW: ispScale factors simplification done on device, other bugfixes

The change in discussion (--parallel 8 in Readme) was already cherry-picked to develop (so no longer appears on this PR)

…ings)

Update shared: stereo_fixes merged

alex-luxonis added 6 commits March 9, 2021 06:08

ColorCamera: add isp, raw outputs

db02740

ColorCamera: add setIspScale/setIspScaleFull API

ea9a98b

StereoDepth: deprecate setOutputDepth/Rectified

0431b74

ColorCamera: add API to get 'isp' scaled output size

5f2f312

make clangformat

e04b257

Update FW: stereo fixes, stereo/ColorCamera improvements

2f9ad0a

alex-luxonis marked this pull request as draft March 9, 2021 05:34

alex-luxonis mentioned this pull request Mar 9, 2021

Fixes and improvements for StereoDepth, ColorCamera luxonis/depthai-python#147

Merged

Add API to configure disparity/depth alignment: left/right/center.

f22ca12

Works with LRcheck or LRcheck+Subpixel for now. The updated FW also fixes some crashes with LRcheck mode enabled

themarpe reviewed Mar 15, 2021

View reviewed changes

alex-luxonis added 3 commits March 25, 2021 11:31

Merge remote-tracking branch 'origin/develop' into stereo_fixes

4fc1433

StereoDepth: add overloaded setDepthAlign(CameraBoardSocket)

974dc98

StereoDepth: remove for now 'setBaselineOverrideCm', 'setFovOverrideD…

4ec7f1f

…egrees', will be refactored when the new calibration structure is integrated

themarpe requested changes Mar 25, 2021

View reviewed changes

alex-luxonis added 4 commits March 30, 2021 14:12

Merge remote-tracking branch 'origin/develop' into stereo_fixes

ca3784b

Update FW: disparity (U8) aligning to RGB works.

69726b3

TODO depth and subpixel (U16)

Address review comments:

f809bb6

- add `isp` and `raw` to ColorCamera list of outputs, add docstrings - overloaded `setIspScale`, with tuple inputs as options - also overloaded `setPreviewSize`, `setVideoSize` and `setStillSize` with tuple inputs

CameraControl: add ranges for extra controls,

d4338e2

remove non-implemented setNoiseReductionStrength. Updated FW: all initial controls can be applied for Mono/ColorCamera (no longer limited to focus settings for Color)

alex-luxonis marked this pull request as ready for review March 31, 2021 13:20

alex-luxonis requested review from SzabolcsGergely and themarpe March 31, 2021 13:46

themarpe approved these changes Mar 31, 2021

View reviewed changes

alex-luxonis added 3 commits April 1, 2021 17:05

Add rgb_depth_aligned example

8c3a8e6

Fix conversion of YUV420p frames to OpenCV BGR,

29ca1c5

the chroma planes were swapped

Merge remote-tracking branch 'origin/develop' into stereo_fixes

757d830

alex-luxonis commented Apr 1, 2021

View reviewed changes

alex-luxonis added 4 commits April 1, 2021 19:06

Examples: remove deprecated API setOutputDepth/Rectified

3964a7e

GitHub CI: limit cmake --parallel to 8 threads,

fefb4b0

to prevent an out-of-memory situation due to too many threads created

README build snippets: limit cmake --parallel to 8

878e4a6

Cleanup, remove some unused variables

b72bb07

themarpe requested changes Apr 1, 2021

View reviewed changes

Update FW: optimize depth align, make it work with subpixel/U16

7df7402

(still not optimized)

SzabolcsGergely mentioned this pull request Apr 12, 2021

SpatialCalculator fixes #106

Merged

alex-luxonis and others added 10 commits April 19, 2021 11:44

Merge remote-tracking branch 'origin/develop' into stereo_fixes

df43775

stereo_example: rectified flip no longer needed with LR-check on,

b0dd5d6

don't link depth in pipeline if not used, and other cleanup. Update FW: ispScale factors simplification done on device, other bugfixes

spatial_object_tracker example: remove deprecated setOutputDepth

6e11c1f

fixed getting size of video/still when setting isp scaling

1e05f8c

Address review comments. Note:

3f2fb9d

The change in discussion (--parallel 8 in Readme) was already cherry-picked to develop (so no longer appears on this PR)

clangformat cleanup

6d84ac9

Update FW: fix still capture with scaling, add FPS capping (with warn…

f62304d

…ings)

Merge remote-tracking branch 'origin/develop' into stereo_fixes

2e25a1e

Update FW: fix ImageManip U16 crop (for depth/subpixel disparity)

bf1cbdd

Update shared: stereo_fixes merged

Update FW, fix CI build: depthai-shared PR merged

bffa915

alex-luxonis merged commit 8b7eaf7 into develop Apr 28, 2021

alex-luxonis deleted the stereo_fixes branch April 28, 2021 11:28

Luxonis-Brandon added the Gen2 Feature for or issue with Gen2 label Apr 28, 2021

Luxonis-Brandon added this to In progress in Gen2 Phase I Delivery via automation Apr 28, 2021

Luxonis-Brandon moved this from In progress to Done in Gen2 Phase I Delivery Apr 28, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes and improvements for StereoDepth, ColorCamera #82

Fixes and improvements for StereoDepth, ColorCamera #82

alex-luxonis commented Mar 9, 2021

themarpe Mar 15, 2021

alex-luxonis Mar 16, 2021

themarpe Mar 17, 2021

alex-luxonis Mar 18, 2021

saching13 Mar 18, 2021

Luxonis-Brandon Mar 25, 2021

alex-luxonis Mar 25, 2021

themarpe left a comment

themarpe Mar 25, 2021

themarpe Mar 25, 2021

themarpe Mar 25, 2021

alex-luxonis Apr 1, 2021

themarpe left a comment

alex-luxonis commented Apr 2, 2021

themarpe commented Apr 2, 2021

		Output isp{*this, "isp", Output::Type::MSender, {{DatatypeEnum::ImgFrame, false}}};
		Output raw{*this, "raw", Output::Type::MSender, {{DatatypeEnum::ImgFrame, false}}};

Fixes and improvements for StereoDepth, ColorCamera #82

Fixes and improvements for StereoDepth, ColorCamera #82

Conversation

alex-luxonis commented Mar 9, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

themarpe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

themarpe left a comment

Choose a reason for hiding this comment

alex-luxonis commented Apr 2, 2021

themarpe commented Apr 2, 2021