Enhance roi depth mapping by alex-luxonis · Pull Request #85 · luxonis/depthai

alex-luxonis · 2020-04-28T13:47:34Z

Enhance average depth ROI and X,Y,Z coordinates calculation. Issue ROI and Depth Only Approximately Mapped Currently #74
Add an option to overlay the NN bounding box on left/right/depth_sipp/depth_color_h streams. Issue Add Option to Overlay Bounding Box On depth_sipp #75

  -bb, --draw-bb-depth  Draw the bounding boxes over the left/right/depth*
                        streams

Add a fix for the horizontal stripes artifacts showing on the right side of previewout with some NN models. Issue Strange artifacts showing on right side of preview image #72

SzabolcsGergely

lgtm

Luxonis-Brandon · 2020-04-29T20:20:50Z

So trying this on Mac and running into problems:

python3 test.py -s metaout left -bb
python3 depthai.py '-s' 'metaout' 'left' '-bb' 
No calibration file. Using Calibration Defaults.
Using Arguments= {'config_overwrite': None, 'board': None, 'field_of_view': 71.86, 'rgb_field_of_view': 68.7938, 'baseline': 9.0, 'rgb_baseline': 2.0, 'swap_lr': True, 'store_eeprom': False, 'clear_eeprom': False, 'override_eeprom': False, 'device_id': '', 'dev_debug': None, 'force_usb2': None, 'cnn_model': 'mobilenet-ssd', 'disable_depth': False, 'draw_bb_depth': True, 'streams': [{'name': 'metaout'}, {'name': 'left'}]}
depthai.__version__ == 0.0.10a
depthai.__dev_version__ == 560396b735544c84935dcaeff2e5fecde135bfef
XLink initialized.
Sending device firmware "cmd_file": /Users/leeroy/depthai/depthai.cmd
Successfully connected to device.
Loading config file
Attempting to open stream config_d2h
watchdog started 1000
Successfully opened stream config_d2h with ID #0!
Closing stream config_d2h: ...
Closing stream config_d2h: DONE.
WARNING: Version (dev) does not match (255b5d738bdfacde1f47d441e7d48b4ffee10942 & 560396b735544c84935dcaeff2e5fecde135bfef)
EEPROM data: valid (v2)
  Board name     : BW1098OBC
  Board rev      : R0M0E0
  HFOV L/R       : 71.86 deg
  HFOV RGB       : 68.7938 deg
  L-R   distance : 7.5 cm
  L-RGB distance : 3.75 cm
  L/R swapped    : yes
  L/R crop region: top
  Calibration homography:
    1.002324,   -0.004016,   -0.552212,
    0.001249,    0.993829,   -1.710247,
    0.000008,   -0.000010,    1.000000,
Available streams: ['meta_d2h', 'left', 'right', 'disparity', 'depth_sipp', 'metaout', 'previewout']
CNN configurations read: /Users/leeroy/depthai/resources/nn/mobilenet-ssd/mobilenet-ssd_depth.json
depthai: Calibration file is not specified, will use default setting;
config_h2d json:
{"_board":{"_homography_right_to_left":[0.988068163394928,0.0029474012553691864,5.067617416381836,-0.008765067905187607,0.9921473264694214,-8.795275688171387,-8.449587767245248e-06,-3.603489403758431e-06,1.0]},"_load_inBlob":true,"_pipeline":{"_streams":[{"name":"metaout"},{"name":"left"},{"name":"meta_d2h"}]},"ai":{"calc_dist_to_bb":true},"board":{"clear-eeprom":false,"left_fov_deg":71.86000061035156,"left_to_rgb_distance_m":0.019999999552965164,"left_to_right_distance_m":0.09000000357627869,"name":"","override-eeprom":false,"revision":"","rgb_fov_deg":68.7938003540039,"stereo_center_crop":false,"store-to-eeprom":false,"swap-left-and-right-cameras":true},"depth":{"padding_factor":0.30000001192092896}}
Attempting to open stream config_h2d
Successfully opened stream config_h2d with ID #1!
Writing 1000 bytes to config_h2d
!!! XLink write successful: config_h2d (1000)
Closing stream config_h2d: ...
Closing stream config_h2d: DONE.
Read: 14514560
Attempting to open stream inBlob
Successfully opened stream inBlob with ID #1!
Writing 14514560 bytes to inBlob
!!! XLink write successful: inBlob (14514560)
Closing stream inBlob: ...
Closing stream inBlob: DONE.
depthai: done sending Blob file /Users/leeroy/depthai/resources/nn/mobilenet-ssd/mobilenet-ssd.blob
Attempting to open stream outBlob
Successfully opened stream outBlob with ID #0!
Closing stream outBlob: ...
Closing stream outBlob: DONE.
CNN input width: 300
CNN input height: 300
CNN input num channels: 3
CNN to depth bounding-box mapping: start(318, 38), max_size(643, 643)
Host stream start:meta_d2h
Opening stream for read: meta_d2h
Attempting to open stream meta_d2h
Successfully opened stream meta_d2h with ID #1!
Starting thread for stream: meta_d2h
Started thread for stream: meta_d2h
Host stream start:left
Opening stream for read: left
Attempting to open stream left
Successfully opened stream left with ID #2!
Starting thread for stream: left
Started thread for stream: left
Host stream start:metaout
Opening stream for read: metaout
Attempting to open stream metaout
Successfully opened stream metaout with ID #3!
Starting thread for stream: metaout
depthai: INIT OK!
Started thread for stream: metaout
watchdog triggered 
Stopping threads: ...
E: [global] [    570179] [] XLinkReadData:165	addEventWithPerf(&event, &opTime) failed with error: 3
Device get data failed: 3
E: [global] [    570179] [] XLinkReadData:165	addEventWithPerf(&event, &opTime) failed with error: 3
Device get data failed: 3
E: [global] [    570179] [] XLinkReadData:165	addEventWithPerf(&event, &opTime) failed with error: 3
Device get data failed: 3
Closing stream meta_d2h: ...
Closing stream left: ...
Closing stream metaout: ...
E: [global] [    570181] [] dispatcherEventSend:53	Write failed (header) (err -99) | event XLINK_CLOSE_STREAM_REQ

E: [xLink] [    570181] [] sendEvents:1036	Event sending failed
E: [global] [    570181] [] dispatcherEventSend:53	Write failed (header) (err -99) | event XLINK_CLOSE_STREAM_REQ

E: [xLink] [    570181] [] sendEvents:1036	Event sending failed
E: [global] [    570181] [] dispatcherEventSend:53	Write failed (header) (err -99) | event XLINK_CLOSE_STREAM_REQ

E: [xLink] [    570181] [] sendEvents:1036	Event sending failed
Closing stream meta_d2h: DONE.
Thread for meta_d2h finished.
Closing stream left: DONE.
Thread for left finished.
Closing stream metaout: DONE.
Thread for metaout finished.
Stopping threads: DONE.
Closing all observer streams: ...
Closing all observer streams: DONE.
XLink initialized.
No USB device [03e7:2485], still looking... 0.077s [FOUND]
Sending device firmware "cmd_file": /Users/leeroy/depthai/depthai.cmd
Return code:-11

Luxonis-Brandon · 2020-04-29T20:23:40Z

And then the following command doesn't overlay the bounding box on the left stream:

python3 test.py -s metaout previewout left -bb
python3 depthai.py '-s' 'metaout' 'previewout' 'left' '-bb' 
No calibration file. Using Calibration Defaults.
Using Arguments= {'config_overwrite': None, 'board': None, 'field_of_view': 71.86, 'rgb_field_of_view': 68.7938, 'baseline': 9.0, 'rgb_baseline': 2.0, 'swap_lr': True, 'store_eeprom': False, 'clear_eeprom': False, 'override_eeprom': False, 'device_id': '', 'dev_debug': None, 'force_usb2': None, 'cnn_model': 'mobilenet-ssd', 'disable_depth': False, 'draw_bb_depth': True, 'streams': [{'name': 'metaout'}, {'name': 'previewout'}, {'name': 'left'}]}
depthai.__version__ == 0.0.10a
depthai.__dev_version__ == 560396b735544c84935dcaeff2e5fecde135bfef
XLink initialized.
Sending device firmware "cmd_file": /Users/leeroy/depthai/depthai.cmd
Successfully connected to device.
Loading config file
Attempting to open stream config_d2h
watchdog started 1000
Successfully opened stream config_d2h with ID #0!
Closing stream config_d2h: ...
Closing stream config_d2h: DONE.
WARNING: Version (dev) does not match (255b5d738bdfacde1f47d441e7d48b4ffee10942 & 560396b735544c84935dcaeff2e5fecde135bfef)
EEPROM data: valid (v2)
  Board name     : BW1098OBC
  Board rev      : R0M0E0
  HFOV L/R       : 71.86 deg
  HFOV RGB       : 68.7938 deg
  L-R   distance : 7.5 cm
  L-RGB distance : 3.75 cm
  L/R swapped    : yes
  L/R crop region: top
  Calibration homography:
    1.002324,   -0.004016,   -0.552212,
    0.001249,    0.993829,   -1.710247,
    0.000008,   -0.000010,    1.000000,
Available streams: ['meta_d2h', 'left', 'right', 'disparity', 'depth_sipp', 'metaout', 'previewout']
CNN configurations read: /Users/leeroy/depthai/resources/nn/mobilenet-ssd/mobilenet-ssd_depth.json
depthai: Calibration file is not specified, will use default setting;
config_h2d json:
{"_board":{"_homography_right_to_left":[0.988068163394928,0.0029474012553691864,5.067617416381836,-0.008765067905187607,0.9921473264694214,-8.795275688171387,-8.449587767245248e-06,-3.603489403758431e-06,1.0]},"_load_inBlob":true,"_pipeline":{"_streams":[{"name":"metaout"},{"name":"previewout"},{"name":"left"},{"name":"meta_d2h"}]},"ai":{"calc_dist_to_bb":true},"board":{"clear-eeprom":false,"left_fov_deg":71.86000061035156,"left_to_rgb_distance_m":0.019999999552965164,"left_to_right_distance_m":0.09000000357627869,"name":"","override-eeprom":false,"revision":"","rgb_fov_deg":68.7938003540039,"stereo_center_crop":false,"store-to-eeprom":false,"swap-left-and-right-cameras":true},"depth":{"padding_factor":0.30000001192092896}}
Attempting to open stream config_h2d
Successfully opened stream config_h2d with ID #1!
Writing 1000 bytes to config_h2d
!!! XLink write successful: config_h2d (1000)
Closing stream config_h2d: ...
Closing stream config_h2d: DONE.
Read: 14514560
Attempting to open stream inBlob
Successfully opened stream inBlob with ID #1!
Writing 14514560 bytes to inBlob
!!! XLink write successful: inBlob (14514560)
Closing stream inBlob: ...
Closing stream inBlob: DONE.
depthai: done sending Blob file /Users/leeroy/depthai/resources/nn/mobilenet-ssd/mobilenet-ssd.blob
Attempting to open stream outBlob
Successfully opened stream outBlob with ID #0!
Closing stream outBlob: ...
Closing stream outBlob: DONE.
CNN input width: 300
CNN input height: 300
CNN input num channels: 3
CNN to depth bounding-box mapping: start(318, 38), max_size(643, 643)
Host stream start:meta_d2h
Opening stream for read: meta_d2h
Attempting to open stream meta_d2h
Successfully opened stream meta_d2h with ID #1!
Starting thread for stream: meta_d2h
Host stream start:left
Opening stream for read: left
Attempting to open stream left
Started thread for stream: meta_d2h
Successfully opened stream left with ID #2!
Starting thread for stream: left
Started thread for stream: left
Host stream start:metaout
Opening stream for read: metaout
Attempting to open stream metaout
Successfully opened stream metaout with ID #3!
Starting thread for stream: metaout
Host stream start:previewout
Opening stream for read: previewout
Attempting to open stream previewout
Started thread for stream: metaout
Successfully opened stream previewout with ID #4!
Starting thread for stream: previewout
Started thread for stream: previewout
depthai: INIT OK!
py: DONE.
Return code:0

Luxonis-Brandon · 2020-04-29T20:33:13Z

OK, using python3 test.py -bb -s metaout previewout depth_sipp works for overlaying on the depth stream:

And this seems to work well:
https://photos.app.goo.gl/hXGFvM9aHgSvRAbK7

As discussed internally I'm wondering if it would make sense to allow an option for the neural networks to get the full field of view of the color camera (to 'squish' the image into the square aspect ratio of the say 300x300 pixel model), such that the whole field of view for color camera is still available even when square models are used.

Luxonis-Brian

I just tested with BW1097 (CM3+) as well, and the bounding boxes on the depth_sipp look good. I did notice the offset in the box locations we had been discussing, but it's not as bad as I was thinking. One thing I did notice was that the previewout and depth_sipp (-bb) streams would display slightly different location values at each frame. I noticed that one was not following the other's values, that they each seemed to be using different values. I figure this is just a normal artifact of the metaout running at 30fps in the background and the two streams grabbing the values at slightly different times. I don't think it's a problem, it's just something I happened to notice after I took a picture of my computer monitor.

LGTM.

…apping

(was missed from previous commits)

Previously was hardcoded to 5m.

alex-luxonis · 2020-05-01T15:14:43Z

I missed overlaying the BB on left/right streams. It's fixed now.
For the failure in this case python3 test.py -s metaout left -bb, looks like there is a high chance of not-working when metaout is enabled without previewout, but sometimes it works. The issue is also on master, need to investigate it.

Luxonis-Brandon · 2020-05-01T15:34:11Z

Thanks. LGTM @alex-luxonis and since this isn't a regression but something already on Master I think we're clear to merge.

Thanks again!

…apping

alex-luxonis · 2020-05-02T01:47:46Z

Added an option to 'squish' the RGB 1920x1080 into the NN resolution set, by using different horizontal and vertical scaling factors, and not cropping:
-ff, --full-fov-nn Full RGB FOV for NN, not keeping the aspect ratio
With the -bb option, added drawing the inner bounding box where the average depth is calculated over, as configured by padding_factor. Helps to check why X,Y,Z values may seem off, if the depth quality is bad over that area:

Luxonis-Brandon · 2020-05-02T01:59:54Z

Sweet thanks!

Enhance roi depth mapping

alex-luxonis added 2 commits April 28, 2020 16:27

Draw NN bounding box on left/right/depth* streams, enabled with -bb

e2382dc

Update depthai-api. Rebuild binaries, todo RPi lib

445bfe8

alex-luxonis requested review from Luxonis-Brandon, Luxonis-Brian and SzabolcsGergely April 28, 2020 13:47

Rebuild RPi lib

6d7e48c

SzabolcsGergely approved these changes Apr 29, 2020

View reviewed changes

Luxonis-Brian approved these changes Apr 29, 2020

View reviewed changes

alex-luxonis added 3 commits May 1, 2020 16:33

Merge remote-tracking branch 'origin/master' into enhance_roi_depth_m…

52439d1

…apping

Draw BB on left/right streams

0442d3f

(was missed from previous commits)

Configurable depth limit for filtering during X,Y,Z calc.

02b1e99

Previously was hardcoded to 5m.

alex-luxonis commented May 1, 2020

View reviewed changes

Comment thread depthai.py

Luxonis-Brandon approved these changes May 1, 2020

View reviewed changes

alex-luxonis added 4 commits May 1, 2020 20:04

Merge remote-tracking branch 'origin/master' into enhance_roi_depth_m…

036109d

…apping

Draw the inner box taken for average depth based on 'padding_factor'

9d844c2

Add option to use full RGB FOV for NN

ed69d2c

Rebuild binaries

b8e5c76

Luxonis-Brandon mentioned this pull request May 4, 2020

Strange artifacts showing on right side of preview image #72

Closed

Update depthai-api after merge, rebuild FW after code cleanup

ef9c097

alex-luxonis merged commit ca53622 into master May 4, 2020

alex-luxonis deleted the enhance_roi_depth_mapping branch May 4, 2020 22:03

SzabolcsGergely pushed a commit that referenced this pull request Oct 30, 2020

Merge pull request #85 from luxonis/enhance_roi_depth_mapping

c3559d6

Enhance roi depth mapping

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Enhance roi depth mapping - #85

Enhance roi depth mapping#85
alex-luxonis merged 11 commits into
masterfrom
enhance_roi_depth_mapping

alex-luxonis commented Apr 28, 2020

Uh oh!

SzabolcsGergely left a comment

Uh oh!

Luxonis-Brandon commented Apr 29, 2020

Uh oh!

Luxonis-Brandon commented Apr 29, 2020

Uh oh!

Luxonis-Brandon commented Apr 29, 2020

Uh oh!

Luxonis-Brian left a comment

Uh oh!

Uh oh!

alex-luxonis commented May 1, 2020

Uh oh!

Luxonis-Brandon commented May 1, 2020

Uh oh!

alex-luxonis commented May 2, 2020

Uh oh!

Luxonis-Brandon commented May 2, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

alex-luxonis commented Apr 28, 2020

Uh oh!

SzabolcsGergely left a comment

Choose a reason for hiding this comment

Uh oh!

Luxonis-Brandon commented Apr 29, 2020

Uh oh!

Luxonis-Brandon commented Apr 29, 2020

Uh oh!

Luxonis-Brandon commented Apr 29, 2020

Uh oh!

Luxonis-Brian left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alex-luxonis commented May 1, 2020

Uh oh!

Luxonis-Brandon commented May 1, 2020

Uh oh!

alex-luxonis commented May 2, 2020

Uh oh!

Luxonis-Brandon commented May 2, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants