Run NN on left/right mono cameras #94

alex-luxonis · 2020-05-06T00:46:22Z

Add a new option to depthai.py:

  -cam {rgb,left,right}, --cnn-camera {rgb,left,right}
                        Choose camera input for CNN (default: rgb)

Fix NN bounding-box lag with depth enabled, for fast NN models like face-detection-retail-0004. TODO: for slower networks, this lowers the FPS of left/right/depth streams, will fix it.
Apply default board-config values for older EEPROM versions (fix depth calc with V1, where RGB FOV was missing).

Usage example (custom model, run NN on Right, show depth map, overlay the bounding box on depth):

python3 test.py -cnn face-detection-retail-0004 -cam right -s previewout metaout depth_sipp -bb

Rebuild libs, TODO RPi

Luxonis-Brandon · 2020-05-06T23:35:57Z

I did notice a couple resets upon booting this... the watchdog properly catches them and tries again, usually 1 or 2 times, until it eventually runs. When it runs it does seem to work well:

alex-luxonis · 2020-05-15T16:44:29Z

Changes in the above FW update:

The couple resets upon booting were caused by a too short watchdog timeout on device. RGB is initialized and started first, followed by the Mono pair. As the frames from RGB were not streamed on XLink (so not feeding the watchdog), the wait for Mono to get ready sometimes caused the watchdog to expire and do reset. I implemented a timer to periodically clear the watchdog during init (for a total of max 9s). The timer is canceled after all sensors are initialized, to recover quickly in case of a subsequent post-init failure. This should also fix some previous known issues:
- Running RGB w/o previewout: python3 depthai.py -s metaout depth_sipp -bb
- Failure to init on some macOS machines (delay until USB comm established).
The system can work now without an RGB camera mounted, so just with the Mono stereo pair. Should fix issue Hang/Stall if No Color Camera Present #44. If RGB is missing, the AI input will be auto-switched to the Right camera. (TODO: should add a detected camera status / warning for this in the host app.)

Luxonis-Brandon · 2020-05-19T02:50:18Z

Just tested this on macOS. I do still see some failure to init -sometimes- (maybe 1 in 5?) when running python3 depthai.py -s metaout depth_sipp -bb. Here is a video showing it (towards the end). Usually it happens a bit more often than that, but it's pretty random.

Below is an example of the output when this occurs:

DESKTOP-OGSI4CI:depthai leeroy$ python3 depthai.py -s metaout depth_sipp -bb
Using Custom Calibration File: depthai.calib
Using Arguments= {'config_overwrite': None, 'board': None, 'field_of_view': 71.86, 'rgb_field_of_view': 68.7938, 'baseline': 9.0, 'rgb_baseline': 2.0, 'swap_lr': True, 'store_eeprom': False, 'clear_eeprom': False, 'override_eeprom': False, 'device_id': '', 'dev_debug': None, 'force_usb2': None, 'cnn_model': 'mobilenet-ssd', 'cnn_camera': 'rgb', 'disable_depth': False, 'draw_bb_depth': True, 'full_fov_nn': False, 'streams': [{'name': 'metaout'}, {'name': 'depth_sipp'}], 'video': None}
depthai.__version__ == 0.0.10a
depthai.__dev_version__ == 30295b558351cd030408e12a220cdd55b5fb450e
XLink initialized.
Sending device firmware "cmd_file": /Users/leeroy/depthai/depthai.cmd
Successfully connected to device.
Loading config file
Attempting to open stream config_d2h
watchdog started 3000
Successfully opened stream config_d2h with ID #0!
Closing stream config_d2h: ...
Closing stream config_d2h: DONE.
EEPROM data: valid (v2)
  Board name     : BW1098OBC
  Board rev      : R0M0E0
  HFOV L/R       : 71.86 deg
  HFOV RGB       : 68.7938 deg
  L-R   distance : 7.5 cm
  L-RGB distance : 3.75 cm
  L/R swapped    : yes
  L/R crop region: top
  Calibration homography:
    1.002324,   -0.004016,   -0.552212,
    0.001249,    0.993829,   -1.710247,
    0.000008,   -0.000010,    1.000000,
Available streams: ['meta_d2h', 'left', 'right', 'disparity', 'depth_sipp', 'metaout', 'previewout', 'jpegout', 'video']
CNN configurations read: /Users/leeroy/depthai/resources/nn/mobilenet-ssd/mobilenet-ssd_depth.json
config_h2d json:
{"_board":{"_homography_right_to_left":[0.9997568726539612,-0.002453562105074525,0.18778356909751892,-0.00013814299018122256,0.9944143295288086,-2.22407865524292,5.636975402012467e-06,-7.338615432672668e-06,1.0]},"_load_inBlob":true,"_pipeline":{"_streams":[{"name":"metaout"},{"data_type":"uint16","name":"depth_sipp"}]},"ai":{"calc_dist_to_bb":true,"camera_input":"rgb","keep_aspect_ratio":true},"board":{"clear-eeprom":false,"left_fov_deg":71.86000061035156,"left_to_rgb_distance_m":0.019999999552965164,"left_to_right_distance_m":0.09000000357627869,"name":"","override-eeprom":false,"revision":"","rgb_fov_deg":68.7938003540039,"stereo_center_crop":true,"store-to-eeprom":false,"swap-left-and-right-cameras":true},"depth":{"depth_limit_mm":10000,"padding_factor":0.30000001192092896}}
Attempting to open stream config_h2d
Successfully opened stream config_h2d with ID #1!
Writing 1000 bytes to config_h2d
!!! XLink write successful: config_h2d (1000)
Closing stream config_h2d: ...
Closing stream config_h2d: DONE.
Creating observer stream host_capture: ...
Attempting to open stream host_capture
Successfully opened stream host_capture with ID #0!
Creating observer stream host_capture: DONE.
Read: 14514560
Attempting to open stream inBlob
Successfully opened stream inBlob with ID #1!
Writing 14514560 bytes to inBlob
!!! XLink write successful: inBlob (14514560)
Closing stream inBlob: ...
Closing stream inBlob: DONE.
depthai: done sending Blob file /Users/leeroy/depthai/resources/nn/mobilenet-ssd/mobilenet-ssd.blob
Attempting to open stream outBlob
Successfully opened stream outBlob with ID #2!
Closing stream outBlob: ...
Closing stream outBlob: DONE.
CNN input width: 300
CNN input height: 300
CNN input num channels: 3
CNN to depth bounding-box mapping: start(318, 78), max_size(643, 643)
Host stream start:depth_sipp
Opening stream for read: depth_sipp
Attempting to open stream depth_sipp
Successfully opened stream depth_sipp with ID #1!
Starting thread for stream: depth_sipp
Started thread for stream: depth_sipp
Host stream start:metaout
Opening stream for read: metaout
Attempting to open stream metaout
Successfully opened stream metaout with ID #2!
Starting thread for stream: metaout
Started thread for stream: metaout
depthai: INIT OK!
E: [global] [    399375] [] XLinkReadData:165	addEventWithPerf(&event, &opTime) failed with error: 3
Device get data failed: 3
Closing stream depth_sipp: ...
E: [global] [    399375] [] XLinkReadData:165	addEventWithPerf(&event, &opTime) failed with error: 3
Device get data failed: 3
Closing stream metaout: ...
E: [global] [    399376] [] dispatcherEventSend:53	Write failed (header) (err -4) | event XLINK_CLOSE_STREAM_REQ

E: [xLink] [    399376] [] sendEvents:1036	Event sending failed
E: [global] [    399376] [] dispatcherEventSend:53	Write failed (header) (err -4) | event XLINK_CLOSE_STREAM_REQ

E: [xLink] [    399376] [] sendEvents:1036	Event sending failed
Closing stream depth_sipp: DONE.
Thread for depth_sipp finished.
Closing stream metaout: DONE.
Thread for metaout finished.
watchdog triggered 
Stopping threads: ...
Stopping threads: DONE 0.000s.
Closing all observer streams: ...
Closing stream host_capture: ...
E: [global] [    401894] [] dispatcherEventSend:53	Write failed (header) (err -4) | event XLINK_CLOSE_STREAM_REQ

E: [xLink] [    401894] [] sendEvents:1036	Event sending failed
Closing stream host_capture: DONE.
Closing all observer streams: DONE.
Reseting device: 0.
E: [global] [    401894] [] dispatcherEventSend:53	Write failed (header) (err -4) | event XLINK_RESET_REQ

E: [xLink] [    401894] [] sendEvents:1036	Event sending failed
Reseting: DONE.
XLink initialized.
Sending device firmware "cmd_file": /Users/leeroy/depthai/depthai.cmd
Successfully connected to device.
Loading config file
Attempting to open stream config_d2h
Successfully opened stream config_d2h with ID #0!
Closing stream config_d2h: ...
Closing stream config_d2h: DONE.
EEPROM data: valid (v2)
  Board name     : BW1098OBC
  Board rev      : R0M0E0
  HFOV L/R       : 71.86 deg
  HFOV RGB       : 68.7938 deg
  L-R   distance : 7.5 cm
  L-RGB distance : 3.75 cm
  L/R swapped    : yes
  L/R crop region: top
  Calibration homography:
    1.002324,   -0.004016,   -0.552212,
    0.001249,    0.993829,   -1.710247,
    0.000008,   -0.000010,    1.000000,
CNN configurations read: /Users/leeroy/depthai/resources/nn/mobilenet-ssd/mobilenet-ssd_depth.json
config_h2d json:
{"_board":{"_homography_right_to_left":[0.9997568726539612,-0.002453562105074525,0.18778356909751892,-0.00013814299018122256,0.9944143295288086,-2.22407865524292,5.636975402012467e-06,-7.338615432672668e-06,1.0]},"_load_inBlob":true,"_pipeline":{"_streams":[{"name":"metaout"},{"data_type":"uint16","name":"depth_sipp"}]},"ai":{"calc_dist_to_bb":true,"camera_input":"rgb","keep_aspect_ratio":true},"board":{"clear-eeprom":false,"left_fov_deg":71.86000061035156,"left_to_rgb_distance_m":0.019999999552965164,"left_to_right_distance_m":0.09000000357627869,"name":"","override-eeprom":false,"revision":"","rgb_fov_deg":68.7938003540039,"stereo_center_crop":true,"store-to-eeprom":false,"swap-left-and-right-cameras":true},"depth":{"depth_limit_mm":10000,"padding_factor":0.30000001192092896}}
Attempting to open stream config_h2d
Successfully opened stream config_h2d with ID #1!
Writing 1000 bytes to config_h2d
!!! XLink write successful: config_h2d (1000)
Closing stream config_h2d: ...
Closing stream config_h2d: DONE.
Creating observer stream host_capture: ...
Attempting to open stream host_capture
Successfully opened stream host_capture with ID #0!
Creating observer stream host_capture: DONE.
Read: 14514560
Attempting to open stream inBlob
Successfully opened stream inBlob with ID #1!
Writing 14514560 bytes to inBlob
!!! XLink write successful: inBlob (14514560)
Closing stream inBlob: ...
Closing stream inBlob: DONE.
depthai: done sending Blob file /Users/leeroy/depthai/resources/nn/mobilenet-ssd/mobilenet-ssd.blob
Attempting to open stream outBlob
Successfully opened stream outBlob with ID #2!
Closing stream outBlob: ...
Closing stream outBlob: DONE.
CNN input width: 300
CNN input height: 300
CNN input num channels: 3
CNN to depth bounding-box mapping: start(318, 78), max_size(643, 643)
Host stream start:depth_sipp
Opening stream for read: depth_sipp
Attempting to open stream depth_sipp
Successfully opened stream depth_sipp with ID #1!
Starting thread for stream: depth_sipp
Host stream start:metaout
Opening stream for read: metaout
Attempting to open stream metaout
Started thread for stream: depth_sipp
Successfully opened stream metaout with ID #2!
Starting thread for stream: metaout
Started thread for stream: metaout
depthai: INIT OK!
Stopping threads: ...
Closing stream depth_sipp: ...
Closing stream depth_sipp: DONE.
Thread for depth_sipp finished.
Closing stream metaout: ...
Closing stream metaout: DONE.
Thread for metaout finished.
Stopping threads: DONE 0.023s.
Closing all observer streams: ...
Closing stream host_capture: ...
Closing stream host_capture: DONE.
Closing all observer streams: DONE.
Reseting device: 0.
E: [xLink] [    407155] [] DispatcherWaitEventComplete:380	waiting is timeout, sending reset remote event
E: [xLink] [    407155] [] DispatcherAddEvent:323	curr == NULL expression failed
Segmentation fault: 11

Note that the watchdog does recover and it does run eventually. It's worth noting that the probability of this occuring seems to go up when the Myriad X is cold booted (i.e. unplugging only 5V power, plugging it back in, and then running python3 depthai.py -s metaout depth_sipp -bb.

(Ignore the macOS segfault at the end, this is a known issue for which Szabi has a workaround.)

On to testing w/out RGB camera.

Luxonis-Brandon · 2020-05-19T04:11:22Z

Just tested w/out the color camera and it works great:

So I'd say this functionality is ready to go.

I'm guessing the Mac initialization is still a bit too slow for the watchdog... not sure if there's some setting in the code or otherwise that could speed up the macOS initialization.

alex-luxonis · 2020-05-19T13:33:19Z

Just tested this on macOS. I do still see some failure to init -sometimes- (maybe 1 in 5?) when running python3 depthai.py -s metaout depth_sipp -bb
...
Note that the watchdog does recover and it does run eventually. It's worth noting that the probability of this occuring seems to go up when the Myriad X is cold booted (i.e. unplugging only 5V power, plugging it back in, and then running python3 depthai.py -s metaout depth_sipp -bb.

I can reproduce this issue on master. Looks like one of the Mono camera doesn't work. The NN metadata + xyz will be missing too (I think this issue was encountered before).
By running python3 depthai.py -s previewout metaout left right (on master or here on this PR), the issue with one of the Mono cameras can be observed more easily.

Luxonis-Brandon · 2020-05-19T14:00:26Z

Ah, very interesting, thanks. So to make sure I'm understanding properly, sometimes on boot up one of the mono cameras just fails to initialize. Is that right?

alex-luxonis · 2020-05-19T14:30:52Z

Yes, one of the mono cameras will fail to initialize. I tested by keeping the python script opened, and resetting or power-cycling the board. Sometimes one of the mono cam windows got frozen, and recovered after one more restart.

Luxonis-Brandon · 2020-05-19T14:33:49Z

Ah, got it. Thanks.

…oding, not when displaying (for a slight improvement in performance)

…on bindings, previously there were crashes and wrong BB association to previewout frames, with the option `-cam left_right`

Luxonis-Brandon · 2020-05-29T22:18:44Z

This seems to be working well for me now:

Tested with python3 test.py -cnn face-detection-retail-0004 -cam left_right -s previewout metaout -bb -dd.

Haven't seen a single issue yet. The only thing to note is that since we haven't optimized to use more resources yet (Szabi is working on that) you can definitely see it's slower.

face-detection-retail-0004.json

…ion-retail-0009

(even if unused for now, used only with decode_mobilenet_ssd)

up to 60fps for H:720 or H:800, up to 120fps for H:400.

max 42 fps for 4K, max 118 fps 1080p. Higher is possible by tweaking the configs.

alex-luxonis · 2020-07-07T00:01:00Z

Known issues for now:

if NN is too slow and can't keep up with camera FPS (note NN FPS is shown in previewout-... windows), due to synchronization in use and frames getting queued, this could cause lag and lower the FPS of left/right/depth. Possible workarounds for now:
- make NN faster, running with 2 NCEs and 10 or 12 shaves: ./depthai.py -s previewout metaout right depth_sipp -bb -sh 10 -cmx 10 -nce 2
- lower the camera FPS using the -monof and/or -rgbf
depth_sipp stream doesn't work with Mono resolution 1280x800, looks like an issue with XLink / the host doesn't read the frames. XYZ or disparity/depth_color_h are fine though.

Luxonis-Brandon · 2020-07-07T17:43:31Z

Just tried building this on Mac to test it and I'm running into the following:

[ 27%] Building CXX object CMakeFiles/depthai.dir/Users/leeroy/depthai/depthai-api/host/core/pipeline/host_pipeline_config.cpp.o
/Users/leeroy/depthai/depthai-api/host/py_module/py_bindings.cpp:489:23: error: variable-sized object may not be initialized
        int size_blob[num_stages] = {0};
                      ^~~~~~~~~~
[ 30%] Building CXX object CMakeFiles/depthai.dir/Users/leeroy/depthai/depthai-api/host/core/pipeline/cnn_host_pipeline.cpp.o
[ 33%] Building CXX object CMakeFiles/depthai.dir/Users/leeroy/depthai/depthai-api/host/core/types.cpp.o
[ 36%] Building CXX object CMakeFiles/depthai.dir/Users/leeroy/depthai/depthai-api/host/core/host_json_helper.cpp.o
[ 38%] Building CXX object CMakeFiles/depthai.dir/Users/leeroy/depthai/depthai-api/host/core/host_data_reader.cpp.o
[ 41%] Building CXX object CMakeFiles/depthai.dir/Users/leeroy/depthai/depthai-api/host/core/host_capture_command.cpp.o
[ 44%] Building CXX object CMakeFiles/depthai.dir/Users/leeroy/depthai/depthai-api/host/core/disparity_stream_post_processor.cpp.o
1 error generated.
make[2]: *** [CMakeFiles/depthai.dir/py_bindings.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/depthai.dir/all] Error 2
make: *** [all] Error 2
[BUILD depthai-api/host/py_module/build] cp *.so ../../../../
cp: *.so: No such file or directory

Commands prior to this:

git reset --hard origin/master   
git pull 
git checkout -f nn_on_left_right --recurse-submodules=yes -f    
git pull     
git submodule update --init --recursive && ./depthai-api/install_dependencies.sh && ./depthai-api/build_py_module.sh --clean

Luxonis-Brandon

This looks great to me! ;-)

https://youtu.be/uAfGulcDWSk

Luxonis-Steven

Tested the face detection with emotion and face detection with landmarks. Both looks great.

Free up another shave for NN; max shaves : 14/1080p, 11/4k

Run NN on left/right mono cameras; add support for second stage NN

Add option to select the camera {rgb,left,right} used for NN

8b2674f

Rebuild libs, TODO RPi

alex-luxonis requested review from iamsg95 and Luxonis-Brandon May 6, 2020 00:46

alex-luxonis marked this pull request as draft May 6, 2020 00:47

alex-luxonis changed the title ~~Run NN on left/right mono cameras. WIP~~ Run NN on left/right mono cameras May 6, 2020

Update device FW

c779688

alex-luxonis added 7 commits May 26, 2020 16:22

Add support for NN on both Left+Right

0abf7b4

Update firmware(USB3) and x86_64 lib

a0ce026

Merge remote-tracking branch 'origin/master' into nn_on_left_right

588815e

Rebuild FW(USB3) and x86_64 lib after merge w/ master

1a90a15

mobilenet_ssd: check confidence_threshold and append to list when dec…

3014ada

…oding, not when displaying (for a slight improvement in performance)

TMP workaround: create a copy of NN data, due to issues with C++/pyth…

c0113f0

…on bindings, previously there were crashes and wrong BB association to previewout frames, with the option `-cam left_right`

Update FW, fix crash with -dd/--disable_depth

9fc89a9

Luxonis-Brandon mentioned this pull request Jun 1, 2020

Video Encoding of left and right in parallel #122

Closed

themarpe force-pushed the master branch from bcc9f0b to 80b779a Compare June 6, 2020 22:05

alex-luxonis added 6 commits June 11, 2020 17:33

Two-stage inference: pass a second NN model with -cnn2

5801d3c

TMP merge landmarks-regression-retail-0009.json into

820b368

face-detection-retail-0004.json

Rebuild USB3 FW and Linux x86_64 lib

3f22f02

Draw the face landmarks, works with the patched JSON

16289cb

Compiled for 10 Shaves: face-detection-retail-0004, landmarks-regress…

a119259

…ion-retail-0009

TMP update FW: run on 10 shaves, disable RGB and Depth

3544784

alex-luxonis added 8 commits July 6, 2020 16:42

For two-stage NN, recompile second-model too (shv,cmx,nce)

3543a44

Add **kwargs to all other decode_NN functions

d448f5a

(even if unused for now, used only with decode_mobilenet_ssd)

Rebuild FW and Linux x86_64 lib, update depthai-api

e50c12e

Update mono cam resolution and FPS support:

c508862

up to 60fps for H:720 or H:800, up to 120fps for H:400.

Count NN FPS per camera source, display in previewout bottom-left

7606e42

Display BBox over left/right when cnn_camera == 'rgb'

f0b6ab3

Update FW: fix for non-mobilenetssd single-stage

0d1a5f0

Implement configurable FPS for RGB:

f961183

max 42 fps for 4K, max 118 fps 1080p. Higher is possible by tweaking the configs.

Update depthai-api: fix a clang compilation error on macOS

f6f0008

Luxonis-Brandon approved these changes Jul 7, 2020

View reviewed changes

Luxonis-Brandon requested a review from Luxonis-Steven July 7, 2020 18:43

This was referenced Jul 7, 2020

Example scripts have blurry video output #31

Closed

Hang/Stall if No Color Camera Present #44

Closed

SzabolcsGergely added 4 commits July 9, 2020 15:05

merge master into nn_on_left_right

db0d900

Merge remote-tracking branch 'origin/master' into nn_on_left_right_merge

38dfce2

synchronize NN packets with latest depth packets

92f6064

fix 12mp full resolution

7f9a1b9

Luxonis-Steven approved these changes Jul 9, 2020

View reviewed changes

SzabolcsGergely and others added 3 commits July 9, 2020 23:56

free up another shave for NN; max shaves : 14/1080p, 11/4k

235802d

Merge pull request #152 from luxonis/14shave_nn

77928f3

Free up another shave for NN; max shaves : 14/1080p, 11/4k

recompile lib for x86_64-linux

d53f034

iamsg95 force-pushed the nn_on_left_right branch from ca19819 to d53f034 Compare July 20, 2020 10:40

SzabolcsGergely added 3 commits July 20, 2020 04:52

rebuild RPI lib

07e74c7

Merge remote-tracking branch 'origin/master' into HEAD

28c8a6b

Force default NN engines to 2 in case inference is done on 2 cameras

84aa59c

SzabolcsGergely merged commit 99a60b3 into master Jul 20, 2020

SzabolcsGergely deleted the nn_on_left_right branch October 7, 2020 11:06

SzabolcsGergely added a commit that referenced this pull request Oct 30, 2020

Merge pull request #94 from luxonis/nn_on_left_right

31e78bc

Run NN on left/right mono cameras; add support for second stage NN

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run NN on left/right mono cameras #94

Run NN on left/right mono cameras #94

alex-luxonis commented May 6, 2020

Luxonis-Brandon commented May 6, 2020

alex-luxonis commented May 15, 2020

Luxonis-Brandon commented May 19, 2020 •

edited

Luxonis-Brandon commented May 19, 2020

alex-luxonis commented May 19, 2020

Luxonis-Brandon commented May 19, 2020

alex-luxonis commented May 19, 2020

Luxonis-Brandon commented May 19, 2020

Luxonis-Brandon commented May 29, 2020

alex-luxonis commented Jul 7, 2020

Luxonis-Brandon commented Jul 7, 2020

Luxonis-Brandon left a comment

Luxonis-Steven left a comment

Run NN on left/right mono cameras #94

Run NN on left/right mono cameras #94

Conversation

alex-luxonis commented May 6, 2020

Luxonis-Brandon commented May 6, 2020

alex-luxonis commented May 15, 2020

Luxonis-Brandon commented May 19, 2020 • edited

Luxonis-Brandon commented May 19, 2020

alex-luxonis commented May 19, 2020

Luxonis-Brandon commented May 19, 2020

alex-luxonis commented May 19, 2020

Luxonis-Brandon commented May 19, 2020

Luxonis-Brandon commented May 29, 2020

alex-luxonis commented Jul 7, 2020

Luxonis-Brandon commented Jul 7, 2020

Luxonis-Brandon left a comment

Choose a reason for hiding this comment

Luxonis-Steven left a comment

Choose a reason for hiding this comment

Luxonis-Brandon commented May 19, 2020 •

edited