Colorized depth map + fps limit and multi tensor output #38

iamsg95 · 2020-02-28T20:37:25Z

This pr contains:
-multi tensor output support
-fix for occasional crashes

…colorized depth

…depthai-python-extras into multiple_output_tensors

itsderek23 · 2020-02-28T22:08:43Z

Thanks @GergelySzabolcs!

-fix for occasional crashes

Clarifying: does this addresses crashes you've seen on our current master branch, or crashes introduced in development of this branch?

Luxonis-Brandon · 2020-02-28T22:43:34Z

Yes, thanks @GergelySzabolcs !

rebuild libs; limit previewout, depth sipp to 10 fps

So for our previewout-only example script (the default test.py), will this now be limited to 10FPS by default as well?

I ask because it's compelling initial demo for that to show the full 30FPS. Although 10FPS isn't bad, so that might be fine as well.

Either way, just wanting to make sure I understand.

Thanks again!

iamsg95 · 2020-02-29T09:00:40Z

Thanks @GergelySzabolcs!

-fix for occasional crashes

Clarifying: does this addresses crashes you've seen on our current master branch, or crashes introduced in development of this branch?

Crashes seen on master

iamsg95 · 2020-02-29T09:02:42Z

Yes, thanks @GergelySzabolcs !

rebuild libs; limit previewout, depth sipp to 10 fps

So for our previewout-only example script (the default test.py), will this now be limited to 10FPS by default as well?

I ask because it's compelling initial demo for that to show the full 30FPS. Although 10FPS isn't bad, so that might be fine as well.

Either way, just wanting to make sure I understand.

Thanks again!

Limiting the fps it is supposedly just an example of usage.
We should decide which streams we enable by default.
In test.py depth aipp, previewout and metaout are enabled, that's why the limiting to 10 fps, so we don't saturate usb2 bandwidth. If depth sipp is not enabled we can do 30 fps.

Luxonis-Brandon · 2020-02-29T20:45:56Z

Fantastic. Thanks.

This is a great feature to have - the capability to limit the framerate so as to not saturate USB2 bandwith.

I'm going to move this to complete on the roadmap. Thanks for knocking out this feature in the process of solving the calibration/colorized-depth display problem.

For the default display:
I'm thinking we stick w/ previewout w/ the metadata visualized on it. As it's nice and 30FPS even on a Pi w/ USB2. CC: @itsderek23 for any thoughts on this.

Thanks again!

itsderek23

I've added comments. Biggest issues:

test.py doesn't run for me

On the 1097, I'll see an initial frame for some of the streams. However, almost immediately the log is full of "Data queue is full previewout/metaout/depth_sipp/left/right".

test.py is outdated

This is using an outdated version of test.py, so a number of important things were dropped (argument parsing, 1097-based defaults, etc.

test.py

itsderek23 · 2020-03-03T13:41:12Z

example_run_age_gender_recognition_retail.py

@@ -0,0 +1,115 @@
+import sys


This example runs for me. 👍

itsderek23 · 2020-03-03T13:42:02Z

example_run_landmarks_recognition.py

@@ -29,7 +29,8 @@

        # 35 landmarks
        'blob_file_config': consts.resource_paths.prefix + 'nn/object_recognition_4shave/landmarks/landmarks-config-35.json',
-        'blob_file': consts.resource_paths.prefix + 'nn/object_recognition_4shave/landmarks/facial-landmarks-35-adas-0002.blob'
+        'blob_file': consts.resource_paths.prefix + 'nn/object_recognition_4shave/landmarks/facial-landmarks-35-adas-0002.blob',


This works for me. 👍

itsderek23 · 2020-03-03T13:47:17Z

example_run_vehicle_attributes_recognition_barrier.py

@@ -0,0 +1,142 @@
+import sys


This runs for a bit, but classifies me as a red car when I'm wearing a blue sweatshirt.

After a minute or so, it crashes with:

ValueError: array is too big; `arr.size * arr.dtype.itemsize` is larger than the maximum possible size. Traceback (most recent call last): File "example_run_vehicle_attributes_recognition_barrier.py", line 121, in <module> data0 = data[0,:,:] IndexError: too many indices for array Stopping threads: ...

This one was done by yurii.
I just merged his latest work with multiple_outpout_tensors branch.

Inference still seems incorrect:

...I am pale/white for sure, but not a car :).

Good news is it hasn't crashed for me over a period of a couple of minutes ... going to let it run longer.

I’d be curious if we fed that same image into the same network running on a host computer for inference it it would have the same output.

itsderek23 · 2020-03-03T13:47:52Z

...ect_recognition_4shave/age_gender_recognition_retail/age-gender-recognition-retail-0013.json

+            "output_properties_type": "f16"
+       }
+    ],
+    "mappings":


Why are these labels included?

itsderek23 · 2020-03-03T13:48:02Z

...have/vehicle_attributes_recognition_barrier/vehicle-attributes-recognition-barrier-0039.json

+    ],
+    "mappings":
+    {
+        "labels":


Why are these labels included?

I don't know. It was done by yurii.

iamsg95 · 2020-03-03T14:21:18Z

@itsderek23 i just updated the branch to match master. Also merged yurii's multiple output tensors branch, which might contain fixes for models that are not working?

itsderek23 · 2020-03-03T20:08:31Z

@GergelySzabolcs - test.py still freezes for me on the 1097 after showing the first frame w/data queue full for metaout/preview. I haven't yet dug through the host-side Python code to see if we've introduced any performance changes there.

itsderek23 · 2020-03-03T20:13:11Z

Yes - I wouldn't be shocked if this was an issue w/the NN itself. This does appear to be stable now though. Thanks!

…

On Tue, Mar 3, 2020 at 1:08 PM Luxonis-Brandon ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In example_run_vehicle_attributes_recognition_barrier.py <#38 (comment)> : > @@ -0,0 +1,142 @@ +import sys I’d be curious if we fed that same image into the same network running on a host computer for inference it it would have the same output. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#38>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAB5SEHB3AAMOUZ7PXQXLDRFVPUNANCNFSM4K54GIZQ> .

Luxonis-Brandon · 2020-03-03T21:19:55Z

Yes. Could be.

I think we completely punt on it (example_run_vehicle_attributes_recognition_barrier.py) for now and remove it from any all current efforts and planned future efforts and/or tests.

And also remove example scripts which run anything other than an object detector from the repository.

With the focus on the following instead:

Object detector models that don't have any additional/special metadata. So that normally just return class and bounding box locations.
Augmentation of these object detector models with disparity depth results - to allow 3D object localization (i.e. the object class, and the XYZ locations of the centroid of the bounding box).
Check of calibration - so the capability to visualize the depth map on a Raspberry Pi 3B+ with JET color map. The check of calibration is necessary for users to enable 2.

So I'm thinking we focus all PRs/efforts going forward to cover those until we get that dialed.

Once we have that dialed, we can move on to support of other network types, multi-output-tensors, etc. But I'm thinking we should really make it a conscious decision, with a check-off list that those are dialed.

Thoughts?

itsderek23 · 2020-03-05T20:21:49Z

Superseded by #41.

ImageManip: rotated/arbitrary cropping, add camera controls

YuriiLaba and others added 12 commits February 25, 2020 18:03

added tests for multiple output tensors

1e14bc1

Update README.md

cd119e6

Update README.md

0d15267

Update example_run_landmarks_recognition.py

f6f2d22

fix colorized depth performance issue

8b0d855

remove import termcolor not present on pi

415bab7

add support for colorized,grayscale,raw depth map; by default enable …

0d75a96

…colorized depth

rebuild libs; limit previewout, depth sipp to 10 fps

1703b69

added new parser

ce0061c

Merge branch 'multiple_output_tensors' of https://github.com/luxonis/…

a56cf55

…depthai-python-extras into multiple_output_tensors

Update README.md

6b6b829

update binary with metad2h hotfix

f3c7782

iamsg95 requested a review from itsderek23 February 28, 2020 20:38

itsderek23 suggested changes Mar 3, 2020

View reviewed changes

gsz95 added 3 commits March 3, 2020 16:07

Merge remote-tracking branch 'origin/master' into HEAD

7e4106b

resolve conflicts after merge

c46b951

Merge remote-tracking branch 'origin/multiple_output_tensors' into HEAD

f471027

Luxonis-Brandon mentioned this pull request Mar 3, 2020

Depth color JET range only shows dark/light blue #39

Closed

iamsg95 closed this Mar 5, 2020

SzabolcsGergely deleted the colorized_depth branch April 9, 2020 21:00

jdavidberger pushed a commit to constructiverealities/depthai that referenced this pull request May 26, 2022

Merge pull request luxonis#38 from luxonis/gen2_imagemanip_rotate

ad670be

ImageManip: rotated/arbitrary cropping, add camera controls

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Colorized depth map + fps limit and multi tensor output #38

Colorized depth map + fps limit and multi tensor output #38

iamsg95 commented Feb 28, 2020

itsderek23 commented Feb 28, 2020

Luxonis-Brandon commented Feb 28, 2020

iamsg95 commented Feb 29, 2020

iamsg95 commented Feb 29, 2020

Luxonis-Brandon commented Feb 29, 2020

itsderek23 left a comment

itsderek23 Mar 3, 2020

itsderek23 Mar 3, 2020

itsderek23 Mar 3, 2020

iamsg95 Mar 3, 2020

itsderek23 Mar 3, 2020

Luxonis-Brandon Mar 3, 2020

itsderek23 Mar 3, 2020

itsderek23 Mar 3, 2020

iamsg95 Mar 3, 2020

iamsg95 commented Mar 3, 2020

itsderek23 commented Mar 3, 2020

itsderek23 commented Mar 3, 2020 via email

Luxonis-Brandon commented Mar 3, 2020 •

edited

itsderek23 commented Mar 5, 2020

Colorized depth map + fps limit and multi tensor output #38

Colorized depth map + fps limit and multi tensor output #38

Conversation

iamsg95 commented Feb 28, 2020

itsderek23 commented Feb 28, 2020

Luxonis-Brandon commented Feb 28, 2020

iamsg95 commented Feb 29, 2020

iamsg95 commented Feb 29, 2020

Luxonis-Brandon commented Feb 29, 2020

itsderek23 left a comment

Choose a reason for hiding this comment

test.py doesn't run for me

test.py is outdated

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iamsg95 commented Mar 3, 2020

itsderek23 commented Mar 3, 2020

itsderek23 commented Mar 3, 2020 via email

Luxonis-Brandon commented Mar 3, 2020 • edited

itsderek23 commented Mar 5, 2020

Luxonis-Brandon commented Mar 3, 2020 •

edited