Out network support

[Erol444]

This returns dai.NNData objects:

from depthai_sdk import OakCamera

with OakCamera() as oak:
    color = oak.create_camera('color')
    # List of models that are supported out-of-the-box by the SDK:
    # https://docs.luxonis.com/projects/sdk/en/latest/features/ai_models/#sdk-supported-models
    nn = oak.create_nn('yolov7tiny_coco_640x352', color)
    oak.visualize([nn.out.main], fps=True)
    def cb(packet):
        print(packet.nn_data)
    oak.callback(nn.out.nn_data, callback=cb)
    oak.start(blocking=True)

[moratom]

LGTM, thanks @Erol444!

@PetrNovota do you mind testing this and checking if it will work for you? (This is not yet merged in rvc3_support/develop branches, so you can test on any RVC2 device).

If all looks good, I'll merge it into rvc3_develop.

[PetrNovota]

@moratom thank you for adding this feature so fast. I tested you example and:

1. oak.sync([nn.out.nn_data, nn.out.main], callback=cb]) does not work for me i get:

File "/Users/petrnovota/programming_projects/depthai_testing/merged_model_support/depthai_sdk/classes/output_config.py", line 179, in setup
    if self.visualizer:
AttributeError: 'SyncConfig' object has no attribute 'visualizer'

2. Using callback works for me, at least in a sense that i can see both processed and raw model outputs, i did not validate them

[Erol444]

@PetrNovota fixed, this now works:

from depthai_sdk import OakCamera
from typing import Dict

with OakCamera() as oak:
    color = oak.create_camera('color')
    # List of models that are supported out-of-the-box by the SDK:
    # https://docs.luxonis.com/projects/sdk/en/latest/features/ai_models/#sdk-supported-models
    nn = oak.create_nn('yolov7tiny_coco_640x352', color)
    oak.visualize([nn.out.main], fps=True)
    def cb(data: Dict):
        for name, packet in data.items():
            print(f'{name}: {packet}')
    oak.sync([nn.out.nn_data, nn.out.main], callback=cb)
    oak.start(blocking=True)

[Erol444]

@daniilpastukhov - I'm not exactly sure, but why do we have visualizers inside the oak.sync()?

[Erol444]

@PetrNovota does everything work as expected? Can we merge the PR?

[daniilpastukhov]

LGTM

[PetrNovota]

@Erol444 which depthai version should I used with this new sdk? Currently, I am using
depthai==2.19.1.0.dev0+f184dc652e2d6a059abb32ea8495a3baf1a62f4d And I get a bunch of errors

dai.ColorCameraProperties.SensorResolution.THE_1440X1080: (1440, 1080),
AttributeError: type object 'depthai.SensorResolution' has no attribute 'THE_1440X1080'. Did you mean: 'THE_240X180'?

and when I delete the line then I get another error

File "/Users/petrnovota/programming_projects/testing_ground/merged_model_test/depthai_sdk/components/stereo_component.py", line 278, in StereoComponent
   def set_colormap(self, colormap: dai.Colormap):
AttributeError: module 'depthai' has no attribute 'Colormap'

[daniilpastukhov]

@PetrNovota I think the minimum version is 2.21 (or equivalent to it)

[PetrNovota]

@daniilpastukhov oh I get it i cannot test it on rvc3. my bad

[PetrNovota]

Setup:

    def cb(msgs):
        something...
    with OakCamera() as oak:
        rgb = oak.create_camera(source="color")
        nn = oak.create_nn('yolov7tiny_coco_640x352', rgb)
        oak.visualize([nn.out.main], fps=True)

        oak.sync([nn.out.nn_data, nn.out.main, rgb.out.main], callback=cb)
        oak.start(blocking=True)

Packets have attribute msg instead of imageFrame this could be cause by an error that i get when I use the depthai-sdk without any changes:

  File "/Users/petrnovota/programming_projects/depthai_testing/merged_model_support/depthai_sdk/classes/output_config.py", line 5, in <module>
    from attr import has
ModuleNotFoundError: No module named 'attr'

When I delete the import, I can run the code but it looks like the Packets ImageFrame attribute has wrong name

There seems to be an performance issue. If I run the above example without the visualizer and without the nn_data output, then it runns at 20FPS. When I add the nn_data to the callback, the FPS drops to 5FPS

[PetrNovota]

cc: @Erol444 and @daniilpastukhov ^^^^

[Erol444]

@PetrNovota My bad, I accidentally pushed this import.

Packets have attribute msg instead of imageFrame

Yes, this PR changes so the "main" dai message is saved as msg.

[PetrNovota]

@Erol444 and what about the performance drop. Why is my FPS capped at 5FPS when its working at 20FPS without the NNData output?

[Erol444]

@PetrNovota depending on the model, NNData can be huge. Perhaps check its size:)

[PetrNovota]

@Erol444 I am using the model from your example: yolov7tiny_coco_640x352 and indeed, the output lists are large. 1 list with 800k 1 list with 250k and 1 with 50k FP16 values.

1. Why is the output in a form of a python list? seems inefficient. I am already using a model where I decode its outputs and I get them by calling output.getTensor output being dai.NNData
2. Why are the outputs so large? I dont think any nn model has such a large output. I dont believe that the yolo model has indeed 1000k outputs. Why am I getting so many values?
3. I want to use it with a merged model, it has 6 outputs. 3 of them are yolo outputs and can be processed on camera. 3 of them need to be processed manually. Will this work? And Will the NNData only contain the 3 outputs that cannot be processed on camera?

[PetrNovota]

@Erol444 In any case. Having the NNData store so many values and causing it such a large performance drop is not useful. What can we do about it?

[Erol444]

@PetrNovota please check with the ML team - I know it's possible to prune the output size/length, so eg. instead of getting 10000 detections, only get top 100.

[PetrNovota]

Changes tested and it looks good