Device Discovery and Board Configuration

[themarpe]

Why
Currently, DepthAI devices run unified FW with little to no upfront knowledge about a particular board it is running on. With future designs and more permutations, knowing the board and its capabilities is crucial.

How
To enable this, a couple of aspects have to be strung together.

• XLink improvements which allow querying unbooted device about its EEPROM contents (or just the board name specified in EEPROM)
• Using the board name, host can lookup an applicable BoardConfig to use.
• BoardConfig resembles Linux's device tree - it talks about exposed MIPI lines, can downselect which camera drivers are connected to it (to allow zero probing), which GPIOs are available, SPI, IMU, and other HW features...
• The BoardConfig is sent to device when booting, using preboot mechanism. (*might be split into preboot and regular part)

There are going to be existing (included) BoardConfigs which will specify configurations of known boards - OAK-1, OAK-D, ..., but along included configurations, will be a way of creating your own configuration, which allows to specify a BoardConfig for a custom board. This enables potential optimization of boot up time by pinning cameras, etc...

What & API
Main aspects of (public) API will be:

• Expose board name (& SOM name where applicable) from XLink along existing infromation
• Create BoardConfig structure that describes needed HW features
• Add a lookup table for internal BoardConfigs (known boards)
• Optional: Add mechanism for adding custom BoardConfigs to lookup table (eg being able to register it and use rest of API in same manner)
• Add API to DeviceBase/Device that connects to a device with a specified BoardConfig (skips lookup, uses specified config)

Conclusion
Side effect of this feature is capability of knowing the board config (when a known board is connected) before booting. This information can generally distinguish between specific OAK devices, but also enables feature lookup, as configurations will specify this upfront (OAK-1 exposes only 1 camera of type X, OAK-D exposes 3 cameras, ...). In some cases, this isn't adequate and runtime (after booting FW) querying is required. (eg knowing exact sensor type on a board which supports multiple, etc...)

[diablodale]

Hi. Below is one of my real-world use cases. I wrote this use case before I read this issue so I wasn't biased by it.
It represents not me, but instead represents my customer.
I build a solution that hides the tech details. To create my customer's experience below,
I need APIs that make such possible. I build the plugin called "dp.oak".

My customer

Customer uses Cycling74 Max patching environment. This environment is always
running, always active. There is no stop/compile. A "patch" is a series of nodes
which are connected in a web of patchcords; defining in/outs of the nodes.

Customer is a musician and researcher at a university. They want to use Max to create
an interactive system that "watches" them play the violin and reacts by dynamically
generating music and visuals.

Customer uses Max, OAK-D, and a plugin called "dp.oak". dp.oak provides in/out to the OAK series
of devices. dp.oak can output in native Max formats the OAK outputs of depth, color, body
pose, and pointclouds. Max itself is interactive. The customer is changing patchcords,
processing the output of dp.oak, sending that processed output to graphics+music nodes all
realtime...all interactive.

Customer continues to iterate on their system. They purchase an OAK-1 and the new OAK-D-Lite.
Customer knows some features are not available on OAK-1 (e.g. depth) and
OAK-D-Lite (e.g. IMU, highres depth). dp.oak works with all three models of sensors. Customer did
not have to download a new version of dp.oak to use these new sensors.

dp.oak and Max together have a UI and that UI enables/disables elements that represent features
the sensor has/not. For example, the UI has a checkbox for depth output and depth alignment.
Both of these UI elements are disabled for the OAK-1. The UI for depth resolution changes
between the OAK-D and OAK-D-Lite. The latter does not have the 1280x720 highres depth listed.

Max and dp.oak can also present an overview of all OAK sensors attached. This UI presents a
pulldown/radio buttons for the first 5 attached devices. It also presents a search box where
any serial number of a device can be typed. The OAK series has POE devices so there may be
hundreds of devices "attached". It is possible to search for devices that match requirements.
The search UI has a method to select features (e.g. depth+fixed focus+poe) and will then
only list "attached" devices that match those requirements.

After a device is selected, the UI presents a picture of the device and an extensive
list of descriptors for all features and settings supported by the specific device, e.g.

• serial number
• model
• memory
• color output: true/false
• color resolutions: 1920x1080, 640x480,
• color frame format: NV12, RGB, BGR, YUY2,
• color camera settings possible: iso, exposure time, auto/manual, etc.
• color camera fps
• lens focus: sauto, auto+controllable, manual, fixed, ...)
• depth output: true/false, or none/myriad-disparity/nn-depth-v1/etc.
• depth resolution: 640x480, 1280x720,
• depth alignment: left, center, right, rgb, etc.
• mono/depth camera fps
• imu: true/false
• usb speed
• poe: true/false
• poe speed
• etc.

The musician continues to evolve their system. Next year, they add three of the
(yet unreleased) OAK POE v2 devices. The musician connects the ether cables and opens
the UI of their system. The UI shows the extensive list of features, similar to the older
models. The musician did not have to download an updated version of dp.oak
to use their new POEv2 devices. The musician adds the new POEv2 devices to their Max
patch and they immediately work. Features POEv2 does/not have are enabled/disabled
in the UI. Customers tunes the settings for these new devices (focus, exposure,
resolution, etc.). Now their setup is OAK-D, OAK-1, OAK-D-Lite, and
three OAK-POEv2. They click "go" in their Max UI...and all six devices bootup and
run their pipeline+patches. 🎶🎇

The musician uses their system during live performances. They sometimes adjust
NN parameters, or camera exposure to deal with a problem -or- to make amazing
improvised chaos. Happy artist, happy customer!

[diablodale]

@themarpe , related... is there a published list of today's return results/mapping of calibration.getEepromData().boardName for each of the public/common models oak-1, oak-d, oak-d-poe, oak-1-poe, oak-d-lite, etc.

I see https://github.com/luxonis/depthai-hardware but... my OAK-D returns BW1098OBC which is
https://github.com/luxonis/depthai-hardware/tree/master/BW1098OBC_DepthAI_USB3C
rather than
https://github.com/luxonis/depthai-hardware/tree/master/BW1098OAK_USB3C

[themarpe]

Thanks for the use case. Hopefully it'll enable most of your required functionality.

Regarding a mapping, @saching13 if we have a up to date list somewhere.
There are some mappings which have different name but point to same model, as in your case.

We'll consolidate that mapping with above feature being implemented as well

[diablodale]

In your OP you write "lookup table for internal BoardConfigs (known boards)"
This suggests to me that the board config's are always host-side. So when any new Luxonis device is released, the host-side app is out of date and will not have a board config for the newly released device.
Can the device itself contain a (default) board config? Perhaps retail devices have factory-stored board config in EEPROM alongside the cali data? This allows new retail devices to declare themselves.
A host app can use local "known boards" config, a custom config, or use the default config from EEPROM.

[themarpe]

So when any new Luxonis device is released, the host-side app is out of date and will not have a board config for the newly released device.

Yes, this is the initial idea. Note that also with new devices, (usually) updated FW is released which (usually) requires library update.

It would be possible, in same manner as its currently. Existing devices don't yet have board config information in EEPROM and space is also a bit limited. TBD. I think best course of action here is a mix of all:

A host app can use local "known boards" config, a custom config, or use the default config from EEPROM.

*or use default config from EEPROM IFF exists otherwise fallback to generic (as it is done currently)

Though, as you can see now, this isn't a must have, but more of an optimization & UX (there might be cases with later HW revisions, where its a must).

[saching13]

List of Board Names in EEPROM in factory :

1. OAK-D (Old Name: BW1098OBC)
2. OAK-D-CM3 (Old Name: BW1097)
3. OAK-D-IoT-40 (Old Name: BW1092)
4. OAK-D-CM4
5. OAK-D-IoT-75 (Old Name: DM1098OAKD_WIFI)
6. OAK-1-POE
7. OAK-1 (Old Name: BW1093OAK)
8. OAK-D-LITE
9. OAK-D-POE

[diablodale]

@saching13 thank you! That is immediately helpful for me. And to verify... there are devices, for example the OAK-D, sold to people around with world that have calibration.getEepromData().boardName equal to the exact case-sensitive string OAK-D or BW1098OBC

@themarpe , I might be getting a distinction. 🤔

• Luxonis device firmware presents a set of public interfaces. These firmware public interfaces are common across a wide set of Luxonis devices.
• The internal closed-source code of this firmware will evolve to add support for changed/new device components. In general, the firmware will hide these changes and continue to present the consistant public interfaces. To support drastically changed/new components, the firmware must internally change.
• The "depthai" set of c++ APIs presents a set of public c++ interfaces. These will work across a wide set of Luxonis devices now+future.
• This depthai c++ API and its open-source implementation code uses the XLink and firmware public interfaces.
• This depthai API is manifested in static libs or shared libs (e.g. DLL or LIB). Both of these forms include the binary data of the Luxonis firmware...it is included in the DLL/LIB itself as a binary resource/artifact.

Did I get all that right?

If so, this implies to me that while the depthai c++ api interface supports current+future Luxonis devices...the firmware does not.
Since the firmware is included within the depthai DLL/LIB itself as a means of distribution, that causes depthai DLL/LIB/pywheel/etc (as a distribution package) to not support current+future. Future support is bound by the abilities of the firmware that was included within the library at the time of packaging.

[Luxonis-Brandon]

I think so. But I'm not technical enough to confirm exactly. So will wait on @themarpe when he's online (in Europe, so late there now).

[saching13]

And to verify... there are devices, for example the OAK-D, sold to people around with world that have calibration.getEepromData().boardName equal to the exact case-sensitive string OAK-D or BW1098OBC

Yes.

[themarpe]

@diablodale
Correct overview. TLDR: At the end of the day, a "now" version of depthai supports past+now devices (with exceptions)

We do exploit a couple of points, because we use packed FW approach:

• While the public FW interface is common between devices (as we have unified FW), it is not versioned and iterates.
• DepthAI internals might change to accommodate differences in communication between FW and library as well

The reason I explicitly point the above is that for the most part we use the packed FW approach into our advantage of iterating as quickly as one can with an embedded product. We can release a FW improvement with same zero inconvenience to the user as a library update (not an attribute of packed FW but that of not requiring a explicit FW upgrade, by eg. reflashing)

The FW being packed into the lib isn't a requirement per se. Initially the build system was designed in a manner of being able to read it from filesystem while installing it along side into eg. /usr/share. So this can still be done if desired. Note, that there is again a level of interoperability between "FW" (actually FWP - firmware package) and a specific library version. So it might be that later FWPs won't work with todays library. (same reason, it iterates every now and then)

To combat this, preferably, the abstraction is made at API level of DepthAI. (ABI might work temporarily, but we don't intend to be ABI stable). API stability is something we are more careful about. And library is updated to accommodate new devices and improvements, etc..

[diablodale]

Thanks @saching13 and @themarpe. 😃
I now have code that detects specific devices, maps device->features, and persists info. When BoardConfig is official, I can likely map your schema to mine to easily digest.

[diablodale]

@saching13 are you still on the team? or can @themarpe point me to someone else?
I request access to a formatted list (excel, json, csv, etc.) of all the OAK models and programmatic identifiers.

A customer of mine reported their OAK-D-PRO is not identifying itself as such and instead id's itself as DM9098. Probably changed behavior from factory burned firmware...perhaps part of the BoardConfig progression.

Looking at https://github.com/luxonis/depthai-shared/blob/f03e9d9b08df2c0a50b8a928901ce95b14f9c174/include/depthai-shared/device/BoardConfig.hpp#L29 and https://github.com/luxonis/depthai-shared/blob/f03e9d9b08df2c0a50b8a928901ce95b14f9c174/include/depthai-shared/common/EepromData.hpp#L21 I see a productName however that doesn't have a string which matches the string list above, layoutformat (spaces or dashes) is different, and I can find no SDK documentation on the topic. I see an issue #651 (comment) with one of these strings and I can see it has FF and spaces in it.

As discussed since 2021, it is not realistic for customers to constantly track model releases. We still need discovery and config info as in my OP.

1. What is the current state of this in release v2.22.0? I'm exploring some newer discovery info with CameraFeatures
2. Where is the documentation so we known how to use and understand these fields? For example, EepromData doesn't have anything that will reliably declare that a OAK device has an IMU. Maybe BoardConfig::imu has it, however I can find no public or private API to get a BoardConfig from a device. It seems to only be one-way to the device. I only see PipelineImpl::getBoardConfig() and that does not get the config of an hardware device. Perhaps this is a dying struct since CameraFeatures has some info.

[themarpe]

device.getDeviceName() would be the way to go - exposes "ALL-CAPS-WITH-DASHES" naming and covers some edge cases of older devices that did not have this information available.

WRT IMU, its under device.getConnectedIMU() which reports the currently found IMU if available.

Correct, BoardConfig is one way to the device, to configure it, not vice versa. Pipeline carries it just in case it has to configure the device in some manner (eg openvino version, etc...)

The BoardConfig from host did not happen, but rather FW was updated to carry this information. Nowadays devices are much more discoverable (eg getConnectedIMU, getConnectedCameraFeatures, etc...) to allow for dynamic use of various new devices.

[diablodale]

Got it. While exploring I read in SDK code

For some sensors it's not possible to determine if they are color or mono(e.g. OV9782 and OV9282), so this could return more than one entry".

Is this fixed/resolved...or does std::vector<CameraSensorConfig> CameraFeatures::configs have entries that can distinguish a color "camera" from some other camera? The intention is to programmatically discover the color camera since the enum is deprecated. And I would prefer to get out of the constant chase of keeping my private mapping of DASH-MODEL-NAMES to cameras+features.

I see std::string CameraFeatures::name. Is this string exactly always the same string used with ::setCamera(string);? So for the common retail units it will always have "left", "right", and "????" for color? Is the definitive method to discover the color camera to walk the DeviceBase::getConnectedCameraFeatures() vector looking for name == "?????"? Or is there a structured field that doesn't have the multi-entry issue above?