open source the depthengine

[mindbeast]

It's clear that microsoft isn't actively supporting the azure kinect SDK anymore, but the hardware still has many active users. I think the community would still like to keep using and maintaining the hardware. This isn't really possible without all of the SDK being totally open: https://github.com/microsoft/Azure-Kinect-Sensor-SDK/blob/develop/docs/depthengine.md

Please, microsoft, publish the source for the small remaining parts of the SDK for decoding the depth images from the sensor. This will foster a lot of open source development on this old hardware and, frankly, generate a lot of support for microsoft in the robotics community.

[mindbeast]

I know this a large ask, but ideally one that leaves less of a long term support burden on microsoft and keeps the remaining users of this hardware happy that they invested in this equipment in the first place.

[mindbeast]

Anyone from microsoft still around? My support thread on azure isn't getting attention either.

[mindbeast]

I'm still trying to get some attention on the azure forums, but no one from microsoft has gotten back yet.

[MaxPalmer-UH]

It's very sad, such a good product but seemingly 'done' on the SW side....

[mindbeast]

Tell me about it. The only response I can get from microsoft right now is "buy something else"

https://learn.microsoft.com/en-us/answers/questions/1303648/azure-kinect-sdk-is-unmaintained

[mindbeast]

From the forums:

The Depth Engine algorithm is Microsoft IP, and we currently have no plans to open-source this component. The firmware of Azure Kinect DK is authored by a 3^rd party (Vendor of the SoC used in AKDK) to requirements provided by Microsoft, I don't think we can open-source this code either. We've licensed our ToF technology to Analog Devices, they have sensors, module and dev kit offerings if someone wants a custom design: 3D Time of Flight (ToF) | Analog Devices Similarly, we've also partnered with Orbbec and they have a camera (available for purchase) using the same ToF Depth module as Azure Kinect DK: Femto Mega - ORBBEC - 3D Vision for a 3D World

[mindbeast]

I've managed to also DM azure support on twitter and they have indicated they are going to respond in some capacity.

[jncronin]

So I managed a partial open source implementation of the depth library: https://github.com/jncronin/Azure-Kinect-Sensor-SDK/tree/no-de

• It currently only creates the depth map (i.e. no point cloud) and only works for NFOV unbinned (although shouldn't be too difficult to get working with other modes).
• Uses cuda rather than opengl (so can run true headless on NVIDIA Jetson Nano which was my primary reason for creating). CPU version is also available.
• Uses a slow brute force algorithm for calculating depth rather than the patented Microsoft version. Still, the speed is adequate - ~2ms per 640x576 frame on RTX 4060 or 10-20ms per frame on Jetson Nano.
• No filtering so the depth images aren't as pretty
• Needs further calibration - the frequencies of emitted light are not published so they are all based upon rough estimates

[dbruning]

This is an amazing effort. I'm not a C developer (C#) but please let me know if I can help.

[dbruning]

@jncronin You mention a "patented Microsoft version" of the depth calculation algorithm. Do you mean this one and if so, did you know it was abandoned in 2015?

[jncronin]

I got the details from https://medium.com/chronoptics-time-of-flight/phase-wrapping-and-its-solution-in-time-of-flight-depth-sensing-493aa8b21c42 and you are absolutely correct it links to the abandoned patent.

However, there is a new application https://image-ppubs.uspto.gov/dirsearch-public/print/downloadPdf/20210033731 which seems to cover similar stuff.

The current approach seems to work reasonably well especially with cuda assistance and therefore I will probably stick with it.

[dbruning]

Good info, thankyou.
Any interest in developing a drop-in open-source replacement for the depth engine? I know that your solution involves modifications to the SDK, do you think it could be done as a re-implementation of the depthengine_2_0.dll ? I'm willing to sponsor work in this area.
Sad that the eventual reply to the Q&A post was a non-answer ("We licensed our IP!")

[jncronin]

It should already be a drop in replacement (but for the entire libk4a rather than just the depthengine), at least for the NFOV unbinned and WFOV binned modes.

There is a new dependency on CUDA. Unfortunately some sort of GPU compute is essential. The original version uses OpenGL and therefore is a little bit more portable to non-CUDA devices but I created this expressly to allow use on headless Nvidia Jetson Nano hence the choice of CUDA.

[dbruning]

Understood. I need it to be just an alternative depthengine_2_0.dll, and I'm OK with the CUDA dependency. If you're interested in a contract to create that, please get in touch via darren@bruning.net.nz

[joshi-bharat]

Has anyone figure about how the transform engine works? The transformation from depth image to color image. They say it uses triangle mesh warping.

This transformation function is more complex than simply calling k4a_calibration_2d_to_2d() for every pixel. It warps a triangle mesh from the geometry of the depth camera into the geometry of the color camera. The triangle mesh is used to avoid generating holes in the transformed depth image. A Z-buffer ensures that occlusions are handled correctly. GPU acceleration is enabled for this function by default.

Does anybody know about the implementation details? Any idea @jncronin