Corrupted JPEG Stream

[jmachowinski]

Describe the bug
We are experiencing a rather strange bug with the color stream of the kinect.
On an AMD EPYC 7302 machine, the color stream is corrupted.
This is reproducible using the k4aviewer and the ros wrapper.

I patched in a better debug message in case the MJPEG decoding fails and
it gave me this:
DecodeMJPEGtoBGRA32(). MJPEG decode failed, dropping image: Not a JPEG file: starts with 0xfe 0x77
DecodeMJPEGtoBGRA32(). MJPEG decode failed, dropping image: Not a JPEG file: starts with 0xf8 0x54
DecodeMJPEGtoBGRA32(). MJPEG decode failed, dropping image: Corrupt JPEG data: premature end of data segment
DecodeMJPEGtoBGRA32(). MJPEG decode failed, dropping image: Corrupt JPEG data: 1283 extraneous bytes before marker 0xd3
DecodeMJPEGtoBGRA32(). MJPEG decode failed, dropping image: Corrupt JPEG data: 161 extraneous bytes before marker 0xd4
DecodeMJPEGtoBGRA32(). MJPEG decode failed, dropping image: Not a JPEG file: starts with 0xfb 0xbf
DecodeMJPEGtoBGRA32(). MJPEG decode failed, dropping image: Corrupt JPEG data: 9005 extraneous bytes before marker 0xd6
DecodeMJPEGtoBGRA32(). MJPEG decode failed, dropping image: Corrupt JPEG data: found marker 0xd5 instead of RST3
DecodeMJPEGtoBGRA32(). MJPEG decode failed, dropping image: Corrupt JPEG data: 841 extraneous bytes before marker 0xd6

This then leads to the error :
replace_sample(). capturesync_drop, releasing capture early due to full queue TS: 57300211 type:Depth
as there is no color image available for the depth measurement.

Note usbfs_memory_mb is set to 1024

To Reproduce
On an AMD EPYC 7302 :

1. Start k4aviewer
2. Enable color camera
3. Disable all other streams
4. Choose any resolution / fps (720p / 5 FPS also fails)
5. Start
6. Log shows a bunch of MJPEG decode failed, dropping image messages
7. After a few seconds a popup 'Camera failure : timed out!' appears

Desktop

• OS with Version: Ubuntu 18.04
• Kernel : 5.4.0-47-generic
• SDK Version: 1.4.1
• Firmware version: 1.6.110079014
• GPU : NVidia 2070 Super
• CPU AMD EPIC 7302
• Mainboard : Supermicro Mainboard H11DSi-NT

Additional context
Note, on our normal desktop machines everything is working fine.

We also tried using an external PCI-E USB Host Controller (DeLock 90492)
but this did not work either. The controller also gave us a bunch of wired
kernel warnings, so it might be a driver issue with this card.

Is there any PCI-E USB Host Controller with at least two ports, that is known
to work under linux ?

Grabbing only the depth stream seems to work. Out of curiosity, is there some
sort of CRC on the depth data, to detect corruption ?

[jmachowinski]

We debugged this issue further, it seems, the Azure is extremely
sensitive to the used USB host controller.
So far we tested :
Asmedia 3142

• Not working

Renesas/NEC - µPD720202

• Not working

Renesas/NEC - µPD720201

• Working
• Only one Azure per pcie card
• Minor frame drops
• Ros driver failure after ~20 minutes

Fresco Logic FL1100

• Working
• Two Azure per pcie card
  • Ros driver failure after ~10 minutes

[RoseFlunder]

Yes it definetly is. Also the choice of USB cabel matters. I hab cables labeled as USB 3.0 that did fail after some minutes. The cable that comes with the kinect itself works the best but it is a bit short.

Have already seen this page?
https://docs.microsoft.com/en-us/azure/kinect-dk/troubleshooting

"For the Azure Kinect DK on Windows, Intel, Texas Instruments (TI), and Renesas are the only host controllers that are supported. The Azure Kinect SDK on Windows platforms relies on a unified container ID, and it must span USB 2.0 and 3.0 devices so that the SDK can find the depth, color, and audio devices that are physically located on the same device. On Linux, more host controllers may be supported as that platform relies less on the container ID and more on device serial numbers."

[qm13]

Yes, only select USB controllers are supported per text quoted by @RoseFlunder.

Also make sure you are using Sensor SDK 1.4.1 as we made some changes to improve resilience to dropped color frames. See issue #1194.

[jmachowinski]

I was aware of the troubleshooting page. But lets just say the information
regarding linux is sparse...

We are using the latest firmware and SDK 1.4.1

As for our problem, we are currently testing the Fresco Logic
FL1100 with only one camera per host controller. This ran
stable for 3 hours with one camera, yesterday.
We are trying multiple cards with a camera each now.

One thing I noticed while testing, is that only the color
stream is unstable. The depth and imu stream seems
to be stable all the time.

Also I suspect a bug in the ros node, as it does not detect
a broken color stream.

[RoseFlunder]

I haven't measured the time at our PCs running the kinects but I think we kept them running for 24h without an error on Ubuntu, with default cable and connected to USB Ports on the Mainboard with Intel Chipset.
Not the ROS Node though but we don't do anything different regarding getting captures compared to the ROS node. Just calling "getCapture" using infinite timeout.
But we also use only one Kinect per PC which send their data to a central PC for processing.

[qm13]

@jmachowinski I have passed you ROS node issue to the Microsoft ROS team.

[ooeygui]

@jmachowinski When you say the ROS node fails after x amount of time, can you post the failure? Does it crash or start throwing errors?

[jmachowinski]

@ooeygui The node does not crash, but it stops publishing any data.
The log is full of

[2020-09-18 08:52:14.087] [error] [t=8620] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/capturesync/capturesync.c (142): replace_sample(). capturesync_drop, releasing capture early due to full queue TS:3874968522 type:Depth
[2020-09-18 08:52:14.119] [error] [t=8620] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/capturesync/capturesync.c (142): replace_sample(). capturesync_drop, releasing capture early due to full queue TS:3875001866 type:Depth

As the color stream broke down completely.

I already looked into the code, and have the guess (only a guess, not verified yet) that if
k4a::device::get_capture()
is called with timeout infinite the function never returns in case the color stream broke down.

I just looked at the code again, and now I am pretty sure this is the case. With an infinite timeout,
we wait for a push to the capture queue. As the color stream broke down, this is never happening
and the ros node just stalls. I think a good workaround for this, would be to give a timeout
of double the fps time, and shutdown the node if we did not receive X consecutive captures.

[jmachowinski]

As for the long time tests :
2 of 3 cameras stopped sending color images after
some hours. I'll patch in some additional debug messages,
to determine the exact cause of this.

I also noticed, that this is CPU load dependent. Without subscribing to
the topics of the node, the color stream did not drop out (I verified in the code,
that the capture from the camera actually happens in this case). This behavior
puzzles me, as I run the tests on a 64 core machine, and every thread
has basically its own cpu.

[ooeygui]

@jmachowinski Thank you for the investigation!
I need to profile the code to see where the slowdown is occurring. We do know that there are some problems with the OpenCV version being used with Melodic, and there are some inefficiencies in the Kinect ROS node. Additionally, since it is using ROS pub/sub without a quality of service monitor, the publishing rate itself may be too fast for the network stack - We may need to throttle upstream of it.

I'll post findings when I am able to carve off time to work on this ROS node.

[jmachowinski]

To answer my own question :

Out of curiosity, is there some sort of CRC on the depth data, to detect corruption ?

The depth data is transmitted in usb bulk mode, so it should be always complete.

I debugged this further and I am pretty sure, we are dealing with a firmware issue
here. My findings so far :
The color stream is an usb isochronous transfer.
Depth and IMU are transmitted as usb bulk transfer.

The isochronous transfer is started in the libuvc part and keeps on
going. Using wireshark, one can see, that the usb transfers are
send out to the device.
At start the color device correctly fills in the payload of the transfer,
and everything is working. At some point, the color device just stops
doing this.
Also note, there is no error transmitted to the usb host.
If one adds debug to the function :
https://github.com/microsoft/libuvc/blob/5fc483d596c63f1bcd36be35d512468c0b75c5f3/src/stream.c#L601
this behavior is visible.

Any ideas ?