D435i Aligning IMU and RGB Frames

[JBBee]

---

Required Info
Camera Model	D435i
Firmware Version	05.11.01.00
Operating System & Version	Win 10
Kernel Version (Linux Only)
Platform
SDK Version	2.17.1
Language	C++
Segment	AR

Issue Description

I'm capturing RGB and Depth frames at 30 fps using one pipe and then using a second pipe with a callback to capture accelerometer and gyro readings at 63 and 200 hz. I'm then trying to align the IMU frame timestamps with the timestamp of the RGB frame. However, the timestamps of the IMU readings are in the hardware clock domain and the timestamps of the RGB frames are in the system time domain. How do I align the two using their timestamps?

Thanks.

[dorodnic]

You need to enable frame metadata to get hardware timestamp for the color stream:
Enable hardware timestamps on Windows / Enable hardware timestamp on Linux

[JBBee]

This worked, so now both the color stream and IMU stream are in the hardware clock domain. But there appears to be a large offset between the two timestamps even when collected concurrently. Are they timestamped using the same clock, or how do I compare the two values?

Thanks.

[dorodnic]

@JBBee - yes, they should have the same clock.
It is reasonable for them to have some offset. You should expect the IMU data to be ~40 ms ahead of depth data. This is due to IMU having ~2ms latency, while UVC frames have roughly at least one frame of latency (and more on Windows).
If you see the situation being other way around (IMU lagging behind depth) - it is probably a rare firmware bug we are investigating right now.
To clarify, if you get motion_frame and depth_frame in the same frameset, you should expect motion_frame.timestamp < depth_frame.timestamp. - Actually, I'm not so sure about the last part, because when using wait_for_frames the pipeline will most likely try to synchronize between them.
However, if you add printfs to callback example you should see this effect.

[JBBee]

@dorodnic I'm actually collecting the IMU samples using a second pipe with a callback, and I'm getting timestamp differences between the IMU samples and the color frames such as 4.12596e+06. Is this because they're collected on a different pipes?

[dorodnic]

Hmm... And both are hardware timestamps?
That's strange. Using two pipelines with callbacks (or one with wait for frames and one with callbacks) should not introduce any problems.
What about when starting streaming in the RealSense Viewer? You can see timestamps of each stream under (i) icon of each stream.

[dorodnic]

Thank you. Did you try this with later pre-release? There are some issues with 2.18 and 5.11.1 firmware, but it would be interesting to check with newer viewer. This looks like something we have encountered and already resolved (note that the FPS is also completely wrong)

[JBBee]

The timestamps are still different with that pre-release. Should I try downgrading the firmware?

[dorodnic]

Could you please share the exact Windows build you are using?
Downgrading firmware is probably not recommended, but we are all waiting for the new firmware to be released for several weeks now (should be published any day now) and it might help with this problem.

[JBBee]

Ok great, I'll lookout for the new firmware. I'm using Win 10 build 1703.

[RealSenseCustomerSupport]

---

Hello @JBBee,

New firmware has been released, please see latest RealSense Camera Firmware located here:

https://downloadcenter.intel.com/download/28568/Latest-Firmware-for-Intel-RealSense-D400-Product-Family?product=128255

Windows Device Firmware Update Tool (required to upgrade firmware):

https://downloadcenter.intel.com/download/27514/Windows-Device-Firmware-Update-Tool-and-Latest-Firmware-for-Intel-RealSense-D400-Product-Family?v=t

Also please use latest LibRealSense Release:

https://github.com/IntelRealSense/librealsense

Please update ticket with your results.
Thank you!

[JBBee]

Hi @RealSenseCustomerSupport

After following the instructions to enable frame metadata I ran it with this latest firmware release and the timestamps now match.

Thank you!