PlusServer OpenCVVideo stream freezes in Slicer, and timestamp buffer error when mixing US image with NDI tracking

[patigarciaa]

Hi everyone,

I am working with 3D Slicer and Plus Toolkit for an ultrasound spatial calibration setup. My hardware setup includes an NDI Polaris tracker, a tracked ultrasound probe rigid body, and a tracked stylus rigid body. The ultrasound image is streamed as an MJPEG video stream using an OpenCVVideo device in PlusServer.

I tested three different Plus configuration files:

1. US image only
   This configuration only streams the ultrasound image to Slicer. It works at first, but after some time the image in Slicer freezes. PlusServer does not show any clear error message when this happens.

2. US image + US probe rigid body + stylus rigid body
   In this configuration, the image and both transforms are sent to Slicer. The image and transforms appear correctly in Slicer, but PlusServer prints the following error repeatedly:

   Failed to get video buffer item by timestamp 151.246

   The error comes from:

   vtkPlusChannel.cxx(790)

3. Tracking only: US probe rigid body + stylus rigid body
   This configuration works correctly. Both transforms are sent to Slicer without problems.

During the test, the ultrasound probe was fixed and kept in contact with water, while the stylus was moved. Therefore, the video image itself was almost static, but the stylus transform was changing.

My question is: could this be caused by a synchronization or buffering problem between the OpenCVVideo MJPEG stream and the tracker stream when using the VirtualMixer? Is there any recommended way to configure the timestamps, buffer size, acquisition rate, or LocalTimeOffsetSec when combining an MJPEG ultrasound stream with NDI Polaris tracking?

I attach the three XML configuration files. Any advice would be very helpful.

Thank you!
PlusDeviceSet_Server_US-vero.xml
CaliEspacialPrueba2-vero.xml
PlusDeviceSet_Server_NDIPuntero_USRB-vero.xml