diff --git a/docs/source/_static/images/examples/depth_video_synced.gif b/docs/source/_static/images/examples/depth_video_synced.gif new file mode 100644 index 000000000..1c37fd5e3 Binary files /dev/null and b/docs/source/_static/images/examples/depth_video_synced.gif differ diff --git a/docs/source/components/messages/message_group.rst b/docs/source/components/messages/message_group.rst new file mode 100644 index 000000000..dd56d616e --- /dev/null +++ b/docs/source/components/messages/message_group.rst @@ -0,0 +1,32 @@ +MessageGroup +============ + +The MessageGroup message type is a versatile container used in DepthAI pipelines to group together a map of arbitrary DepthAI messages. It serves as the primary output of the :ref:`Sync` node, effectively synchronizing various input streams, and acts as the input to the :ref:`Demux` node for subsequent disaggregation and processing. + +Creating MessageGroup +##################### + +MessageGroup can be created automatically by the Sync node as it aligns and groups messages from different sources based on their timestamps. Alternatively, it can be manually constructed in a host application or within a :ref:`Script` node to create custom groupings of DepthAI messages. + +Reference +######### + +.. tabs:: + + .. tab:: Python + + .. autoclass:: depthai.MessageGroup + :members: + :inherited-members: + :noindex: + + .. tab:: C++ + + .. doxygenclass:: dai::MessageGroup + :project: depthai-core + :members: + :private-members: + :undoc-members: + + +.. include:: ../../includes/footer-short.rst diff --git a/docs/source/components/nodes/demux.rst b/docs/source/components/nodes/demux.rst new file mode 100644 index 000000000..1b92374c1 --- /dev/null +++ b/docs/source/components/nodes/demux.rst @@ -0,0 +1,122 @@ +Demux +===== + + +The Demux (Demultiplexer) node is used to separate a `MessageGroup` into individual outputs. It currently serves as way to demultiplex the output of :ref:`Sync` node. + +How to Place It +############### + +.. tabs:: + + .. code-tab:: py + + pipeline = dai.Pipeline() + demux = pipeline.create(dai.node.MessageDemux) + + .. code-tab:: c++ + + dai::Pipeline pipeline; + auto demux = pipeline.create(); + + +Inputs and Outputs +################## + +.. code-block:: + + ┌───────────────────┐ + input │ │ + ──────────────►│ │ + │ Demux │ output1 + │ ├───────────► + │ │ output2 + │ ├───────────► + │ │ ... + └───────────────────┘ + +**Message types** + +- :code:`input` - :ref:`MessageGroup` +- :code:`output1`, :code:`output2`, ... - Individual output messages + +Usage +##### + +The Demux node is particularly useful for handling different types of data coming from a single source. +For example, when the :ref:`Sync` node is used to synchronize the outputs of multiple nodes, the output of the Sync node is a :ref:`MessageGroup` containing all the messages from the synchronized nodes. The Demux node can be used to separate the messages into individual streams. + +.. tabs:: + + .. code-tab:: py + + # Create sync node and set sync threshold + sync = pipeline.create(dai.node.Sync) + sync.setSyncThresholdMs(timedelta(milliseconds=100)) + + # Create demux node + demux = pipeline.create(dai.node.MessageDemux) + + # Sync the outputs of multiple nodes + rgb.preview.link(sync.inputs["rgb"]) + stereo.depth.link(sync.inputs["depth"]) + script.outputs["out"].link(sync.inputs["script"]) + + sync.out.link(demux.input) # Sync output is a MessageGroup containing all the messages from the synchronized nodes + + # Demux the MessageGroup into individual messages + demux.outputs["rgb"].link(xout1.input) + demux.outputs["depth"].link(xout2.input) + demux.outputs["script"].link(xout3.input) + + + + + .. code-tab:: c++ + + // Create sync node and set sync threshold + auto sync = pipeline.create(); + sync->setSyncThreshold(std::chrono::milliseconds(100)); + + // Create demux node + auto demux = pipeline.create(); + + // Sync the outputs of multiple nodes + rgb.preview.link(sync->input["rgb"]); + stereo.depth.link(sync->input["depth"]); + script.outputs["out"].link(sync->input["script"]); + + sync->out.link(demux->input); // Sync output is a MessageGroup containing all the messages from the synchronized nodes + + // Demux the MessageGroup into individual messages + demux->outputs["rgb"].link(xout1.input); + demux->outputs["depth"].link(xout2.input); + demux->outputs["script"].link(xout3.input); + + +Examples of Functionality +########################## + +- :ref:`Demuxing Synchronized Script Outputs` + +Reference +######### + +.. tabs:: + + .. tab:: Python + + .. autoclass:: depthai.node.MessageDemux + :members: + :inherited-members: + :noindex: + + .. tab:: C++ + + .. doxygenclass:: dai::node::MessageDemux + :project: depthai-core + :members: + :private-members: + :undoc-members: + +.. include:: ../../includes/footer-short.rst diff --git a/docs/source/components/nodes/sync_node.rst b/docs/source/components/nodes/sync_node.rst new file mode 100644 index 000000000..9656adbf4 --- /dev/null +++ b/docs/source/components/nodes/sync_node.rst @@ -0,0 +1,96 @@ +Sync +==== + +The Sync node is used for synchronizing multiple input streams based on their timestamps. It outputs a grouped message containing synchronized frames from the input streams. +The output message is a :ref:`MessageGroup` containing synchronized messages from all the input streams. These can be demultiplexed using the :ref:`Demux` node. + +How to Use +########## + +.. tabs:: + + .. code-tab:: py + + pipeline = dai.Pipeline() + sync = pipeline.create(dai.node.MessageDemux) + + # Configure threshold for timestamp alignment + sync.setSyncThreshold(timedelta(milliseconds=50)) + + # Configure inputs to be synchronized + sync.inputs["input1"] + sync.inputs["input2"] + + # ... + + .. code-tab:: c++ + + dai::Pipeline pipeline; + auto sync = pipeline.create(); + + // Configure threshold for timestamp alignment + sync->setSyncThreshold(std::chrono::milliseconds(50)); + + // Configure inputs to be synchronized + sync->inputs["input1"]; + sync->inputs["input2"]; + + // ... + + +Inputs and Outputs +################## + +.. code-block:: + + ┌───────────────────┐ + input1 │ │ + ──────────────►│ │ + input2 │ │ out + ──────────────►│ Sync ├───────────► + │ │ + ... │ │ + ──────────────►│ │ + └───────────────────┘ + +**Message types** + +- :code:`input1`, :code:`input2`, ... - any message type from :ref:`Messages` +- :code:`out` - :ref:`MessageGroup` + + +Message Synchronization +######################## + +The Sync node aligns incoming messages based on their timestamps. The synchronization criteria and behavior can be configured using the :code:`depthai.node.Sync.setSyncThreshold` and :code:`depthai.node.Sync.setSyncAttempts` method. More info in the :ref:`API Reference `. + + +Examples of Functionality +########################## + +- :ref:`Depth and Video Sync` +- :ref:`Multiple Scripts Sync` +- :ref:`IMU and Video Sync` +- :ref:`Demuxing Synchronized Script Outputs` + +Reference +######### + +.. tabs:: + + .. tab:: Python + + .. autoclass:: depthai.node.Sync + :members: + :inherited-members: + :noindex: + + .. tab:: C++ + + .. doxygenclass:: dai::node::Sync + :project: depthai-core + :members: + :private-members: + :undoc-members: + +.. include:: ../../includes/footer-short.rst diff --git a/docs/source/samples/Sync/demux_message_group.rst b/docs/source/samples/Sync/demux_message_group.rst new file mode 100644 index 000000000..5cbd08be8 --- /dev/null +++ b/docs/source/samples/Sync/demux_message_group.rst @@ -0,0 +1,78 @@ +Demuxing Synchronized Script Outputs +==================================== + +This example demonstrates the use of the DepthAI Sync node in conjunction with the Demux node to synchronize and then demux outputs from two separate script nodes. Each script node generates data buffers at different intervals, which are first synchronized by the Sync node and then demultiplexed by the MessageDemux node. + +.. rubric:: Similar samples: + +- :ref:`Multiple Scripts Sync` +- :ref:`Depth and Video Sync` +- :ref:`IMU and Video Sync` + +Demo +#### + + +.. code-block:: + + ~/depthai-python/examples/Sync $ python3 demux_message_group.py + Start + Buffer 1 timestamp: 0:00:03.581073 + Buffer 2 timestamp: 0:00:03.591084 + ---------- + Buffer 1 timestamp: 0:00:04.583100 + Buffer 2 timestamp: 0:00:04.497079 + ---------- + Buffer 1 timestamp: 0:00:06.587174 + Buffer 2 timestamp: 0:00:06.611154 + ---------- + Buffer 1 timestamp: 0:00:07.589147 + Buffer 2 timestamp: 0:00:07.517125 + ---------- + Buffer 1 timestamp: 0:00:09.593076 + Buffer 2 timestamp: 0:00:09.631089 + ---------- + Buffer 1 timestamp: 0:00:10.595106 + Buffer 2 timestamp: 0:00:10.537082 + + +Setup +##### + +.. include:: /includes/install_from_pypi.rst + + +Source code +########### + +.. tabs:: + + .. tab:: Python + + Also `available on GitHub `__ + + .. literalinclude:: ../../../../examples/Sync/demux_message_group.py + :language: python + :linenos: + + .. tab:: C++ + + Also `available on GitHub `__ + + .. literalinclude:: ../../../../depthai-core/examples/Sync/demux_message_group.cpp + :language: cpp + :linenos: + +How it Works +############ + +#. Initialize a DepthAI pipeline. +#. Create two Script nodes, with each script generating and sending data buffers at different intervals. +#. Set up a Sync node with a synchronization threshold. +#. Integrate a MessageDemux node to separate the synchronized data streams. +#. Link the outputs of the Script nodes to the Sync node, and then from the Sync node to the MessageDemux node. +#. Start the pipeline and continuously receive demultiplexed data from the MessageDemux node. +#. Print the timestamps of the demultiplexed data for comparison. + + +.. include:: /includes/footer-short.rst diff --git a/docs/source/samples/Sync/depth_video_sync.rst b/docs/source/samples/Sync/depth_video_sync.rst new file mode 100644 index 000000000..a9b62c4e3 --- /dev/null +++ b/docs/source/samples/Sync/depth_video_sync.rst @@ -0,0 +1,59 @@ +Depth and Video Sync +==================== + +This example demonstrates the use of the DepthAI Sync node to synchronize output from StereoDepth and Color Camera nodes. It showcases how to process and display disparity maps from stereo cameras and video frames from a color camera in real time. + + +.. rubric:: Similar samples: + +- :ref:`IMU and Video Sync` +- :ref:`Multiple Scripts Sync` + + +Demo +#### + +.. image:: ../../../../docs/source/_static/images/examples/depth_video_synced.gif + :alt: Depth and Video Sync Demo + :width: 100% + :align: center + + +Setup +##### + +.. include:: /includes/install_from_pypi.rst + + +Source code +########### + +.. tabs:: + + .. tab:: Python + + Also `available on GitHub `__ + + .. literalinclude:: ../../../../examples/Sync/depth_video_synced.py + :language: python + :linenos: + + .. tab:: C++ + + Also `available on GitHub `__ + + .. literalinclude:: ../../../../depthai-core/examples/Sync/depth_video_synced.cpp + :language: cpp + :linenos: + +How it Works +############ + +#. Initialize MonoCamera nodes for left and right cameras. +#. Set up a ColorCamera node. +#. Create a StereoDepth node for depth perception. +#. Configure the Sync node to synchronize disparity from the StereoDepth node and video frames from the ColorCamera node. +#. Display the synchronized frames using OpenCV. Frames are synchronized to threshold of 50 milliseconds. + + +.. include:: /includes/footer-short.rst diff --git a/docs/source/samples/Sync/imu_video_sync.rst b/docs/source/samples/Sync/imu_video_sync.rst new file mode 100644 index 000000000..ab2dbb1b9 --- /dev/null +++ b/docs/source/samples/Sync/imu_video_sync.rst @@ -0,0 +1,82 @@ +IMU and Video Sync +================== + +This example demonstrates the use of the DepthAI Sync node to synchronize IMU (Inertial Measurement Unit) data with video frames from a color camera. It highlights the capability to process and display the latest rotation vector from the IMU alongside the video stream in real-time. + +.. rubric:: Similar samples: + +- :ref:`Depth and Video Sync` +- :ref:`Multiple Scripts Sync` + +Demo +#### + +.. code-block:: + + ~/depthai-python/examples/Sync $ python3 imu_video_synced.py + IMU type: BNO086, firmware version: 3.9.7 + + Device timestamp imu: 0:00:05.379914 + Device timestamp video:0:00:05.385096 + Quaternion: i: -0.0549 j: -0.0335 k: 0.0018 real: 0.9979 + + + Device timestamp imu: 0:00:05.410274 + Device timestamp video:0:00:05.418425 + Quaternion: i: -0.0549 j: -0.0334 k: 0.0018 real: 0.9979 + + + Device timestamp imu: 0:00:05.445439 + Device timestamp video:0:00:05.451753 + Quaternion: i: -0.0548 j: -0.0334 k: 0.0018 real: 0.9979 + + + Device timestamp imu: 0:00:05.475084 + Device timestamp video:0:00:05.485082 + Quaternion: i: -0.0547 j: -0.0334 k: 0.0018 real: 0.9979 + + + Device timestamp imu: 0:00:05.510046 + Device timestamp video:0:00:05.518411 + Quaternion: i: -0.0546 j: -0.0334 k: 0.0018 real: 0.9979 + + +Setup +##### + +.. include:: /includes/install_from_pypi.rst + + +Source code +########### + +.. tabs:: + + .. tab:: Python + + Also `available on GitHub `__ + + .. literalinclude:: ../../../../examples/Sync/imu_video_synced.py + :language: python + :linenos: + + .. tab:: C++ + + Also `available on GitHub `__ + + .. literalinclude:: ../../../../depthai-core/examples/Sync/imu_video_synced.cpp + :language: cpp + :linenos: + +How it Works +############ + +#. Initialize the DepthAI device. +#. Check the connected IMU type and firmware version. +#. Create a pipeline and add a ColorCamera and IMU node. +#. Set up the Sync node to synchronize the IMU data with the video frames. +#. Link the output of the ColorCamera and IMU nodes to the Sync node. +#. Start the pipeline and continuously receive synchronized data. +#. Display the video frames and print the IMU rotation vector data, including quaternion values. + +.. include:: /includes/footer-short.rst diff --git a/docs/source/samples/Sync/scripts_sync.rst b/docs/source/samples/Sync/scripts_sync.rst new file mode 100644 index 000000000..73236d48b --- /dev/null +++ b/docs/source/samples/Sync/scripts_sync.rst @@ -0,0 +1,74 @@ +Multiple Scripts Sync +===================== + +This example illustrates the use of the DepthAI Sync node to synchronize outputs from two separate script nodes. Each script generates and sends data buffers at different intervals, and the Sync node aligns these outputs based on their timestamps. + +.. rubric:: Similar samples: + +- :ref:`Depth and Video Sync` +- :ref:`IMU and Video Sync` + +Demo +#### + +.. code-block:: + + ~/depthai-python/examples/Sync $ python3 sync_scripts.py + Start + Received s1 with timestamp 0:00:02.420089 + Received s2 with timestamp 0:00:02.461076 + Time interval between messages: 40.987ms + ---------- + Received s1 with timestamp 0:00:03.422108 + Received s2 with timestamp 0:00:03.367069 + Time interval between messages: 55.039ms + ---------- + Received s1 with timestamp 0:00:05.426088 + Received s2 with timestamp 0:00:05.481086 + Time interval between messages: 54.998ms + ---------- + Received s1 with timestamp 0:00:06.428106 + Received s2 with timestamp 0:00:06.387129 + Time interval between messages: 40.977ms + ---------- + +Setup +##### + +.. include:: /includes/install_from_pypi.rst + + +Source code +########### + +.. tabs:: + + .. tab:: Python + + Also `available on GitHub `__ + + .. literalinclude:: ../../../../examples/Sync/sync_scripts.py + :language: python + :linenos: + + .. tab:: C++ + + Also `available on GitHub `__ + + .. literalinclude:: ../../../../depthai-core/examples/Sync/sync_scripts.cpp + :language: cpp + :linenos: + + +How it Works +############ + +#. Initialize a DepthAI pipeline. +#. Create two Script nodes, each generating and sending data buffers at different intervals. +#. Set up a Sync node with a synchronization threshold. +#. Link the outputs of the Script nodes to the Sync node. +#. Start the pipeline and continuously receive synchronized data from the Script nodes. +#. Print the received data along with timestamps and the interval between messages. + + +.. include:: /includes/footer-short.rst diff --git a/docs/source/tutorials/code_samples.rst b/docs/source/tutorials/code_samples.rst index eed30756a..cbb25f600 100644 --- a/docs/source/tutorials/code_samples.rst +++ b/docs/source/tutorials/code_samples.rst @@ -23,6 +23,7 @@ Code Samples ../samples/Script/* ../samples/SpatialDetection/* ../samples/StereoDepth/* + ../samples/Sync/* ../samples/SystemLogger/* ../samples/ToF/* ../samples/VideoEncoder/* @@ -159,6 +160,15 @@ are presented with code. - :ref:`Stereo Depth Video` - An extended version of **Depth Preview** - :ref:`RGB Depth alignment` - Displays RGB depth aligned frames + +.. rubric:: Sync Node + +- :ref:`Depth and Video Sync` - Synchronizes depth and video streams for real-time processing and display. +- :ref:`IMU and Video Sync` - Aligns IMU data with video frames, showcasing real-time rotation vector display alongside video. +- :ref:`Multiple Scripts Sync` - Demonstrates synchronization of data from two script nodes emitting data at different intervals. +- :ref:`Demuxing Synchronized Script Outputs` - Features synchronization of script outputs followed by demultiplexing for distinct processing streams. + + .. rubric:: SystemLogger - :ref:`System information` - Displays device system information (memory/cpu usage, temperature) diff --git a/docs/source/tutorials/message_syncing.rst b/docs/source/tutorials/message_syncing.rst index bf8dceefa..da03b0fd1 100644 --- a/docs/source/tutorials/message_syncing.rst +++ b/docs/source/tutorials/message_syncing.rst @@ -67,6 +67,11 @@ As opposed to sequence number syncing, **timestamp syncing** can sync: - **IMU** results with other messages - messages with **other devices connected to the computer**, as timestamps are :ref:`synced to the host computer clock ` + +.. note:: + DepthAI 2.24 introduces **Sync node** which can be used to sync messages from different streams, or messages from different sensors (eg. IMU and color frames). See :ref:`Sync node ` for more details. + The sync node does not currently support multiple device syncing, so if you want to sync messages from multiple devices, you should use the manual approach. + Feel free to check the `demo here `__ which uses timestamps to sync IMU, color and disparity frames together, with all of these streams producing messages at different FPS. diff --git a/docs/source/tutorials/standalone_mode.rst b/docs/source/tutorials/standalone_mode.rst index df90a7470..680137b13 100644 --- a/docs/source/tutorials/standalone_mode.rst +++ b/docs/source/tutorials/standalone_mode.rst @@ -27,6 +27,9 @@ To "communicate" with the outside world (eg. a server), POE cameras can use :ref - `HTTP client `__ - `MQTT client `__ +.. note:: + Standalone mode is missing a DNS resolver, so you will need to use IP addresses instead of domain names. + Converting a demo to standalone mode ####################################