Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
32 changes: 32 additions & 0 deletions docs/source/components/messages/message_group.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
MessageGroup
============

The MessageGroup message type is a versatile container used in DepthAI pipelines to group together a map of arbitrary DepthAI messages. It serves as the primary output of the :ref:`Sync` node, effectively synchronizing various input streams, and acts as the input to the :ref:`Demux` node for subsequent disaggregation and processing.

Creating MessageGroup
#####################

MessageGroup can be created automatically by the Sync node as it aligns and groups messages from different sources based on their timestamps. Alternatively, it can be manually constructed in a host application or within a :ref:`Script` node to create custom groupings of DepthAI messages.

Reference
#########

.. tabs::

.. tab:: Python

.. autoclass:: depthai.MessageGroup
:members:
:inherited-members:
:noindex:

.. tab:: C++

.. doxygenclass:: dai::MessageGroup
:project: depthai-core
:members:
:private-members:
:undoc-members:


.. include:: ../../includes/footer-short.rst
122 changes: 122 additions & 0 deletions docs/source/components/nodes/demux.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
Demux
=====


The Demux (Demultiplexer) node is used to separate a `MessageGroup` into individual outputs. It currently serves as way to demultiplex the output of :ref:`Sync` node.

How to Place It
###############

.. tabs::

.. code-tab:: py

pipeline = dai.Pipeline()
demux = pipeline.create(dai.node.MessageDemux)

.. code-tab:: c++

dai::Pipeline pipeline;
auto demux = pipeline.create<dai::node::MessageDemux>();


Inputs and Outputs
##################

.. code-block::

┌───────────────────┐
input │ │
──────────────►│ │
│ Demux │ output1
│ ├───────────►
│ │ output2
│ ├───────────►
│ │ ...
└───────────────────┘

**Message types**

- :code:`input` - :ref:`MessageGroup`
- :code:`output1`, :code:`output2`, ... - Individual output messages

Usage
#####

The Demux node is particularly useful for handling different types of data coming from a single source.
For example, when the :ref:`Sync` node is used to synchronize the outputs of multiple nodes, the output of the Sync node is a :ref:`MessageGroup` containing all the messages from the synchronized nodes. The Demux node can be used to separate the messages into individual streams.

.. tabs::

.. code-tab:: py

# Create sync node and set sync threshold
sync = pipeline.create(dai.node.Sync)
sync.setSyncThresholdMs(timedelta(milliseconds=100))

# Create demux node
demux = pipeline.create(dai.node.MessageDemux)

# Sync the outputs of multiple nodes
rgb.preview.link(sync.inputs["rgb"])
stereo.depth.link(sync.inputs["depth"])
script.outputs["out"].link(sync.inputs["script"])

sync.out.link(demux.input) # Sync output is a MessageGroup containing all the messages from the synchronized nodes

# Demux the MessageGroup into individual messages
demux.outputs["rgb"].link(xout1.input)
demux.outputs["depth"].link(xout2.input)
demux.outputs["script"].link(xout3.input)




.. code-tab:: c++

// Create sync node and set sync threshold
auto sync = pipeline.create<dai::node::Sync>();
sync->setSyncThreshold(std::chrono::milliseconds(100));

// Create demux node
auto demux = pipeline.create<dai::node::MessageDemux>();

// Sync the outputs of multiple nodes
rgb.preview.link(sync->input["rgb"]);
stereo.depth.link(sync->input["depth"]);
script.outputs["out"].link(sync->input["script"]);

sync->out.link(demux->input); // Sync output is a MessageGroup containing all the messages from the synchronized nodes

// Demux the MessageGroup into individual messages
demux->outputs["rgb"].link(xout1.input);
demux->outputs["depth"].link(xout2.input);
demux->outputs["script"].link(xout3.input);


Examples of Functionality
##########################

- :ref:`Demuxing Synchronized Script Outputs`

Reference
#########

.. tabs::

.. tab:: Python

.. autoclass:: depthai.node.MessageDemux
:members:
:inherited-members:
:noindex:

.. tab:: C++

.. doxygenclass:: dai::node::MessageDemux
:project: depthai-core
:members:
:private-members:
:undoc-members:

.. include:: ../../includes/footer-short.rst
96 changes: 96 additions & 0 deletions docs/source/components/nodes/sync_node.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
Sync
====

The Sync node is used for synchronizing multiple input streams based on their timestamps. It outputs a grouped message containing synchronized frames from the input streams.
The output message is a :ref:`MessageGroup` containing synchronized messages from all the input streams. These can be demultiplexed using the :ref:`Demux` node.

How to Use
##########

.. tabs::

.. code-tab:: py

pipeline = dai.Pipeline()
sync = pipeline.create(dai.node.MessageDemux)

# Configure threshold for timestamp alignment
sync.setSyncThreshold(timedelta(milliseconds=50))

# Configure inputs to be synchronized
sync.inputs["input1"]
sync.inputs["input2"]

# ...

.. code-tab:: c++

dai::Pipeline pipeline;
auto sync = pipeline.create<dai::node::MessageDemux>();

// Configure threshold for timestamp alignment
sync->setSyncThreshold(std::chrono::milliseconds(50));

// Configure inputs to be synchronized
sync->inputs["input1"];
sync->inputs["input2"];

// ...


Inputs and Outputs
##################

.. code-block::

┌───────────────────┐
input1 │ │
──────────────►│ │
input2 │ │ out
──────────────►│ Sync ├───────────►
│ │
... │ │
──────────────►│ │
└───────────────────┘

**Message types**

- :code:`input1`, :code:`input2`, ... - any message type from :ref:`Messages`
- :code:`out` - :ref:`MessageGroup`


Message Synchronization
########################

The Sync node aligns incoming messages based on their timestamps. The synchronization criteria and behavior can be configured using the :code:`depthai.node.Sync.setSyncThreshold` and :code:`depthai.node.Sync.setSyncAttempts` method. More info in the :ref:`API Reference <reference>`.


Examples of Functionality
##########################

- :ref:`Depth and Video Sync`
- :ref:`Multiple Scripts Sync`
- :ref:`IMU and Video Sync`
- :ref:`Demuxing Synchronized Script Outputs`

Reference
#########

.. tabs::

.. tab:: Python

.. autoclass:: depthai.node.Sync
:members:
:inherited-members:
:noindex:

.. tab:: C++

.. doxygenclass:: dai::node::Sync
:project: depthai-core
:members:
:private-members:
:undoc-members:

.. include:: ../../includes/footer-short.rst
78 changes: 78 additions & 0 deletions docs/source/samples/Sync/demux_message_group.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
Demuxing Synchronized Script Outputs
====================================

This example demonstrates the use of the DepthAI Sync node in conjunction with the Demux node to synchronize and then demux outputs from two separate script nodes. Each script node generates data buffers at different intervals, which are first synchronized by the Sync node and then demultiplexed by the MessageDemux node.

.. rubric:: Similar samples:

- :ref:`Multiple Scripts Sync`
- :ref:`Depth and Video Sync`
- :ref:`IMU and Video Sync`

Demo
####


.. code-block::

~/depthai-python/examples/Sync $ python3 demux_message_group.py
Start
Buffer 1 timestamp: 0:00:03.581073
Buffer 2 timestamp: 0:00:03.591084
----------
Buffer 1 timestamp: 0:00:04.583100
Buffer 2 timestamp: 0:00:04.497079
----------
Buffer 1 timestamp: 0:00:06.587174
Buffer 2 timestamp: 0:00:06.611154
----------
Buffer 1 timestamp: 0:00:07.589147
Buffer 2 timestamp: 0:00:07.517125
----------
Buffer 1 timestamp: 0:00:09.593076
Buffer 2 timestamp: 0:00:09.631089
----------
Buffer 1 timestamp: 0:00:10.595106
Buffer 2 timestamp: 0:00:10.537082


Setup
#####

.. include:: /includes/install_from_pypi.rst


Source code
###########

.. tabs::

.. tab:: Python

Also `available on GitHub <https://github.com/luxonis/depthai-python/examples/demux_message_group.py>`__

.. literalinclude:: ../../../../examples/Sync/demux_message_group.py
:language: python
:linenos:

.. tab:: C++

Also `available on GitHub <https://github.com/luxonis/depthai-core/blob/main/examples/Sync/demux_message_group.cpp>`__

.. literalinclude:: ../../../../depthai-core/examples/Sync/demux_message_group.cpp
:language: cpp
:linenos:

How it Works
############

#. Initialize a DepthAI pipeline.
#. Create two Script nodes, with each script generating and sending data buffers at different intervals.
#. Set up a Sync node with a synchronization threshold.
#. Integrate a MessageDemux node to separate the synchronized data streams.
#. Link the outputs of the Script nodes to the Sync node, and then from the Sync node to the MessageDemux node.
#. Start the pipeline and continuously receive demultiplexed data from the MessageDemux node.
#. Print the timestamps of the demultiplexed data for comparison.


.. include:: /includes/footer-short.rst
59 changes: 59 additions & 0 deletions docs/source/samples/Sync/depth_video_sync.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
Depth and Video Sync
====================

This example demonstrates the use of the DepthAI Sync node to synchronize output from StereoDepth and Color Camera nodes. It showcases how to process and display disparity maps from stereo cameras and video frames from a color camera in real time.


.. rubric:: Similar samples:

- :ref:`IMU and Video Sync`
- :ref:`Multiple Scripts Sync`


Demo
####

.. image:: ../../../../docs/source/_static/images/examples/depth_video_synced.gif
:alt: Depth and Video Sync Demo
:width: 100%
:align: center


Setup
#####

.. include:: /includes/install_from_pypi.rst


Source code
###########

.. tabs::

.. tab:: Python

Also `available on GitHub <https://github.com/luxonis/depthai-python/blob/main/examples/depth_video_synced.py>`__

.. literalinclude:: ../../../../examples/Sync/depth_video_synced.py
:language: python
:linenos:

.. tab:: C++

Also `available on GitHub <https://github.com/luxonis/depthai-core/blob/main/examples/Sync/depth_video_synced.cpp>`__

.. literalinclude:: ../../../../depthai-core/examples/Sync/depth_video_synced.cpp
:language: cpp
:linenos:

How it Works
############

#. Initialize MonoCamera nodes for left and right cameras.
#. Set up a ColorCamera node.
#. Create a StereoDepth node for depth perception.
#. Configure the Sync node to synchronize disparity from the StereoDepth node and video frames from the ColorCamera node.
#. Display the synchronized frames using OpenCV. Frames are synchronized to threshold of 50 milliseconds.


.. include:: /includes/footer-short.rst
Loading