SDK Operation and Synchronization
The Sync Board supports two communication methods: serial and Ethernet.
- Obtain the current system reference time in the SDK and perform real-time software time alignment with the Sync Board
- Obtain IO pulse trigger moments in real-time, as well as IMU feedback data.
Where:
- The Type-C serial port supports both power supply and communication, making it simple to use. However, limited by the serial communication baud rate, software sync accuracy is lower, and IO trigger moments can currently only be obtained at up to 30Hz.
- The Ethernet port does not support power supply (requires serial power or 12V power supply), but offers high communication speed and high time sync accuracy, with tests reaching up to 100Hz for obtaining IO trigger moments.
In previous documentation, considering testing convenience, we all defaulted to using the serial port for synchronization. However, when hardware supports it, we strongly recommend using the Ethernet port as the communication method to achieve the highest accuracy and operational efficiency.
IMPORTANT !!!!: Once the Sync Board is connected to the Ethernet port in hardware, the serial port function will be disabled until a power cycle. Therefore, if both Ethernet and serial ports are connected simultaneously, only the Ethernet port will be used, and only the Ethernet port SDK can be used.
Different communication methods are selected through the SDK configuration parameters described later.
If you choose to use the Ethernet port for communication, some configuration is needed. First, connect the Sync Board to the host computer with an Ethernet cable and ensure the LED on the Ethernet port is blinking.
After the Ethernet cable is plugged into the Sync Board, power cycle the Sync Board and it will automatically use the Ethernet port to transmit information.
If the Sync Board IP is configured as 192.168.1.188, you can check if the Ethernet port configuration is successful or if the Sync Board is online by pinging:
ping 192.168.1.188Users can use the nc command to view uploaded data. The nc command in Linux is a powerful network tool, short for netcat.
sudo apt-get install netcat # DownloadOpen any terminal and enter the following command:
# Before use, modify the host IP to 192.168.1.xxx. (Any IP on the same subnet is fine)
# 192.168.1.188 8888 represents the Sync Board's IP address and port
echo "Hello InfiniteSense" | nc -u 192.168.1.188 8888 
sudo apt-get install libzmq3-dev # Default version 4.3.4
git clone git@github.com:InfiniteSenseLab/SimpleSensorSync.git -b main
cd SimpleSync
mkdir build && cd build
cmake ..
make -jIf the git clone command fails and you cannot download, you can directly download from the project page via Code/Download ZIP.
Since there are many different sensor synchronization combinations, we provide some basic SDK usage processes and commonly used example references for convenience.
Our SDK provides three main functions:
- Acquire and publish IMU data
- Obtain camera trigger moments
- Acquire camera images and publish them
Therefore, for a VIO system, simply running our SDK provides synchronized image and IMU data. If LiDAR synchronization is needed, you need to additionally run the corresponding manufacturer's LiDAR driver on top of our SDK.
For different cameras and IMUs, different SDKs need to be run. We provide support for mainstream camera models and some IMU models on the market, organized as examples.
Currently, some cameras and IMUs are still not supported and require some C++ secondary development based on the examples we provide. They are all supported at the hardware level.
If your sensor synchronization requirements match those below, you can jump directly to the corresponding section. We will continue updating to provide more solution examples.
Usage Examples:
- Multiple USB plug-and-play cameras and Sync Board IMU forming a VIO system
- MID360/RoboSense LiDAR and multiple USB plug-and-play cameras synchronization
- MID360 and multiple USB plug-and-play cameras synchronization — FAST-LIVO method
- Multiple industrial cameras and Sync Board IMU forming a VIO system
- MID360 and multiple industrial cameras synchronization
- MID360 and multiple industrial cameras synchronization — FAST-LIVO method
- Multiple industrial cameras and pulse-triggered external IMU synchronization
TODO: 8. RoboSense LiDAR and multiple industrial cameras synchronization 9. RoboSense LiDAR and multiple USB plug-and-play cameras synchronization
Synchronization requirements beyond these usage examples are all achievable. Since our SDK is open source and supports secondary development, users need a certain level of development capability. If needed, contact the corresponding technical support. We will also continuously expand our example library while meeting user requirements.
If using ROS1, open the example/VideoCam/main_ros.cpp file; if using ROS2, open the example/VideoCam/main_ros2.cpp file.
At the beginning of the file, the following configuration code is included. The configuration parameters in the file need to be modified as follows:
struct video_config {
bool onboard_imu = true;// Use internal IMU
bool extern_imu = false;// Use external IMU, TODO
std::string imu_name = "imu_1";
// Camera name + Device ID
std::pair<std::string, int> CAM3_Video = {"left_cam", 0}; // 0 represents device /dev/video0
std::pair<std::string, int> CAM4_Video = {"right_cam", -1};
std::pair<std::string, int> CAM1_Video = {"front_cam", -1};
std::pair<std::string, int> CAM2_Video = {"rear_cam", -1}; // -1 represents none
// Communication method
int type = 0; // 0 --> Serial communication 1 --> Ethernet communication
std::string uart_dev = "/dev/ttyACM0";
std::string net_ip = "192.168.1.188";
int net_port = 8888;
};Select based on needs:
- Communication method and parameters
int type = 0; // 0 --> Serial communication 1 --> Ethernet communication
std::string uart_dev = "/dev/ttyACM0"; // Serial device name
std::string net_ip = "192.168.1.188"; // Ethernet address, consistent with configuration parameters
int net_port = 8888; // Ethernet port, consistent with configuration parameters- Connected camera position and device name
// Camera name + Device ID
std::pair<std::string, int> CAM3_Video = {"left_cam", 0}; // 0 represents device /dev/video0, USB camera-specific device name
std::pair<std::string, int> CAM4_Video = {"right_cam", -1}; // -1 represents none
std::pair<std::string, int> CAM1_Video = {"front_cam", -1}; // -1 represents none
std::pair<std::string, int> CAM2_Video = {"rear_cam", -1}; // -1 represents noneAfter modification, recompile and run the SDK:
# Device permissions (only needed when using serial sync)
sudo chmod 777 /dev/ttyACM*
# Enter the build directory
cd SimpleSensorSync/build
# Compile
make -j
# Run ROS1 SDK
./example/VideoCam/video_cam_ros
# Run ROS2 SDK
./example/VideoCam/video_cam_ros2The example in this section shows the most common synchronization method. If you are unsure about the difference between your required sync method and the FAST-LIVO method, use the sync method in this section.
Preparation
- Correctly wire and set the camera trigger frequency to the specified frequency, then observe using the host computer software to ensure correct camera triggering.
- Correctly wire, run the Sync Board's SDK, and after opening the corresponding LiDAR host computer software, observe the GPS sync indicator
Camera Sync
First, refer to 3.1 Multiple USB Plug-and-Play Cameras and Sync Board IMU Forming a VIO System to complete camera synchronization.
If your LiDAR includes IMU data, we recommend using the LiDAR's internal IMU instead of our onboard IMU. The only difference is disabling the internal IMU in the configuration parameters. That is, changing bool onboard_imu = true;// Use internal IMU to false.
Note: After modifying parameters, you need to recompile the SDK and then run the SDK.
# Device permissions (only needed when using serial sync)
sudo chmod 777 /dev/ttyACM*
# Enter the build directory
cd SimpleSensorSync/build
# Compile
make -j
# Run ROS1 SDK
./example/VideoCam/video_cam_ros
# Run ROS2 SDK
./example/VideoCam/video_cam_ros2LiDAR Sync
After running the SDK, the cameras will be correctly synchronized. LiDAR sync also requires this SDK to remain running.
Then, depending on the LiDAR you are using, run the corresponding LiDAR driver normally to achieve correct synchronization of cameras, IMU, and LiDAR.
This synchronization method satisfies most LiDAR synchronization application requirements. The timestamp signal source is consistent, but the data is not acquired at the exact same moment. The special FAST-LIVO synchronization method is described below.
The FAST-LIVO synchronization method is special. It requires that the timestamp signal source is consistent, and that camera and LiDAR data acquired at the same moment are associated. Therefore, the timestamps of LiDAR and camera data will be exactly matched.
Preparation
- Correctly wire and set the camera trigger frequency to 10Hz, then observe using the host computer software to ensure correct camera triggering.
- Correctly wire, run the Sync Board's SDK, and after opening the LiDAR host computer software, observe the GPS sync indicator
LiDAR Driver Execution
The official Livox driver cannot be used directly; you need to use the Livox driver modified by the original author
SDK Modification
If using ROS1, open the example/VideoCam_LVI/main_ros.cpp file; if using ROS2, open the example/VideoCam2_LVI/main_ros2.cpp file.
Refer to 3.2 MID360/RoboSense LiDAR and Multiple USB Plug-and-Play Cameras Synchronization to modify configuration parameters.
Recompile the SDK and run:
# Device permissions (only needed when using serial sync)
sudo chmod 777 /dev/ttyACM*
# Enter the build directory
cd SimpleSensorSync/build
# Compile
make -j
# Run ROS1 SDK
./example/VideoCam_LVI/video_cam_ros
# Run ROS2 SDK
./example/VideoCam_LVI/video_cam_ros2cd build/examples/NetCam
./net_cam
Running the internal topic monitor helps check whether data is being received, whether the data frequency is normal, etc. The monitor can also display the names of internal SDK topics, helping you quickly find the corresponding topics.
cd build/tools
./monitor
Where cam_1_trigger (num: 20) represents the image trigger signal, and cam_1 (num: 20) represents the image frame data. num indicates the number of image frames received per second, i.e., a frequency of 20Hz.
# Create workspace directory
mkdir -p ~/ros2_ws/src
cd ~/ros2_ws/src
# Download code
git clone git@github.com:InfiniteSenseLab/SimpleSync.git
# Return to workspace root
cd ~/ros2_ws
# Build workspace
colcon build
# Source environment variables
source install/setup.bash
# Find the run node
cd install/infinite_sense/lib/net_cam
# Start ROS2 node
./net_cam_ros2
ros2 topic echo /imu_1The Time Sync Board is equipped with the ICM42688P 6-axis IMU, supporting attitude estimation suitable for non-tightly-coupled scheme applications, viewable in rviz2.
sudo apt-get install ros-${ROS_DISTRO}-imu-tools
# Launch rviz2
rviz2
# Subscribe to topic /imu_1 in rviz2
With correct sensor connections and correct SDK execution, sensors will be correctly synchronized. You can verify this using the following methods:
Method 1:
- Set the camera frequency to a low frequency, e.g., 1Hz, and place multiple cameras in parallel
- Use the camera software to open multiple camera streams
- Wave an object in front of the lenses and observe whether the position of the moving object is consistent across the multiple captured images.
Method 2:
- Set the camera frequency to a low frequency, e.g., 1Hz, and place multiple cameras in parallel
- Use the camera software to open multiple camera streams
- Display a rapidly changing clock on a phone or other device, and observe whether the times captured by multiple cameras are consistent
- Connect the cable between the LiDAR and the Sync Board
- Run the SDK
- Open the LiDAR software or host computer to check synchronization status
Taking MID360 as an example, the correctly synchronized status is as follows:
This requires the introduction of time statistics tools such as ROS. Taking ROS1 as an example:
- Record camera and LiDAR timestamps
rosbag record --use-header-time -O lidar_camera /camera_topic /lidar_topic
- View timestamps
rqt_bag lidar_camera.bag
- The LiDAR and camera times are consistent and aligned
Similar to the image below:
To better adapt to different cameras, we support custom cameras. Taking OpenCV reading from a video device as an example, here is an interpretation of writing a custom camera under ROS2.
Create a folder under the example directory, e.g., VideoCam, including the following files:
- CMakeLists.txt
- main_ros2.cpp
- video_cam.h
- video_cam.cpp
Four files in total.
Add the corresponding path to the CMakeLists.txt file under the example directory:
cmake_minimum_required(VERSION 3.10)
project(example)
add_subdirectory(NetCam)
add_subdirectory(VideoCam)
add_subdirectory(CustomCam) # Added path
The video_cam.h file contains the following code, where the overridden functions are the parts we need to implement:
#pragma once
#include "infinite_sense.h"
#include <opencv2/opencv.hpp>
#include <atomic>
#include <thread>
#include <vector>
#include <mutex>
namespace infinite_sense {
// Inherits from the Sensor class, which automatically calls initialization, start, stop, etc. functions
class VideoCam final : public Sensor {
public:
~VideoCam() override;
bool Initialization() override; // Camera initialization function
void Stop() override; // Camera stop acquisition function
void Start() override;// Camera start function
private:
// Specific camera reading implementation
void Receive(void* handle, const std::string&) override;
// OpenCV video device reading function
cv::VideoCapture cap_;
// Initialization flag
std::atomic<bool> is_initialized_{false};
// Multi-threading for reading image devices, one thread per camera
std::mutex cap_mutex_;
std::vector<std::thread> cam_threads;
};
} // namespace infinite_sense
In the video_cam.cpp file, write the following code:
#include "video_cam.h"
#include "infinite_sense.h"
namespace infinite_sense {
// Call stop reading function
VideoCam::~VideoCam() { Stop(); }
// Initialize camera
bool VideoCam::Initialization() {
std::lock_guard<std::mutex> lock(cap_mutex_);
if (!cap_.open(0)) { // Open default camera
// Error on failure to open
std::cerr << "[OpenCVCam] Failed to open camera!" << std::endl;
return false;
}
// Set camera frame rate. Note: the frame rate here needs to be 1. supported by the camera 2. preferably greater than the trigger frame rate to ensure timestamp matching
cap_.set(cv::CAP_PROP_FPS, 30);
is_initialized_ = true;// Initialization flag
std::cout << "[OpenCVCam] Camera initialized." << std::endl;
return true;
}
// Stop reading function
void VideoCam::Stop() {
is_running = false;// Running flag
{
std::lock_guard<std::mutex> lock(cap_mutex_);
if (cap_.isOpened()) {
cap_.release();// Release camera device
std::cout << "[OpenCVCam] Camera released." << std::endl;
}
}
std::this_thread::sleep_for(std::chrono::milliseconds(100));
for (auto& t : cam_threads) {
if (t.joinable()) {
t.join();
}
}
// Close threads
cam_threads.clear();
cam_threads.shrink_to_fit();
}
// Camera reading function
void VideoCam::Receive(void *handle, const std::string &name) {
// Custom camera class, containing 1. image data 2. timestamp 3. camera name
CamData cam_data;
// Unified message management class
Messenger &messenger = Messenger::GetInstance();
// Initialization complete, in running state
while (is_running) {
// 1. Read a frame from the camera
cv::Mat frame_bgr;
{
std::lock_guard<std::mutex> lock(cap_mutex_);
if (!cap_.isOpened() || !cap_.read(frame_bgr)) {
std::cerr << "[OpenCVCam] Failed to read frame!" << std::endl;
std::this_thread::sleep_for(std::chrono::milliseconds(5));
continue;
}
}
// The time_stamp_us here is the camera trigger time. Call GET_LAST_TRIGGER_STATUS(params[name], time) to get the latest trigger time for the device corresponding to name
if (params.find(name) == params.end()) {
LOG(ERROR) << "cam: " << name << " not found!";
} else {
if (uint64_t time; GET_LAST_TRIGGER_STATUS(params[name], time)) {
cam_data.time_stamp_us = time;
} else {
LOG(ERROR) << "Trigger cam: " << name << " not found!";
}
}
// Camera name
cam_data.name = name;
// Camera data, send the raw data directly
cam_data.image = GMat(frame_bgr.rows, frame_bgr.cols,
GMatType<uint8_t, 3>::Type, frame_bgr.data).Clone();
messenger.PubStruct(name, &cam_data, sizeof(cam_data));
std::this_thread::sleep_for(std::chrono::milliseconds{2});
}
}
// Start reading function
void VideoCam::Start() {
if (!is_initialized_) {
std::cerr << "[OpenCVCam] Start failed: not initialized." << std::endl;
return;
}
// Running flag
is_running = true;
// Start a thread and tell it the camera name
std::string name = "video_cam";
cam_threads.emplace_back(&VideoCam::Receive, this, nullptr, name);
std::cout << "[OpenCVCam] Camera capture started." << std::endl;
}
} // namespace infinite_sense
Then call the provided video_camera class:
#include "infinite_sense.h"
#include "video_cam.h"
#include "rclcpp/rclcpp.hpp"
#include <image_transport/image_transport.hpp>
#include <cv_bridge/cv_bridge.h>
#include <sensor_msgs/msg/imu.hpp>
// Inherits from rclcpp::Node
class CamDriver final : public rclcpp::Node {
public:
CamDriver()
: Node("ros2_cam_driver"), node_handle_(std::shared_ptr<CamDriver>(this, [](auto *) {})), transport_(node_handle_) {
// Camera name, ensure consistency with the camera name in video_cam.cpp
std::string camera_name_1 = "video_cam";
// IMU device name, can be customized
const std::string imu_name = "imu_1";
// Use serial device for synchronization. Different devices can be selected here
synchronizer_.SetUsbLink("/dev/ttyACM0", 921600);
// synchronizer_.SetNetLink("192.168.1.188", 8888);
// Construct the VideoCam class
const auto video_cam = std::make_shared<infinite_sense::VideoCam>();
// IMPORTANT!!! Register the camera name video_cam, and bind it to the CAM_1 interface, meaning the device corresponding to video_cam must be connected to CAM_1
video_cam->SetParams({{camera_name_1, infinite_sense::CAM_1}
});
// Synchronizer class adds the camera class
synchronizer_.UseSensor(video_cam);
// Call start functions one by one
synchronizer_.Start();
// IMU and image information publishing functions
imu_pub_ = this->create_publisher<sensor_msgs::msg::Imu>(imu_name, 10);
img1_pub_ = transport_.advertise(camera_name_1, 10);
// Callback functions after SDK obtains IMU and image information
{
using namespace std::placeholders;
infinite_sense::Messenger::GetInstance().SubStruct(imu_name, std::bind(&CamDriver::ImuCallback, this, _1, _2));
infinite_sense::Messenger::GetInstance().SubStruct(camera_name_1,
std::bind(&CamDriver::ImageCallback1, this, _1, _2));
}
}
// Callback function after SDK obtains IMU information
void ImuCallback(const void *msg, size_t) const {
// Publish IMU data in ROS2 format
const auto *imu_data = static_cast<const infinite_sense::ImuData *>(msg);
sensor_msgs::msg::Imu imu_msg;
// The time unit obtained is us, needs conversion to ms
imu_msg.header.stamp = rclcpp::Time(imu_data->time_stamp_us * 1000);
imu_msg.header.frame_id = "map";
imu_msg.linear_acceleration.x = imu_data->a[0];
imu_msg.linear_acceleration.y = imu_data->a[1];
imu_msg.linear_acceleration.z = imu_data->a[2];
imu_msg.angular_velocity.x = imu_data->g[0];
imu_msg.angular_velocity.y = imu_data->g[1];
imu_msg.angular_velocity.z = imu_data->g[2];
imu_msg.orientation.w = imu_data->q[0];
imu_msg.orientation.x = imu_data->q[1];
imu_msg.orientation.y = imu_data->q[2];
imu_msg.orientation.z = imu_data->q[3];
imu_pub_->publish(imu_msg);
}
// Callback function after SDK obtains image information
void ImageCallback1(const void *msg, size_t) const {
// Raw image data
const auto *cam_data = static_cast<const infinite_sense::CamData *>(msg);
std_msgs::msg::Header header;
// The time unit obtained is us, needs conversion to ms
header.stamp = rclcpp::Time(cam_data->time_stamp_us * 1000);
header.frame_id = "map";
// Construct OpenCV image. This is a color image, therefore CV_8UC3 and bgr8
const cv::Mat image_mat(cam_data->image.rows, cam_data->image.cols, CV_8UC3, cam_data->image.data);
const sensor_msgs::msg::Image::SharedPtr image_msg = cv_bridge::CvImage(header, "bgr8", image_mat).toImageMsg();
img1_pub_.publish(image_msg);
}
private:
infinite_sense::Synchronizer synchronizer_;
SharedPtr node_handle_;
image_transport::ImageTransport transport_;
image_transport::Publisher img1_pub_,img2_pub_;
rclcpp::Publisher<sensor_msgs::msg::Imu>::SharedPtr imu_pub_;
};
// main function
int main(const int argc, char *argv[]) {
rclcpp::init(argc, argv);
// Construct the CamDriver class for reading camera, IMU, etc. information
rclcpp::spin(std::make_shared<CamDriver>());
return 0;
}
Finally, the overall CMakeLists.txt file is as follows:
cmake_minimum_required(VERSION 3.16)
project(video_cam VERSION 1.0)
message(STATUS "System Processor: ${CMAKE_SYSTEM_PROCESSOR}")
find_package(Threads REQUIRED)
find_package(OpenCV QUIET)
if(OpenCV_FOUND)
message(STATUS "Found OpenCV version: ${OpenCV_VERSION}")
message(STATUS "OpenCV include dirs: ${OpenCV_INCLUDE_DIRS}")
message(STATUS "OpenCV libraries: ${OpenCV_LIBS}")
# **************************Build Standard Node*****************************
add_executable(${PROJECT_NAME}
main.cpp
video_cam.cpp
video_cam.h
)
target_link_directories(${PROJECT_NAME} PRIVATE )
target_link_libraries(${PROJECT_NAME} PRIVATE
infinite_sense_core
${OpenCV_LIBS}
Threads::Threads
)
target_include_directories(${PROJECT_NAME} PRIVATE ${OpenCV_INCLUDE_DIRS})
set_target_properties(${PROJECT_NAME} PROPERTIES INSTALL_RPATH "$ORIGIN")
# **************************Build ROS2 Node*****************************
find_package(rclcpp QUIET)
if (NOT rclcpp_FOUND)
message(STATUS "rclcpp not found. ROS2 demo will not be built")
else ()
message(STATUS "rclcpp found. ROS2 demo will be built")
find_package(ament_cmake REQUIRED)
find_package(std_msgs REQUIRED)
find_package(sensor_msgs REQUIRED)
find_package(image_transport REQUIRED)
find_package(OpenCV REQUIRED)
find_package(cv_bridge REQUIRED)
add_executable(${PROJECT_NAME}_ros2
main_ros2.cpp
video_cam.cpp
video_cam.h
)
target_link_directories(${PROJECT_NAME}_ros2 PRIVATE ${OpenCV_LIBS})
target_include_directories(${PROJECT_NAME}_ros2 PUBLIC ${OpenCV_INCLUDE_DIRS})
target_link_libraries(${PROJECT_NAME}_ros2
infinite_sense_core
${OpenCV_LIBRARIES}
)
set(CMAKE_BUILD_WITH_INSTALL_RPATH TRUE)
ament_target_dependencies(${PROJECT_NAME}_ros2
rclcpp
std_msgs
sensor_msgs
image_transport
OpenCV
cv_bridge
)
install(TARGETS ${PROJECT_NAME}_ros2
DESTINATION lib/${PROJECT_NAME}
)
install(TARGETS infinite_sense_core
DESTINATION lib/${PROJECT_NAME}
)
ament_package()
endif ()
# **************************Build ROS1 Node*****************************
find_package(catkin QUIET)
if (NOT catkin_FOUND)
message(STATUS "catkin not found, ROS1 demo will not be built")
else ()
message(STATUS "catkin found. ROS1 demo will be built")
find_package(catkin REQUIRED COMPONENTS
roscpp
sensor_msgs
cv_bridge
image_transport
geometry_msgs
)
file(GLOB_RECURSE SOURCE_FILES
main_ros.cpp
video_cam.cpp
video_cam.h
)
add_executable(${PROJECT_NAME}_ros ${SOURCE_FILES})
target_link_directories(${PROJECT_NAME}_ros PRIVATE ${OpenCV_LIBS})
target_link_libraries(${PROJECT_NAME}_ros
infinite_sense_core
Threads::Threads
${catkin_LIBRARIES}
)
target_include_directories(${PROJECT_NAME}_ros PRIVATE
${catkin_INCLUDE_DIRS} ${OpenCV_INCLUDE_DIRS}
)
target_compile_definitions(${PROJECT_NAME}_ros PUBLIC -DROS_PLATFORM)
endif ()
endif()Recompile the entire project to generate executable files for specific camera models, then run the file.