Depth Anything 3

This guide explains how auv-software uses Depth-Anything-3 in two different environments.

• Local development uses Depth-Anything-3/zmq_server.py and the ROS client in auv_vision.
• Jetson deployment uses the TensorRT ROS node in auv_vision.
• Both paths publish the same ROS outputs so downstream nodes can stay the same.

What auv-software Expects

Both DA3 front-ends publish the same core outputs:

• depth_anything/raw_depth
• depth_anything/scaled_camera_info

Both can also publish an optional colored point cloud:

• depth_anything/rgb_point_cloud as sensor_msgs/PointCloud2

Both also expect the same camera inputs:

• Input image topic: remapped from image_raw
• Camera info topic: ${camera_namespace}/camera_info

Important details:

• depth_anything/raw_depth is published as a sensor_msgs/Image with 32FC1 depth values.
• The ZMQ server itself returns DA3 raw output. Metric scaling happens inside auv-software after the node scales the camera intrinsics to the DA3 output size.
• Always use depth_anything/scaled_camera_info together with the depth image. DA3 resizes the image before inference, so the original camera intrinsics no longer match the published depth resolution.
• If you are going to match the depth map with an object detection (bbox, segmentation, etc.) don't forget depth map's camera info is different from the original image. You need to scale one of them, check the object_plane_fitter_node.py for an example

Local Development: ZMQ Server + ROS Client

1. Set up Depth-Anything-3

Use a separate Python environment for the DA3 repo.

cd ~/catkin_ws/src/Depth-Anything-3
uv venv .venv
source .venv/bin/activate
uv pip install "torch>=2" torchvision xformers pyzmq
uv pip install -e .

2. Start the DA3 ZMQ server

cd ~/catkin_ws/src/Depth-Anything-3
source .venv/bin/activate
python zmq_server.py --model da3metric-large --device cuda --port 5555

3. Start the ROS ZMQ client from auv-software

roslaunch auv_vision depth_anything_zmq.launch

Enable the optional RGB point cloud only when needed:

roslaunch auv_vision depth_anything_zmq.launch publish_rgb_point_cloud:=true

4. Verify the DA3 outputs

rostopic hz /taluy/depth_anything/raw_depth
rostopic echo -n 1 /taluy/depth_anything/scaled_camera_info

If scaled_camera_info never appears, check the camera namespace and make sure the camera publishes ${camera_namespace}/camera_info.

Jetson: TensorRT ROS Node

On Jetson, we do not use the external ZMQ server/client split. We run the TensorRT node directly inside auv-software.

TODO: write about how to export trt engine

1. Prepare the TensorRT environment

The ROS Python environment on the Jetson must be able to import:

• tensorrt
• pycuda

The node also needs a serialized TensorRT engine file.

Important details:

• The node preprocesses every frame to 644x476.
• The default launch file points to $(find auv_detection)/models/DA3METRIC-LARGE_476x644_sim.engine.

2. Start the TensorRT node

roslaunch auv_vision depth_anything_trt.launch

Enable the optional RGB point cloud only when needed:

roslaunch auv_vision depth_anything_trt.launch publish_rgb_point_cloud:=true

Using DA3 with object_plane_fitter

object_plane_fitter is the main downstream consumer of the DA3 depth output in this repo.

Its launch file expects:

• depth_topic:=depth_anything/raw_depth
• scaled_camera_info_topic:=depth_anything/scaled_camera_info
• original_camera_info_topic:=cameras/cam_torpedo/camera_info
• detections_topic:=/yolo_detections_torpedo

auv_navigation/auv_mapping/launch/start.launch already includes object_plane_fitter.launch.

The node starts disabled by default. Enable it when you actually want pose updates:

rosservice call /taluy/object_plane_fitter/enable "data: true"

Why the Sim Sync Queue Is Bigger

object_plane_fitter.launch uses different ApproximateTime sync queue sizes:

• sync_queue_size_sim:=90
• sync_queue_size_real:=30

This bigger sim queue exists because DA3 usually runs slower on development computers than the rest of the sim pipeline. A larger queue gives the synchronizer more buffered depth and detection messages, which makes it easier to still match one pair of messages when DA3 lags behind.

If the sync queue is too small in sim:

• Depth and detection messages may fail to synchronize
• object_plane_fitter may produce few or no updates even though both topics exist

Do not set the queue arbitrarily high:

• A larger queue keeps more depth images and detection messages in memory
• Depth images are large, so RAM usage can climb quickly if the queue is much bigger than necessary

For faster setups, especially Jetson with TensorRT, the smaller real-hardware queue is usually enough.

Troubleshooting

• No depth output from the ZMQ path: check the zmq_server.py terminal first.
• ZMQ client connects but no scaled_camera_info: make sure ${camera_namespace}/camera_info exists.
• Depth exists but plane fitter does nothing: check the enable service and the sync queue size.
• Plane fitter is running in sim but rarely updates: increase sync_queue_size_sim.
• Jetson TensorRT node fails on startup: verify engine_path, tensorrt, and pycuda in the ROS Python environment.