Using Airsim stereo camera image with stereo_image_proc (ros)

[ryoma310]

Question

What's your question?

I try to use stereo_image_proc to depth estimation from images obtained by airsim_ros_pkg.
It seems to work well as I see disparity image.
But when I check the point cloud, it shows only one point.

Include context on what you are trying to achieve

Context details

This is my setting.json

{
  "SeeDocsAt": "https://github.com/Microsoft/AirSim/blob/master/docs/settings.md",
  "SettingsVersion": 1.2,
  "SimMode": "Multirotor",
  "RecordUIVisible": true,
  "LogMessagesVisible": true,
  "ViewMode": "",
  "RpcEnabled": true,
  "PhysicsEngineName": "",
  "EnableCollisionPassthrogh": false,
  "SubWindows": [
    {"WindowID": 0, "CameraName": "front_left", "ImageType": 0, "VehicleName": "", "Visible": false, "External": false},
    {"WindowID": 1, "CameraName": "front_center", "ImageType": 0, "VehicleName": "", "Visible": false, "External": false},
    {"WindowID": 2, "CameraName": "front_right", "ImageType": 0, "VehicleName": "", "Visible": false, "External": false}
  ],
  "CameraDefaults": {
    "CaptureSettings": [
      {
        "ImageType": 0,
        "Width": 1936,
        "Height": 1296
      }
    ]
  },
  "SimMode": "Multirotor",
    "ClockType": "SteppableClock",
    "Vehicles": {
      "PX4": {
        "VehicleType": "PX4Multirotor",
        "AllowAPIAlways": true,
        "UseSerial": false,
        "LockStep": true,
        "UseTcp": true,
        "TcpPort": 4560,
        "ControlPortLocal": 14540,
        "ControlPortRemote": 14580,
        "Sensors":{
          "Barometer":{
              "SensorType": 1,
              "Enabled": true,
              "PressureFactorSigma": 0.0001825
          }
        },
        "Parameters": {
          "NAV_RCL_ACT": 0,
          "NAV_DLL_ACT": 0,
          "COM_OBL_ACT": 1,
          "COM_RCL_EXCEPT": 4,
          "LPE_LAT": 47.641468,
          "LPE_LON": -122.140165
        },
        "Cameras": {
          "front_center_custom": {
            "CaptureSettings": [
              {
                "PublishToRos": 1,
                "ImageType": 0,
                "FOV_Degrees": 27
              }
            ],
            "X": 0.50, "Y": 0, "Z": 0.10,
            "Pitch": 0, "Roll": 0, "Yaw": 0
          },
          "front_left_custom": {
            "CaptureSettings": [
              {
                "PublishToRos": 1,
                "ImageType": 0,
                "FOV_Degrees": 90,
                "Width": 1280,
                "Height": 720
              }
            ],
            "X": 0.50, "Y": -0.06, "Z": 0.10,
            "Pitch": 0.0, "Roll": 0.0, "Yaw": 0.0
          },
          "front_right_custom": {
            "CaptureSettings": [
              {
                "PublishToRos": 1,
                "ImageType": 0,
                "FOV_Degrees": 90,
                "Width": 1280,
                "Height": 720
              }
            ],
            "X": 0.50, "Y": 0.06, "Z": 0.10,
            "Pitch": 0.0, "Roll": 0.0, "Yaw": 0.0
          }
        },
        "X": 0, "Y": 0, "Z": 0,
        "Pitch": 0, "Roll": 0, "Yaw": 0
      }
    }
}

• Airsim version: v1.8.1 - Linux
• ROS: noetic
• stereo_image_proc launch file

<launch>
  <node name="transform" pkg="topic_tools" type="transform"
      args="/stereo/disparity /stereo/disparity_image sensor_msgs/Image 'm.image'" />

  <!-- Launch stereo_image_proc node which runs OpenCV's block matching  -->
  <node ns="stereo" pkg="stereo_image_proc" type="stereo_image_proc" name="stereo_image_proc">
    <param name="stereo_algorithm" type="double" value="1.0" />
    <param name="correlation_window_size" type="double" value="19.0" />
    <param name="disparity_range" type="double" value="32.0" />
    <param name="uniqueness_ratio" type="double" value="40.0" />
    <param name="speckle_size" type="double" value="1000.0" />
    <param name="speckle_range" type="double" value="2.0" />
    <param name="approximate_sync" value="true" />
  </node>

  <node ns="/" pkg="topic_tools" type="relay" name="left_scene" args="/airsim_node/PX4/front_left_custom/Scene /stereo/left/image_raw" />
  <node ns="/" pkg="topic_tools" type="relay" name="left_scene_info" args="/airsim_node/PX4/front_left_custom/Scene/camera_info /stereo/left/camera_info" />
  <node ns="/" pkg="topic_tools" type="relay" name="right_scene" args="/airsim_node/PX4/front_right_custom/Scene /stereo/right/image_raw" />
  <node ns="/" pkg="topic_tools" type="relay" name="right_scene_info" args="/airsim_node/PX4/front_right_custom/Scene/camera_info /stereo/right/camera_info" />


  <!-- Launch static transform publishers --> 
  <!--
  <node pkg="tf" type="static_transform_publisher" name="tf_front_camera"
        args="0 0 0 -1.57 0 -1.57 fcu front_camera_link 10"/>
  <node pkg="tf" type="static_transform_publisher" name="tf_right_camera"
        args="0 0 0 -3.14 0 -1.57 fcu right_camera_link 10"/>
  <node pkg="tf" type="static_transform_publisher" name="tf_left_camera"
        args="0 0 0 0 0 -1.57 fcu left_camera_link 10"/> -->

</launch>

---

• RVis
  

---

• camera info output

$ rostopic echo /stereo/right/camera_info 
header: 
  seq: 531
  stamp: 
    secs: 1658384189
    nsecs: 637780428
  frame_id: "front_right_custom_optical"
height: 720
width: 1280
distortion_model: ''
D: []
K: [640.0, 0.0, 640.0, 0.0, 640.0, 360.0, 0.0, 0.0, 1.0]
R: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
P: [640.0, 0.0, 640.0, 0.0, 0.0, 640.0, 360.0, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi: 
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: False
---
$ rostopic echo /stereo/left/camera_info 
header: 
  seq: 536
  stamp: 
    secs: 1658384201
    nsecs: 632036448
  frame_id: "front_left_custom_optical"
height: 720
width: 1280
distortion_model: ''
D: []
K: [640.0, 0.0, 640.0, 0.0, 640.0, 360.0, 0.0, 0.0, 1.0]
R: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
P: [640.0, 0.0, 640.0, 0.0, 0.0, 640.0, 360.0, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi: 
  x_offset: 0
  y_offset: 0
  height: 0
  width: 0
  do_rectify: False
---

Include details of what you already did to find answers

I search about it and found that camera_info may have same error.
I checked the topic detail and find that the projection matrix of left and right camera are same.
(because of this, stereo_image_proc node cannot calculate the depth.)

---

• If you have some idea, please let me know.

[ryoma310]

I resolved it by myself!

[CWC107753035]

For those who struggling on this question I find out the problem.

Because in airsim ros-node it doesn't provide baseline in camera_info. So if you want to get the point cloud through disparity image, you need to provide the Tx parameter manually in one of projection matrix in camera_info.
see also sensor_msgs/CameraInfo.msg for more information. Cheers.

[ryoma310]

@CWC107753035
That's exactly what I did to solve this issue.
Thanks.

[kanishkanarch]

@ryoma310, how did you arm and takeoff this drone? I cannot see any service or topic to do so:

[CWC107753035]

I think you can use ros service call to takeoff. Maybe /airsim_node /PX4/takeoff ? Kanishk Vishwakarma ***@***.***>於 2023年6月23日 週五，15:32寫道：

…

@ryoma310 <https://github.com/ryoma310>, how did you arm and takeoff this drone? I cannot see any service or topic to do so: [image: image] <https://user-images.githubusercontent.com/52484751/248202631-0cf0d0cb-c25e-42bf-b3d6-8c2b6f0778a0.png> [image: image] <https://user-images.githubusercontent.com/52484751/248202524-b9edf858-2c3a-4083-87db-1683e9249be8.png> — Reply to this email directly, view it on GitHub <#4634 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOHFC7R7YTRGWLMOICTKWI3XMVBCXANCNFSM54GP3TZQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[kanishkanarch]

I am running a rosservice call /airsim_node/PX4/takeoff "waitOnLastTask: false" but drone is not taking off (not even arming).

[kanishkanarch]

There also seems to be no option to set the state (OFFBOARD/ALTITUDE/POSITION etc.)

[CWC107753035]

According what you said, i think your mavlink doesn't connect to your airsim. Does your QGC can connect to your airsim? If so , maybe you can try to takeoff by using QGC GUI?

[kanishkanarch]

I am new to PX4, I did not setup PX4 sitl for AirSim. Now I did and it's working perfectly fine!!
Thank you very much!