Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

robot_localization high cpu usage and no output #329

Closed
natejgardner opened this issue Dec 15, 2016 · 8 comments
Closed

robot_localization high cpu usage and no output #329

natejgardner opened this issue Dec 15, 2016 · 8 comments

Comments

@natejgardner
Copy link

I have an IMU (orientation only) and rtabmap connected to robot_localization ukf. When I include the pose from the IMU and the odom from rtabmap, robot_localization works fine and uses 1-2% CPU. However, when I add the twist messages from the IMU, robot_localization only produces about 10 messages before ceasing to have any output. When I watch it in htop, it uses 100-105% CPU. I assume that whatever it's doing is blocking indicated by no messages being published and ctrl-c failing to work. I examined the twist data going into robot_localization from the IMU and they look fine. They're coming in at about 375 Hz like the pose messages. rtabmap is coming in at about 10Hz with its odom message. Transforms have been applied, but I'm not sure if they are correct in the way they relate the rtabmap reference frame with the IMU frame. Any idea why this would be happening?

@ayrton04
Copy link
Collaborator

Do you have a question for this on ROS answers? If so, can you direct me to it? I've been neglecting it lately, sorry. In any case, this typically points to an issue with one of your covariance matrices. If the filter's estimate error covariance becomes singular (via bad measurement covariances or bad covariance parameters), I find the filter slows to a crawl and stops producing data. Can you post a sample IMU message and your configuration?

@natejgardner
Copy link
Author

natejgardner commented Dec 15, 2016 via email

@ayrton04
Copy link
Collaborator

Can you just paste your entire config file for r_l along with the sample IMU messages when you get them? Thanks.

@natejgardner
Copy link
Author

Here is my launch file:

<launch>

  <node name="odom_transform" pkg="tf" type="static_transform_publisher"
      args="0 0 0 0.506 0.52 0.08 camera_link base_link 100" />

  <node pkg="covariance_applicator" 
        type="covariance_applicator_node" 
        name="covariance_applicator"
        clear_params="true"
        output="screen">
        <param name="theta" value="0.0000000003302221698534690" />
        <param name="phi" value="0.0000000003302221698534690" />
        <param name="psi" value="0.0000000003302221698534690" />
        <param name="dtheta" value="0.0000000173106564426725" />
        <param name="dphi" value="0.0000000173106564426725" />
        <param name="dpsi" value="0.0000000173106564426725" />
  </node>
        
  <node pkg="robot_localization" 
        type="ukf_localization_node" 
        name="ukf_localization"
        clear_params="true"
        output="screen">
        <param name="frequency" value="30" />
        <param name="sensor_timeout" value="1" />
        <param name="map_frame" value="map" />
        <param name="odom_frame" value="odom" />
        <param name="base_link_frame" value="base_link" />
        <param name="world_frame" value="map" />
        <param name="transform_time_offset" value="0.0" />

        <param name="pose0" value="/covariance_applicator/pose_with_generated_covariance" />
        <param name="twist0" value="/covariance_applicator/twist_with_generated_covariance" />
        <param name="odom0" value="/rtabmap/odom" />

        <rosparam param="pose0_config">
          [false, false, false,
           true, true, true,
           false, false, false,
           false, false, false,
           false, false, false]</rosparam>

        <rosparam param="twist0_config">
          [false, false, false,
           false, false, false,
           false, false, false,
           true, true, true,
           false, false, false]</rosparam>
          
        <rosparam param="odom0_config">
         [true, true, true,
          true, true, true,
          true, true, true,
          true, true, true,
          false, false, false]</rosparam>
        
        <param name="pose0_differential" value="false" />
        <param name="odom0_differential" value="false" />
        <param name="twist0_differential" value="false" />
        <param name="pose0_relative" value="true" />
        <param name="odom0_relative" value="true" />
        <param name="twist0_relative" value="true" />
        <param name="print_diagnostics" value="true" />

        <param name="pose0_queue_size" value="500" />
        <param name="odom0_queue_size" value="500" />
        <param name="twist0_queue_size" value="500" />
  </node>
  

</launch>

Here are a few IMU pose messages:

rostopic echo /covariance_applicator/pose_with_generated_covariance
header: 
  seq: 30371
  stamp: 
    secs: 1481863645
    nsecs: 202669101
  frame_id: map
pose: 
  pose: 
    position: 
      x: 0.0
      y: 0.0
      z: 0.0
    orientation: 
      x: -0.385437011719
      y: -0.616821289062
      z: 0.430236816406
      w: 0.53466796875
  covariance: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.30222169853469e-10, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.30222169853469e-10, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.30222169853469e-10]
---
header: 
  seq: 30372
  stamp: 
    secs: 1481863645
    nsecs: 202743884
  frame_id: map
pose: 
  pose: 
    position: 
      x: 0.0
      y: 0.0
      z: 0.0
    orientation: 
      x: -0.386596679688
      y: -0.615661621094
      z: 0.429260253906
      w: 0.536010742188
  covariance: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.30222169853469e-10, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.30222169853469e-10, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.30222169853469e-10]
---
header: 
  seq: 30373
  stamp: 
    secs: 1481863645
    nsecs: 202784921
  frame_id: map
pose: 
  pose: 
    position: 
      x: 0.0
      y: 0.0
      z: 0.0
    orientation: 
      x: -0.387817382812
      y: -0.614440917969
      z: 0.42822265625
      w: 0.537353515625
  covariance: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.30222169853469e-10, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.30222169853469e-10, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.30222169853469e-10]

Here are a few IMU twist messages:

rostopic echo /covariance_applicator/twist_with_generated_covariance
header: 
  seq: 16668
  stamp: 
    secs: 1481863610
    nsecs: 502735699
  frame_id: map
twist: 
  twist: 
    linear: 
      x: 0.0
      y: 0.0
      z: 0.0
    angular: 
      x: 0.000410602829054
      y: -0.00207957168262
      z: 0.00726565388067
  covariance: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.73106564426725e-08, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.73106564426725e-08, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.73106564426725e-08]
---
header: 
  seq: 16669
  stamp: 
    secs: 1481863610
    nsecs: 502780801
  frame_id: map
twist: 
  twist: 
    linear: 
      x: 0.0
      y: 0.0
      z: 0.0
    angular: 
      x: 6.54603515098e-05
      y: -0.0015071277041
      z: 0.00585367047805
  covariance: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.73106564426725e-08, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.73106564426725e-08, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.73106564426725e-08]
---
header: 
  seq: 16670
  stamp: 
    secs: 1481863610
    nsecs: 502820408
  frame_id: map
twist: 
  twist: 
    linear: 
      x: 0.0
      y: 0.0
      z: 0.0
    angular: 
      x: -0.000711396528358
      y: -0.00105999786758
      z: 0.00497927153915
  covariance: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.73106564426725e-08, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.73106564426725e-08, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.73106564426725e-08]
---
header: 
  seq: 16671
  stamp: 
    secs: 1481863610
    nsecs: 502861026
  frame_id: map
twist: 
  twist: 
    linear: 
      x: 0.0
      y: 0.0
      z: 0.0
    angular: 
      x: -0.000749539231974
      y: -0.000682462728371
      z: 0.00391130287678
  covariance: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.73106564426725e-08, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.73106564426725e-08, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.73106564426725e-08]

Here are a few rtabmap odom messages:

rostopic echo /rtabmap/odom
header: 
  seq: 39905
  stamp: 
    secs: 1481863744
    nsecs: 235916534
  frame_id: odom
child_frame_id: camera_link
pose: 
  pose: 
    position: 
      x: 0.0390180945396
      y: 0.00428201491013
      z: -0.0198945067823
    orientation: 
      x: -0.0146936458139
      y: 0.0447869416632
      z: -0.0325307512123
      w: 0.998358638616
  covariance: [0.00019712561334017664, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.00019712561334017664, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.00019712561334017664, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.00019712561334017664, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.00019712561334017664, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.00019712561334017664]
twist: 
  twist: 
    linear: 
      x: 0.0441059432924
      y: 0.0754606723785
      z: 0.0191122014076
    angular: 
      x: -0.0319370143116
      y: 0.0358424596488
      z: 0.123539820313
  covariance: [9.856280667008832e-05, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 9.856280667008832e-05, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 9.856280667008832e-05, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 9.856280667008832e-05, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 9.856280667008832e-05, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 9.856280667008832e-05]
---
header: 
  seq: 39906
  stamp: 
    secs: 1481863744
    nsecs: 335520603
  frame_id: odom
child_frame_id: camera_link
pose: 
  pose: 
    position: 
      x: 0.029379889369
      y: -0.00234850170091
      z: -0.0415699779987
    orientation: 
      x: -0.0114554956508
      y: 0.0317452560334
      z: -0.0328300864379
      w: 0.998891014354
  covariance: [0.0002865927235689014, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0002865927235689014, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0002865927235689014, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0002865927235689014, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0002865927235689014, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0002865927235689014]
twist: 
  twist: 
    linear: 
      x: -0.0725080370903
      y: -0.0655384361744
      z: -0.227154746652
    angular: 
      x: 0.073962919414
      y: -0.259697675705
      z: -0.00754591543227
  covariance: [0.0001432963617844507, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0001432963617844507, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0001432963617844507, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0001432963617844507, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0001432963617844507, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0001432963617844507]
---
header: 
  seq: 39907
  stamp: 
    secs: 1481863744
    nsecs: 434483313
  frame_id: odom
child_frame_id: camera_link
pose: 
  pose: 
    position: 
      x: 0.026605527848
      y: -0.00220267544501
      z: -0.0453459247947
    orientation: 
      x: -0.00664620087931
      y: 0.0325705992017
      z: -0.033428701615
      w: 0.998888135233
  covariance: [0.00016321238945238292, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.00016321238945238292, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.00016321238945238292, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.00016321238945238292, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.00016321238945238292, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.00016321238945238292]
twist: 
  twist: 
    linear: 
      x: -0.0256240703166
      y: 0.000604405649938
      z: -0.0398366712034
    angular: 
      x: 0.0969137921929
      y: 0.0200346689671
      z: -0.00871383678168
  covariance: [8.160619472619146e-05, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 8.160619472619146e-05, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 8.160619472619146e-05, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 8.160619472619146e-05, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 8.160619472619146e-05, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 8.160619472619146e-05]

Here is the entire output of r_l before it stops publishing:

rostopic echo /odometry/filtered 
header: 
  seq: 0
  stamp: 
    secs: 1481863964
    nsecs: 519217491
  frame_id: map
child_frame_id: base_link
pose: 
  pose: 
    position: 
      x: -0.00118760676038
      y: -0.00221552931645
      z: 0.00608548355159
    orientation: 
      x: -0.00136972757593
      y: 0.00428033626892
      z: 0.00186730468976
      w: 0.9999881578
  covariance: [5.4222779588477144e-05, -1.6868290478265478e-29, 7.439130414337916e-30, 2.1780532450033566e-30, -1.938588312620143e-29, 2.2555819530257642e-29, -1.686829047785884e-29, 5.4222779588477144e-05, -6.1535716888842054e-30, 1.4561216566545772e-28, -5.004895760682738e-29, 2.976812461928287e-29, 6.66187182856474e-30, -6.153571688884205e-30, 5.4331045955375266e-05, 4.588683648936677e-29, -1.0037483577276516e-30, -1.9485394273572768e-29, -4.002183763332119e-30, 1.5487481739617264e-28, 4.5883754977945867e-29, 5.3796595556091474e-05, -5.83371094414702e-29, 1.118385423838971e-28, -1.938588314971132e-29, -5.931777236308079e-29, -1.780003060344963e-30, -5.83371094414702e-29, 5.3796595556091474e-05, -4.607106566718901e-29, 2.1775479433063565e-29, 2.9761961617598345e-29, -2.571482901408449e-29, 1.1802392791364831e-28, -4.6071065667189005e-29, 5.433104595537353e-05]
twist: 
  twist: 
    linear: 
      x: -1.30596995364e-06
      y: -2.43634226206e-06
      z: 6.68306149413e-06
    angular: 
      x: -2.99736587129e-06
      y: 9.42691545745e-06
      z: 4.07024052049e-06
  covariance: [0.0016777823479639967, -8.894572917666133e-37, -8.698658031704616e-37, 1.4220053905091253e-45, -1.265606926173083e-44, 7.414384684703603e-45, -8.894572917666133e-37, 0.0016777823479639967, -1.0461899574372401e-36, 9.505963402476968e-44, -3.2673746176814375e-44, 9.785647096995182e-45, -8.698658034265843e-37, -1.0461899574372401e-36, 0.002684451154040047, 2.499975825608169e-44, -5.4722617869151315e-46, -5.3468401495256835e-45, -2.610607093949694e-45, 1.0110855274905928e-43, 2.4999758256081685e-44, 0.0006711135373840257, -6.29629860938646e-44, 6.079609981515371e-44, -1.2656069261730828e-44, -3.8722664902418975e-44, -9.684596050133412e-46, -6.296298609386458e-44, 0.0006711135373840257, -2.5041523244692314e-44, 7.158837526697871e-45, 9.785647096221691e-45, -7.055231507320724e-45, 6.415839547035372e-44, -2.5041523244692314e-44, 0.00134222607476806]
---
header: 
  seq: 1
  stamp: 
    secs: 1481863964
    nsecs: 685341120
  frame_id: map
child_frame_id: base_link
pose: 
  pose: 
    position: 
      x: -0.000328303095486
      y: -0.00405944864023
      z: 0.00831629624722
    orientation: 
      x: -0.00177834449936
      y: 0.00575986233378
      z: 0.00294158872596
      w: 0.999977504013
  covariance: [4.198104728050066e-05, 6.615418728299239e-16, 2.0490352662247204e-13, -5.559198662865039e-25, -9.449802356584182e-25, -1.1788658019390678e-24, 6.615418727976121e-16, 4.198104727865665e-05, 6.599358203171804e-14, 1.0340759002913375e-23, 1.9526329896917108e-23, 7.664422307794503e-24, 2.0490352662247204e-13, 6.599358203254522e-14, 4.200740774890077e-05, -1.0921744938415632e-23, -1.9897349504226274e-23, -5.216145569687941e-24, -5.535795519077324e-25, 1.0340758969166712e-23, -1.0915326500101388e-23, 4.1876279739577575e-05, 1.3610871577371871e-15, 2.522134032931778e-13, -9.385131180346818e-25, 1.952633864523477e-23, -1.989573012016568e-23, 1.3610871577662677e-15, 4.187627973585399e-05, 8.124020287308249e-14, -1.182080493639571e-24, 7.658791259702197e-24, -5.216141987586049e-24, 2.5221340329483217e-13, 8.124020287349608e-14, 4.2007380755630305e-05]
twist: 
  twist: 
    linear: 
      x: -0.0262780938663
      y: -0.0453231487448
      z: 0.117106306662
    angular: 
      x: -0.0284828844293
      y: 0.151705849613
      z: 0.0575076826239
  covariance: [0.0016951597397805353, 1.0922074358402892e-15, 7.64538903503356e-15, 2.9940762920356505e-24, -1.8831227842929853e-22, 6.562529429671821e-25, 1.0922074358402778e-15, 0.0016951597397769033, -1.0501152316128596e-14, 4.975407949249108e-25, -3.1288907073229983e-23, 1.0906162530782703e-25, 7.645389035033295e-15, -1.0501152316128609e-14, 0.0027034258895126686, -3.1175162443402456e-24, 1.960343962740558e-22, -6.828429926536926e-25, 2.994994140553549e-24, 4.975407949225095e-25, -3.117236951828786e-24, 0.0006863419042537942, 2.281450470632591e-15, -1.3684808497232686e-13, -1.883147260251118e-22, -3.129013087113633e-23, 1.9603439627400334e-22, 2.281450470635279e-15, 0.0006863419042463856, -1.0566756068404783e-13, 6.562529429042224e-25, 1.0906162518084885e-25, -6.831399635176096e-25, -1.3684808497232681e-13, -1.0566756068405043e-13, 0.0013583201043499756]

@natejgardner
Copy link
Author

We found the problem. It had nothing to do with r_l. We were inadvertently publishing bad data on our twist topic. The odom output of r_l was accidentally being routed into the twist message... ROS took the odom data type and evidently interpreted it as a twist message. Yes, our covariance matrix is technically singular, but I believe that's okay due to the values in the boolean matrix. In any case, it now works, and it's totally not a problem with r_l.

@ayrton04
Copy link
Collaborator

Cool. One thing you should fix, though, is that your IMU twist data has a frame_id of map. That should be changed to base_link (assuming the IMU isn't mounted with none-zero roll or pitch).

@ayrton04
Copy link
Collaborator

Two other things:

  1. You have relative mode turned on for twist data. Relative mode is only used for pose variables. It doesn't hurt anything as you have it; it's just being ignored.

  2. You are fusing orientation from both rtabmap and your IMU. You may see some jitter as it jumps back and forth between their orientation estimates. I'd turn on differential mode for whichever one provides the poorer orientation estimate.

Can you please update your ROS answers question?

@natejgardner
Copy link
Author

ROS Answers is closed and links back here. Thanks for noticing those. We caught the frame issue and switched it to base_link and it works much better now. We are noticing the jitter, but rtabmap is jittery anyway. The orientation ends up looking very stable, but due to the lack of translational data from the IMU, position jitters a lot. This is probably partially due to the transform closely linking the IMU to the map. Thanks so much for your help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants