Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix invalid timer handle exception #474

Merged

Conversation

z80020100
Copy link
Contributor

  • This PR try to fix invalid timer handle exception caused by removing handle repeatedly.
  • This issue is found when using Nav2 and set a small transform_tolerance for local_costmap and global_costmap.
  • Specific error messages are as follows:
    [server-9] [planner_server-4] terminate called after throwing an instance of 'tf2_ros::InvalidTimerHandleException'
    [server-9] [planner_server-4]   what():  Invalid timer handle in remove()
    
  • We found this exception is thrown from
    throw InvalidTimerHandleException("Invalid timer handle in remove()");
  • If adding more messages for debug, we found that there is a high probability that handle will be removed repeatedly.
    [server-9] [planner_server-4] [INFO] [1636089969.121297887] [tf2_buffer]: [cliff] remove@217, it->first=286, this->timer_interface_=0x55eee89ff140
    [server-9] [planner_server-4] [INFO] [1636089969.126648738] [tf2_buffer]: [cliff] remove@286, timer_handle=286, timer_interface_=0x55eee89ff140
    [server-9] [planner_server-4] terminate called after throwing an instance of 'tf2_ros::InvalidTimerHandleException'
    [server-9] [planner_server-4]   what():  Invalid timer handle in remove()
    
  • Therefore, we have made a change to ensure that timer_handle only can be removed when it can be found in timer_to_request_map_

Copy link
Member

@aprotyas aprotyas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable to me. Is it possible to add a test case that demonstrates this?

@z80020100
Copy link
Contributor Author

How should I add a test case?

@aprotyas
Copy link
Member

aprotyas commented Nov 6, 2021

How should I add a test case?

Can you write a code snippet that uses the tf2_ros interface in a way that throws this timer handle exception without the patch (but not with it)? If so, that snippet should may live as a test case in tf2_ros/test/test_buffer.cpp.

@z80020100
Copy link
Contributor Author

Thank you for the feedback. I will try to write a test case and push it later.

@z80020100 z80020100 force-pushed the hotfix/invalid_timer_handle_exception branch from 59d746e to 46bcb33 Compare November 9, 2021 07:15
@aprotyas
Copy link
Member

aprotyas commented Nov 9, 2021

@z80020100 there are linter errors in the Rpr job, can you address them please?
https://build.ros2.org/job/Rpr__geometry2__ubuntu_focal_amd64/372/testReport/

@z80020100
Copy link
Contributor Author

@aprotyas Sure, I will fix them.

@clalancette clalancette self-assigned this Feb 4, 2022
@z80020100 z80020100 force-pushed the hotfix/invalid_timer_handle_exception branch from 5d3f011 to c85bc8b Compare February 9, 2022 09:57
@audrow audrow changed the base branch from ros2 to rolling June 28, 2022 14:18
@cmraaron
Copy link

cmraaron commented Mar 1, 2023

status: works for me 👍

@swarajpeppermint
Copy link

swarajpeppermint commented Apr 28, 2023

Will this be merged into humble? @aprotyas I'm sure other people will also face this issue (we did and we independently identified the same bug). If this is good, then we should merge this into the code base or at least mark it as an issue so that others can find this.

@aprotyas
Copy link
Member

@swarajpeppermint thanks for flagging this. Since I’m not a maintainer I can’t actually merge the code, but I’m tagging @clalancette who can hopefully get this PR across the finish line

Copy link
Contributor

@clalancette clalancette left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry for the very long delay here. In short, after looking at this more closely, I agree with this change. Thank you so much for adding the test case as well, that is very helpful.

I'm going to approve, rebase it onto the latest, and go ahead and run CI on it.

@clalancette clalancette force-pushed the hotfix/invalid_timer_handle_exception branch from c85bc8b to b3326c2 Compare July 20, 2023 14:06
@clalancette
Copy link
Contributor

CI:

  • Linux Build Status
  • Linux-aarch64 Build Status
  • Windows Build Status

@clalancette clalancette merged commit 2eaab25 into ros2:rolling Jul 20, 2023
2 checks passed
@cmraaron
Copy link

fantastic!, can we backport to iron and humble?

@ahcorde
Copy link
Contributor

ahcorde commented Jul 21, 2023

https://github.com/Mergifyio backport iron humble

@mergify
Copy link
Contributor

mergify bot commented Jul 21, 2023

backport iron humble

✅ Backports have been created

mergify bot pushed a commit that referenced this pull request Jul 21, 2023
* Add class MockCreateTimerROS for testing

* Add a test case to reproduce invalid timer handle exception

* Fix invalid timer handle exception caused by removing handle repeatedly

(cherry picked from commit 2eaab25)
mergify bot pushed a commit that referenced this pull request Jul 21, 2023
* Add class MockCreateTimerROS for testing

* Add a test case to reproduce invalid timer handle exception

* Fix invalid timer handle exception caused by removing handle repeatedly

(cherry picked from commit 2eaab25)
clalancette pushed a commit that referenced this pull request Jul 24, 2023
* Add class MockCreateTimerROS for testing

* Add a test case to reproduce invalid timer handle exception

* Fix invalid timer handle exception caused by removing handle repeatedly

(cherry picked from commit 2eaab25)

Co-authored-by: Cliff Wu <z800201002005@gmail.com>
clalancette pushed a commit that referenced this pull request Jul 24, 2023
* Add class MockCreateTimerROS for testing

* Add a test case to reproduce invalid timer handle exception

* Fix invalid timer handle exception caused by removing handle repeatedly

(cherry picked from commit 2eaab25)

Co-authored-by: Cliff Wu <z800201002005@gmail.com>
@tonynajjar
Copy link
Contributor

tonynajjar commented Aug 29, 2023

This issue was independently reported to Nav2, ROS answers and Robotics Stack Exchange and I couldn't find a fix for a long time. Thanks for this fix, it was really a frequent problem for our production robots.

at least mark it as an issue

In retrospect, this would have probably been a good idea to help find this issue/PR more easily

PGotzmann pushed a commit to kin-dergarten/geometry2 that referenced this pull request Sep 21, 2023
* Add class MockCreateTimerROS for testing

* Add a test case to reproduce invalid timer handle exception

* Fix invalid timer handle exception caused by removing handle repeatedly

(cherry picked from commit 2eaab25)

Co-authored-by: Cliff Wu <z800201002005@gmail.com>
@anath93
Copy link

anath93 commented Jan 18, 2024

@z80020100 @swarajpeppermint @clalancette Can this be backported to Galactic or Foxy ?

@ahcorde
Copy link
Contributor

ahcorde commented Jan 18, 2024

@anath93 Galactic and Foxy are EOL we are not merging more PRs in these branches.

@anath93
Copy link

anath93 commented Jan 18, 2024

@ahcorde these are the only LTS version on 20.04, there is still a lot of community and would be helpful for the community. Can this be considered still ?

@clalancette
Copy link
Contributor

Can this be considered still ?

No, sorry. We have no way to deliver fixes for those distributions anymore.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants