Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to start UR5 under ROS kinetic with process [ur_hardware_interface-3] died #327

Closed
chenjq20 opened this issue Mar 3, 2021 · 12 comments

Comments

@chenjq20
Copy link

chenjq20 commented Mar 3, 2021

Summary

Hello,
I am using external control with corresponding IP address. I have met the problem to bring up the robot (UR5) at the point of Quick start. I have got the error "REQUIRED process [ur_hardware_interface-3] has died!". While I have checked the previous issues reported this problem and checked that the "EtherNet/IP" is disabled, I still could not fix this problem.

Introduction to the issue

Versions

  • ROS Driver version: Ubuntu 16.04 with ROS kinetic 1.12.13
  • Affected Robot Software Version(s): URSoftware 3.9.1.64192
  • Affected Robot Hardware Version(s): UR5 CB3
  • Robot Serial Number: 2019350758
  • URCaps Software version(s):externalcontrol-1.0.4

Impact

I have met the problem with "roslaunch ur_robot_driver ur5_bringup.launch robot_ip:=192.168.56.101". Before this quick start step, I have completed the installation of URCap and the calibration process successfully.

Issue details

The error I got with roslaunch is as followed.

process[rosout-1]: started with pid [12848]
started core service [/rosout]
process[robot_state_publisher-2]: started with pid [12865]
process[ur_hardware_interface-3]: started with pid [12866]
process[ros_control_controller_spawner-4]: started with pid [12867]
process[ros_control_stopped_spawner-5]: started with pid [12876]
process[controller_stopper-6]: started with pid [12886]
process[ur_hardware_interface/ur_robot_state_helper-7]: started with pid [12896]
[ INFO] [1614764092.226656300]: Waiting for controller manager service to come up on /controller_manager/switch_controller
[ INFO] [1614764092.228815445]: waitForService: Service [/controller_manager/switch_controller] has not been advertised, waiting...
[ INFO] [1614764092.232160537]: Initializing dashboard client
[ INFO] [1614764092.234232145]: waitForService: Service [/ur_hardware_interface/dashboard/play] has not been advertised, waiting...
[ INFO] [1614764092.247935112]: Initializing urdriver
Warning: No realtime capabilities found. Consider using a realtime system for better performance
at line 396 in /tmp/binarydeb/ros-kinetic-ur-client-library-0.1.1/include/ur_client_library/comm/pipeline.h
[ INFO] [1614764092.255319592]: waitForService: Service [/ur_hardware_interface/dashboard/play] is now available.
[INFO] [1614764092.457543]: Controller Spawner: Waiting for service controller_manager/load_controller
[INFO] [1614764092.458991]: Controller Spawner: Waiting for service controller_manager/load_controller
Error: The calibration parameters of the connected robot don't match the ones from the given kinematics config file. Please be aware that this can lead to critical inaccuracies of tcp positions. Use the ur_calibration tool to extract the correct calibration from the robot and pass that into the description. See [https://github.com/UniversalRobots/Universal_Robots_ROS_Driver#extract-calibration-information] for details.
at line 45 in /tmp/binarydeb/ros-kinetic-ur-client-library-0.1.1/src/ur/calibration_checker.cpp
Warning: No realtime capabilities found. Consider using a realtime system for better performance
at line 396 in /tmp/binarydeb/ros-kinetic-ur-client-library-0.1.1/include/ur_client_library/comm/pipeline.h
Warning: No realtime capabilities found. Consider using a realtime system for better performance
at line 396 in /tmp/binarydeb/ros-kinetic-ur-client-library-0.1.1/include/ur_client_library/comm/pipeline.h
[ INFO] [1614764094.337737221]: Loaded ur_robot_driver hardware_interface
[ INFO] [1614764094.375721815]: waitForService: Service [/controller_manager/switch_controller] is now available.
[ INFO] [1614764094.375750433]: Service available.
[ INFO] [1614764094.375771205]: Waiting for controller list service to come up on /controller_manager/list_controllers
[ INFO] [1614764094.376254377]: Service available.
[ INFO] [1614764094.452807943]: Robot's safety mode is now NORMAL
[ INFO] [1614764094.453124970]: Robot mode is now RUNNING
[INFO] [1614764094.580918]: Controller Spawner: Waiting for service controller_manager/switch_controller
[INFO] [1614764094.582525]: Controller Spawner: Waiting for service controller_manager/switch_controller
[INFO] [1614764094.584551]: Controller Spawner: Waiting for service controller_manager/unload_controller
[INFO] [1614764094.586107]: Controller Spawner: Waiting for service controller_manager/unload_controller
[INFO] [1614764094.588020]: Loading controller: joint_state_controller
[INFO] [1614764094.589543]: Loading controller: pos_joint_traj_controller
[INFO] [1614764094.610542]: Loading controller: scaled_pos_joint_traj_controller
[INFO] [1614764094.714315]: Loading controller: joint_group_vel_controller
[INFO] [1614764094.810611]: Loading controller: speed_scaling_state_controller
[INFO] [1614764094.826449]: Controller Spawner: Loaded controllers: pos_joint_traj_controller, joint_group_vel_controller
[INFO] [1614764094.834219]: Loading controller: force_torque_sensor_controller
[INFO] [1614764094.850443]: Controller Spawner: Loaded controllers: joint_state_controller, scaled_pos_joint_traj_controller, speed_scaling_state_controller, force_torque_sensor_controller
[INFO] [1614764094.858671]: Started controllers: joint_state_controller, scaled_pos_joint_traj_controller, speed_scaling_state_controller, force_torque_sensor_controller
================================================================================REQUIRED process [ur_hardware_interface-3] has died!
process has died [pid 12866, exit code -11, cmd /home/liaolab/catkin_ws/devel/lib/ur_robot_driver/ur_robot_driver_node __name:=ur_hardware_interface __log:=/home/liaolab/.ros/log/b42510cc-7c03-11eb-a7db-e4a471d4e81f/ur_hardware_interface-3.log].
log file: /home/liaolab/.ros/log/b42510cc-7c03-11eb-a7db-e4a471d4e81f/ur_hardware_interface-3*.log
Initiating shutdown!
================================================================================
[ur_hardware_interface/ur_robot_state_helper-7] killing on exit
[controller_stopper-6] killing on exit
[ros_control_stopped_spawner-5] killing on exit
[ros_control_controller_spawner-4] killing on exit
[ur_hardware_interface-3] killing on exit
[robot_state_publisher-2] killing on exit
[INFO] [1614764095.140237]: Shutting down spawner. Stopping and unloading controllers...
[INFO] [1614764095.141125]: Stopping all controllers...
[INFO] [1614764095.141830]: Shutting down spawner. Stopping and unloading controllers...
[INFO] [1614764095.142681]: Stopping all controllers...
[WARN] [1614764095.145965]: Controller Spawner error while taking down controllers: unable to connect to service: [Errno 111] Connection refused
[WARN] [1614764095.148275]: Controller Spawner error while taking down controllers: unable to connect to service: [Errno 111] Connection refused
[controller_stopper-6] escalating to SIGTERM
[rosout-1] killing on exit
[master] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete
done

Use Case and Setup

I have followed the instruction steps to download and build the Universal_Robots_ROS_Driver. However, I followed the instructions on the net to replace the "ur_driver" with "ur_modern_driver". I wonder if this step caused this error or if there was some edition conflits between Universal_Robots_ROS_Driver and ur_modern_driver.

Thank you for any help you can offer!

@chenjq20
Copy link
Author

chenjq20 commented Mar 4, 2021

Hi,

I have also tried different versions of Universal_Robots_ROS_Driver and fmauch_universal_robot. I found that the only combination of the version that could succeed in the catkin_make process was "master" of "Universal_Robots_ROS_Driver” and "calibration-devel" of "fmauch_universal_robot".

Also, I have replaced the "ur_driver" in "fmauch_universal_robot" with "ur_modern_driver". The version of "ur_modern_driver" is "kinetic-devel".

I wonder how I could solve the problem mentioned above, which was described as "[ur_hardware_interface-3] died". Thank you for any help you can offer!

@fmauch
Copy link
Collaborator

fmauch commented Mar 7, 2021

Also, I have replaced the "ur_driver" in "fmauch_universal_robot" with "ur_modern_driver". The version of "ur_modern_driver" is "kinetic-devel".

I don't understand, what that should have to do with anything. ur_driver and ur_modern_driver are separate (deprecated) drivers that won't affect the ur_robot_driver from this repository.

I have also tried different versions of Universal_Robots_ROS_Driver and fmauch_universal_robot. I found that the only combination of the version that could succeed in the catkin_make process was "master" of "Universal_Robots_ROS_Driver” and "calibration-devel" of "fmauch_universal_robot".

Yes, as the documentation states.

The interesting question is indeed why your hardware interface dies. You could start the driver in debug mode which might give a hint.:

roslaunch ur_robot_driver ur5_bringup.launch robot_ip:=192.168.56.101 debug:=true

This starts the driver inside a gdb session. Once it crashed, you'll get a prompt, where you can type bt and then Enter to see the stack trace of the most recent function calls. You can post the output here so we can get a clue what's going wrong.

@chenjq20
Copy link
Author

chenjq20 commented Mar 8, 2021

Hi,

I have tried to start the driver inside a gdb session. It would stop at this following message and could only be crashed by keyboard interrupt:

[INFO] [1615173483.148452]: Started controllers: joint_state_controller, scaled_pos_joint_traj_controller, speed_scaling_state_controller, force_torque_sensor_controller

And in the gdb session, I got these messages and the stack trace of the most recent function calls:

Before interrupt:

[Thread debugging using libthred_db enabled]
Using host libthred_db library "/lib/x86_64-linux-gnu/libthread_db.so.1"
[New Thread 0x7fffee573700 (LWP 17507)]
[New Thread 0x7fffedd72700 (LWP 17508)]
[New Thread 0x7fffed571700 (LWP 17509)]
[New Thread 0x7fffecd70700 (LWP 17514)]
[New Thread 0x7fffe7fff700 (LWP 17531)]

After interrupt:

Thread 1 "robot_state_pub" received signal SIGINT, Interrupt.
pthread_cond_timewait@@GLIBC_2.3.2()
at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
225 ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S: No such file or directory.
(gdb) bt
#0 pthread_cond_timedwait@@GLIBC_2.3.2 ()
at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S: 225
#1 0x00007ffff725b3b7 in ros::CallbackQueue::callAvailable(ros::WallDuration)
()from /opt/ros/kinetic/lib/libroscpp.so
#2 0x00007ffff72b7ef9 in ros::SingleThreadeSpinner::spin(ros::CallbackQueue*)
()from /opt/ros/kinetic/lib/libroscpp.so
#3 0x00007ffff729cedb in ros::spin() () from /opt/ros/kinetic/lib/libroscpp.so
#4 0x00000000000409148 in main ()

Also, I wonder which customer port number I should set. I have tried the customer port number of 30001 and 50002 and this change could not solve the problem.

Thank you very much for your help!

@fmauch
Copy link
Collaborator

fmauch commented Mar 8, 2021

In a normal setup you should not have to change the port configuration. Use that only if you really need to (such as having two arms connected to one ROS machine or in case of port conflicts.)

You might try to

  1. Update your ROS installation
  2. delete & rebuild your workspace

There might be ABI incompatibilities causing your crashes.

@chenjq20
Copy link
Author

chenjq20 commented Mar 9, 2021

Hi,

I have tried to create and rebuild a workspace. When I started the robot driver with debug mode, I got these tracks:

[INFO] [1615252649.100486]: Started controllers: joint_state_controller, scaled_pos_joint_traj_controller, speed_scaling_state_controller, force_torque_sensor_controller
Thread 1 "ur_robot_driver" received signal SIGSEGV, Segmentation fault.
0x00007ffff6d4ddda in ros::TimeBase<ros::Time, ros::Duration>::operator+(ros::Duration const&) const () from /opt/ros/kinetic/lib/librostime.so
(gdb) bt
#0 0x00007ffff6d4ddda in ros::TimeBase<ros::Time, ros::Duration>::operator+(ros::Duration const&) const () from /opt/ros/kinetic/lib/librostime.so
#1 0x00007fffc425d874 in force_torque_sensor_controller::ForceTorqueSensorController::update(ros::Time const&, ros::Duration const&) ()
from /opt/ros/kinetic/lib//libforce_torque_sensor_controller.so
#2 0x00007ffff7b9a837 in controller_manager::ControllerManager::update(ros::Time const&, ros::Duration const&, bool) ()
from /opt/ros/kinetic/lib/libcontroller_manager.so
#3 0x000000000067ae57 in main (argc=1, argv=0x7fffffffd818)
at /home/liaolab/catkin_ur/src/Universal_Robots_ROS_Driver/ur_robot_driver/src/hardware_interface_node.cpp:144

I also tried to update my ROS installation by

rosdep install --from-paths src --ignore-src -y

And it seemed that the dependencies have already be installed. I wonder if I should mannually update the dependencies or I should upgrade all the dependencies that "sudo apt update -qq" has found.

I wonder if the real-time kernel setup is necessary or I should try the alternative build way "All-source build".

Thank you very much for your help!

@chenjq20
Copy link
Author

Hi,

I have also tried the "All-source build" in a new workspace and it got the same error messages. I wonder if I should upgrade all the dependencies by "rosdep upgrade", though it may cost long time or other problems.

Thank you for any help you can offer!

@fmauch
Copy link
Collaborator

fmauch commented Mar 10, 2021

Updating all your packages using

sudo apt update && sudo apt upgrade is what I meant with updating your ROS installation. As the fts controller seems to fail in it's update loop it might me an abi incompatibility inside ros control.

@chenjq20
Copy link
Author

Thank you very much for your kind help! I have solved the problem mentioned above by "sudo apt-get update && sudo apt-get upgrade". I got these messages with ""roslaunch ur_robot_driver ur5_bringup.launch...":

[INFO] [1616142238.818495]: Started controllers: joint_state_controller, scaled_pos_joint_traj_controller, speed_scaling_state_controller, force_torque_sensor_controller
[ INFO] [1616142252.652934965]: Robot requested program
[ INFO] [1616142252.653049066]: Sent program to robot
[ INFO] [1616142252.689212907]: Robot ready to receive control commands.

After solving this problem, I tried to control UR5 by Moveit! using:

roslaunch ur5_moveit_config ur5_moveit_planning_execution.launch limited:=true

This came with these messages before further operations:

[ INFO] [1616142396.140481414]: MoveGroup context using planning plugin ompl_interface/OMPLPlanner
[ INFO] [1616142396.140523810]: MoveGroup context initialization complete
You can start planning now!

But when I tried to use Rviz to control UR5, I came up with this problem. I used:

roslaunch ur5_moveit_config moveit_rviz.launch config:=true

Then I tried to "plan and execute" in Rviz window. This operation came with the following messages in the terminal of "roslaunch ur5_moveit_config ur5_moveit_planning_execution.launch limited:=true":

[ERROR] [1616142685.874778525]: Unable to identify any set of controllers that can actuate the specified joints: [ elbow_joint shoulder_lift_joint shoulder_pan_joint wrist_1_joint wrist_2_joint wrist_3_joint ]
[ERROR] [1616142685.874812194]: Known controllers and their joints:

And the following messages in the terminal of "roslaunch ur5_moveit_config moveit_rviz.launch config:=true"

[ INFO] [1616142633.012892180]: rviz version 1.12.17
[ INFO] [1616142633.012925399]: compiled against Qt version 5.5.1
[ INFO] [1616142633.012935189]: compiled against OGRE version 1.9.0 (Ghadamon)
[ INFO] [1616142633.257078613]: Stereo is NOT SUPPORTED
[ INFO] [1616142633.257154826]: OpenGl version: 3 (GLSL 1.3).
[ INFO] [1616142636.665562368]: Loading robot model 'ur5'...
[ WARN] [1616142636.665602909]: Skipping virtual joint 'fixed_base' because its child frame 'base_link' does not match the URDF frame 'world'
[ INFO] [1616142636.665616045]: No root/virtual joint specified in SRDF. Assuming fixed joint
[ INFO] [1616142636.693847988]: Loading robot model 'ur5'...
[ WARN] [1616142636.693870130]: Skipping virtual joint 'fixed_base' because its child frame 'base_link' does not match the URDF frame 'world'
[ INFO] [1616142636.693879114]: No root/virtual joint specified in SRDF. Assuming fixed joint
[ INFO] [1616142636.760798903]: Starting scene monitor
[ INFO] [1616142636.763222939]: Listening to '/move_group/monitored_planning_scene'
[ INFO] [1616142637.042767694]: Constructing new MoveGroup connection for group 'manipulator' in namespace ''
[ INFO] [1616142637.988832681]: Ready to take commands for planning group manipulator.
[ INFO] [1616142637.988975958]: Looking around: no
[ INFO] [1616142637.989061743]: Replanning: no
[ WARN] [1616142638.022015248]: Interactive marker 'EE:goal_ee_link' contains unnormalized quaternions. This warning will only be output once but may be true for others; enable DEBUG messages for ros.rviz.quaternions to see more details.
[ INFO] [1616142677.990041145]: ABORTED: Solution found but controller failed during execution

And this following with UR5 robot could not be moved while the planning process executed normally in Rviz window.

I wonder how I could fixed this "Unable to identify any set of controllers that can actuate the specified joints:" and "ABORTED: Solution found but controller failed during execution" problem. Thank you for any help you can offer!

@fmauch
Copy link
Collaborator

fmauch commented Mar 22, 2021

Did you remap the controller namespace inside the moveit configuration as explained here?

@chenjq20
Copy link
Author

Hi,

Thank you very much for you kind help! I have fixed the problem by remap the controller namespace as

scaled_pos_joint_traj_controller/follow_joint_trajectory

Sorry for not browsing other issues thoughtfully and carefully.

Now I can control the UR5 robot with Moveit! and Rviz smoothly. But several warnings still exit without troubling the control process.

[ INFO] [1616669740.705901919]: Loading robot model 'ur5'...
[ WARN] [1616669740.705960391]: Skipping virtual joint 'fixed_base' because its child frame 'base_link' does not match the URDF frame 'world'
[ INFO] [1616669740.705976544]: No root/virtual joint specified in SRDF. Assuming fixed joint
[ INFO] [1616669740.734545308]: Loading robot model 'ur5'...
[ WARN] [1616669740.734604317]: Skipping virtual joint 'fixed_base' because its child frame 'base_link' does not match the URDF frame 'world'
[ INFO] [1616669740.734620190]: No root/virtual joint specified in SRDF. Assuming fixed joint
[ INFO] [1616669740.799083091]: Starting scene monitor
[ INFO] [1616669740.801394791]: Listening to '/move_group/monitored_planning_scene'
[ INFO] [1616669741.077248773]: Constructing new MoveGroup connection for group 'manipulator' in namespace ''
[ INFO] [1616669742.089346722]: Ready to take commands for planning group manipulator.
[ INFO] [1616669742.089502370]: Looking around: no
[ INFO] [1616669742.089584827]: Replanning: no
[ WARN] [1616669742.120872502]: Interactive marker 'EE:goal_ee_link' contains unnormalized quaternions. This warning will only be output once but may be true for others; enable DEBUG messages for ros.rviz.quaternions to see more details.
[ WARN] [1616669785.405992933]: Link [base_link] does not exist
[ WARN] [1616669785.406076065]: Link [shoulder_link] does not exist
[ WARN] [1616669785.406130065]: Link [upper_arm_link] does not exist
[ WARN] [1616669785.406177088]: Link [forearm_link] does not exist
[ WARN] [1616669785.406225881]: Link [wrist_1_link] does not exist
[ WARN] [1616669785.406274224]: Link [wrist_2_link] does not exist
[ WARN] [1616669785.406323201]: Link [wrist_3_link] does not exist
[ WARN] [1616669785.406370419]: Link [ee_link] does not exist
[ WARN] [1616669785.406419076]: Link [base_link] does not exist
[ WARN] [1616669785.406466591]: Link [shoulder_link] does not exist
[ WARN] [1616669785.406516230]: Link [upper_arm_link] does not exist
[ WARN] [1616669785.406602633]: Link [forearm_link] does not exist
[ WARN] [1616669785.406661578]: Link [wrist_1_link] does not exist
[ WARN] [1616669785.406706047]: Link [wrist_2_link] does not exist
[ WARN] [1616669785.406753870]: Link [wrist_3_link] does not exist
[ WARN] [1616669785.406799952]: Link [ee_link] does not exist

[ INFO] [1616669713.025641801]: Loading robot model 'ur5'...
[ WARN] [1616669713.025694422]: Skipping virtual joint 'fixed_base' because its child frame 'base_link' does not match the URDF frame 'world'
[ INFO] [1616669713.025704532]: No root/virtual joint specified in SRDF. Assuming fixed joint

I wonder if these warnings could be ignored or some change could be made to the ur5.srdf.

Thank you again for your reply and help!

@fmauch
Copy link
Collaborator

fmauch commented Mar 30, 2021

The default moveit configurations are currently being worked on. See ros-industrial/universal_robot#538.

Anyway, you should create your own moveit configuration matching your actual setup in order to gain the maximum benefits of MoveIt.

@fmauch
Copy link
Collaborator

fmauch commented Mar 30, 2021

As the control setup is working, I'll close this issue. If there's anything related coming up, feel free to comment or reopen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants