[Very Weird] Suddenly Can't Load All Dynamixel Motors Anymore

[msr-peng]

Steps to reproduce

1. When I run PyRobot with Azure Kinect camera (with my own minor modified codes about camera and LoCoBot) on Ubuntu 16.04, sometimes there would be some issues about loading Dynamixel Motors, which caused Rivz and the whole ROS master failed to launch.
   
   But once I disconnected the DYNAMIXEL U2D2 and reconnected it, then the whole PyRobot can work fluently.

2. One day, however, all motors can't be loaded. And I even didn't modify any codes after the last successful run. When I run roslaunch locobot_control main.launch use_arm:=true teleop:=true, the arm motors errors is this:
   
   The same errors would occur on active camera motors when I run roslaunch locobot_control main.launch use_camera:=true:
   
   This error happened on pyrobot/robots/LoCoBot/locobot_control/src/locobot_controller.cpp Line 1408 when launch dynamixel_controllers.launch if I run anything about main.launch

More Info

1. I'm inclined that it may not a hardware issue. Since I tried a new DYNAMIXEL U2D2, DYNAMIXEL Hub and cables, the exact errors would occur.

2. I also don't think it has something to do with Azure Kinect. I didn't install RealSense library and SDK when I followed locobot_install_all.sh, but it shouldn't cause the arm's motors failure.

3. Then I reinstalled the whole PyRobot (Develop branch) on a fresh Ubuntu 18.04 with Python 3, there would be the same errors when lauch main.launch.

4. I can also connect all 9 motors (active camera's and arm's) via DYNAMIXEL Wizard 2.0 and control them fluently. Here is all motors info showed in GUI:
   
   They have correct motor IDs, the same protocol (2.0) and firmware version (43) and the same baud rate (1Mbps).

I got stuck on this issue for several weeks, and already tried all solutions to my best knowledge. So I really need PyRobot teams help. Do you have any clues about the issue and how to debug it?

[kalyanvasudev]

I have few questions,

1. Could you please point me to the repository you are working with ? So that I can take better look to understanding what is happening?
2. Are you able to at least control the arm with use_cam:=false ?

[kalyanvasudev]

One day, however, all motors can't be loaded. And I even didn't modify any codes after the last successful run. When I run roslaunch locobot_control main.launch use_arm:=true teleop:=true, the arm motors errors is this:

Do you mean that you did not change any code and did not pull any changes from the remote and even then it failed (however, it worked the day before)?

[msr-peng]

One day, however, all motors can't be loaded. And I even didn't modify any codes after the last successful run. When I run roslaunch locobot_control main.launch use_arm:=true teleop:=true, the arm motors errors is this:

Do you mean that you did not change any code and did not pull any changes from the remote and even then it failed (however, it worked the day before)?

Hey Kalyan, thank you for quick reply!

Yep. It worked the day before, sometimes the error in the first image will occur. But once I reconnected the Dyanmixel U2D2 for several times, main.launch will run successfully.

The current repository is the same as PyRobot Develop branch, (I git clone it today) except I deleted everything about RealSense in locobot_install_all.sh, then I followed the scrips to install PyRobot on Ubuntu 18.04

Did you mean run roslaunch locobot_control main.launch use_cam:=false?
I did it, and on another terminal tried to control the arm, but this error occured:

[kalyanvasudev]

Did you mean run roslaunch locobot_control main.launch use_cam:=false?
I did it, and on another terminal tried to control the arm, but this error occured:

Oh I meant roslaunch locobot_control main.launch use_arm:=true use_cam:=false use_base:=false

Using just roslaunch locobot_control main.launch use_cam:=false wouldn't as arm is not enabled.

Also, How are you running the Azure Kinect? Precisely, what is the associated launch file?

[msr-peng]

Hey Kalyan, once I run roslaunch locobot_control main.launch with use_arm:=true, the Can't find Dynamixel ID X error (image 2) will appear.

1. To run the Azure Kinect, I modified main.launch file, add camera arg, it has realsense and azurekinect two options.

main.launch screenshot:

2. Once pass the camera:=azurekinect to main.launch, the main.launch will launch pyrobot/robots/LoCoBot/locobot_navigation/orb_slam2_ros/launch/orb_slam2_azurekinect.launch, which start the Azure Kinect ROS node.
   orb_slam2_azurekinect.launch screenshot
   

3. I also added Azure Kinect configuration and yaml files, and did slight modification on pyrobot/src/pyrobot/cfg/locobot_config.py and pyrobot/src/pyrobot/cfg/core.py to make them match the Azure Kinect's ROS topic name. But all of them have nothing do to with motors directly.

All of these modifications are on my Ubuntu 16.04 environment.

I currently run PyRobot Develop branch on Ubuntu 18.04. And I didn't apply any of azure kinect modifucations on it, except I didn't install realsense installation parts in locobot_install_all.sh. The motors error is the same as that on Ubuntu 16.04.

[msr-peng]

Hey Kalyan, thank you for helping me out! I partially solved this issues tonight. It turns out a weird motor-level hardware problem.

I firstly made LoCoBotController::loadDynamixels function's for loop sleep 2 seconds after loading every Dynamixel motor. This made motors can be loaded without errors.

Then the reason why main.launch failed is that my motors misreported the Hardware Error Status (Overload Error, Electrical Shock Error, Motor Encoder Error, etc.). Like the errors below:

It's very similar to what happen in Issue 32.

To protect the motors from hardware damage, this hardware error status is designed to trigger the ROS_ERROR when LoCoBotController::hardwareStatusPublish is called. That's why main.launch is forced to be stopped.

However, from DYNAMIXEL Wizard 2.0 I learnt that my motors were actually in normal condition, so I commented out LoCoBotController::hardwareStatusPublish to force the main.launch keep going on.

Then I can control my arm, although the arm command wouldn't work if misreport happened at the same time.

After several minutes motors "warmup", now my arm motors can run fluently without misreport hardware error status.

I'll also follow Issue 32's solutions to replace all arm's cables.

I hope this solution can help others who run similar issues with Dynamixel motors. ROBOTIS products are actually not robust.

[kalyanvasudev]

Thank you very much for the detailed analysis! This will definitely help others who would probably run into the similar issue.

However, it is still very weird that the Dynamixel motors are misreporting the errors. Let me look into this.

I firstly made LoCoBotController::loadDynamixels function's for loop sleep 2 seconds after loading every Dynamixel motor. This made motors can be loaded without errors.
So the sleep for 2 seconds, is still needed in this?

[kalyanvasudev]

Also, there is an unrelated different bug in your main.launch
azure_kinect_ros_driver/launch/ddriver.launch, as mentioned here

Overwrites, the LoCoBot's URDF rosparam. This would break things like IK FK, TF etc.

IMO, just create a sperate launch similar to azure_kinect_ros_driver/launch/ddriver.launch and store it in locobot_control/launch

[msr-peng]

Also, there is an unrelated different bug in your main.launch
azure_kinect_ros_driver/launch/ddriver.launch, as mentioned here

Overwrites, the LoCoBot's URDF rosparam. This would break things like IK FK, TF etc.

IMO, just create a sperate launch similar to azure_kinect_ros_driver/launch/ddriver.launch and store it in locobot_control/launch

Thank you for suggestion. I created a launch file locobot_control/launch/azurekinect_driver.launch which doesn't have rosparam robot_description, and make main.launch start this launch file instead of that in Azure Kinect ROS Driver.

[kalyanvasudev]

I take it that this issue is resolved for now? Closing this issue. Please feel free to re-open if you have further questions.

[msr-peng]

I take it that this issue is resolved for now? Closing this issue. Please feel free to re-open if you have further questions.

Yep.