Fix issue #690

[caguero]

See #690 .

I tracked down the issue in the SimpleHydrodynamics plugin. If you create enough movement in the WAM-V, either spinning in circles or pitching up and down constantly, eventually, the plugin produces a very large combination of force and torque ending up with a crash in the physics Step() function.

To mitigate this issue I made the following changes:

1. I increased the damping of the hydrodynamics parameters.
2. I added an extra parameter and logic to limit the amount of force that the plugin applies.

With these changes I haven't been able to produce a crash, although we have modified the WAM-V behavior a bit as a side effect.

Note: A better solution would be to understand why these high intensity forces are created when violent WAM-V movements are generated. I leave that for a future exercise because it requires a lot of debugging time.

Update (July 19): I don't think the problem is the hydrodynamics plugin, the problem is an abnormal velocity and acceleration in the vehicle. The hydrodynamics is just responding to that. I've decided to remove the cropping for this reason.

How to test this PR?

Run a world and spin in circles in place. You normally will get a crash without this patch but the simulation should carry on with the patch in this PR.

[M1chaelM]

It is definitely harder to produce a crash now, but I'm still able to do it. In addition, it seems that some of the interactions between the wamv and buoys has changed--the buoys appear to produce more resistance now when we run into them--sometimes enough to stop the WAMV.

Here's a video of a crash with the current branch.

crash_wamv.mp4

Note that by spinning one direction and then the other, I still get a surprising acceleration, though it's harder to produce.

EDIT: I can also flip buoys upside down, but I have confirmed this happens on the main branch also. This is a more minor issue, but I'm not sure whether buoys do this (my expectation was that they would be designed to make it impossible to flip them over).

[caguero]

@M1chaelM , I've tweaked a bit the parameters. Could you give it a shot please?

[M1chaelM]

In the sydney_regatta world I'm still able to produce a crash, but it is more difficult to produce now, and the buoy behavior seems mostly back to normal. The WAM-V just moves a little slower, which is fine.

Unfortunately, the acoustic tracking task appears to be a very bad test case for this particular bug. I was not able to teleop through the task (doing my best to follow the beacon) without producing a crash. I did better than previously, but by the time I reached the second lap the wamv's movement started getting less stable, and it was rocking forward and back more than it should before it finally seemed to slip under the water and crash.

Qualitatively, it seems like part of the issue is exacerbated by collisions, and part may have to do with moving the wamv in small bursts of acceleration, which produces accumulating pitch that eventually leads to a crash.

A couple notes towards tracking down the root of this problem. The crash outputs the following error message:

ODE INTERNAL ERROR 1: assertion "aabbBound >= dMinIntExact && aabbBound < dMaxIntExact" failed in collide() [collision_space.cpp:460]

This error is also referenced in the following open issue:

• ODE crash on rolling_shapes.sdf gazebosim/gz-sim#15
  In addition, it looks like it may be related to this bug report in ardupilot that Rhys has already done a nice writeup of:
• Error: [ruby $(which gz) sim-1] ODE INTERNAL ERROR 1: assertion "aabbBound >= dMinIntExact && aabbBound < dMaxIntExact" failed in collide() [collision_space.cpp:460]  ArduPilot/ardupilot_gazebo#45

If I'm understanding correctly, Rhys's conclusion is that a change in the quadratic damping formula that gazebosim uses ultimately have the effect of causing an overflow in the angular velocity about x. The issue says the problem occurred with an upgrade to Gazebo 7.2, and Rhys appears to have a workaround for the rov in which the issue was reported:

• Update quadratic damping coefficients clydemcqueen/bluerov2_ignition#5

I'm still using gazebo 7.0.0, but this sounds a lot like what we're seeing, so I wonder if the change might have propagated down to 7.0.

[caguero]

@M1chaelM , could you give it one more shot? I think we found the root of the problem and a fix. It shouldn't crash anymore.

[M1chaelM]

@caguero With this update, I'm unable to move the WAM-V at all. Gazebo seems to be receiving teleop updates, so my guess is that the WAMV is staying locked.

[caguero]

@caguero With this update, I'm unable to move the WAM-V at all. Gazebo seems to be receiving teleop updates, so my guess is that the WAMV is staying locked.

Hmm, which world are you running? And not after 10 seconds?

The behavior when the WAM-V is locked is slightly different than before but I don't see that issue...

[M1chaelM]

Hmm, which world are you running? And not after 10 seconds?

It was happening in both the acoustic_tracking_task world and the sydney_regatta world. I tried wiping the environment and building from scratch, but no luck. Changing back to the main branch restores the teleop capability. If there's a specific world I should try let me know.

[caguero]

Hmm, which world are you running? And not after 10 seconds?

It was happening in both the acoustic_tracking_task world and the sydney_regatta world. I tried wiping the environment and building from scratch, but no luck. Changing back to the main branch restores the teleop capability. If there's a specific world I should try let me know.

Sorry, could you try again now? I think this was a problem related with the scale of the teleoperation config.

[M1chaelM]

Teleop did come back after that last change, and I was able to test. I ran sydney_regatta world and was unable to produce a crash. I was also able to complete acoustic_tracking_task.sdf without a crash. Before, I could only get half way through that task before I triggered a crash. Now I was able to do it without worrying about trying to move the WAMV carefully. It looks to me like the problem is gone and the WAMV behavior is now much more stable.

[toorxx]

Hello @caguero and @M1chaelM,

I have been working with the WAM-V using VRX (Gazebo) along with OpenAI Gym to train a series of actions, and I came across the problem described in this PR. I implemented the proposed changes in my setup, but unfortunately, I am still facing the issue.

I am using the same version of Gazebo mentioned in this PR and have tested the WAM-V in the sydney_regatta world. Despite the WAM-V holding up for more training steps, I still experience the same error.

I wonder if you could consider reopening this case or provide me with additional guidance on how I might resolve this issue.
I am willing to collaborate on any further testing or in solving this problem if necessary. I greatly appreciate your help and your work on this project.

Thanks!