Copter: Detect and handle motor loss for improved redundancy

[lthall]

Motor mixers are based on a fixed mixing that limits the roll, pitch, yaw and thrust to prevent saturation of the motor outputs. When a esc/motor/propeller fails there that motor tends to go to maximum limiting the average motor output (thrust) and the roll, pitch, and yaw authority.

The motor mixers are designed to prioritise roll and pitch first, then yaw, followed by thrust. This then represents the order in which control is lost. First the aircraft will start to descend, more severe loss will result in loss of yaw control, followed finally by complete loss of attitude control.

The severity may be limited by ensuring the aircraft is well tuned with the correct CG location and excess thrust. Yaw control can be given up earlier by reducing MOT_YAW_HEADROOM to 0. This may mean that yaw control is temporarily lost during aggressive maneuvers or extreme disturbances.

The most vulnerable frame is the hex as any loss of motor control will make yaw control limited or impossible depending on the precise location of the CG. This may be mitigated using a new propeller configuration:
Copter: Additional Hex propeller configuration #7524

The solution has two parts, detection and handling.

Detection:
-This is most reliably done using ESC feedback. This should be a basic requirement of any CAN esc implementation (both failure and maximum thrust limitation).
-This could also be done much less reliably by looking at the maximum thrust output to any motor and trigger a failure if a motor stays at full throttle for an extended period. This may have many other implications in the case of a false positive and I do not recommend this approach.

Handling:

• The simplest approach is to remove the ability for the compromised motor to limit the thrust of the remaining motors. This will ensure we achieve a significantly higher maximum thrust and also maximise roll and pitch authority.
• The more complex and complete solution is to completly recalculate the motor mixing matrix based on the remaining motors potentially even including the limited thrust output of the compromised motor. This is a much more ambitions goal and I would not recommend this approach in the first instance.

This is an issue that encompases this issue:
Copter: ensure hexacopter with failed motor can make safe landing #838
3.3.1 Hexacopter motor redundancy #3294

To quote @rmackay9 -

I had a chat with @lthall and there's two things we need:

single motor failure detection. Leonard suggested we modify the AP_Motors_Matrix class so that it looks for a single motor >95% while all other motors are <75% (those percentages are guesses and could be modified based on testing). We should probably use a counter and/or bitmask so that the situation must continue for some amount of time (0.5sec?) before we decide the motor has failed and avoid false positives.
based on the above checks, exclude the "failed" motor from the saturation handling code. So for example, we should exclude from the calculation of "rpy_low" and "rpy_high" here in AP_MotorsMatrix.cpp. https://github.com/ArduPilot/ardupilot/blob/master/libraries/AP_Motors/AP_MotorsMatrix.cpp#L231
According to Leonard, doing this will theoretically allow the overall throttle of the motors to climb to 66% instead of the current 33% while still maintaining attitude control.

[lthall]

Related #2666

[hsteinhaus]

Regarding motor saturation: I am flying a quad with the patch mentioned in #2666 since a few years now. The notification causes a vibration alert in the TX and has proven very valuable in daily use, even with no motor redundancy available, to inform the pilot that the vehicle fights with it's performance limits (e.g. caused by nasty weather conditions or a weak battery). If motor redundancy is available, the notification might be even more important, as the pilot will probably not notice that a motor is failing, if the copter is far away and flying in AUTO.

The 95% limit is IMHO ok, as the weak motor usually "sticks" at 100% anyway for extended time. Important: do not forget to implement some kind of warning rate limiting or hysteresis, otherwise the user would get flooded with warning on/off messages during gusts, turns and other maneuvers.

I do not apply a negative limit condition (like "only warn if all other motors below 75%"), as in my philosophy a motor running full power is never considered to be a normal operating condition. However, e.g. for a racer, this could be seen differently.

[lthall]

@hsteinhaus Can you link to the code you are using?

[hsteinhaus]

The code has been PRed here #3672 a long time ago.
The latest revision that I am flying currently : c0ead6d, caa3331, a8fe4cf (part of https://github.com/DuraCopter/ardupilot/commits/DuraCopter-3.5)

BTW, I also warn about throttle below a certain limit, as this is notorious for loss of control issues with extremely light copters (e.g. on test flights with payload removed and smaller-than-usual battery).

[lthall]

Cool, thanks!!

[mark97564]

This would be nice! Last year my x8 copter running arducopter on pixhawk had 1 motor fail and it dropped like a brick 70 meters to the ground, lipo ruptured and had a fire and lots of smoke! I'm happy I carry a few gallons of salt water with me when I fly or it could have been real bad.. Frame was shot but I managed to save everything else on the copter since the lipo was strapped underneath. I was able to pour salt water on it and cut the Velcro straps to get all the electronics away from the fire. But honestly, for a x8 to drop like a brick when 7 working motors are left is just tragic... After looking at the logs that motor was at 100% for 10 plus seconds before it hit the ground, so pixhawk could have easily shut off the remaining 3 motors and landed the copter as a x4' or a x6 with the bottom motors across from each other used and the bad motor and the once across from it shut down... Once I realized my x8 came down over 1 motor I got sad, that was one of the reasons I built a x8, for redundancy.. So any work on keeping copter air born in emergency landing mode in case of motor failure would definitely be a huge huge improvement...

[xiongmaoer]

Really interested in the situation where an octocopter has a motor failed. Can we use another structure of controller that has stronger robustness to deal with the problem , such as ADRC which actually i am little familiar with.

[lthall]

This has been completed to my satisfaction.

[rmackay9]

I've added a new issue to cover reporting to the pilot when an ESC with telemetry has failed. #11947