Critical failure during "Failsafe"

[fermiums22]

Current Behavior

Steps to Reproduce

1. Be in flight in turn mode angle
2. Enable Mode FS
3. Watch the drone die. =)

Expected behavior

Smooth landing on the ground.

Suggested solution(s)

Give a detailed description of the construction of the landing mode.

LOG00011.TXT
INAV_6.0.0_clean_my.txt
https://youtu.be/NmOX9EetNTE

• FC Board name and vendor:
  iflight blitz f7 pro
  iflight blitz E80 Pro 4-IN-1 ESC
  Iflight Chimera 5
• INAV version string:
  6.0.0 stable

[breadoven]

Looks like the roll, pitch and yaw attitude outputs all switched to 0 immediately after Failsafe mode was enabled. PIDS went ballistic, nothing in control any more, hence the crash. Very odd.

Just checked one of my logs from testing an emergency landing back in March to be sure my eyes weren't deceiving me and attitude remained active throughout ... as expected.

[fermiums22]

first it happened in the field. this video is when I decided to double-check what was wrong and launched the drone next to me. If you claim that it's not inav, then which of the settings did I specify incorrectly? I am not sure about the orientation of the optics flow.

[fermiums22]

During the tast, I saw that something was wrong with arming mode. The drone did not listen to me after FS mode. After reviewing the log, I saw that disabling the arm didn't stop the drone either, which is very scary. Only the kill switch helped.

[breadoven]

first it happened in the field. this video is when I decided to double-check what was wrong and launched the drone next to me. If you claim that it's not inav, then which of the settings did I specify incorrectly? I am not sure about the orientation of the optics flow.

Not sure what the problem is other than the attitude and PIDS didn't behave as expected when Failsafe mode was activated. What did it do in the field before this crash flight that made you think something was wrong ?

One thing I did notice is you have RTH and Failsafe modes triggered together by the same switch. This doesn't make sense given you have Failsafe set to Land. Need to check the code to see if this might cause some unforeseen interaction between RTH and Failsafe.

Also, the normal disarm switch won't work in Failsafe, including Failsafe mode. This is protect against false disarms when there are problems with the RC signal which could send a false disarm command. This doesn't apply for the Kill switch. Not sure that makes sense really given the Kill switch could also be affected by a false disarm signal. Might be worth a separate discussion to see what others think.

Would also be useful if you could provide a Diff, it makes it much easier to see what settings have been changed from defaults.

[fermiums22]

Ok I'll provide the diff and additionally the same case as before. A little later in the evening.

[breadoven]

Actually looking at the log PID values again (time 00:13.964) when this glitch happens they're switching to values of 2,147,483,648 which is the limit for a signed 32 bit number (there's also 4,294,967,285 popping up though which is almost an unsigned 32 bit but not quite) . Which suggests something is causing an overflow. Any ideas @DzikuVx ?

[fermiums22]

Here is the first accident in the fifth minute.
Video from glasses, during the crash. Video from the camera the entire flight. And diff as requested.
INAV_6.0.0_cli_diff.txt
https://youtu.be/UH2IihaCmNM
https://youtu.be/kLp-9HGMHi4
LOG00001.TXT

[Jetrell]

You have failsafe_procedure = LAND.. With 1000uS as the FS throttle value for 20 seconds. i.e. NO SET-THR value to control the descent, in the case of an altitude sensor failure and the landing procedure reverts to the old method. I'm not sure what the effect would be, with Airmode permanently enabled under this condition.

I wonder what would occur if the Optical flow sensor lost sight of the ground in a tumble ?

[fermiums22]

I wanted to set the throttle to a minimum so as not to get the effect of lifting. In the us, I didn't know the meaning. I know that somewhere 15% of the deviation of the throttle stick. the only one where there is a value in us is in the BLHelli ESC settings. It is not entirely clear how to calculate.

[fermiums22]

Has anyone tested this mode? Please give an example of a log, config and a small description of the frame?

[breadoven]

I've tested emergency landing before, and more recently, and it worked as expected. Only difference is I started it manually not from Failsafe although I don't think that should make any difference. As far as your log is concerned it seemed to switch to Failsafe and Emergency landing as expected so I'm not sure this is the issue.

Looking at the second log it's notable that one of the motors ramps up significantly higher than the other motors at the point the Emergency Landing starts (40% vs 10%). The throttle is set to 0 at this point due to the altitude controller being reset. The throttle then gets set to 1500 once the Nav emergency landing controller kicks in and starts controlling. At this point the motor that was higher than the others ramps up immediately to 80% vs 50% on the other 3 motors. I'm guessing the thrust differential at this higher speed would have been more significant than before the motors sped up possibly causing the quad to flip the way it did. It's possible that the one motor was running higher than the other three to correct a slight pitch up and right roll rotation if I read it right which probably I'm not because rather than correcting the rotation it made it worse. The other notable point is the raw gyro data (gyroRaw) in the log doesn't seem to correlate to the filtered gyro data (gyro). I'd have thought there should be some correlation. Would need to spend more time understanding what it was doing because right now it doesn't make much sense.

It's also obvious it was raining in the video. Is the quad proofed against water affecting the electronics ? Seems a stretch given this didn't affect the rest of the flight but then again this happened at the end obviously when things might have got a bit damp.

[fermiums22]

I've tested emergency landing before, and more recently, and it worked as expected. Only difference is I started it manually not from Failsafe although I don't think that should make any difference. As far as your log is concerned it seemed to switch to Failsafe and Emergency landing as expected so I'm not sure this is the issue.

Looking at the second log it's notable that one of the motors ramps up significantly higher than the other motors at the point the Emergency Landing starts (40% vs 10%). The throttle is set to 0 at this point due to the altitude controller being reset. The throttle then gets set to 1500 once the Nav emergency landing controller kicks in and starts controlling. At this point the motor that was higher than the others ramps up immediately to 80% vs 50% on the other 3 motors. I'm guessing the thrust differential at this higher speed would have been more significant than before the motors sped up possibly causing the quad to flip the way it did. It's possible that the one motor was running higher than the other three to correct a slight pitch up and right roll rotation if I read it right which probably I'm not because rather than correcting the rotation it made it worse. The other notable point is the raw gyro data (gyroRaw) in the log doesn't seem to correlate to the filtered gyro data (gyro). I'd have thought there should be some correlation. Would need to spend more time understanding what it was doing because right now it doesn't make much sense.

It's also obvious it was raining in the video. Is the quad proofed against water affecting the electronics ? Seems a stretch given this didn't affect the rest of the flight but then again this happened at the end obviously when things might have got a bit damp.

I like that at least someone is trying to help me.

The first thing I would like to say is that all the boards are well smeared with a protective varnish. I had to sweat dismantling all the boxes, and dismantle all the protective covers and then mount them back. As you can see in the video, the drone remains operational even after diving into a puddle. The only thing that has not been sealed is the barometer chip, and the optics on the matek.

You noticed that the engine did not work normally in log1. Logs are numbered in chronological order. In log 1, when the first crash happened, I started rechecking all systems. After checking, I saw that the lower left screw was unclenched. When I tightened all the nuts and made a couple of test flights, between 1 and 11 I decided to repeat the landing procedure. Thinking that the unclenched screw ruined the autopilot.

To exclude the possibility of an open nut, flight 11 was carried out, which proved that the problem was in the algorithms, although the battery and camera became the victim. I'm so upset.

I also don't think it's possible that low throttle could cause these flips.

I really hope you can help find the problem. The whole assembly was hope for INAV.

[fermiums22]

Would need to spend more time understanding what

I really appreciate the time. Maybe $50 for Job - code analysis, and finding errors that led to accidents. Transfer to PayPal

[breadoven]

OK the only conclusion here is the filtered gyro output did something strange. It seemed to be indicating a roll to the right and pitch up with motor 1 (right rear) correctly powered up to adjust back to level. However, it's clear from the video the quad rolled left when this happened so the filtered gyro roll output seemed to be reversed for some reason leading to a runaway with a full initial rotation in around 0.2s based on the video ... and faster thereafter. The raw and filtered gyro values bare no correlation with each other but then the raw gyro output doesn't make sense either given the roll started the instant Failsafe was selected based on the OSD video.

Don't know if testing this on SITL with Realflight would be much use given it can't replicate the real world behaviour of the gyro dynamics. Might reveal a bug related to Failsafe or Emergency Landing perhaps. Flagging as a bug given there's no obvious reason for this other than wet in the electronics.

[fermiums22]

Thanks for the screenshots, I saw that you can track the modes that were running. True, I do not see a couple of them, but I think this is a feature of blackbox.
I think the following happened.
I. At the time of the activation of the emergency landing, the function of initialization of this landing was activated. In it, you can observe the reset of the PID controller. I think that at this moment the gyroscope settings failed and it was inverted. Since the deviation of the copter on the model does not correspond to reality. Tried to show with arrows. This error builds up for the first 10 milliseconds before turning on the height control.


II. After 10 milliseconds, when we entered the main loop, my opinion (an unreasonable assumption) that the structures were written incorrectly and corrupted the PID, a common failure with incorrect optimization.


I'm also still trying to come up with a safer test for landing mode. It is interesting if you turn on this mode with the blades removed, whether the copter will correctly determine its orientation in space. Am I speaking correctly?

[Jetrell]

I think that at this moment the gyroscope settings failed and it was inverted. Since the deviation of the copter on the model does not correspond to reality.

I'm not sure about Breadoven, but when I tested this feature a while back. I never encountered this behavior.
But l never tested it with Lidar or Optical flow sensors either. Only with a Barometer.
So I wonder how fast the altitude fusion data can update in a case like this, without producing a conflict.

@b14ckyy Have you tested emergency landing with Optical flow ?

I'm also still trying to come up with a safer test for landing mode.

I tested over thick long grass.. I never like flying over concrete or bitumen. If something goes wrong, it always trashes your gear, as you've experienced.

[b14ckyy]

@Jetrell I don't fly quads with INAV so far and only have LIDAR on a test plane but no Opt Flow. Sorry

[fermiums22]

Inav 6.1.0 Landing after RTH work is good 😊. I think this problem in bad PID tune. Close issues. But Lidar has no affect in new version. I will try tune nav_pid...