Emergency landing tumbled crash? -- inav 6.1.1

[breadoven]

Discussed in #9183

^{Originally posted by lida2003 July 21, 2023}
There are quite a lot of discussions about my crash in What will happen, when RC & GPS signal lost? -- inav 6.1.1 #9167

And Thanks for @Jetrell @breadoven patience , I think I have got much more info than before.

Due to our discussion about what will happen in emergency landing when fail throttle set to 1000 with airmode enabled? #9181, I think it'll be really not that easy to talk about this crash, as:

1. airmode seems to keep the attitude even if zero throttle
2. old doc says it'll be a supprise when failsafe_throttle = 1000 ( what I think it should probably write before airmode is implemented )
3. my crash, the quad is tumbled NOT attitude hold drop

So I decided to do an emergency landing test.

And It crashed, crashed, crashed!!!

Preset:

• failsafe_throttle = 1150
• disable GPS in gps tab
• RC channel trigger failsafe, which set to LAND

PS: detailed steps, please consult: https://blog.csdn.net/lida2003/article/details/131849539

Result:

Video: https://www.bilibili.com/video/BV1tu411V7JY?p=1
Log: blackbox_log_2023-07-21_070855.TXT

I'm NOT sure about exact order of the crash. But it seems has something to do with gyro and logic.

1. failsafe switch from althold
2. throttle reset to 0
3. gyro suddenly enlarged from ~1 to ~35
4. gyro further enlarged (out of range?????)
5. throttle adjusted to ~57% ???? try to save the quad???
6. gyro dead????? (Maybe on the ground as it's only about 1meter high for test.)
7. Now it's a mess

10. I wanna fly back to where I was, it flipped when take off. And all gyro data is zero.
    

Any ideas, please kindly let me know!

[breadoven]

This looks like the same issue as #8950.

The Emergency Landing starts at 00:44:235. The same issue as above happens at 00:44:288, Attitude values drop to 0, PIDS go crazy ... although the motor outputs look wrong even before that.

[fermiums22]

I received a notice about this topic. I want to say that I closed the topic, because. landing after returning home suited me and I was tired of thinking about this problem for 3 months. I do not have enough data to analyze the situation and my assumptions can only get in the way. I dealt with more pressing issues that I had. I've also verified that the flow optics don't work on most surfaces, as it does on your grass video.

Lidar works fine up to 1.5 meters, but without flow optics, most likely the drone will not be able to land on the grass in such conditions. Although nothing prevents inva team from trying, the inav policy is contrary to solving these problems =) (for example, another problem, remove the compass when you return home - the team has no time to try, or fix the camera power settings - I don’t understand why this hard to fix)

Also, all these tests are hindered by the fact that during this time, after several crashes, the barometer was broke. I am now waiting for a new part. Without a barometer, nothing in hold mode also works. The GPS works great though.

I in turn figured out how to test all the landing algorithms using STM debugging, pulled out the SWD wires and will soon try to write tests inside the inav code. But since I am not the developer of the inav code, this is just a personal challenge and only my efforts will be directed to finding a problem that most likely will not give anything, but will only give pleasure from fiddling with the code.

I still believe that the algorithm that landing the drone at a height of up to 1.5 meters does not work correctly due to various problems caused by incorrect optics data or one of the spatial regulators.

Indirect observation - with certain filter settings - I can’t say which ones, because changed a lot. The behavior of the copter in the configurator (first tab) becomes similar to what happened then during the accident. For unknown reasons, the copter began to spin where it is displayed in confgurator.

While I fly with the return home mode, I only worry about the compass =). Lidar and Optic flow is my goal. I would like to make it so that the drone can hang several of these sensors and it can fly around obstacles at a slow speed, such as a ceiling or a door frame.

P.S. Sorry for a lot of irrelevant text, but I covered everything in one place, what I think on this topic.

[breadoven]

Looking at the PID numbers they suddenly jump in the space of 1 loop from reasonable values to values that indicate an issue with a 32 bit overflow:

Why it happens during a FS Emergency Landing is a bit of a mystery. It doesn't happen with a manually triggered Emergency Landing with no FS. This is nothing to do with rcCommand values because, with the exception of the Throttle, they remain 0 throughout, which is as expected.

[lida2003]

Ah~~~ discusion here ！ OK

@breadoven I have read #8950

It's closed by @fermiums22, but I didn't get the point of "Inav 6.1.0 Landing after RTH work is good". Maybe @fermiums22 was using RTH to land! (Well, I now know the answer, @fermiums22 Thanks.)

What if in my case (gps glitch), oh no~~~

Two issues have a similar data trend(just forget about vibration of my quad):

It was found in the log that the gyro raw data is inconsistent with gyro filtered data in the same situation(#8950 & #9184), which you (@breadoven ) have mentioned data consistency in #8950

[lida2003]

Looking at the PID numbers they suddenly jump in the space of 1 loop from reasonable values to values that indicate an issue with a 32 bit overflow:

Humm... PID numbers and gyro are both strange data. Memory out of bounds or pointer issues??? In my case, it's 100% replicable.

Why it happens during a FS Emergency Landing is a bit of a mystery. It doesn't happen with a manually triggered Emergency Landing with no FS.

@breadoven What do you mean by "manually triggered Emergency Landing"? I think RC channel switch for FAILSAFE (configured to land) should be called manual trigger, is there any ohter way?

[breadoven]

@breadoven What do you mean by "manually triggered Emergency Landing"? I think RC channel switch for FAILSAFE (configured to land) should be called manual trigger, is there any ohter way?

You can manually trigger an Emergency Landing by rapidly toggling the PosHold mode switch 5 times. Really just intended as a last resort function. Still needs to be mentioned somewhere in the documentation.

#8686

I'd reduce the default value of nav_emerg_landing_speed if testing emergency landing. The default 5m/s is quite a bang into the ground, probably OK to get down quickly in a real emergency but too high for testing.

[lida2003]

Thanks @breadoven .

I get 5 times posHold for triggering emergency landing in checkManualEmergencyLandingControl. And it's a good idea for lowing nav_emerg_landing_speed to 200 (2m/s) for testing emergency landing.

I found there is no big difference between poshold/failsafe triggering emergency landing, so I don't want to do emergency landing test any more, unless we have found some clues.

Currently, following fw version has been tested, I'm NOT sure if it's fw version related (API data type, such as double/float/int32 etc.).

• Mar 26 INAV 6.0.0 (7e7efe4) IFLIGHT_BLITZ_F7_PRO @fermiums22 's FC
• Jun 14 INAV 6.1.1 (42ced25) AOCODARCH7DUAL @lida2003 's FC

PS: I hope there is a tutorial (which I didn't notice before) for RealFlight SITL simulation setup? #9187 .

[fermiums22]

Sorry, I don’t quite understand where this request came from, I somehow collected an inav for a f405 controller, but I didn't push it to git, it’s some kind of error(> * Mar 26 INAV 6.0.0 ([7e7efe4]) =)

Thanks @breadoven .

I get 5 times posHold for triggering emergency landing in checkManualEmergencyLandingControl. And it's a good idea for lowing nav_emerg_landing_speed to 200 (2m/s) for testing emergency landing.

I found there is no big difference between poshold/failsafe triggering emergency landing, so I don't want to do emergency landing test any more, unless we have found some clues.

Currently, following fw version has been tested, I'm NOT sure if it's fm version related (API data type, such as double/float/int32 etc.).

• Mar 26 INAV 6.0.0 (7e7efe4) IFLIGHT_BLITZ_F7_PRO @fermiums22 's FC
• Jun 14 INAV 6.1.1 (42ced25) AOCODARCH7DUAL @lida2003 's FC

PS: I hope there is a tutorial (which I didn't notice before) for RealFlight SITL simulation setup? #9187 .

[lida2003]

Mar 26 INAV 6.0.0 (7e7efe4) IFLIGHT_BLITZ_F7_PRO

This is from your log file.

Since tumbled crash of emergency landing is 100% replicable issue in my case. I suspect the issue is due to some git code changes.

@breadoven Which version did you use for manul emergency landing?

[lida2003]

I also noticed a behavior when GPS and RC signal lost in #8673

* Drone armed without gps fix (motors working at 10%)

* Drone are on the ground (not fly)

* Power off radio

* Motors are spinning up after failsafe (I hold it by hand on the ground), sometimes are only 2 motors are spinning (in normal flight will be flip). I feel on hand that is no a landing mode. Throttle had very high values.

sometimes are only 2 motors are spinning (in normal flight will be flip).

GPS lost + RC link fail triggered motor spin issue (sometimes 2 motors???).

[breadoven]

Are you able to test multirotors with SITL @shota3527 ? Any chance of checking failsafe emergency landing if you can ? Be interesting to see if it's possible replicate this issue with SITL, try and work out what's happening.

[lida2003]

Yeah! @shota3527 Is there any reference about how to setup sitl for multicopter?

Since @breadoven has checked in plane, #9167 (comment)

Hope there is some guide for #9187

[shota3527]

Yes, I am able to test the multi rotor in sitl with realflight.
I have already triggered some sudden loss of trust in emergency landing in my vtol development which might be the same issue as this one. But I forget how to reproduce it.

I figured out how to test multi rotor in sitl by my self and I have to say the instructions is not enough now.
Should I make a pull request to improve sitl documents? I am not good at writing documents

First download the archive/model from here:
https://github.com/ArduPilot/SITL_Models/tree/master/RealFlight
And import the archive file in the realflight
A multi rotor of this have 4 "serial input", map m01 m02 m03 m04 to them on the sitl bootup

[shota3527]

INAV_7.0.0_cli_SIMQUAD_20230807_222925.txt

I will share my cli file for a quickest setup
1234 need to be mapped to m1234 on the sitl boot.
The motor3 and motor4 trust is reduced in mmix because realflight model seems to have a very unbalanced trust

[lida2003]

@shota3527 Thanks for the time and effort, and quick response.

Yes, I am able to test the multi rotor in sitl with realflight.
I have already triggered some sudden loss of trust in emergency landing in my vtol development which might be the same issue as this one. But I forget how to reproduce it.

Good news for locating the issue.

I figured out how to test multi rotor in sitl by my self and I have to say the instructions is not enough now.
Should I make a pull request to improve sitl documents? I am not good at writing documents

Please if you have time. I think a 101 lessons or step by step commands will be more than enough.
If there is a step by step ops, I will be glad to try and verify the process.

If you don't have time, just kick in commands here then I'll try.

First download the archive/model from here:
https://github.com/ArduPilot/SITL_Models/tree/master/RealFlight
And import the archive file in the realflight
A multi rotor of this have 4 "serial input", map m01 m02 m03 m04 to them on the sitl bootup

Is this the steps to simulate?

a) First download the archive/model from here: https://github.com/ArduPilot/SITL_Models/tree/master/RealFlight
b) And import the archive file in the realflight, which is a multi rotor of this have 4 "serial input", map m01 m02 m03 m04 to them on the sitl bootup

Any further steps that we should do?

[shota3527]

@lida2003 @breadoven
I have just made a video to show how to use sitl with realflight in wsl
https://youtu.be/9inhWvQCuHk

[Diver6691]

Hi everyone ,
Recently I flashed my SpeedyBee F7 V3 flight controller with a custom firmware in order to have servo functionality and tilt the camera of my Chimera 7 built.
Everything was working fine (RTH , PosHold) and I decided to test emergency landing (descend rate 5 Km/h)
After I triggered the procedure with 5 fast posHold on-off , the drone freaked out. The damage was massive .
Unfortunately FC’s black box is not working properly and I can’t post any data from there.
But I have some data from the telemetry, if this is something that can help.
I manage to recover the last 20 seconds from my goggles, using a smart phone’s camera.
One more thing that I forgot to mention . When emergency landing was triggered , I heard the motors spinning rapidly . You can see the amperage climbing in the telemetry log at the end of the file
https://youtu.be/eTM4M1S0BjA
chimera 7-2023-08-06-065441.csv

Edit..... If you need any additional information , please tell me to post them. Thanks

[breadoven]

Can you provide a Diff file @Diver6691.

So this happened when there was no failsafe active ? Was it using surface terrain following by any chance ?

I did test emergency landing using the Poshold switch and also triggering it using the Multifunction trigger (not yet merged) and it seemed to work OK ... certainly didn't end up in a death spin which is what seems to happen when it goes wrong. In my case the motors dropped in speed to start the descent (similar to start of RTH landing) and controlled normally thereafter. Not obvious why this seems to only affect emergency landing and not other modes.

Edit: I just found the DVR video of the Emergency Landing test from march of this year. Emergency Landing was started using the Multifunction trigger and it behaved as expected, descending at 1m/s and also held position.

[Diver6691]

INAV_6.1.1_cli_CHIMERA_7_20230809_141140.txt

It was triggered manually , no surface terrain following.

Is it possible that the changes in custom firmware caused this ?

target.c.txt

[breadoven]

The problem as mentioned before seems to be related to the filtered gyro rates changing abruptly to strange values that don't seem to correlate to the raw gyro values as can be seen in the following BB screenshot for the log given above:

Filtered gyro values jump from single figures to 37 degs/s when the throttle drops to 0, then shoot up to nonsense values of 1780 deg/s shortly after the throttle increases up to 57% before they drop back down to single figures again ... causing PID meltdown.

This doesn't happen when switching into Althold even though throttle behaviour is the same, drops to 0 before shooting back up to 57% as shown in following screenshot. Bit baffling as to why Emergency Landing causes problems with the filtered gyro values compared to Althold given there's no obvious connection. @DzikuVx ?

There are differences in filter related settings between the quad I used to test Emergency Landing in March this year and those for @Diver6691 and @lida2003 so maybe something here is causing an issue. My quad used defaults for these settings and has MPU6000 hardware.

Diver6691 and lida2003 settings:
set gyro_main_lpf_hz = 110
set gyro_main_lpf_type = PT1
set gyro_use_dyn_lpf = ON
set gyro_dyn_lpf_min_hz = 85
set gyro_dyn_lpf_max_hz = 300
set gyro_dyn_lpf_curve_expo = 3
set dynamic_gyro_notch_q = 250
set dynamic_gyro_notch_min_hz = 100
set dynamic_gyro_notch_mode = 3D (= 2D lida2003)
set setpoint_kalman_q = 200
set acc_hardware = BMI270

[breadoven]

Well I did find one "bug", if you want to call it that, related to #7845.

This changed the calculation reference for posControl.rcAdjustment[THROTTLE] but didn't update the value from 0 to currentBatteryProfile->nav.mc.hover_throttle in resetMulticopterAltitudeController. This results in a throttle of 0 rather than the previous value of Hover throttle in the places where now rcCommand[THROTTLE] = posControl.rcAdjustment[THROTTLE].

Still doesn't explain why this might be a problem during an Emergency Landing rather than Althold but needs fixing anyway.

[Jetrell]

With my settings, I haven't yet been able to replicate this issue, even with MC Cruise mode.

I was wonder how gyro_use_dyn_lpf = ON works if the throttle becomes zero. Does it also turn off the filter ?
If it does, and the altitude target changes as emergency land is triggered. Then a mechanically noisy quad could loose its gyro filtering, producing a difference between baro altitude and estimated altitude, causing the skyrocket effect.
But in this case it maybe just enough for the motors on the low bank side to spin-up for a split second and start the spin.

[DzikuVx]

@Jetrell interesting... there is no constrain on dynamicLpf, so maybe this is indeed related

[DzikuVx]

The quad was indeed using set gyro_use_dyn_lpf = ON
I will have to retest it after I rebuild it. There is some damage that prohibits this thing from flying right now

[DzikuVx]

OK, I did the math, dynlpf in such case fucks up filtering. I will make a fix to master

[DzikuVx]

#9242

[Jetrell]

OK, I did the math, dynlpf in such case fucks up filtering. I will make a fix to master

@DzikuVx Thanks for the fix.

I think the only way to test the gyro_use_dyn_lpf constraint, will be to try it on a MC thats a little noisy... i.e. It can't get by, without filtering... My test quads are well balanced and don't experience sensor fusion issues without filtering, hence why I couldn't reproduce it.
And I don't imagine the SITL can simulate gyro noise.

[fermiums22]

It's been about half a year since my first crash on this issue. After dynamic filter bug fix, did the fix "mixer stile landing" problem or have not been able to test yet? Is it possible to further work in the landing code using lidar?

[lida2003]

@fermiums22 Sit tight and keep tuned, I think the latest stable version is NOT good for us. And there is a progress of the bug, which have been verified by developer and they are working on this. I hope 7.0 stable release will include this. I have switched to Ardupilot for long range fly, which is safe.

[lida2003]

I have saw discussion in Multirotor course hold/cruise mode #9213.

Well, I don't know details about the code, thought discussion I have found desciptions like below:

• gyro filtering
• race condition (attitude control and gyro filtering)
• flight mode transitions

So I wanna ask if we have found the root case for this issue, and if "Add constrain for DynLPF computation #9242" ultimate solve the "Emergency landing tumbled crash? -- inav 6.1.1#9184 "?

[Jetrell]

So I wanna ask if we have found the root case for this issue, and if "Add constrain for DynLPF computation #9242" ultimate solve the "Emergency landing tumbled crash? -- inav 6.1.1#9184 "?

It would appear that it has been fixed, from limited testing... I haven't personally experienced it with the Dynamic LPF turned on.. But that may not be totally conclusive..
If you're willing. You could do another close to the ground landing test. As you did in the park before. When you originally confirmed the issue.
You would have to flash a copy of the current master, that includes the fix.. Or even just run a test using 6.1.1 But have the gyro_use_dyn_lpf = OFF.. This will also confirm it was the cause.

I know this may not be the answer you were looking for.. But the more tests that can be done, the better. Because not all quads are built equal.. With some requiring more filtering than others.. Which in turn made the landing bug more pronounced on less than perfect builds.

[lida2003]

@Jetrell OK, thanks for the reply.

Now is the Asian Games, during which all aircraft are prohibited from flying. If possible, could you help me confirm which patches based on 6.1.1 can be tested?

I estimate that the relevant verification can be conducted after October 29th.

PS: Currently, I'm stuck using TX12 RC contorller "How to confiugre TX12 (openTx 2.3.11)". I hope that I can do simulation(realflight) before October 29th.

--
Patches for test:

1. 6.1.1 inav release
2. Add constrain for DynLPF computation #9242

[lida2003]

@shota3527

I did TX12 configuration and got Quadcopter(Classic) fly fine.

When I switch to Quadcopter(Flight Axis)

1. connect simulator

$ ./inav_6.1.1_SITL --path quad.bin --sim=rf --chanmap=M01-01,M02-02,M03-03,M04-04 --simip=192.168.68.52

inav_6.1.1_SITL: 192.168.68.53
realflight: 192.168.68.52 // Realflight Evolution from steam(latest version)

2. configuration tab: IMU/baro/mag FAKE
3. receiver tab: SIM(SITL)
4. fly with TX12, It just fliped over. // reboot though configurator before fly :)

Any thing I have missed here? BTW, I can't see any movement in receiver tab, when I operate sticks.

PS: Configure OSD later.