Does upgrading 3.7 to 3.8, downgrading back to 3.7, and upgrading back to 3.8 lose servo REV values?

[marcmerlin]

@magicrub after filing #6510 , I fixed up the aircraft from that crash, did a full 3 axis recalibration,
I then actuated FBWA on the ground, made sure that the plane was keeping itself level.
I think I also tried auto to see whether the aircraft actuated the ailerons and the elevator to have the proper takeoff pitch.
It did, but I didn't trust it enough to do another auto takeoff since I hadn't heard back on my previous bug mentioned above.
So, I took off in manual and flew in manual for a while, everything was fine.

My plan was to do another autotune and I switched from manu to autotune.
The moment I did that the aircraft pitched over, I was not fast enough and hit RTL instead of disabling ATUN, but probably by then recovery would have been very hard given that it when full throttle straight into the ground, nose first.
The aircraft if of course destroyed beyond repair, and sadly some of the avionics were also damaged, the cameras shattered and even the airspeed sensor, broken :(

Video:
https://youtu.be/EDe0IbeDjVE?t=2m27s

00000048_bix2_autotune_crash.bin.gz

In hindsight, I guess I could have tried FBWA first before AUTOTUNE, but the loss of attitude from AUTOTUNE was so violent that even if I were prepared for it and had my aircraft again, I'm not sure if I could recover the 2nd time either.
I suppose next time I'll AUTOTUNE FPV from 300m altitude, but it's not my prefered way of doing it obviously.
Was this a bad bug in AP, or did the pixfalcon fail in some way and cause this to happen?
More importantly, why did everything look fine on the ground when I tried FBWA before takeoff?

[magicrub]

I'll take a look.

[marcmerlin]

Any idea if it was a hardware bug giving bad sensor input, or an ardupilot bug?
I'd love to know if I should burn that pixfalcon or if my problem was a software bug and I should just avoid 3.8 for now (and by that I mean future aircrafts since I've lost all 3 I had , now)

[tridge]

hi Marc, really glad you posted this with the same aircraft as #6510. It allows me to compare manual flight to the RC and SERVO settings.
It really looks like it is as simple as having incorrect servo reversals. When you takeoff in MANUAL I see that you pull the pitch stick in a direction that decreases the PWM value. Assuming you are in fact pulling back on the stick for takeoff, and the fact that the plane climbs out, means that the elevator is reversed.
ArduPilot has always has the convention that a non-reversed elevator means a higher PWM value is up elevator. That means your elevator is reversed, so you need SERVO2_REVERSED=1

the same goes for aileron. At takeoff in MANUAL the plane is rolled left (negative ATT.Roll). I can see you putting in aileron stick input to correct. The correction you put in is to move the aileron to have a smaller PWM value. That means that right aileron is using small PWM values. That means you have reversed aileron, so you need SERVO2_REVERSED=1.

Unless you change your transmitter settings you will also need to change RC1_REVERSED and RC2_REVERSED both to 1.
I think you've been fooled by the common error that testing control direction in MANUAL mode isn't sufficient, or even checking in a stabilised mode with pilot input. You need to check that the plane does correct attitude itself with no pilot input (usually in FBWA mode).
I do hope I haven't mis-read the log!

[tridge]

It also looks like the trims in your transmitter are very different from the RC trims in ArduPilot. For example, it looks like your transmitter RC trim is 1393, but RC2_TRIM is 1500.
Perhaps check the RC calibration too?

[marcmerlin]

@tridge thanks for the look.
First, on RC trim, I had trim in manual mode to compensate for the fact that the plane wasn't flying quite straight due to it not being quite pristine. The RC trim values at 1500 in AP were likely correct for servos being centered while the trim on my RC controller was simply to correct for the non perfect airframe and flight straight with some aileron correction.
For RC2, which is the elevator, the CG on that plane was way off, so I had back elevator to pitch the nose up on level flight, but I didn't want to tell AP that 1393 was the elevator being straight (because it was not, 1500 was straight).

Ok, this is perplexing since I've had this flying for a while without issues and things weren't reversed or it wouldn't have flown without issues for close to a year.
I took the wreckage of the plane on my desk, powered it back on in FBWA mode and moved the pixfalcon. Sure enough the aileron and elevator servos (from the AP side) were reversed :(
At this point I think that in my FBWA level test before takeoff, I focused on whether level was level for the ailerons and elevator while apparently missing that they were going in the wrong direction :(
How they got reversed after being right for a long time, I'm not 100% certain, but I'm going to guess that when I upgraded to AP 3.8 I had to recalibrate with mission planner several times (especially as I went back down from 3.7 and back up to 3.8) and at some point the wrong wizard may have been running, and my RC control settings messed up

Then again, unless mission planner did something bad (with my help, or not), look at this diff from my settings between 3.7 and 3.8.
RCx_REV were all renamed to RCx_REVERSED and all reset to 0
Is there any chance at all that the migration of setting names did something wrong and lost the REV values?
Mmmh, how about this. I went to 3.8, then was asked to flash back to 3.7 to test something, and I went back to 3.8 after that.
Maybe REV got converted to REVERSED on 3.7 -> 3.8 but then got lost when I went back from 3.8 to 3.7, and the 2nd time I went back from 3.7 to 3.8, all my REV/REVERSED values had been lost?

I just had a look at an old and newer parameter file, and found this.
RC1_DZ,30
-RC1_MAX,2006
-RC1_MIN,982
-RC1_REV,-1
+RC1_MAX,1900
+RC1_MIN,1100
+RC1_REVERSED,0
RC1_TRIM,1500
RC2_DZ,30
-RC2_MAX,2006
-RC2_MIN,982
-RC2_REV,-1
-RC2_TRIM,1532
+RC2_MAX,1900
+RC2_MIN,1100
+RC2_REVERSED,0
+RC2_TRIM,1500
RC3_DZ,30
-RC3_MAX,2006
-RC3_MIN,982
-RC3_REV,0
-RC3_TRIM,982
+RC3_MAX,1900
+RC3_MIN,1100
+RC3_REVERSED,0
+RC3_TRIM,1500
RC4_DZ,30
-RC4_MAX,2006
-RC4_MIN,982
-RC4_REV,-1
-RC4_TRIM,1493
+RC4_MAX,1900
+RC4_MIN,1100
+RC4_REVERSED,0
+RC4_TRIM,1500
RC5_DZ,0
-RC5_FUNCTION,18
-RC5_MAX,2006
-RC5_MIN,982
-RC5_REV,-1
+RC5_MAX,1900
+RC5_MIN,1100
+RC5_REVERSED,0

[marcmerlin]

The main reason why I suspect the downgrade as being the issue is:

1. it's not well tested, if at all
2. I had 2 other aircrafts I upgraded from 3.7 to 3.8 and their REV values were not lost (but I don't think I downgraded them to 3.7).

[marcmerlin]

And just to make sure my memory isn't wonky, in #6261 @peterbarker and @WickedShell asked me for GPS init logs on 3.7 vs 3.8. I did indeed downgrade my bixler from 3.8 back to 3.7 to get those GPS logs, and then back up to 3.8.
At this point, I'm starting to be reasonably sure that the 3.8 to 3.7 lost all my reversed values, causing ailerons and elevator to be reversed and the fatal crash that ensued.
All that said, I should have paid much more attention to my pre takeoff FBWA level test and verified that the servos were moving in the right direction. I totally failed there :(

[tridge]

I'm sort of glad it was reversal, as that is better than it being a insta-crash bug!

At this point, I'm starting to be reasonably sure that the 3.8 to 3.7 lost all my reversed values, causing ailerons and elevator to be reversed and the fatal crash that ensued.

the upgrade code for the parameters from 3.7 to 3.8 is very complex, so there may be scenarios where it gets it wrong. It does do the right thing for most cases, but really all I can do is stress the need for careful ground tests when people upgrade.

All that said, I should have paid much more attention to my pre takeoff FBWA level test and verified that the servos were moving in the right direction. I totally failed there :(

it happens to everyone sometime!

[marcmerlin]

So, the release notes should have a warning that people recheck their RC* settings after upgrade given the new SERVO* variables that were added, and a bigger warning that if they downgrade back to 3.7, their RC settings will likely be wrong, up to inverted servos and crashes.

[OXINARF]

The reversed setting shouldn't be affected in any way, we went with a new index (given the new name and values), so 3.8 doesn't override or change the setting in 3.7. Now, like @tridge said it is possible something is wrong in the upgrade code - but that is hard to look at without a defined set of steps.

I completely agree that release notes should have a warning about people checking their parameters.

Can we close this? Given this was an unfortunate configuration error I think there is nothing left to do.

[marcmerlin]

@OXINARF we can close this as long as you get my point that the bug is likely on a 3.8 to 3.7 downgrade. The upgrade works fine from what I can tell, the downgrade is what I'm pretty certain caused my issue.
As you said a warning in the release notes is good, but it should spell out specifically that RC settings get potentially lost on downgrade and that someone should save their parameters in pre 3.8 before upgrading to 3.8 in case they want to revert.
Once you've seen this, feel free to close this issue :)

[juvinski]

Hi,

I tried the downgrade from 3.8 to 3.7.1 and I'm having the same issue with servos. I'm using bbbmini and this is the steps I made.
First, I move the folder /var/APM calling /var/APM371
Installed from scratch Plane 3.8 and made all calibration process. A new folder /var/APM was created.
To downgrade I simply move /var/APM to /var/APM38 and move back /var/APM 371 to /var/APM
On each move operation I'm rebooting the linux to ensure there was no open files, and for my surprise (with previous configuration folder 371) the servos are reversed. I tried a 3.7.1 installations and configuration from scratch and even with this scenarios the servos are reversed - If I put back folder /var/APM 38 to /var/APM the servos are fine.
I really need to downgrade to 3.7.1 and I don't know if someone has any clue what can be happening.

Regards

[marcmerlin]

Thanks @juvinski for confirming that downgrades are basically dangerous and should be avoided.
@OXINARF @tridge please make sure you have a very clear warning about not downgrading from 3.8 without a backup of all settings from 3.7, or other people will also get crashes

[juvinski]

Just an update.
On 3.7.1 I was using ELEVON_OUTPUT = 3, on 3.8 was with the same value.
After removing the folder with configuration and running 3.7.1 with fresh configuration I had to use ELEVON_OUTPUT with 1 to work again. I will test tomorow.

[WickedShell]

This is a doc problem, opened an item on the wiki to track the warning: ArduPilot/ardupilot_wiki#999