Serious problems when upgrading from INAV 7 to INAV 9 !

[dzaro-dev]

Current Behavior

When going into any autonomous flight mode after updating from INAV7 to INAV9, the plane does not fly as it should. It's almost like a ghost is flying when there no stick input. I've done 4 flights, two with INAV7 and two after updating to INAV9. Only the first two flight are relevent though.
The video is long, but it covers everything that I thought would be necessary:

https://youtu.be/2iv1w4tqykU

You might want to skip the third and fourth flight of the video as I'm only testing to see if stock PIDs can fix these issues and if going back to INAV 7 worked fine (as well as re-tuning the plane).

Steps to Reproduce

1. Take an INAV 7 diff and modify it so that you can paste it into INAV 9 CLI.
2. Paste the diff into a freshly flashed MATEK F765 running INAV 9.0.1.
3. Fly the plane in Acro and notice the OSD elements not refreshing in time.
4. Put the plane in any mode like CRUISE or RTH and notice how it starts going all over the place.

Expected behavior

Obviously, this should not happen! If it let the RTH continue for example, the plane would have crashed in a matter of minutes. If you look at the last part of my YouTube video where I show the goggles recording with the BBL data overlayed, you can see that some strange things are happening in autonomous flight modes.

Additional context

Link to original goggles recordings and BB logs are here:
https://drive.google.com/drive/folders/1fSlFeH6ZTXIODtx9AC2-qF4-Vw7wO1s3?usp=sharing

---

• FC Board name and vendor: MATEK F765
• INAV version string: 7.1.2 and 9.0.1

[breadoven]

Looking at the V9 Log there's an obvious problem with the Yaw angle and the Course over Ground. They won't be exactly the same, diverging as cross wind speed increases, but they should be similar which they aren't in this case. Having said that Cruise uses ground course to control direction with ground course being derived from navVel[0] and navVel[1]. Checking these velocities shows they roughly match GPS ground course so it's not clear why it fails to control to the desired heading navTgtHdg. It doesn't get close, all over the place. Needs further checking to understand what's causing this and also why the Yaw angle is nonsense.

[breadoven]

Bit of an odd one. During Cruise the plane seems to be setting roll control input, rcCommand[roll], correctly to achieve a turn in the right direction towards navTgtHdg. But in spite of the plane banking in the right direction and ground course initially behaving as expected this stops when within around 30 - 40 degs of the desired heading with ground course suddenly moving in the wrong direction, opposite to the bank angle in fact, e.g. right roll commanded but ground course moves anti clockwise. The ground course error builds up in the wrong direction until it triggers a turn reversal causing the plane to bank in the opposite direction where initially it starts to head in the right direction again but eventually stops before achieving the desired heading and heads in the wrong direction again leading to another turn reversal when the error gets too large. This cycle then repeats.

Doesn't make sense that ground course would head in the right direction initially then move in the wrong direction when close to the desired heading when the roll command and bank angle are still rolling in the right direction to achieve the desired heading. There seems to be a bit of wind during the flight but it doesn't appear strong enough to cause these problems.

[b14ckyy]

What I noticed in the log comparison between both.
When in INAV 7 the stick (raw RC) is centered, the TrgtHdg updates accordingly to lock in to the new heading.
But in INAV 9 it does not. The TrgtHdg is completely stuck and not updating after manual turns and then the plane tries to hunt it but goes all over the place. Rates look good though so the plane movement is actually commended but the commands from the nav controller don't make any sense.

I asked him to do another flight on 9 and remove CRSH and SOAR modes from his setup to see if it still happens. the DIFF does not look crazy except that messed up YAW rate but yaw setpoint is 0 the whole time anyway (as far as I checked). So should not cause an issue.

[Jetrell]

@dzaro-dev I also suggest you look over the 9.0 release notes.
You haven't adjusted the required fw altitude PID's. Nor added any tuning for the dynamic PID attenuation.. A fair bit has changed between those two releases.

[breadoven]

I think these problems are wind related. The wind estimator is showing windspeeds of 30+ km/h in the V9 video when the plane is only cruising at 40-60 km/h which makes the wind significant whereas wind speeds during the V7 flight seem much lower. The problematic Cruise mode courses seem to be crosswind (if you believe the wind estimator). So the plane is banking in the right direction to correct the course error but even with the max 35 degs bank it can't turn enough to counter the cross wind and I'm thinking that the cross wind drift gets worse the more bank you apply given it exposes more of that large wing area on a Ranger to the wind. It's the only thing that explains why the GPS ground course drifts off in the wrong direction even though the plane is banked hard over in the right direction to correct the course error.

One thing that would probably help significantly would be to enable nav_use_fw_yaw_control. I've always found turns a lot more effective on these planes when rudder is used. It will have been used to an extent because Turn Assistance was active but using Yaw control should be much more effective again.

[b14ckyy]

@breadoven but what about the stuck Target Heading in course hold or cruise mode? That never changes after stick center like it does in INAV 7?
In one part of the flight the plane flew the opposite direction of the target heading and was snaking 90° left/right/left instead of turning around to the target heading either.

[breadoven]

Target Heading navTgtHdg seems to work as expected on both versions during Cruise. It only updates when the stick is centred which it does do in the V9 flight. Perhaps it's not so obvious because the course was only adjusted a couple of times.

The reason the plane is flying around all over the place and in the completely wrong direction at times is because the turn reverses when the course error exceeds 170 degs. So it flips around and tries to turn in the opposite direction to try and get back on course by turning the other way as it were.

There are some odd things going on with the V9 flight in general though especially with the GPS. The GPS speed during launch makes little sense and it's why the speed (and also altitude) shown in the video gets stuck at 0 for 10s or so even though the plane is clearly already up and flying. The GPS simply doesn't update correctly for 10s after launch even though the Baro does. Does make you wonder how reliable the GPS is in this case. Also makes you wonder how reliable the wind estimations are which hit >40 km/h at times toward the end of the flight.

[Jetrell]

Also makes you wonder how reliable the wind estimations are which hit >40 km/h at times toward the end of the flight.

I thought the same. Surely the wind wasn't that high. And he said here in the video, that we don't have such a windy day today.
But being that the model has this much gyro noise. It was likely adding to the wind estimator providing a bad reading.

[breadoven]

He does also say at around 14 mins in to that video just before launch with V9 loaded that the wind is a "bit" stronger. The problem does seem to be GPS related in some way as mentioned above which is very obvious in the log. It seems to be delayed in it's response by many seconds at launch. If the same issue was happening throughout flight none of the Nav stuff will work properly.

[b14ckyy]

@breadoven I just doublechecked and yeah the Target Heading does indeed change. It just confused me that it was all over the place.
But I just chatted with @dzaro-dev and he pointed me at the serial baud rate for GPS that is set to 9600. because thats printed on his M8Q from Matek as the default and he thought it has to be set like that.

I wonder if that could be the issue. before INAV 8 we did not check what update rate the GPS is capable of and UBLOX was always limited to 5Hz and UBLOX7 only allowed 10Hz with 1 or 2 GNSS. He runs 3 GNSS according to settings.

I don't know if we can see the actual GPS update rate in the logs. But a too low baud rate for the GPS could cause GPS data to pile up in the GPS serial output right? Depending on its buffer size? That would explain the delay and maybe even jumps in speed or position if the buffer overflows and gets wiped.

[breadoven]

@b14ckyy the GPS Lat and Lon values are changing slightly during the launch so something is getting through from the GPS but the changes are tiny, just wandering around the launch point position even though the plane is long gone. The GPS Altitude remained stuck at the Arm value until >13s after the plane was launched at which point it shot up all of a sudden. Maybe @sensei-hacker would have some idea, he seems to be the GPS expert.

I guess if the GPS is unreliable the GPS ground course will be inaccurate and also the position estimator ground course since it's calculated from x, y velocities which fundamentally come from the GPS.

[b14ckyy]

Yep exactly.
So I was wondering... at 5 or 10 Hz how much data would the GPS actually send to the FC? Is it possible that there is a buffer big enough to buffer ~13s of data if the throughput over serial is insufficient?
@dzaro-dev also told me that sometimes he gets GPS HW failure on/off in intervals and has to power cycle the plane to make it work without error. And sometimes even gets GPS sensor lost warning on his radio on INAV 9.

Lets wait until he does another test flight, to confirm that as the root cause.

If that's really the case that the UBLOX messages are buffered for so long, maybe a sanity check would be an option. The FC gets the GPS time precisely as soon as the first satellite data are there. Maybe the GPS timestamps could be compared to the system time of the FC and if there is a discrepancy over a certain threshold, a system message (GPS unreliable) could be shown. Could technically also reduce the GPS update rate on the fly.