Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

xtrack error does not always converge to zero #2650

Closed
magicrub opened this issue Aug 3, 2015 · 48 comments
Closed

xtrack error does not always converge to zero #2650

magicrub opened this issue Aug 3, 2015 · 48 comments

Comments

@magicrub
Copy link
Contributor

magicrub commented Aug 3, 2015

L1 controller xtrack (cross track) error does not always converge to zero. As you can see from this SITL run, this long stretch is about a meter or two off. It will always be a positive error. This is with zero wind and zero turbulence and a totally stock -w sitl plane. @WickedShell has seen much larger xtrack errors.
xtrackerror

@magicrub
Copy link
Contributor Author

magicrub commented Aug 3, 2015

Issue #1538 would come in handy here

@WickedShell
Copy link
Contributor

_crosstrack_error looks to be a valid computation, and the reporting difference should just be the sign (although we might as only compute it the one time and use the value we report in the other calculations).

I'm a bit at a loss as to why this is happening, working to try and reproduce the behavior in SITL on an order of magnitude like I've seen in flight (where the reported xtrack_error was very far from the true value)

@magicrub
Copy link
Contributor Author

@hemav posted a duplicate issue of this (#2684) with some great plots showing a great exaggerated example of the problem. He posted his params, would be great to get that exact mission too so we can try and duplicate the huge xtrack error. Hopefully the problem will be easier to fix with a "more broken" example to examine.

@hemav
Copy link

hemav commented Aug 10, 2015

@magicrub
The mission wasn't anything special, but I'll post it tomorrow. What I want to try now is to port as many of the parameters of the real thing to a SITL simulation without breaking it, and hopefully reproduce the problem in all its glory.

@magicrub
Copy link
Contributor Author

@WickedShell had a huge 25m offset once, which kicked off this whole investigation. At first I suspected a float angle round-off that a double precision would fix since small missions didn't trigger it and large ones did but now we think it's a true logical math error that we're missing. @hemav , can you duplicate this in SITL?

@WickedShell
Copy link
Contributor

I've had bad ones before, the recent 25m one was the first time I discovered that xtrack_error was reported over MAVLink and that the autopilot didn't really know how far off it actually was. Unfortunately I haven't been able to recreate it in SITL yet. (Still struggling to get the mission in question to load correctly).

I've seen very similar results to @hemav's before as well where it's over an entire flight plan. It's also been enough of a problem I've had to increase sidelap factor by 5% to ensure that I have enough sidelap if the lines start to pair.

@magicrub
Copy link
Contributor Author

I wasn't clear before, my example image from SITL, as well as other missions, were having that positive 1m offset. However, I can't get mine to have a large offset like you guys. not sure if that's a param thing or not. I haven't tried tweaking the NAVL1_PERIOD and _DAMP params, would be a good start to see if that changes the behavior

@hemav
Copy link

hemav commented Aug 10, 2015

@magicrub I use 13 and 0.8 on the real thing, while the SITL defaults are 15 and 0.8, but it doesn't seem to change anything notable in the simulation.

I haven't managed to reproduce any big offset in simulation, only around 1 meter (and usually less than that).

@WickedShell
Copy link
Contributor

@hemav Are you flying the exact same mission in SITL as you were in the air? (primarily the same wp's?)

@hemav
Copy link

hemav commented Aug 10, 2015

@WickedShell not yet, I don't have them with me. I'll test with them tomorrow.

@hemav
Copy link

hemav commented Aug 11, 2015

I have run the missions in SITL, with almost very parameter mirrored from the ones that we use on the plane, and I was unable to reproduce any considerable xtrack_error. It is true, however, that it floats around 1 meter positive along the paths (that is, 1 meter to the right). Which is what @magicrub had managed to reproduce. I don't think the parameters that I haven't ported to SITL would increase the error, but I'll post them just in case.

Parameters used in the simulation:
http://pastebin.com/FzKHfkM2

Parameters that differ from the simulation, with the values used in the pixhawk:
http://pastebin.com/eYKa6d2Q

Plot of the SITL xtrack_error:
sitl_1m_xtrack_error

@WickedShell
Copy link
Contributor

So the 25 meter error being reported as 5 meters from my logs has been shown to be related to changing modes from AUTO to AUTO which reset the previous waypoint location. f2c1010 fixes this if changing from auto to auto. Thanks to tridge for finding and fixing that.

I suspect the 1 meter error is a rounding precision problem, which I'm testing here.

@hemav
Copy link

hemav commented Aug 12, 2015

@WickedShell I wonder if that's related to the behaviour I got. How do you know that you're changing from AUTO to AUTO? Could you share the associated .tlog?

@WickedShell
Copy link
Contributor

Tlog is available here http://files.swiftradioplanes.com/2015-07-28%2015-25-36.tlog.gz

In that log the problematic leg showed up as the leg from way point 25 to 26 (or 27 I forget how the turn worked). Mode changes were done on channel 7 for this plane, and if you graph the RC channel you could see it flip between modes. (And modes 1/2/3 were all auto on this plane)

@hemav
Copy link

hemav commented Aug 12, 2015

Ah well, then sadly my problem must have another source, since my mode channel (5) stays at 1503-1504 during the entire auto part.

@WickedShell
Copy link
Contributor

What was your distance between intended flight lines? I was looking through the graphs in your log, and you have a constant error of ~6 meters reported is that a correct assessment of how far off the line the aircraft is? If you look through my log as well I have a similar problem with a 5 meter offset. The mode change problem was addressing the fact that my log showed a time where the estimated error in position, was not matching the expected error at all.

@hemav
Copy link

hemav commented Aug 12, 2015

Yeah, in my case the xtrack_error reported is true to the reality. I thought your 5m error was also caused by the constant mode switches, but if it isn't, I'm glad I'm not alone having that issue.

@magicrub
Copy link
Contributor Author

If the problem is indeed a decimal place round off, should be easy to
detect with longer flight legs. How long are the legs you guys are talking
about?
On Aug 12, 2015 3:44 PM, "hemav" notifications@github.com wrote:

Yeah, in my case the xtrack_error reported is true to the reality. I
thought your 5m error was also caused by the constant mode switches, but if
it isn't, I'm glad I'm not alone having that issue.


Reply to this email directly or view it on GitHub
#2650 (comment)
.

@hemav
Copy link

hemav commented Aug 12, 2015

500 - 700 meters.

Does the SITL use different floating point precision from the embedded version? What about different boards/board versions?

@magicrub
Copy link
Contributor Author

Should be all the same round off in SITL.
On Aug 12, 2015 4:17 PM, "hemav" notifications@github.com wrote:

500 - 700 meters.

Does the SITL use different floating point precision from the embedded
version? What about different controllers 🐗🐗


Reply to this email directly or view it on GitHub
#2650 (comment)
.

@magicrub
Copy link
Contributor Author

@WickedShell bad news, I just re-wrote update_waypoint() and changed everything over to use a newly added vector2d. There is absolutely no float transactions happening in that function and I get the same exact xtrack error of about +1 meter.

See for your self:
https://github.com/Airphrame/ardupilot/tree/master_L1_double

@WickedShell
Copy link
Contributor

Arguably good news, since all build targets support hardware floating
point, but not all of them have double. Haven't gotten to check it out yet
though.
On Sep 18, 2015 10:27 PM, "Tom Pittenger" notifications@github.com wrote:

@WickedShell https://github.com/WickedShell bad news, I just re-wrote
update_waypoint() and changed everything over to use a newly added
vector2d. There is absolutely no float transactions happening in that
function and I get the same exact xtrack error of about +1 meter.

See for your self:
https://github.com/Airphrame/ardupilot/tree/master_L1_double


Reply to this email directly or view it on GitHub
#2650 (comment)
.

@magicrub
Copy link
Contributor Author

Yeah, I suppose it's good. However, now I'm at a loss of what to try next.
Must be up/downstream somewhere. Got any ideas?

I think I'll cherry pick the vector2d stuff just to fill out the math
library a little.
On Sep 18, 2015 10:45 PM, "WickedShell" notifications@github.com wrote:

Arguably good news, since all build targets support hardware floating
point, but not all of them have double. Haven't gotten to check it out yet
though.
On Sep 18, 2015 10:27 PM, "Tom Pittenger" notifications@github.com
wrote:

@WickedShell https://github.com/WickedShell bad news, I just re-wrote
update_waypoint() and changed everything over to use a newly added
vector2d. There is absolutely no float transactions happening in that
function and I get the same exact xtrack error of about +1 meter.

See for your self:
https://github.com/Airphrame/ardupilot/tree/master_L1_double


Reply to this email directly or view it on GitHub
<
https://github.com/diydrones/ardupilot/issues/2650#issuecomment-141622414>
.


Reply to this email directly or view it on GitHub
#2650 (comment)
.

@magicrub
Copy link
Contributor Author

magicrub commented Feb 6, 2017

This bug has been observed by @jmachuca77 in Plane v3.7.0, reopening.

@magicrub magicrub reopened this Feb 6, 2017
@magicrub
Copy link
Contributor Author

magicrub commented Feb 6, 2017

The big issue here is the AHRS_TRIM_X (or y?) can cause this problem if it's not perfectly aligned. Why can't that be detected, accounted for, and saved during flight?

@magicrub
Copy link
Contributor Author

magicrub commented Feb 6, 2017

Here's a log from @jmachuca77 showing the problem:
DwingLog.zip

@magicrub
Copy link
Contributor Author

magicrub commented Feb 6, 2017

problem seems to be sensitive to mission heading: when same mission on same aircraft is rotated 45degs the problem is much less. @jmachuca77 will post logs of this soon

@WickedShell
Copy link
Contributor

Is it possible the L1_I is simply constrained to low in the code? There is no param for it, and we constrain the I term to 0.1 radians or just under 6 degrees.

@jmachuca77
Copy link
Contributor

This is the log of the same area same plane, with the mission rotated about 45 deg.

22.BIN.zip

@jmachuca77
Copy link
Contributor

I ran both missions on SITL and they seem to track correctly. very different from the actual flights.

@magicrub
Copy link
Contributor Author

magicrub commented Feb 8, 2017

Loosening the constraint is one option, although the NTUN.XTi log value will tell you what it's at and I don't think it was near .1. I'll check in the morning.

@WickedShell
Copy link
Contributor

XTi max's at -0.1 for quite a lot of 22.BIN, in 21.BIN it doesn't appar to be a problem, but it is maxed ~ half the time in 22.BIN

@magicrub
Copy link
Contributor Author

magicrub commented Feb 10, 2017 via email

@WickedShell
Copy link
Contributor

WickedShell commented Feb 10, 2017 via email

@jmachuca77
Copy link
Contributor

I have not flown plane in a few weeks but I wanted to bring this issue back. How can we go about reproducing this on the simulator? I tried running the mission I had an issue with on the simulator, but it worked fine there. Nothing like what I saw on the field.

@WickedShell
Copy link
Contributor

@magicrub managed to reproduce it in SITL here all he did was simply offset the trim, which the integrator ran out of authority to correct for. My proposed solution was to simply bump the allowed XTi. However this only showed up as a problem in one of your two logs so I don't think it will fully resolve this. If you can reliably reproduce this in flight then we can see if bumping it is a workable solution.

We should probably throw a AHRS_TRIM_X >= 0.1 as a prearm warning that the autopilot is to far off level and might be unable to correct persistent navigation errors... (Or we just raise the XTi limit behind the scenes or make it a parameter)

@jmachuca77
Copy link
Contributor

But that was before I found this error again. And my AHRS_Trims are all < 0.1, So Im not sure this is the issue.

@WickedShell
Copy link
Contributor

How much less? It will still limit the amount of correction that can happen. If you are seeing XTi max out in a log then its still the limiting factor.

@magicrub
Copy link
Contributor Author

ping @priseborough regarding enforcing AHRS_TRIM_X >= 0.1 as a prearm warning

@ahelten
Copy link

ahelten commented Apr 7, 2021

I am seeing this problem with a skid-steer rover that must be auto-navigated with centimeter precision. The problem is seen with a dev version from only a few weeks ago (e.g. based on/around commit 526065a but with some u-blox HPPOSLLH changes). The issue isn't 100% reproducible and, when it does occur, it isn't the entire mission but is typically several meters (e.g. 5-10).

The rover is typically able to achieve the desired centimeter accuracy except for this occasional navigation offset. In those cases, the rover still tracks the offset line with high accuracy but the mere presence of the offset can lead to collisions with obstacles that are >4cm from the desired track. I’ve seen the offset be as large as 60cm but it is typically closer to +/-10cm.

Regarding the above discussion on this issue:

  1. A comment in the original issue description (from 2015) states the error is always positive but that is not true in my case. I've seen the xtrack error remain "constant" positive and negative (possibly within the same mission but I can't say for certain).
  2. Much of the original discussion concerns large errors to be 25m or 5m but in our case 25cm or even 5cm are considered large errors.

I’m fairly new to ArduPilot but am very interested in learning techniques to troubleshoot this issue. In particular, I'd like to look closer at the internals of the L1 during this navigation offset. Thanks for any help!

@magicrub
Copy link
Contributor Author

magicrub commented Apr 7, 2021

We need to figure out why a known and logged xtrack error is not being actively brought to zero. That's the whole point of the xtrack error: To know how much you need to correct. If it's non-zero then it should be trying to do something about it.

@danvel9
Copy link

danvel9 commented Apr 23, 2021

Have you seen the investigation carried out by LiamClarkZA here?:
https://discuss.ardupilot.org/t/issue-with-constant-xtrack-error/66595/4?u=daniel_fernandez
He's found that if certain function is not called often enough the XTintegrator is reset, and thus it always shows very small (close to zero) values.
I guess this could be hardware dependent and maybe it does not happen in the SIL.

@magicrub
Copy link
Contributor Author

Seems this commit solves the problem. Would be nice to get that into a release
fd9068d

@magicrub
Copy link
Contributor Author

Here's PR #17262 to cherry pick that commit into the next 4.0.x release

@peterbarker
Copy link
Contributor

There almost certainly won't be a another 4.0.x release, and I doubt we'd pull the cross-track fix back if we did. Should've a long time ago, probably, but....

I'm closing this as "fixed" - even 'though people will need to run the beta to get the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

8 participants