Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Improved Lat/Long precision and accuracy #13

merged 1 commit into from Aug 7, 2014


None yet
4 participants

koehn commented Feb 26, 2013

I patched the base code to use longs rather than floats for lat/long, and to put them in the correct format (lat/long are in units of 1/10,000,000 of a degree with the minutes converted to 1/100,000 of a degree). Using a common integer format is more accurate, and using longs instead of floats is both faster and more precise. I'm uploading some geo-fencing code to GitHub that uses this version of the library and is much faster and extremely precise with WAAS enabled (within 3m).

For example, the library would return a lat/long of approximately 407,098,514N / 740,079,168W for AdaFruit HQ.

Any reasons why this patch returns strange lat/lng values?

Here's info from U-Center
Parameter Value Unit Description
Lat 5434.51664 ddmm.mmmm Latitude
Lon 02511.38718 dddmm.mmmm Longitude

and here's the patched result:
Location: 200201320110031N, 33000321032121E

This patch works perfectly for me.

@tdicola tdicola merged commit 7a3956f into adafruit:master Aug 7, 2014


tdicola commented Aug 7, 2014

Thanks for submitting the pull request and apologies it took so long to be reviewed. It looks like a great way to switch the logic to fixed point arithmetic for better speed and precision.

One thing that could be a problem is switching the latitude and longitude values from floating point to fixed point. Like flegmatoid saw with the parse sketch (and probably other sketches) there are some side effects of calls like print having their expected type switched from float to int--in this case the print call now sends the precision value of 4 as the int printing enum and has some unexpected behavior.

The easiest fix I think is to put the new fixed point value in latitude_fixed and longitude_fixed members, and then use those to calculate the floating point value in the original latitude and longitude members. I updated the pull with this behavior, gave it a few tests with the examples, and merged it in. Thanks again for submitting the change!

tdicola added a commit that referenced this pull request Aug 8, 2014

Remove scaling from minute calulcation.
Pull #13 added scaling of minute values by 50 / 3, but this seems to cause problems like minute values going over 60.  Removing this calculation to take the exact minute value given by the module.

tdicola commented Aug 8, 2014

Actually it looks like the minute calculation has some unexpected behavior. Someone noticed a string longitude/latitude value from the module like 3051.8300 is getting interpreted as 86.383 minutes, which is over 60 and unexpected. The scaling of minutes by 50 / 3 seems to cause the problem--is there a reason minutes were scaled by this constant?

To be safe for now I backed out the scaling and just multiply minutes by 10 so they fit in the fixed point value without any change.


koehn commented Aug 10, 2014

The minutes are converted to decimal fractions of a degree, thus minutes 0..59 mapped to 0..983. If you don't convert degrees/minutes/seconds into a single common unit (1/10,000,000 of a degree) then you can't do any math on them.


tdicola commented Aug 11, 2014

Ahh, that makes sense. Looks like I need to do a little more conversion when I convert it to a floating point value. Will take a look at fixing it tomorrow and adding it back in--thanks for clarifying!


tdicola commented Aug 11, 2014

Just merged the minute scaling back in to what the pull originally had. Fixed up the float conversion so it should be correct too. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment