Analysis920XT
- Background
- Devices
- Recording
- The Trail
- Hints
- Analysis - Position Based - Time Based Complete Lap - Time Based KM Post - Detailed Track Differences
- Results - Complete Laps - Manual km Splits - Activities - Detailed Track Differences
I got a Garmin ForeRunner 920XT and was not too impressed by the distance calculations. The device functionality is good, but the position seem hard filtered to give distance calculations close to ideal in good conditions. On trails and in worse weather conditions. The distance calculation always seem to be short compared to the Garmin 305, but that is not necessarily an error. What seem worse is that variation was relative big.
I set out to test the accuracy with 4 devices:
- Garmin 920XT. Firmware both 5.20 and 6.10, Glonass on, 4 activities with Smart recording and 3 with per second recording on (should only affect GPS distance, not Garmin calculations.) Worn on left arm.
- GlobalSat 625XT. SirfStar IV based with high position accuracy, but less good filtering (and general software). Worn on left arm.
- Garmin ForeRunner 305. Failing battery, could not record all laps, not record at all for last two activities. All results are not fair to the 305 due to limited soak time. Worn on right arm.
- Sony Xperia Z3 with IpBike. Per second recording without enough filtering (a basic GPS filter improves the accuracy). The device was worn in the windbreaker pocket, less ideal than the arm. Included as a reference, but not intended as a pure sports device.
Recording was done on 7+ occasions with 5 times. The GPS was locked for a couple of minutes on the 920XT so other devices was able to be stable too. The (running) transportation to/from the start were recorded as separate activities. The exception is the 305, that got a lock, then was turned off before starting recording at the trail again (so less time to stabilze, this can be seen in the activities).
In each recording the lap times were marked with a lap on the 920XT, pauses were set on the 920XT. The other devices mostly had a free running timer, some including the transportation and pauses. (I did an attempt to put laps on the kilometer postings a few times too, but the lap markingss were not exact enough.) The 920XT activities were imported as My Activities, the other as My Friend's Activities.
The following measurement errors are expected:
- Incorrect lap timing. Maximum a few meters. In one situation stop was pressed instead of lap, adjusted after, in another the 920XT was not lapped (but another device).
- Two dimensional distance vs elevation. If the climb is 80m and all climb in one km and all downhill in another, the reported distance is 7m too short. Do not affect the variance though.
- Different path taken when meeting/passing other runners. Cannot be many meters, the track was not crowded.
- Soak time. A problem for the 305 due to the failing battery (including some 2011 runs to compare with). As the 920XT has no detailed GPS accuracy reporting, I cannot say how the 920XT was affected.
Other effects:
- Weather - no rain but one activity had rain.
- Device position: Left arm vs right arm vs pocket. (The 920XT and 625XT on the same arm though.)
I have done some tests on various trails, but there are too many factors to take in to account comparing. The reported tests was done on a trail "Femman" in Skatås, Göteborg, Sweden. The trail has been controlled measured by a federation official with a measurement wheel to 4763m. I have also seen 4820m and 4845m reported as an official result (the 4845m is posted in the trail centre). The exact length is not really the point here anyway, it is the variation.
The trail is mostly sidelined with a mix of spruce and birches. No leaves, no dense cover. Weather mostly cloudy but no rain, relative similar.
A hand edited trail on top of an outdoor map (gpx shows that the outdoor map very closely floows the shortest possible path. After separating start/stop 10m, the GPS distance in ST is 4,76 km, Garmin Connect 4.74 km.
There is a Garmin Connect Segment 5:an Från Klockan medsols for the trail. Note that only 16 of 35 laps (in 4 out of 7 activities) matches the trail at all. The segment length is listed as 4.74 km.
The Strava Segment Skatås - 5:an is measured as 4.8 km, about 25 m too short (there is a similar Skatås - 5:an where the start/end differ another 200m). Strava matches 30 of 35 laps for the 4.8 km as the start position for the segment is 15 m after the "watch". The 920XT has occasionally some additional positioning error and Strava do not pick up all activities (this is why the Split based analysis is more accurate). The 4.6 km segment matches all 35 laps.
Some hints when working with the list:
- A trail was created to identify the trail in question. The detection must not be perfect.
- Possibly delete unwanted results by selecting and pressing Delete (or from context menu)
- If analysing many or big results, see After setting up the trail, set map to None or some other service that is not caching using a browser (so no Google sevice). Google maps slows down performance considerably.
- Select the remaining activities as results in trails, for instance by using the Splits trail and Shift select them. Shift-Ctrl-g to get a popup to set the import source for the selected results.
- Select all actities with the mouse or Ctrl-a.
- You can save selected results with Shift-Alt-g, select the results again with Alt-g or if selection has changed, use Shift-Ctrl-Alt-g to reinsert the activities and results again.
- Set Standard deviation for Summary in the Trail menu (the triangle right of the trail selector) to show how much the results differs.
After import, the import sources were corrected, to simplify selecting later. (SportTracks does not set the import source when importing from files.) This canbe done in the plugin.
- Add the Activity - Import Source column in the result list
- Double click the Import Source column to set the source for one activity, set it as Reference Result (Right click context menu)
Some manipulation after import to simplify comparing Splits. This is not necessary for comparing normal position based trails, just to use Splits to get a little more precise results. A few split times that were incorrectly set were adjusted, some pauses and presentation were set as rest laps in the SportTracks Splits activity page and Trails Split Trail to find outliers. The number of laps were set to exactly 5 for each lap, also to simplify split-time based analysis.
Step by step how analysis was done.
- Select one of the activities to compare
- Select the trail (may be automatically selected)
- Insert similar activities with keyboard shortcut Ctrl-i or Context menu (right-click list) Insert Activities - Category: My Activities: Running.
- Select activities with same time as the selected activities with Alt-t. This will also set the inserted results to use same pauses as the refererence results, see t related shortcuts. (As the 920XT GPS has its limits, let the inserted results use the time of position rather than the 920XT time that Shift-Alt-t would set).
- Set Use device distances in the Trail menu (the triangle right of the trail selector).
- Select one of the 920XT results (see Import Source column to see which), press Ctrl-g to select all similar results. Ctrl-c (or context menu) to copy results). In this case, it is Distance and standard deviation that is interesting.
- Repeat previous for all devices
- Set Use device distances to false and get results again.
Since the start/stop was marked with a manual lap, it is possible to make a time based analysis too. This improves the accuracy.
- Select the activities to compare.
- This can be done by selecting the limiting trail, inserting activities and limiting the selected activities to the results (context menu).
- Another alternative is to select the activities in Reports view.
- Select Splits trail
- Insert related activities with Alt-t
- Since I am only interested in the complete lap, merge subresults by select last lap (5) for one of the results, then Shift-Ctrl-l.
- To get the summary for all laps instead of activities, use Alt-s.
- Get the summaries similar to above.
For a few activities also laps were taken at km postings. The postings are not exactly matchin the trail, the purpose is to check the variance on separate parts of the trail.
The standard way to perform the analysis is to select the splits one by one. To avoid this, some undocumented features were used.
- The behavior of "Select Similar Splits" is changed from selecting by split number to match displayed number modulo (so paused laps that increases split number are skipped). This is done with Alt-h five times. This means that by clicking lap 1, the first, sixth etc lap isselected for all results.
- Select the base activities, insert the related other sources.
- Delete all results except one (reinsert the activities to repeat for other sources).
- Select lap to analyze
- Select Similar Splits with Space, to select all related laps. (It is possible to activate the setting too, then it should be deactivated before deselecting).
- Delete laps that are off (late etc) with Ctrl-Click
- Note the distance and variance, possibly screenshots for grouping of end positions.
- Possible change to measure both Device and GPS distance calculations
- Repeat for laps, then activities.
To get detailed information where distance differs, you need a reference track to compare a single track to. This information is more informative, no mathematical analysis.
- Set the number of results to see in Settings. By default limited as the chart handling is slow.
- Select/insert activities
- Select the reference result by right-click on the result and select it. The reference must have high quality to be meaningful and should be for same time.
- Select the result to compare to.
- Select the Diff chart, zoom in to parts that differs the most. View on map.
- Distance diff over time show the best information for results covering the same time. To compare results not over the same time, Difference Time over Distance show better information.
Some brief step to step to compare splits in the previous step:
- Select/insert activities
- Select the reference result by right-click on the result. In this case one of the 625XT.
- Expand one of the results
- Select the top level result to compare (in this case the 920XT). Ctrl-g cannot be used as the reference is a 625XT.
- Use hidden feature to select similar by modulo: Alt-h five times.
- Ctrl-click the split index to compare.
- Press Space to Select Similar Splits. Every fifth result is selected.
- De-select the top level results with Ctrl-click
- Select summary line, to get average chart
- Select Distance X-axis
- Select Difference graph
The summary line will only give meaningful results if the splits are very short and have the same pace.
Distances in km, deviation in meters. Distances are calculated from device or from GPS track. (a few older activities too, gives a fairer results for the 305).
Both Position and Time based.
| Device | Total Laps | Splits, Device | Splits, GPS | Position, Device | Position, GPS |
|---|---|---|---|---|---|
| 920XT | 35 | 4,65 s53,3 | 4,66 s52,6 | 4,63 s55,2 | 4,66 s46,6 |
| 920XT, smart | 20 | 4,66 s52,2 | 4,66 s52,9 | 4,65 s48,8 | 4,66 s43,2 |
| 920XT, 1s | 15 | 4,65 s54,4 | 4,66 s51,9 | 4,60 s48,6 | 4,65 s50,5 |
| 920XT, 1s with 2s/10m filter | 15 | - | 4,64 s50,2 | ||
| 920XT, fw6.10 | 25 | 4,67 s49,7 | 4,67 s48,6 | ||
| GlobalSat 625XT, 3s | 35 | - | 4,88 s30,9 | - | 4,88 s31,9 |
| Sony Z3 IpBike, 1s | 35 | 4,74 s65,0 | 4,84 s75,1 | 4,73 s63,4 | 4,84 s72,9 |
| Sony Z3 IpBike, 1s with 2s/10m filter | 35 | - | 4,74 s64,6 | ||
| GF 305, smart | 16 | 4,79 s41,0 | 4,80 s40,2 | 4,79 s39,1 | 4,80 s39,9 |
| GF 305, smart (2011) | 12 | - | - | - | 4,75 s18,6 |
| GlobalSat 625XT (2011) | 12 | - | - | - | 4,85 s19,0 |
The difference between smart and per-second recording is as expected low, no significant difference. No difference was expected for Garmin calculations, smart should just be a filter to give GPS points similar to the internal calculations. (I therefore would have expected per-second GPS to be slightly higher than is like for the Sony Z3.) Both the 625XT and especially the Z3 would improve by a GPS filter.
The shortest 920XT device distance is 4.52 km, the longest 4.76 km
For the activities at 2015-11-07, 2015-11-08 and 2015-11-15 laps were taken at km posts. The km passings are not exactly matching this trail (the second km has no specific post, just estimated).
The second leg has most turns and the 920 has most problems here, but the 920 has problems on other legs too.
| Split | Count | Device | Splits, Device | Splits, GPS |
|---|---|---|---|---|
| 1 | 14 | 920XT | 0,99 s15,9 | 0,99 s16,4 |
| 625XT | - | 1,03 s4,9 | ||
| Z3 | 1,00 s18,2 | |||
| 2 | 14 | 920XT | 0,96 s23,7 | 0,96 s24,1 |
| 625XT | - | 1,07 s7,8 | ||
| Z3 | 1,00 s44,0 | |||
| 3 | 14 | 920XT | 0,93 s11,8 | 0,94 s21,9 |
| 625XT | - | 1,03 s4,9 | ||
| Z3 | 0,93 s25,9 | |||
| 4 | 13 | 920XT | 0,88 s20,4 | 0,89 s21,2 |
| 625XT | - | 0,93 s6,0 | ||
| Z3 | 0,91 s26,2 | |||
| 5 | 14 | 920XT | 0,90 s10,5 | 0,90 s8,8 |
| 625XT | - | 0,92 s6,5 | ||
| Z3 | 0,90 s18,9 |
The 920XT positions (especially for the 4 km) is a little behind the actual lap point.
Individual 920XT results, just Splits.
| Date | GPS Points | Firmware | Device | GPS |
|---|---|---|---|---|
| 2015-10-25 | 1680 | 5.20 | 4,63 s44,6 | 4,63 s42,4 |
| 2015-11-01 | 8099 | 5.20 | 4,60 s38,4 | 4,61 s37,2 |
| 2015-11-07 | 1647 | 6.10 | 4,67 s80,3 | 4,67 s80,5 |
| 2015-11-08 | 8298 | 6.10 | 4,70 s30,7 | 4,70 s24,3 |
| 2015-11-15 | 8084 | 6.10 | 4,64 s42,0 | 4,65 s43,2 |
| 2015-11-22 | 1642 | 6.10 | 4,68 s7,4 | 4,69 s6,9 |
| 2015-11-28 | 1597 | 6.10 | 4,65 s29,8 | 4,66 s29,1 |
See where the 920xt differs to the 625.
Most difference for the second split, there is also where the standard deviation is the biggest.
All 920XT tracks for the activities for the split analysis, on the outdoor map that closely follows the shortest path, corresponding for 625XT. The difference chart for the individual laps 920XT with the 625XT as reference is not very interesting, 920XT 151108 lap2, the 920XT looses consistently to the 625XT.
The 6.10 firmware seem to be a little better than 5.20, both decreased variance and distance calculation closer to expected. As the GPS firmware is the same, I would have expected changes only for Decice distances so this is maybe just a coincidence.
On the route, the 920XT track has some other inaccuracy indications:
- The track points are in a quite wide "band", normally about 30 m, see linls above for more information. The 625XT has less wide band, but no dramatical differences.
- There are some parts of tracks that are really of the taken path for the 920XT (that does not make much error actually).
- The 920XT cuts corners. The center of registered trails are 15m from the taken path in many corners. This is more visible on the outdoor map. Another way is to see where the 920xt differs to the 625. The loss of distance is quite consistent lap to lap, mostly where the trail has corners.
- The lap point was clocked with a meter or so difference. On the map, the 920XT lap point diff is about 50m. The 625XT (and other) has a difference of about 15m.
- The difference for the 4th autolap is another way to view the inaccuracy. There are over 200m from first to last autolap (for the 4*5 laps whare I used autolaps). As the laps differs from 4.54 km to 4.76 km, the difference is not strange. That corresponds to 0:15 /km. Only used "pure" autolap for 2 625XT activities (25m difference), the same activities for 920XT ("only" 60m there).
- When running steady in about 5:45 min/km, the 920XT could display the pace at 12 min/km, then 7 min/km, then 4 min/km. Much worse than I have seen on the 305.
920XT distance calculations is shorter than expected. The distance is at least 2% (average 100m in less than 5 km) too short, where the 305 is close to ideal. (The 625XT and Z3 calculates too long distances.) The actual difference is of minor interest to me, it is the variability that is the issue.
The 920XT has its merits but it comes with a premimum price with a relative poor GPS. It is made worse by the SW that desparately tries to compensate to the shortcomings. The SW makes a relative good work when the taken path is straight, the weather is good and the sky is not covered.
The 920XT does better in other situations. Another example where the 920XT does quite well, but the shortcomings are very obvious 920XT. The GF305 activity does not have good GPS accuracy at the start but picks up quite quick.
A diff graph to the GF 305:
The differences at 6:00, 25:00, 29:00. The 920XT is purple/green with points, the 305 yellow and the Z3 is red.
At the turnaround, the 920XT cuts the track about 35m too early. The 305 itself is about 10m early too. In addition, the 920 position off by 5 m in general, the 305 is very close to the taken path.
The 920 cuts a turn with about 30m. The 305 cuts with a couple of meters.
The 920 believes I was swimming, 40m from the taken path. The 305 is within meters.
A great source for GPS accuracy tests can be found on fellrnr site.