What changes in localization performance across seasons: NCLT seasonal analysis #65
Closed
manankharwar
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
The NCLT benchmark covers nine sequences spanning all four seasons. Running the same config across all of them reveals which failure modes are environmental and which are algorithmic.
Results by season
Winter (2 sequences, FC wins both)
Winter sequences have consistent GPS quality. Snow cover changes the visual environment but not GPS reception. The NCLT campus is relatively open, so multipath is low. RL-EKF loses both sequences because of GPS covariance miscalibration: the sensor reports
var_xy = 9but actual noise on 2012-02-04 is significantly worse. RL-EKF rejects valid fixes as outliers. FusionCore'sgnss.base_noise_xyfloor keeps the gate calibrated to real noise.2012-01-08 is the longest sequence at 92 minutes. FC at 18.6 m is the third-best absolute result in the set.
Spring (2 sequences, FC wins both)
2012-05-11 is the cleanest sequence in the set: FC 9.7 m, RL 11.5 m, both filters performing well. This is what good GPS looks like: high fix density, accurate covariance, no long blackouts. The 16% FC margin is mostly noise.
2012-03-31 is the opposite: RL loses catastrophically at 156.5 m because of the same GPS covariance problem as winter. FC at 22.0 m is the benchmark for what the filter can do on a long sequence with realistic GPS.
Summer (2 sequences, RL wins both)
Both FC losses occur in summer. This is not a seasonal effect on GPS physics. It is a coincidence of sequence-specific data quality issues.
2012-06-15 has the lowest GPS density in the entire set (15,594 fixes vs 30,000-46,000 elsewhere) and a 461-second blackout. The filter dead-reckons for 7.7 minutes. Heading drift accumulates beyond what the bias-estimation pipeline corrects during that duration. Full analysis: issue #63.
2012-08-20 contains 105 mode-3 GPS fixes that are 720-840 m off RTK ground truth, clustered in a 24-second window at the end of the second GPS blackout. The chi2 gate is relaxed for re-acquisition at exactly that moment. Full analysis: issue #64.
Outside the two transients in 2012-08-20, FC achieves 5-10 m error for the remaining 15 minutes. The 98.3 m ATE RMSE is dominated by a 2-minute window.
Fall (3 sequences, FC wins all three)
Fall sequences cover a wide range: 10.8 m to 60.1 m ATE for FC. 2012-09-28 is the best absolute result in the set after 2012-05-11. 2012-11-04 is harder: FC 60.1 m vs RL 122.0 m, both filters struggling, but FC degrades more gracefully.
RL-EKF loses all three fall sequences by large margins (55.7 m, 60.0 m, 122.0 m). The same GPS covariance miscalibration pattern observed in winter and spring repeats here.
What the seasonal data actually shows
The season label is a proxy for two variables that actually drive performance:
GPS fix density and covariance accuracy. When the NCLT GPS receiver reports accurate covariance, both filters perform comparably (2012-05-11). When it reports underconfident covariance (too tight), RL-EKF rejects valid fixes and loses GPS coverage. FusionCore's noise floor prevents this.
Blackout duration and GPS data quality at re-acquisition. Long blackouts stress any filter running on IMU and encoder alone. The 2012-06-15 and 2012-08-20 issues are blackout-adjacent. Both are active areas of work.
If you are deploying on a robot with well-calibrated GPS covariance and no long blackouts, you will likely see FusionCore and RL-EKF perform similarly on straightforward sequences. The gap opens when the GPS data gets difficult.
Full per-sequence results, trajectory plots, and ATE-over-time plots are in
benchmarks/nclt/<date>/results_full/.All reactions