Knowing the alignment works

Knowing the alignment works

Every change to a sensor-fusion pipeline arrives with a claim attached: this is better. The uncomfortable question is how anyone knows.

The default evidence is a walk. Someone changes a parameter, goes outside, comes back and reports that it felt more stable. That is a real observation with an enormous error bar and a strong bias — it was taken once, on one route, in one set of conditions, by the person who wanted it to be better.

What makes this hard

There is no ground truth. To know exactly how wrong the alignment is, you would need to know exactly where the phone was, which is the problem you were trying to solve. Survey equipment gives you that for a few metres of a prepared course, not for a walk through a park.

The system is also stochastic: two walks along the same path produce different GPS noise, different tracking quality and different results from the identical code. And the quantity users care about — whether content looks like it is sitting on a spot — is perceptual, non-linear, and not obviously any of the numbers you can compute.

What can be measured

Internal consistency. The fused trajectory should agree with the odometry about the shape of the path even where it disagrees with GPS about its absolute position. Divergence between the two is a signal that needs no ground truth to interpret.

Residuals against held-out observations. Fit the alignment while withholding some GPS readings, then ask how well it predicts them. Cross validation, applied to a trajectory.

Convergence behaviour. How long, and how much movement, before the estimate stops changing materially. This is the claim most often made publicly about systems like this — "give it fifteen seconds of walking" — so it deserves to be a measured number rather than a remembered impression.

Here it now is one, and the measurement is a useful corrective. Across 51 recordings, the median time for an anchor to come within a metre of the estimate's own final answer was about two minutes, with the ninetieth percentile beyond four. The intermediate checkpoints are measured in observations rather than seconds — at roughly one every half-second, the median remaining error was about 15 m after twenty observations and about 7 m after forty.

That measures "distance to where this configuration eventually settles", which is not the same quantity as "visible drift versus raw GPS" — but it is the number we actually have, and it is a good deal less comfortable than the rule of thumb.

Publishing the uncomfortable version is the point. A convergence claim is the first thing a sceptical reader will test.

Stability under replay. Run the same recording twice and the result must be identical. Sounds trivial; catches real bugs where uninitialised state or ordering leaks into the estimate.

Regression across a corpus. The most valuable one. With a set of recorded sessions, a change can be evaluated against all of them at once, and the question becomes "better on how many, worse on how many, by how much" rather than "better on the one walk I took after making it".

Independent contradiction. Where a second sensing path exists — depth geometry versus solved marker pose, reconstruction-derived trajectory versus fused trajectory — disagreement is an error signal that costs nothing extra.

What cannot be measured this way

• Perceived stability. No metric captures the difference between content that drifts slowly and content that jumps occasionally, though users react to them very differently. The metric may prefer the one people hate.
• Behaviour in conditions not in the corpus. A corpus records where developers walked, which is a biased sample of the world in a way that is invisible from inside it.
• Interaction with the device's own tracking. Layer one differs by phone and by conditions, and every measurement here is entangled with it.

The arrangement that follows

• Keep a corpus and grow it deliberately, especially toward conditions you suspect are weak rather than toward more of what you already have.
• Evaluate changes across the whole corpus, and report the distribution rather than the best case.
• Publish the convergence claim as a measured number and re-measure it, since it is the claim outsiders will test first.
• Keep going outside. The corpus covers what it covers; a real walk is the sampling mechanism for everything it does not — and recordings are how a walk becomes something you can re-run.
• Say "unverified" when it is unverified. The most damaging failure in this area is not a wrong number — it is a number quoted with more confidence than its evidence supports, which then gets repeated until nobody remembers where it came from.

That last one is a discipline rather than a technique, and it is the one that decides whether the rest of the measurements are worth taking.