Repository navigation
A room-accuracy benchmark you can rerun in your own house #101
gbroeckling
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
I wanted a number for how well PadSpan actually picks rooms that I couldn't fool myself with, so I built a small benchmark you can rerun yourself. It's in the repo under
tools/benchmark, with a guide in docs/BENCHMARK.md.The main thing it does differently from a quick self-test is how it splits the data. It never tests on readings that sit right next to the ones it learned from. It holds out a whole walk, a whole stay in a room, or a spot plus everything near it. That's the part that makes most "95%" style numbers look better than they are, including PadSpan's own old self-test, which I've since fixed.
First run is on a public dataset, a real house with 6 beacons and 10 rooms (Zenodo 4783076). Trained only on the calibration session, then tested on two later sessions:
To be fair about what that is: in that dataset the beacons are fixed and the receiver moves, which is the reverse of a PadSpan house. It also doesn't say where the beacons are, so the distance-based method can't be scored, and every reading is scored on its own without PadSpan's smoothing over time. So it's a floor, not a promise for your house.
The more useful bit is that you can run it in your own place. Turn on the RSSI capture, walk around marking which room you're in, export it, and the script scores PadSpan against the simple baseline. It can also score any other Home Assistant room sensor from a history export, Bermuda or whatever you use, as long as each is set up its own documented way.
If you try it, I'd love to see your numbers, good or bad. The more houses, the more it actually means.
All reactions