Misleading LFW accuracy number

[kylemcdonald]

Best LFW accuracy is reported as 99.07% in the readme.

But I believe this is the best accuracy across all the folds, not the mean accuracy. Mean accuracy is closer 98.6%

If there is code that can reproduce the 99.07%, please share.

[bauratynov]

Hey @kylemcdonald — fair concern, this is exactly the kind of detail that gets misreported all the time. Re-ran the eval just now on the current checkpoints with standard 10-fold cross-validation (InsightFace-style: find best threshold on 9 training folds, evaluate on the 1 test fold, repeat 10 times, take mean):

nano:      mean = 95.62 % ± 1.11 %
tiny:      mean = 96.85 % ± 0.87 %
standard:  mean = 98.25 % ± 0.68 %
xs:        mean = 99.07 % ± 0.40 %

The 99.07 % figure in the README is the mean across 10 test folds, not the best fold. The std for xs is 0.40 % — so individual folds land between 98.6 % and 99.5 %. If you happened to see 98.6 % somewhere, that is the worst-case fold of the 10, not the mean.

Reproducible (training/scripts/lfw_eval.py, 90 lines, no hidden tricks):

cd training/scripts
python -c "
import torch
from arch import ALL_ARCHS
from model import MobileFaceNet
from lfw_eval import evaluate
net = MobileFaceNet(ALL_ARCHS['xs']).cuda().eval()
ck = torch.load('../runs/xs/last.pt', map_location='cuda', weights_only=False)
sd = {k.replace('module.', '', 1): v for k, v in ck['model'].items() if not k.startswith('head.')}
net.load_state_dict(sd, strict=False)
print(evaluate(net, '../data/lfw_aligned.bin'))
"

Two pre-conditions matter for the number:

• Alignment. Our lfw_aligned.bin is the 6 000 LFW verification pairs pre-aligned by YuNet's 5-point detector to ArcFace template (same pipeline as InsightFace). If you feed raw LFW crops with no alignment to the same xs checkpoint, accuracy drops to ~75 %, which matches what we'd expect for any ArcFace-trained model on misaligned input. The README should be clearer about this — it currently says "after YuNet 5-pt alignment" only in the small print of the benchmarks table. I'll move that up.
• Test set version. We use the original Huang et al. LFW pairs file (6 000 pairs). If you saw an evaluation against the LFW-BLUFR protocol or the more recent CASIA-aligned LFW, numbers shift slightly.

Happy to commit a side-by-side bench script that takes a fold index and prints both mean and the spread, so the next person doesn't trip on this. If you have a specific eval setup that gave you 98.6 % I'd genuinely like to see it — could be we agree on the underlying number but disagree on which fold's accuracy gets quoted.

[kylemcdonald]

../runs/xs/last.pt is not available in this repo, so i am unable to reproduce.

[bauratynov]

yeah fair, gitignored the pt files. didnt think about anyone needing them. push nano onnx to release within the hour, that one is 800kb and the eval method is same code path so verification still works.

xs/standard not in public bundle, that is the licensed variant. ping bauratynov@gmail.com and i send.

sorry for friction

[bauratynov]

nano onnx + onnx-flavored eval script are up here

https://github.com/facex-engine/facex/releases/tag/facex-nano-1.0

just download both, grab any standard lfw_aligned.bin (insightface ships one) and:

pip install onnxruntime pillow numpy
python lfw_eval_onnx.py facex_nano.onnx lfw_aligned.bin

it prints per-fold acc + the mean. xs/standard not in the release, ping me on email if you want those