Discussion: Classification results slightly off compared to benchmark/README.md results

[TobyB1702]

When I ran classification_benchmark.py i got the following results

Method,ID Acc,ID ECE,Shift Acc,OOD AUROC,Time (s)
Single Net,0.928600013256073,0.055205754935741425,0.14390000700950623,0.5,1695.0
Deep Ensemble,0.9383000135421753,0.024391435086727142,0.156700000166893,0.8970877400891211,36589.6
MC Dropout,0.9315000176429749,0.05082382634282112,0.1412999927997589,0.9094335510141365,1717.7
Laplace (K-FAC),0.9298999905586243,0.053301066160202026,0.1665000021457672,0.8891253591733251,1721.6
VI (ELBO),0.9279000163078308,0.019974060356616974,0.11670000106096268,0.7206589447602949,1765.5
NESBS (SVGD),0.9376000165939331,0.02559744566679001,0.1665000021457672,0.8941837161954518,6349.4
NES-RS,0.9379000067710876,0.025853179395198822,0.16380000114440918,0.8885998982022127,6544.6
NES-RE,0.9381999969482422,0.023850273340940475,0.1687999963760376,0.8988708800706822,5712.7

However comparing the metrics against https://github.com/intsystems/bensemble/blob/master/benchmarks/README.md

Method | ID Acc | ECE ↓ | OOD AUROC ↑
-- | -- | -- | --
Single Net | 92.72% | 0.0569 | 0.500
Deep Ensemble | 93.80% | 0.0267 | 0.898
MC Dropout | 92.90% | 0.0525 | 0.899
Laplace (K-FAC) | 92.78% | 0.0550 | 0.848
VI (ELBO) | 92.91% | 0.0197 | 0.749
NES-BS | 93.73% | 0.0239 | 0.907
NES-RS | 93.94% | 0.0237 | 0.907
NES-RE | 93.82% | 0.0243 | 0.891

Most results seem to be off by a few percent points, why is this? Is there some randomness/seed within the benchmarking code that i have missed? Would be good to fix the seed if it hasn't been defined. Thanks

Part of the JOSS review openjournals/joss-reviews#11055 (comment)

[Dimundel]

Thanks for the issue! It seems that we didn't initialize all random seeds properly. I'll fix this, then run both benchmarks twice to verify that the results are reproducible, and open a PR with the fix.

[Dimundel]

We've made both benchmarks deterministic in #47, repeated runs on the same machine now should give identical results.

Your numbers still won't match ours exactly: bit-identical results across different GPUs and CUDA or PyTorch versions aren't achievable. We'll rerun over several seeds and republish the table as mean +- std so deviations can be judged against the actual spread.

[TobyB1702]

Thanks @Dimundel that makes sense, will rerun regression and classification multiple times each to confirm

[Dimundel]

I reran the classification benchmark over five seeds. The table now reports mean +- std