need quantitative unit tests

[davidssmith]

It would be nice if validate() produced some consolidated numbers that could be checked before and after patching.

[elijahrockers]

I'm a little confused how to interpret the validation results... for instance, when I run validate(4):

Kt RMSE (%): 15.23076798142528
Kt max error (%): 99.99999999999977
Kt CCC: 0.9044129212108128
ve RMSE (%): NaN
ve max error (%): 99.99999999999996
ve CCC: NaN
vp RMSE (%): 25.490496906451867
vp max error (%): 99.99999999999956
vp CCC: 0.9040289643750475

Why are the max error (%) close to 100? What is CCC?

[davidssmith]

The max error is the error for any one voxel. An error of 100% means there was at least one voxel that was off by 100%. The CCC is the concordance correlation coefficient (https://en.wikipedia.org/wiki/Concordance_correlation_coefficient).

[elijahrockers]

So just to see if I've got it:
Kt RMSE of 15%, is this maximum MSE for all voxels?
So the CCC for Kt of 0.904 is actually not too bad then?

I have to say the max error % of 100 is kind of concerning..., thoughts?

[davidssmith]

RMSE is the root mean square error, so it is across all voxels (in the
mask).

A CCC of 0.904 is pretty good in the presence of noise.

A max error of 100% isn't too bad. It just means at least one voxel was
unable to fit the model. You can check the error distribution if you are
concerned to make sure most voxels look ok.

On Fri, Feb 12, 2016 at 12:40 PM, Elijah Rockers notifications@github.com
wrote:

So just to see if I've got it:
Kt RMSE of 15%, is this maximum MSE for all voxels?
So the CCC for Kt of 0.904 is actually not too bad then?

I have to say the max error % of 100 is kind of concerning..., thoughts?

—
Reply to this email directly or view it on GitHub
#23 (comment)
.