Evaluate against multiple measures

[ayush1999]

Initial fix for : #98

[ablaom]

A note: TunedModel in src/tuning.jl makes a call to fit Resampler objects. It must now call using measures= instead of measure but note that in this case it always calls with a single measure. Do not change the keyword argument of TunedModel to measures, keep it as measure. When we tune, we must fix just one measure.

[ayush1999]

@ablaom Sure, it'd be easier to just change the constructor for Resampler to use the measures keyword argument instead of measure, right?

[ablaom]

Yes, the Resample constructor should have measures instead of measure - just a name change. However, this will break fit!(::EitherTunedModel{Grid}, which constructs a Resampler object (from src/tuning.jl):

    resampler = Resampler(model=clone,
                          resampling=tuned_model.resampling,
                          measure=measure,
                          operation=tuned_model.operation)

I think all that is necessary to fix this is to change the third line to measures=measure.
Clear?

[ayush1999]

Got it. Will push the changes.

[ayush1999]

@ablaom Please review.

[ablaom]

Great, thanks for that.

• There is a minor bug on line 90 of resampling.jl (I don't understand why travis does not detect it; maybe because it is in logging??). The line should read:

        "measures=$_measures \n"*

• And a similar problem on line 137.

• Also the reporting of cv results is not exactly as specified. For easy aggregation later, we want a named tuple of vectors, not a vector of named tuples. For example, for this code:

x1 = ones(10)
x2 = ones(10)
X = DataFrame(x1=x1, x2=x2)
y = [1.0, 1.0, 2.0, 2.0, 1.0, 1.0, 2.0, 2.0, 1.0, 1.0]

cv=CV(nfolds=5)
model = ConstantRegressor()
mach = machine(model, X, y)
evaluate!(mach, resampling=cv, measures=[rms, rmslp1])

we get

 (MLJ.rms = 0.5, MLJ.rmslp1 = 0.22314355131420982)
 (MLJ.rms = 0.75, MLJ.rmslp1 = 0.287682072451781) 
 (MLJ.rms = 0.5, MLJ.rmslp1 = 0.22314355131420982)
 (MLJ.rms = 0.75, MLJ.rmslp1 = 0.287682072451781) 
 (MLJ.rms = 0.5, MLJ.rmslp1 = 0.22314355131420982)

But we want

(MLJ.rms=[0.5, 0.75, 0.5, 0.75, 0.5], MLJ.rmslp1=[0.223, 0.287, ...., ])

• For testing the multi-measure case, please add the following after line 23 of test/resampling.jl:

result = evaluate!(mach, resampling=holdout, measures=[rms, rmslp1])
@test result isa NamedTuple

And, after the current line 37, add

result = evaluate!(mach, resampling=cv, measures=[rms, rmslp1])
@test result isa NamedTuple

[ablaom]

Ah, my mistake. My new test mucks up the consequent test. How about you just comment out the new test lines and I will fix this later?

[ablaom]

Looks like your error this time. You have an array where a number is expected. Let me know if you want me to have a look at it.

[ayush1999]

Ah, my bad. I forgot to test cases when single measure is used. Fixed it this time.

[ablaom]

Thanks!