corner case: resample printer output on train data when no train preds are there

[berndbischl]

meas = setAggregation(mmce, train.mean)
r = resample("classif.rpart", iris.task, cv2, measures = meas)
print(r)

this creates NA for the aggregated value, but strangely displays a mean and an sd?

Resample Result
Task: iris-example
Learner: classif.rpart
mmce.aggr: NA
mmce.mean: 0.05
mmce.sd: 0.01
Runtime: 0.0179381

[berndbischl]

i also dont know whether i really like that long output. "aggr" and "mean" are usually redundant?

[PhilippPro]

I just looked over it. The only measure, where it is different is rmse, where we make a \sqrt after the mean, all other have test.mean which is just the mean.
Example:

lrn = makeLearner("regr.rpart")
rdesc = makeResampleDesc("CV", iters = 30)
r = resample(lrn, bh.task, rdesc, measures = list(rmse, mse))
print(r)

For me, I would delete aggr (or just not print it, maybe somebody is using it in real application), the one who is interested can make his/her own measure.

Moreover we want to print out messages, after each CV iteration, so there is already a guess, how good is the prediction (similar to xgboost).

Maybe it would be also interesting to measure on all predicted observations and not only on the specific folds. (e.g. auc over all observations, and not only mean on the folds; I am not sure now, which is the more exact measure... (drawback would be no sd))
edit: just googled it and found that post from catastrophic-failure: http://stackoverflow.com/questions/33246049/calculating-auc-leave-one-out-cross-validation-in-mlr?rq=1
performance(r$pred, auc)

[berndbischl]

PR is here #1187

[giuseppec]

Seems that #1187 has been merged, can this issue be closed?