Aggregate benchmark results on the repetition level

[pat-s]

In mlr this is only possible on the fold level using getBMRAggPerformances() because the repetition ID is not tracked. Can we have an aggregation on the repetition level as well? This would require a tracking of the repetition ID throughout the calls similar to the fold ID.

[berndbischl]

Could you give a somehwat more precise example what you want? Also I am not sure - if I have to guess - that what you want is impossible with mlr

[berndbischl]

I mean can you specify a concrete setup / usecase and say exactly what type of numbers you want pls

[pat-s]

I want to aggregate performances on the repetition level, e.g. to plot them in Boxplots. Right now you either get the mean of all performances on the fold level (getBMRAggrPerformances()) or all fold performances (getBMRPerformances()).
An argument in getBMRAggrPerformances() with aggr = "fold" | "rep" would be great.

E.g. in a 100 rep 5 folds setup with 500 performances in the end I would end up with 100 aggregated values for visualization. On effect of this method is that the variance in the resulting boxplots is reduced.

[mllg]

Can I assume that you have one task, one resampling and multiple learners? And you want to aggregate the performances for each repetition, across learners?

[pat-s]

I think the number of learners, resamplings and tasks does not matter in this case.
The case also applies to resample(): Track the repetition ID to be able to aggregate the performance on the repetition level.

Currently, you only get the fold performances returned.
One can then take the mean of all fold performances (e.g. by using getBMRAggrPerformances() or its mlr3 successor) or make a boxplot of the fold performances.

getBMRAggrPerformances() should be able to aggregate on the repetition level. Not only for BMR objects but also for single resample() objects.

[mllg]

With #191 you can translate resampling iterations to folds and repeats. This is the first step towards custom aggregations.

[berndbischl]

i dont even think we should implement this request here. isnt this the great thing about datatable that the user can now EASILY compute this himself?
@pat-s have you tried this? pls show your code? how lengthy is it?