eval_metrics in R

[annaveronika]

The python library has eval_metrics method, which calculates metric value on every iteration and returns for each requested metric its values on each iteration. We need to do the same for R. This will also allow for visualisation of metric values.

[PhilippPro]

http://mlr-org.github.io/mlr-tutorial/devel/html/measures/index.html

Probably not a good idea, if each package implement their own metrics. ;)

There is also the Metrics package.

[annaveronika]

The calculation of metrics needs to be in C++, it's time critical and needs to be parallelized. That's why we write for metrics our own code.
But this issue is not about code of metrics calculation. I have updated the description, so that it's more clear.

[PhilippPro]

The calculation of metrics in general is not time consuming (at least the measures I know, maybe you can prove me wrong), so this is not the problem. It is even important for the user to be able to look how the measure is implemented which is a reason for implementing it in R.

But I see, that this issue is for something else. ;)

[annaveronika]

It is bottleneck in many cases, depends on the metric. Especislly querywise (groupwise) metrics can be time consuming.
We calculate metrics during training and after training also (which this issue is about).
Training is on c++, if you call on every iteration function from R to calc metrics, it will slow down your training even for simple metrics for some datasets (with little amount of features).

[PhilippPro]

Ok, from this side it makes sense. On the other hand, the user maybe wants to specify his own metric.

[annaveronika]

This is possible for python, see documentation. But in this case it might work slower.