[Metrics] Sklearn metrics

[justusschock]

Do we want to include metrics of sklearn?

The following constraints would apply if we had an sklearn metric interface:

• All imports would be within the class space, so sklearn would only remain an optional dependency (due to it's size)
• All metrics will be provided by sklearn interface at first, but will be removed later on (once we got a native pytorch implementation)

The Advantages:

• We have a pretty impressive set of metrics directly in the beginning
• the maintenance effort is minimal
• they're well tested

The disadvantages

• For every call to a metric, we need to convert tensors to numpy
• We will have to synchronise GPU(s) whenever we call a metric with inputs lying on GPU, which may be a bottleneck the users aren't aware of

My proposal

• Add this layer in the beginning and try to replace as many of the metrics used there by our own :)
• Explicitly state the downfalls in docs (and maybe also raise a warning on that) to ensure users are aware of this problem.

Thoughts?

cc @Borda @williamFalcon @ethanwharris @PyTorchLightning/core-contributors

[ethanwharris]

The plan looks good :) another option might be to see if there is anything we can use in Skorch - although I don't know to what extent they support on GPU stuff

[justusschock]

IIRC skorch aims on perfect sklearn compatibility, which is why they also use sklearn metrics (not on gpu though).

[ethanwharris]

Yeah, my mileage with skorch was pretty limited (and quite a while ago), so I'm not really sure what it's about haha

[rzepinskip]

The AllenNLP also started with CPU-based metrics to move some of them to GPU later - PR 3851

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.