Probabilistic losses?

[fkiraly]

Hello, this is @fkiraly from MLJ.jl across the street, we were wondering whether there are currently any plans for probabilistic losses, such as (strictly) proper scoring rules (with a negative sign)?
JuliaAI/MLJ.jl#91
For example: log-loss (aka cross-entropy loss), Brier score (aka squared integrated loss), discrete and continuous rank probability score, etc.

Practically, this would mean comparing an observation with a probability distribution that intends to predictively model the distribution the observation will be coming from.

This would be quite important for:
(i) evaluating machine learning models that are capable of returning such predictions - this actually happens relatively frequently, but isn't always properly supported by well-designed interfaces (cough).
(ii) optimizing/fitting/tuning probabilistic machine learning models w.r.t. empirical (possibly penalized) probabilistic risk/loss

Technically, it would mean comparing a vector of predictions to a vector of Distributions.jl (which we've chosen in MLJ.jl).
A nice benefit of this interface is that it also disentangles the usual predict_proba etc probabilistic classification interface, see e.g. here for a discussion a while ago.
JuliaAI/MLJ.jl#34

Of course, it's also different from (a generalization of) your current interface... hence the question.

[joshday]

Looks like we already have some of these things?

LossFunctions.jl/src/supervised/other.jl

Lines 29 to 36 in 84c20d8

	@doc doc"""
	CrossentropyLoss <: SupervisedLoss
	Cross-entropy loss also known as log loss and logistic loss is defined as:
	``L(target, output) = - target*ln(output) - (1-target)*ln(1-output)``
	"""
	struct CrossentropyLoss <: SupervisedLoss end

I don't think there really are any plans for LossFunctions, as it already does a lot and satisfies the use cases of those that use it (@Evizero can chime in if he disagrees).

I would say that we are definitely open to suggestions (and PRs!), but no one is actively working on LossFunctions because it's been stable for a good while now.

[fkiraly]

This is not what I mean: the cross-entropy loss is implemented as a deteriministic loss here.
Which is the usual trick to make it work under the standard interface (for classification), but it wouldn't work for probabilistic regression.
For this, you really need to dispatch on full distribution objects.

Also, the problem that you have with binary vs multi-class for the cross-entropy loss disappears, as in that case you can simply write
L(target, output) = - ln*( pmf(output, target) )

[fkiraly]

Regarding "use cases of those who use it" - this is a bit tautological?
Once something covers a use case well, people will use it. Not the other way round...

[juliohm]

It may be reasonable to include probabilistic losses as well, but if it happens, it will be a separate type with a different interface because this idea of having a loss between distribution and value is exotic and not many people will use it. It is for advanced users with a strong probabilistic background.

I will close the issue for now, and reconsider it in a future roadmap.