Rename default_measure to default_loss and make it a function of the scitype only

[juliohm]

I am starting to use the loss functions in MLJBase that were migrated from MLJ recently in MLJBase 0.5.0. Thank you very much! :) I noticed that the terminology used in the project is "measure" instead of "loss", is there a reason for calling it that way?

Regardless of this naming issue, I would like to understand better the default_measure function. I don't understand how a default loss is a function of the learning model, wouldn't it be just a function of the scitype of the model's output? Imagine that I am trying to compare different learning models consistently. If I end up solving a classification task with a probabilistic and deterministic classifier, these two will be assessed differently in a cross-validation procedure for example that uses the default_measure when the user does not specify the loss explicitly.

I think my question is even more general. A loss function is formally defined as loss(y, f(x)) where f(x) is the output of the model f. So in terms of dispatch this function only sees the output of the model, not the model itself. Can we follow this convention instead and rewrite the losses to be a function of the scitype only?

[ablaom]

From the manual:

"In MLJ loss functions, scoring rules, sensitivities, and so on, are collectively referred to as measures."

The orientation trait of a measure determines whether it is loss, score or other. Do info(rms) to get all traits of a measure:

julia> info(rms)
(name = "rms",
 target_scitype = Union{AbstractArray{Continuous,1}, AbstractArray{Count,1}},
 prediction_type = :deterministic,
 orientation = :loss,
 reports_each_observation = false,
 is_feature_dependent = false,
 supports_weights = true,)

So, in tuning, for example, the measure is maximised if it's orientation is :score, and minimised if it is :loss.

The "measure" terminology comes from MLR.

I agree that default_measure can be function of the target scitype of a model (obtained with target_scitype(model)). Happy to review a PR along those lines.

[juliohm]

Thank you @ablaom , I am happy to submit a PR. Could you please confirm that we can remove the current methods of default_measure involving learning models, and only provide methods for scitypes?

default_measure(val) = default_measure(scitype(val))
default_measure(::Type{<:Finite}) = misclassification_rate
default_measure(::Type{<:Infinite}) = rms
default_measure(::Type{<:Distribution}) = cross_entropy
# etc.

[juliohm]

Also maybe provide an alias default_loss to make it clear in call sites that we are talking about losses specifically? I am not a big fan of mixing together the concepts of scores, sensitivities and losses under a same umbrella. Could we maybe split these into separate types instead of trying to rely on a complex trait system? These are used for very different purposes AFAICT, and the name measure is too generic. When I first read it, I thought of measure theory.

[ablaom]

I would not want to have multiple methods here. I don't see anything wrong with traits and the LossFunctions library is already a case where we need it; these losses are instances of structs defined in an external package.

I have had detailed discussions around loss/score function design with @fkiraly, who is an expert in this area.

Happy with the PR proposal. Pull requests are made onto #dev

[juliohm]

My suggestion is to separate the concepts of losses , scores, etc. and still rely on a trait system to accommodate multiple implementations. For example, there would be a separate trait system for loss functions, and this trait system would be applied to LossFunctions.jl and the MLJBase.jl loss functions like rms. There would be a separate trait system for scores, and another trait system for sensitivities. I am afraid putting these 3 trait systems on a single trait system called "measure" will make code unnecessarily complex? By trait system I mean a collection of function names that are specific to the concept default_loss, default_score, ... as opposed to some general default_measure.

@fkiraly could you please share your opinion regarding this suggestion? Do you think it could add value to the user and/or developer experience?

[juliohm]

I think we could satisfy both worlds by creating separate trait systems loss traits, score traits, sensitivity traits, and then a more abstract trait system measure traits for those who want to talk about all three concepts at the same time while coding.

[ablaom]

Edited

Some minor refactoring along the lines of #51 (comment) are addressed in PR #165, although default_measure must know if a deterministic or probabilistic measure is required.

If someone wants to second the poster's initial suggestion to separate the losses from scores (I vote against) please re-open this issue.