Implementing custom multi quantile loss.

[mat-ej]

Hi Catboost team,
first off, thanks for truly amazing library.

My question has to do with custom losses, I went through the following documentation file:

https://github.com/catboost/catboost/blob/master/catboost/tutorials/custom_loss/custom_loss_and_metric_tutorial.ipynb

This tutorial showcases custom "single regression" loss and custom "multi regression" loss.

Problem:
What I don't understand is how does multi quantile loss fit into this framework, it is indeed fit on a single dimension y vector - hence single regression, but the output is multiple outputs, (multi regression?).

I want to implement my own specific MQ loss that behaves in a similar way, that only single column y is needed but model outputs multiple values.

Q1: Can this be achieved ?

Q2: How exactly is MQ loss implemented ? (How does catboost know to output multiple values?)

Thanks in advance for being helpful. Apologies for asking possibly stupid question : )

catboost version: 1.2.3

Operating System: Mac OS

[Evgueni-Petrov-aka-espetrov]

Hi @mat-ej !
Many thanks for trying CatBoost!

Multi quantile loss is technically similar to multi classification -- single label, multiple predictions

At the moment, it is impossible to train models with user-defined multi quantile loss because there is no way to specify prediction dimension

For now, you may train a model for each quantile independently
However, predictions for quantiles may disagree slightly, and this method is slower than computing all quantiles in a single pass

We will try to add a parameter to specify prediction dimension for user-defined regression losses soon

[mat-ej]

great, looking forward to such an update

[mat-ej]

I was able to hack around this by repeating the y vector as many times as needed and then using MultiTargetCustomObjective, e.g.

class MultiRmseObjective(MultiTargetCustomObjective):
    def calc_ders_multi(self, approx, target, weight):
        assert len(target) == len(approx)
        
        w = weight if weight is not None else 1.0
        der1 = [(target[i] - approx[i]) * w for i in range(len(approx))]
        der2 = [-w for i in range(len(approx))]

        return (der1, der2)

Interestingly, if a MultiTargetCustomObjective is used the approx and target comes in batches of N=1 (?)

(Even though the method arguments are named approxes and targets)

When I use normal CustomObjective with a single target, the approxes and targets come in batches N>>>1.

class RmseObjective(object):
    def calc_ders_range(self, approxes, targets, weights):
        assert len(approxes) == len(targets)
        if weights is not None:
            assert len(weights) == len(approxes)
        
        result = []
        for index in range(len(targets)):
            der1 = targets[index] - approxes[index]
            der2 = -1

            if weights is not None:
                der1 *= weights[index]
                der2 *= weights[index]

            result.append((der1, der2))
        return result

Q: Is this a design choice ?

Now back to the original question of writing my own MQ loss.

Can I hack around with replicating y vector and using multi target objective ?

Just to give some background:
My overall goal is to use cross entropy on a regression task, (a trendy thing recently), it can be direct cross entropy or the cross entropy loss is used to learn quantiles.

[Evgueni-Petrov-aka-espetrov]

Q: Is this a design choice ?

yep

Now back to the original question of writing my own MQ loss.
Can I hack around with replicating y vector and using multi target objective ?

i think yes, if you combine it with appropriate MultiTargetCustomMetric

[mat-ej]

thanks for taking the time to reply to my questions.

Only thing that I see happening is that the cross entropy loss if calculated per data point is very tricky and it is not the same cross entropy that would be calculated per "batch" of data points.

Q1: Doesnt this difference of SingleTargetObjective vs MuiltiTargetObjective affect the behaviour of the classifier significantly ?

Q2: Am I misunderstanding things and the MultiTargetRegressor / Classifier does indeed sum up / mean the per data point losses somewhere after the calculation and hence is indeed optimizing in the correct direction ?

[Evgueni-Petrov-aka-espetrov]

A1: no, this is related to different parallelization for single and multiple predictions losses

A2: multi rmse normalizes by total sample weight
A2: https://catboost.ai/en/docs/concepts/loss-functions-multiregression#MultiRMSE

A2: multi logloss/cross-entropy normalizes by total sample weight times target dimension
A2: https://catboost.ai/en/docs/concepts/loss-functions-multilabel-classification#MultiLogloss