Custom Loss function

[nishai]

Hi✌🏽
Firstly, thanks for all your work on NeuralProphet: it's a very cool tool!

Problem

I'm struggling to train my NeuralProphet model with a custom loss function. I want to use the Mean Pinball Loss, and I've coded it up, replicating the sklearn docs, except using torch methods which will allow for autograd. However, I'm not able to use this custom loss function with NeuralProphet.

I've tested two different ways of passing the loss to NeuralProphet: (1) a simple python function, and (2) a custom class extending torch.nn.modules.loss._Loss, as advised in this discussion.

Loss function: python function

def MeanPinballLoss(y: Tensor, yhat: Tensor, alpha: int = 0.9) -> Tensor:
    y_diff = y - yhat
    yhat_diff = yhat - y

    loss = (
        alpha * torch.max(y_diff, torch.zeros_like(y_diff))
        + (1-alpha) * torch.max(yhat_diff, torch.zeros_like(yhat_diff))
    ).sum().mean()

    return loss

Using this function during model init m = NeuralProphet(loss_func=MeanPinballLoss), I get the following error:

/opt/conda/lib/python3.7/site-packages/neuralprophet/configure.py in __post_init__(self)
...
NotImplementedError: Loss function <function MeanPinballLoss at 0x7f60b1656830> not found

I had a look at the source code, and from my understanding of configure.py

neural_prophet/neuralprophet/configure.py

Lines 40 to 55 in 31051a2

	def __post_init__(self):
	if type(self.loss_func) == str:
	if self.loss_func.lower() in ["huber", "smoothl1", "smoothl1loss"]:
	self.loss_func = torch.nn.SmoothL1Loss()
	elif self.loss_func.lower() in ["mae", "l1", "l1loss"]:
	self.loss_func = torch.nn.L1Loss()
	elif self.loss_func.lower() in ["mse", "mseloss", "l2", "l2loss"]:
	self.loss_func = torch.nn.MSELoss()
	else:
	raise NotImplementedError("Loss function {} name not defined".format(self.loss_func))
	elif callable(self.loss_func):
	pass
	elif hasattr(torch.nn.modules.loss, self.loss_func.__class__.__name__):
	pass
	else:
	raise NotImplementedError("Loss function {} not found".format(self.loss_func))

this should not happen, since callable(MeanPinballLoss) is True (line 50).

Loss function: PyTorch module loss

My next attempt was to extend torch.nn.modules.loss._Loss. I basically mimicked PyTorch's implementation of L1Loss.

class MeanPinballLoss(_Loss):
    __constants__ = ['reduction']
    
    def __init__(self, size_average=None, reduce=None, reduction: str = 'mean', alpha=0.5) -> None:
        super(MeanPinballLoss, self).__init__(size_average, reduce, reduction)
        self.alpha = alpha
        
    def forward(self, input: Tensor, target: Tensor) -> Tensor:
        y_diff = target - input
        yhat_diff = input - target

        loss = (
            self.alpha * torch.max(y_diff, torch.zeros_like(y_diff))
            + (1-self.alpha) * torch.max(yhat_diff, torch.zeros_like(yhat_diff))
        ).sum().mean()
        return loss

Initialising this with NeuralProphet by m = NeuralProphet(loss_func=MeanPinballLoss()), I get the same error:

/opt/conda/lib/python3.7/site-packages/neuralprophet/configure.py in __post_init__(self)
...
NotImplementedError: Loss function MeanPinballLoss() not found

Line 52 in configure.py seems to only check whether PyTorch contains the loss function, rather than checking if the loss is a subclass of torch.nn.modules.loss._Loss. So this error message makes sense to me, but it seems that the wrong check in being done.

End

I could not find any similar issues with custom loss functions, and would appreciate any help to know if I'm doing something wrong, or if there is a bug.

Thanks!

[ourownstory]

Hi @nishai,

Thank you for documenting your issue so nicely!
I think you are assessing the situation right.
You seem to be doing everything correct, but our docstring is different from the actual check.
-> we should relax the check to allow for any loss_func being a subclass of _Loss, not just ones predefined in Torch.

We have one test for a callable function as loss:

neural_prophet/tests/test_integration.py

Line 574 in 31051a2

def test_callable_loss(self):

However, we currently do not have a test covering your case.

I would appreciate if you could open a pull request with:

1. Fix the issue by changing the check as described
2. Contribute your two types of defining it as a two new test cases.

You can find how to do a "dev install" (runs black and tests as hooks):
https://github.com/ourownstory/neural_prophet/blob/master/CONTRIBUTING.md

Hope this helps and I would love to welcome you as a Contributor to the community!

[nishai]

Thanks for the quick response @ourownstory!

Cool! I'm happy to contribute, and relax the check
I will open a pull request with the change and tests as soon as I can.

[ourownstory]

fixed in #407