FEA add poisson loss to MLPRegressor

[lorentzenchr]

Reference Issues/PRs

None

What does this implement/fix? Explain your changes.

This adds the parameter loss to MLPRegressor, same as for HGBT.

Any other comments?

While in the past it was decided to not add any features to the neural nets, see https://scikit-learn.org/dev/faq.html#id9

Note that scikit-learn currently implements a simple multilayer perceptron in sklearn.neural_network. We will only accept bug fixes for this module.

I argue that it is important to support for all main model types (linear, tree-based, neural nets) the 3 most important distributions in statistics as losses:

• binomial (log loss)
• Poisson (Poisson deviance)
• normal (squared error)

We have it for all but the neural nets. Therefore this PR.

[lorentzenchr]

This also needs a decision.

[ogrisel]

But the _loss module doesn't use it (for performance reasons)

Is there any real performance impact of computing this term when training a model?

I think I would favor always including it. If it used for monitoring, displaying something named "Poisson loss" which is actually not the true loss, but the loss with a constant offset can be quite surprising.

[ogrisel]

Alternatively, we could just rename this function to offset_poisson_loss to make it explicit that the computation of the constant terms are removed.

But if we expose the value of the objective function under the name "Poisson loss" via the public API, one way or another, I think we should make sure we re-add the constant terms to avoid any confusion when comparing values on resampled data (e.g. between train and test split to check for overfitting).

[OmarManzoor]

I don't think performence is too much of a concern with this module so we might as well keep the constant terms.

[lorentzenchr]

I don't think optimization is too much of a concern with this module so we might as well keep the constant terms.

The constant term in the loss only matters if we report/save those loss values as we do in MLP (and in GradientBoosting, but not in HistGradientBoosting). The nice thing about adding the constant term is that the minimum of the loss is then 0 exactly. Omitting it saves a bit of time.
Note also that NN libraries like pytorch usually omit the constant term (see https://pytorch.org/docs/stable/generated/torch.nn.PoissonNLLLoss.html#torch.nn.PoissonNLLLoss).

[OmarManzoor]

Yes Tensorflow also omits it https://www.tensorflow.org/api_docs/python/tf/keras/losses/Poisson.

I would say let's keep it for now. Would we accomplish much by saving time in this module when there are other optimal libraries specifically for implementing neural networks?

[github-actions]

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 9352f2b. Link to the linter CI: here}

[OmarManzoor]

From my opinion including this seems to make sense and from an overall look at the changes I don't think it adds much extra maintenance overhead as well.

What do others think @scikit-learn/core-devs

[lorentzenchr]

In the monthly meeting on January 27, 2025, https://github.com/scikit-learn/administrative/blob/master/monthly_meetings/2025-01-27.md, we decided:

Poisson loss for MLPRegressor #30712 We say that NN won't get new features, yet I think having the three most important losses (log loss, squared error, Poisson deviance) in all major supervised learning methods (linear, tree, nn) would complete the picture.

• there is a difference between adding new features and maintaining MLP in good working order which includes making it uniform with the rest of the library. For example:
  • completing the set of popular losses (Poisson, Gamma, Tweedie, Pinball)
  • completing array API support
  • completing metadata routing
• There seems to be consensus on those 3 points, but not more (alternative would be to deprecate and remove nn)

[OmarManzoor]

• There seems to be consensus on those 3 points, but not more (alternative would be to deprecate and remove nn)

Is deprecating this module being considered as a possibility? Because if that is the case, I don't think we should add anything.

[lorentzenchr]

The consensus was keep the module and make it consistent with the rest. Therefore add those few extensions, also the Poisson loss. Nobody has so far raised the voice to deprecate it.
Despite there being much better options for NNs, I personally think it would be a pity to deprecate it altogether. If we keep it, you can install one small library, i.e. scikit-learn, and you get basic GLMs, trees and NNs. From there on, you can start looking for more advanced options.

[OmarManzoor]

Despite there being much better options for NNs, I personally think it would be a pity to deprecate it altogether. If we keep it, you can install one small library, i.e. scikit-learn, and you get basic GLMs, trees and NNs. From there on, you can start looking for more advanced options.

That seems fine. Thanks for explaining. I think we should move forward with reviewing this PR then.

[OmarManzoor]

Thanks for the PR @lorentzenchr

A few comments otherwise looks nice

[OmarManzoor]

I don't think performence is too much of a concern with this module so we might as well keep the constant terms.

[OmarManzoor]

LGTM

[ogrisel]

Besides https://github.com/scikit-learn/scikit-learn/pull/30712/files#r1954012957 and getting the tests to pass with all admissible random seeds, LGTM.

[lorentzenchr]

@ogrisel @OmarManzoor Thanks for your reviews.