[MRG+1] Add new regression metric - Mean Squared Log Error #7655

kdexd · Oct 12, 2016

Reference Issue: None

What does this implement/fix? Explain your changes.

This PR implements a new metric - "Mean Squared Logarithmic Error" (name truncated to mean_squared_log_error). I have added the method alongwith other regression metrics in sklearn.metrics.regression module.
Accompanying the implementation, this PR is complete with User Guide Documentation and API docstring.

Any other comments?

The metric is similar to mean_squared_error and MSE method can be used to calculate MSLE by cleverly passing arguments, but it always required external manual work.
I felt that it would be a nice to have metric due to its frequent requirement.
A lot of regression problems in various competitions, especially Kaggle, evaluate submissions based on this error metric or its square root. A Kaggle wiki page can be found here.

Please fix test failures

jnothman · Oct 13, 2016

Please fix test failures

amueller · Oct 14, 2016

amueller reviewed Oct 14, 2016

View changes

doc/modules/model_evaluation.rst

amueller · Oct 14, 2016

amueller reviewed Oct 14, 2016

View changes

amueller · Oct 14, 2016

sklearn/metrics/regression.py

+    y_type, y_true, y_pred, multioutput = _check_reg_targets(
+        y_true, y_pred, multioutput)
+
+    if not (y_true >= 0).all() and not (y_pred >= 0).all():


It can be used with anything > -1, right?

@amueller It can be, but (1 + log(x)) will give huge negative values which change erratically on little change of x between (-1, 0). This will not make the score look sensible. Looking mathematically it is possible, but in practical usages this metric is used for non negative targets. Although if you suggest I'd change it.

Additionally I just recalled that, I read somewhere - this metric is used for positive values only, still there is log(1 + x) to make everything inside log greater than one, and finally outside the log positive, which would be greater than zero. Making it allowable till -1 will nullify this 😄

Yes, my reading of the equation agrees that it's designed for non-negative values with an exponential trend.

RPGOne · Oct 15, 2016

RPGOne approved these changes Oct 15, 2016

View changes

RPGOne · Oct 17, 2016

RPGOne approved these changes Oct 17, 2016

View changes

Hi @amueller and @jnothman, what more shall I do in this PR ? Also, is @RPGOne is a bot or a service ?

kdexd · Oct 18, 2016

Hi @amueller and @jnothman, what more shall I do in this PR ? Also, is @RPGOne is a bot or a service ?

RPGOne is spam, as far as I know

On 18 October 2016 at 20:49, Karan Desai notifications@github.com wrote:

Hi @amueller https://github.com/amueller and @jnothman
https://github.com/jnothman, what more shall I do in this PR ? Also, is
@RPGOne https://github.com/RPGOne is a bot or a service ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7655 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEz64Yn9iA-2Ob_9lyUJmobNRC007N4ks5q1JYNgaJpZM4KVLS7
.

jnothman · Oct 18, 2016

RPGOne is spam, as far as I know

On 18 October 2016 at 20:49, Karan Desai notifications@github.com wrote:

Hi @amueller https://github.com/amueller and @jnothman
https://github.com/jnothman, what more shall I do in this PR ? Also, is
@RPGOne https://github.com/RPGOne is a bot or a service ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7655 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEz64Yn9iA-2Ob_9lyUJmobNRC007N4ks5q1JYNgaJpZM4KVLS7
.

raghavrv · Oct 30, 2016

raghavrv requested changes Oct 30, 2016

View changes

The code is cleanly written. Thanks!

sklearn/metrics/regression.py

Documentation of mean_squared_error in current master renders like this:

There are some inconsistencies, my build after the changes you suggested looks like this ( mean_squared_log_error ):

To keep the diffs in this PR specific to only one metric, I am leaving other docstrings untouched for a while, I'll be taking them up in a separate documentation cleanup issue. I have rephrased the line you reviewed and reused mean_squared_error as well. Thanks !

kdexd · Oct 31, 2016

Documentation of mean_squared_error in current master renders like this:

There are some inconsistencies, my build after the changes you suggested looks like this ( mean_squared_log_error ):

To keep the diffs in this PR specific to only one metric, I am leaving other docstrings untouched for a while, I'll be taking them up in a separate documentation cleanup issue. I have rephrased the line you reviewed and reused mean_squared_error as well. Thanks !

Thanks for the screenshot of the doc!

I'll be taking them up in a separate documentation cleanup issue.

Much appreciated.

raghavrv · Nov 1, 2016

Thanks for the screenshot of the doc!

I'll be taking them up in a separate documentation cleanup issue.

Much appreciated.

I think it should also be added to the scorer so users can readily refer to it by neg_mean_squared_log_error...

raghavrv · Nov 1, 2016

I think it should also be added to the scorer so users can readily refer to it by neg_mean_squared_log_error...

@raghavrv It looks like there is a renaming scheduled for regression metrics similar to this one. For the sake of uniformity, I have added a deprecation message to mean_squared_log_error_scorer just like mean_squared_error_scorer and others. Let me know if I should not include that, and I will amend the commit accordingly, thanks !

kdexd · Nov 2, 2016

@raghavrv It looks like there is a renaming scheduled for regression metrics similar to this one. For the sake of uniformity, I have added a deprecation message to mean_squared_log_error_scorer just like mean_squared_error_scorer and others. Let me know if I should not include that, and I will amend the commit accordingly, thanks !

No, don't add a deprecated version. That's only there for people using
features in older versions.

On 2 November 2016 at 13:36, Karan Desai notifications@github.com wrote:

@raghavrv https://github.com/raghavrv It looks like there is a renaming
scheduled for regression metrics similar to this one. For the sake of
uniformity, I have added a deprecation message to
mean_squared_log_error_scorer just like mean_squared_error_scorer and
others. Let me know if I should not include that, and I will amend the
commit accordingly, thanks !

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7655 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEz6-edpn5mxkHZ8249LbmS10gyGymAks5q5_cxgaJpZM4KVLS7
.

jnothman · Nov 2, 2016

No, don't add a deprecated version. That's only there for people using
features in older versions.

On 2 November 2016 at 13:36, Karan Desai notifications@github.com wrote:

@raghavrv https://github.com/raghavrv It looks like there is a renaming
scheduled for regression metrics similar to this one. For the sake of
uniformity, I have added a deprecation message to
mean_squared_log_error_scorer just like mean_squared_error_scorer and
others. Let me know if I should not include that, and I will amend the
commit accordingly, thanks !

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#7655 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEz6-edpn5mxkHZ8249LbmS10gyGymAks5q5_cxgaJpZM4KVLS7
.

Hello @jnothman, @raghavrv:
I have made the needful additions / modifications in the PR. I'm up for anything else which is suitable to go in here, please have a look 😄

kdexd · Nov 2, 2016

Hello @jnothman, @raghavrv:
I have made the needful additions / modifications in the PR. I'm up for anything else which is suitable to go in here, please have a look 😄

raghavrv · Nov 2, 2016

raghavrv requested changes Nov 2, 2016

View changes

sklearn/metrics/tests/test_regression.py

Kaggle calls this "[root] mean squared logarithmic error", not "[root] mean squared log error" which sounds like it's a function of the log of the error. I think this is an important distinction. I'm not sure if you need to rename the function and scorer to reflect this, but at least the documentation needs to be absolutely clear.

jnothman · Nov 6, 2016

Kaggle calls this "[root] mean squared logarithmic error", not "[root] mean squared log error" which sounds like it's a function of the log of the error. I think this is an important distinction. I'm not sure if you need to rename the function and scorer to reflect this, but at least the documentation needs to be absolutely clear.

jnothman · Nov 6, 2016

jnothman requested changes Nov 6, 2016

View changes

doc/modules/model_evaluation.rst

jnothman · Nov 6, 2016

sklearn/metrics/regression.py

+    y_type, y_true, y_pred, multioutput = _check_reg_targets(
+        y_true, y_pred, multioutput)
+
+    if not (y_true >= 0).all() and not (y_pred >= 0).all():


Yes, my reading of the equation agrees that it's designed for non-negative values with an exponential trend.

@jnothman Logarithmic made the name too long, but if needed, I'll change the names. But yes atleast I should be clear about it in the docstrings and User Guide. I'll push the required changes soon. Also, MSE and MAE have their square roots used quite frequently, but they are not included in scorer so I dropped [root]. Is it a good choice to provide RMSLE in scorer or it is fine this way ?

kdexd · Nov 6, 2016

@jnothman Logarithmic made the name too long, but if needed, I'll change the names. But yes atleast I should be clear about it in the docstrings and User Guide. I'll push the required changes soon. Also, MSE and MAE have their square roots used quite frequently, but they are not included in scorer so I dropped [root]. Is it a good choice to provide RMSLE in scorer or it is fine this way ?

jnothman · Nov 6, 2016

jnothman reviewed Nov 6, 2016

View changes

sklearn/metrics/regression.py

jnothman · Nov 28, 2016

jnothman requested changes Nov 28, 2016

View changes

doc/modules/model_evaluation.rst

Hi @raghavrv, @jnothman:

I have addressed all of your review comments and cleaned up my commit history to reduce down the whole work into isolated sequential commits containing the implementation, tests and documentation one in each ! Please let me know if there's anything else I should do..

kdexd · Nov 28, 2016

Hi @raghavrv, @jnothman:

I have addressed all of your review comments and cleaned up my commit history to reduce down the whole work into isolated sequential commits containing the implementation, tests and documentation one in each ! Please let me know if there's anything else I should do..

FWIW, cleaning up commit history is superfluous.

jnothman · Nov 29, 2016

FWIW, cleaning up commit history is superfluous.

jnothman · Nov 29, 2016

jnothman requested changes Nov 29, 2016

View changes

Otherwise LGTM

sklearn/metrics/regression.py

sklearn/metrics/tests/test_regression.py

Often we go about testing by asserting invariances between two functions, and then testing the more general of them. A main advantage of that approach is legibility, but also that it should often be easy to pinpoint which part is broken.

…

On 29 November 2016 at 17:18, Karan Desai ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In sklearn/metrics/tests/test_regression.py <#7655>: > @@ -23,6 +24,7 @@ def test_regression_metrics(n_samples=50): y_pred = y_true + 1 assert_almost_equal(mean_squared_error(y_true, y_pred), 1.) + assert_almost_equal(mean_squared_log_error(y_true, y_pred), 0.01915163) @jnothman <https://github.com/jnothman> Done ! I was skeptical about the fact that if mean_squared_error actually gets faulty, then these tests will still pass as we are doing the same thing internally. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7655>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz69SpDk4ogDlHoHVdHIjMB9X3BE8Aks5rC8OjgaJpZM4KVLS7> .

jnothman · Nov 29, 2016

Often we go about testing by asserting invariances between two functions, and then testing the more general of them. A main advantage of that approach is legibility, but also that it should often be easy to pinpoint which part is broken.

…

On 29 November 2016 at 17:18, Karan Desai ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In sklearn/metrics/tests/test_regression.py <#7655>: > @@ -23,6 +24,7 @@ def test_regression_metrics(n_samples=50): y_pred = y_true + 1 assert_almost_equal(mean_squared_error(y_true, y_pred), 1.) + assert_almost_equal(mean_squared_log_error(y_true, y_pred), 0.01915163) @jnothman <https://github.com/jnothman> Done ! I was skeptical about the fact that if mean_squared_error actually gets faulty, then these tests will still pass as we are doing the same thing internally. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7655>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz69SpDk4ogDlHoHVdHIjMB9X3BE8Aks5rC8OjgaJpZM4KVLS7> .

Oh yes, if mean_squared_error would have broken that its test itself would fail !

kdexd · Nov 29, 2016

Oh yes, if mean_squared_error would have broken that its test itself would fail !

jnothman · Nov 29, 2016

jnothman approved these changes Nov 29, 2016

View changes

LGTM

jnothman · Nov 29, 2016

LGTM

amueller · Nov 29, 2016

amueller approved these changes Nov 29, 2016

View changes

LGTM apart from nitpick. Would you mind fixing that?

amueller · Nov 29, 2016

sklearn/metrics/regression.py

+
+    Parameters
+    ----------
+    y_true : array-like of shape = (n_samples) or (n_samples, n_outputs)


nitpick: you should write (n_samples,) because it's a tuple. (also everywhere below where there's a tuple with one element)

@amueller i read your comment after the PR was merged, although I think I will handle this is in a larger routine of documentation consistency, @raghavrv already directed me to a matching issue for the same. I will work on one more PR which is already open before starting that.

raghavrv · Nov 30, 2016

raghavrv approved these changes Nov 30, 2016

View changes

Thanks, @karandesai-96

jnothman · Nov 30, 2016

Thanks, @karandesai-96

Sorry I missed your comment before merging, @amueller :/

…

On 30 November 2016 at 13:40, Karan Desai ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In sklearn/metrics/regression.py <#7655>: > @@ -241,6 +243,73 @@ def mean_squared_error(y_true, y_pred, return np.average(output_errors, weights=multioutput) +def mean_squared_log_error(y_true, y_pred, + sample_weight=None, + multioutput='uniform_average'): + """Mean squared logarithmic error regression loss + + Read more in the :ref:`User Guide <mean_squared_log_error>`. + + Parameters + ---------- + y_true : array-like of shape = (n_samples) or (n_samples, n_outputs) @amueller <https://github.com/amueller> i read your comment after the PR was merged, although I think I will handle this is in a larger routine of documentation consistency, @raghavrv <https://github.com/raghavrv> already directed me to a matching issue for the same. I will work on one more PR which is already open before starting that. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#7655>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz67Xd4D6Ph_5sXuIhpjt4l9RK691Sks5rDOImgaJpZM4KVLS7> .

jnothman · Nov 30, 2016

Sorry I missed your comment before merging, @amueller :/

…

On 30 November 2016 at 13:40, Karan Desai ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In sklearn/metrics/regression.py <#7655>: > @@ -241,6 +243,73 @@ def mean_squared_error(y_true, y_pred, return np.average(output_errors, weights=multioutput) +def mean_squared_log_error(y_true, y_pred, + sample_weight=None, + multioutput='uniform_average'): + """Mean squared logarithmic error regression loss + + Read more in the :ref:`User Guide <mean_squared_log_error>`. + + Parameters + ---------- + y_true : array-like of shape = (n_samples) or (n_samples, n_outputs) @amueller <https://github.com/amueller> i read your comment after the PR was merged, although I think I will handle this is in a larger routine of documentation consistency, @raghavrv <https://github.com/raghavrv> already directed me to a matching issue for the same. I will work on one more PR which is already open before starting that. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#7655>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz67Xd4D6Ph_5sXuIhpjt4l9RK691Sks5rDOImgaJpZM4KVLS7> .

Feels good to contribute to the community, thanks @jnothman @amueller @raghavrv for the review ! 😄

kdexd · Nov 30, 2016

Feels good to contribute to the community, thanks @jnothman @amueller @raghavrv for the review ! 😄

@jnothman no worries, it was a nitpick of the highest order ;)

amueller · Nov 30, 2016

@jnothman no worries, it was a nitpick of the highest order ;)

Hi, I was wondering whether this should go in CHANGELOG for next release.

kdexd · Dec 22, 2016

Hi, I was wondering whether this should go in CHANGELOG for next release.

With apologies, we forgot to ask you to add a changelog entry here. Please submit a new PR with it. THanks.

jnothman · Dec 22, 2016

With apologies, we forgot to ask you to add a changelog entry here. Please submit a new PR with it. THanks.

@jnothman: Sure, I'll do that in a moment, thanks for the headsup.

kdexd · Dec 22, 2016

@jnothman: Sure, I'll do that in a moment, thanks for the headsup.

jnothman added New Feature Needs Review labels Oct 13, 2016

kdexd changed the title from [MRG] Add new regression metric - Mean Squared Log Error to [WIP] Add new regression metric - Mean Squared Log Error Oct 13, 2016

kdexd changed the title from [WIP] Add new regression metric - Mean Squared Log Error to [MRG] Add new regression metric - Mean Squared Log Error Oct 13, 2016

raghavrv removed the Needs Review label Oct 30, 2016

raghavrv added this to the 0.19 milestone Oct 30, 2016

ENH Add neg_mean_squared_log_error in metrics.scorer

Unverified

This user has not uploaded their public key yet.

GPG key ID: 83B358DE921CA4CC Learn about signing commits

Loading status checks…

62f1317

jnothman changed the title from [MRG] Add new regression metric - Mean Squared Log Error to [MRG+1] Add new regression metric - Mean Squared Log Error Nov 29, 2016

kdexd deleted the kdexd:msle-metric branch Dec 2, 2016

kdexd added a commit to kdexd/scikit-learn that referenced this pull request Dec 22, 2016

Make changelog entry for mean_squared_log_error.
Implemented this in #7655.

9c56cd7

kdexd added a commit to kdexd/scikit-learn that referenced this pull request Dec 22, 2016

Add changelog entry for MSLE implemented in #7655.

7a7b166

kdexd referenced this pull request Dec 22, 2016
Merged
[MRG + 1] Add changelog entry for MSLE implemented in #7655. #8104

jnothman added a commit that referenced this pull request Dec 23, 2016

[MRG + 1] Add changelog entry for MSLE implemented in #7655. (#8104)

Loading status checks…

7adeed1

sergeyf added a commit to sergeyf/scikit-learn that referenced this pull request Feb 28, 2017

[MRG + 1] Add changelog entry for MSLE implemented in #7655. (#8104)

2d72037

Przemo10 referenced this pull request Mar 17, 2017
Closed
update fork (#1) #8606

Sundrique added a commit to Sundrique/scikit-learn that referenced this pull request Jun 14, 2017

[MRG + 1] Add changelog entry for MSLE implemented in #7655. (#8104)

cc34503

NelleV added a commit to NelleV/scikit-learn that referenced this pull request Aug 11, 2017

[MRG + 1] Add changelog entry for MSLE implemented in #7655. (#8104)

619a0a6

paulha added a commit to paulha/scikit-learn that referenced this pull request Aug 19, 2017

[MRG + 1] Add changelog entry for MSLE implemented in #7655. (#8104)

cdb964a

maskani-moh added a commit to maskani-moh/scikit-learn that referenced this pull request Nov 15, 2017

[MRG + 1] Add changelog entry for MSLE implemented in #7655. (#8104)

8e9a835

scikit-learn/scikit-learn

Join GitHub today

[MRG+1] Add new regression metric - Mean Squared Log Error #7655

Conversation

kdexd commented Oct 12, 2016 • edited

Reference Issue: None

What does this implement/fix? Explain your changes.

Any other comments?

This comment has been minimized.

jnothman commented Oct 13, 2016

jnothman added New Feature Needs Review labels Oct 13, 2016

kdexd changed the title from [MRG] Add new regression metric - Mean Squared Log Error to [WIP] Add new regression metric - Mean Squared Log Error Oct 13, 2016

kdexd changed the title from [WIP] Add new regression metric - Mean Squared Log Error to [MRG] Add new regression metric - Mean Squared Log Error Oct 13, 2016

amueller reviewed Oct 14, 2016 View changes

amueller reviewed Oct 14, 2016 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

RPGOne approved these changes Oct 15, 2016 View changes

RPGOne approved these changes Oct 17, 2016 View changes

This comment has been minimized.

kdexd commented Oct 18, 2016

This comment has been minimized.

jnothman commented Oct 18, 2016 • edited

raghavrv requested changes Oct 30, 2016 View changes

raghavrv removed the Needs Review label Oct 30, 2016

raghavrv added this to the 0.19 milestone Oct 30, 2016

This comment has been minimized.

kdexd commented Oct 31, 2016

This comment has been minimized.

raghavrv commented Nov 1, 2016

This comment has been minimized.

raghavrv commented Nov 1, 2016

This comment has been minimized.

kdexd commented Nov 2, 2016

This comment has been minimized.

jnothman commented Nov 2, 2016

This comment has been minimized.

kdexd commented Nov 2, 2016

raghavrv requested changes Nov 2, 2016 View changes

This comment has been minimized.

jnothman commented Nov 6, 2016

jnothman requested changes Nov 6, 2016 View changes

This comment has been minimized.

This comment has been minimized.

kdexd commented Nov 6, 2016

jnothman reviewed Nov 6, 2016 View changes

jnothman requested changes Nov 28, 2016 View changes

This comment has been minimized.

kdexd commented Nov 28, 2016

This comment has been minimized.

jnothman commented Nov 29, 2016

jnothman requested changes Nov 29, 2016 View changes

kdexd added some commits Nov 9, 2016

This comment has been minimized.

jnothman commented Nov 29, 2016

This comment has been minimized.

kdexd commented Nov 29, 2016

jnothman approved these changes Nov 29, 2016 View changes

This comment has been minimized.

jnothman commented Nov 29, 2016

jnothman changed the title from [MRG] Add new regression metric - Mean Squared Log Error to [MRG+1] Add new regression metric - Mean Squared Log Error Nov 29, 2016

amueller approved these changes Nov 29, 2016 View changes

This comment has been minimized.

This comment has been minimized.

raghavrv approved these changes Nov 30, 2016 View changes

Hide details View details jnothman merged commit cb6a366 into scikit-learn:master Nov 30, 2016 2 of 3 checks passed

2 of 3 checks passed

This comment has been minimized.

jnothman commented Nov 30, 2016

This comment has been minimized.

jnothman commented Nov 30, 2016

This comment has been minimized.

kdexd commented Nov 30, 2016

This comment has been minimized.

amueller commented Nov 30, 2016

kdexd deleted the kdexd:msle-metric branch Dec 2, 2016

adis300 added a commit to adis300/scikit-learn that referenced this pull request Dec 13, 2016

kdexd commented Oct 12, 2016 •

edited

amueller reviewed Oct 14, 2016

View changes

amueller reviewed Oct 14, 2016

View changes

RPGOne approved these changes Oct 15, 2016

View changes

RPGOne approved these changes Oct 17, 2016

View changes

jnothman commented Oct 18, 2016 •

edited

raghavrv requested changes Oct 30, 2016

View changes

raghavrv requested changes Nov 2, 2016

View changes

jnothman requested changes Nov 6, 2016

View changes

jnothman reviewed Nov 6, 2016

View changes

jnothman requested changes Nov 28, 2016

View changes

jnothman requested changes Nov 29, 2016

View changes

jnothman approved these changes Nov 29, 2016

View changes

amueller approved these changes Nov 29, 2016

View changes

raghavrv approved these changes Nov 30, 2016

View changes

jnothman merged commit `cb6a366` into scikit-learn:master Nov 30, 2016
2 of 3 checks passed