Add MAPE as evaluation metric

[amueller]

Also see #6605.

https://en.wikipedia.org/wiki/Mean_absolute_percentage_error

probably need a neg_mape scorer, too.

[mohamed-ali]

@amueller, I'd like to work on this.

[amueller]

sure, go for it.

[mohamed-ali]

@amueller, all sklearn metrics have an output in the[0,1] interval. Is it okay that this metric will have its output in [0,100]? or should we not multiply with 100 and only keep the fraction?

[amueller]

@mohamed-ali not all. MSE doesn't, for example (though I guess all with a fixed range do). I think we should multiply by 100 given that it's part of the definition, even though it's slightly different from other metrics.

[mohamed-ali]

@amueller okay great. Thanks for the quick reply :)

[mohamed-ali]

@amueller regarding the scorer, based on the deprecation message in the following example, I assume there should be a neg_mean_absolute_percentage_error_scorer instead of mean_absolute_percentage_error_scorer:

neg_mean_squared_error_scorer = make_scorer(mean_squared_error,
                                            greater_is_better=False)
deprecation_msg = ('Scoring method mean_squared_error was renamed to '
                   'neg_mean_squared_error in version 0.18 and will '
                   'be removed in 0.20.')
mean_squared_error_scorer = make_scorer(mean_squared_error,
greater_is_better=False)

https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/metrics/scorer.py#L472-L497

Is my assumption correct?

[jnothman]

if we call it mean_absolute_proportion_error we don't need the 100x

[jnothman]

yes, for the scorer, it should be neg_, but for the metric it should not. Andy proposed the scorer name neg_mape above

[mohamed-ali]

@amueller & @jnothman, I pushed the work in progress at this pull request #10711. We can move the discussion there and probably vote with the core devs on right name to consider.

[mohamed-ali]

@amueller & @jnothman, the work is done in #10711. Could you please do a full review?

Thanks.

[mohamed-ali]

@amueller @jnothman, out of curiosity, why do we prefix the scorer name with neg_?

[jnothman]

because the score is negated to follow the convention that greater is better. but users complained when we negated it without signifying so in the name

[mohamed-ali]

@jnothman thank you!