1463 mean absolute percentage error #1510

ParthivNaresh · 2020-12-07T02:06:34Z

CLAassistant · 2020-12-07T02:06:38Z

All committers have signed the CLA.

codecov · 2020-12-07T15:06:11Z

Codecov Report

Merging #1510 (795f546) into main (b09ea39) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@            Coverage Diff            @@
##             main    #1510     +/-   ##
=========================================
+ Coverage   100.0%   100.0%   +0.1%     
=========================================
  Files         227      228      +1     
  Lines       15592    15629     +37     
=========================================
+ Hits        15584    15621     +37     
  Misses          8        8

Impacted Files	Coverage Δ
evalml/objectives/__init__.py	`100.0% <ø> (ø)`
evalml/objectives/utils.py	`100.0% <ø> (ø)`
evalml/objectives/standard_metrics.py	`100.0% <100.0%> (ø)`
...lml/objectives/time_series_regression_objective.py	`100.0% <100.0%> (ø)`
evalml/tests/conftest.py	`100.0% <100.0%> (ø)`
evalml/tests/objective_tests/test_objectives.py	`100.0% <100.0%> (ø)`
...lml/tests/objective_tests/test_standard_metrics.py	`100.0% <100.0%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b09ea39...795f546. Read the comment docs.

freddyaboulton

@ParthivNaresh This looks great! I think we should resolve the discussion about what to do with division by 0 before merging but other than that I think it's good to merge!

I'm ok with holding off with making this the default objective for time series regression problems because of its noted limitations, namely the bias in selecting estimators that under-estimate the target.

I think for the default objective for automl we should consider these two alternatives:

freddyaboulton · 2020-12-07T18:24:47Z

evalml/objectives/standard_metrics.py

+    score_needs_proba = False
+    perfect_score = 0.0
+
+    def objective_function(self, y_true, y_predicted, X=None):


Maybe we should just raise a ValueError if any y_true are 0? My concern is that 0 can be a meaningful target value (like stock returns) and users may be misled that those records are being used in the evaluation when they are not.

freddyaboulton · 2020-12-07T18:26:13Z

evalml/objectives/standard_metrics.py

+    """Mean absolute percentage error for time series regression."""
+    name = "Mean Absolute Percentage Error"
+    greater_is_better = False
+    score_needs_proba = False


nit-pick: You don't need to specify score_needs_proba since that is specified in RegressionObjective

freddyaboulton · 2020-12-07T18:26:32Z

evalml/objectives/standard_metrics.py

@@ -345,6 +345,17 @@ def objective_function(self, y_true, y_predicted, X=None):
        return metrics.mean_absolute_error(y_true, y_predicted)


+class MAPE(RegressionObjective):


I think we should add to the api reference too

dsherry

@ParthivNaresh looks great!! The only blocking comments I had were about using eps in the impl, and starting a thread with @freddyaboulton about whether or not we should define TimeseriesRegressionObjective right now

dsherry · 2020-12-07T19:02:51Z

evalml/objectives/standard_metrics.py

+    perfect_score = 0.0
+
+    def objective_function(self, y_true, y_predicted, X=None):
+        return (np.abs((y_true - y_predicted) / y_true))[y_true != 0].mean() * 100


@ParthivNaresh nice, looks good!

Sklearn 0.24.2 (a current release candidate) has a MAPE impl (docs).

I like their use of max(y_true, eps) in the denominator. I think we should do that instead of excluding rows where y_true is exactly 0.

I also liked that they had weight support. We don't have a pattern for this yet from other objectives, and its not a requirement for this PR, but would be nice to have.

@dsherry That's an interesting approach, but wouldn't that result in an extremely high value in the overall calculation? @freddyaboulton

Good point @dsherry! I think @ParthivNaresh is right because sklearn's definition of eps is way too small. Even if we used our own I'm worried that there'll always be one use case where the value we pick is "wrong".

I prefer treating this how we treat RootMeanSquaredLogError or MeanSquaredLogError. Those objectives error out if passed targets with any negative values. They are also kept out of the core objectives because the user only uses them when they are defined for their particular dataset. Maybe we should add MAPE to get_non_core_objectives?

sklearn's definition of eps is way too small

Hmm, its an arbitrary threshold, right? Setting it to min numerical precision makes sense to me because any differences smaller than that (more or less) cannot be trusted.

It seems we have a few options for what to do when the target contains values equal to or super close to zero (options not all mutually exclusive):

Have the objective error out.

Have the objective round the zero value up to a small eps in the denominator.

Have automl error out during init if MAPE was selected as the automl objective. Similar treatment for RMSLE/MSLE if the target has negative values.

What behavior is most useful to users?

@freddyaboulton @ParthivNaresh I don't have a strong preference between 1 and 2--if you're both advocating for 1, let's do that in this PR. However to really nail that question I think we should also file 3 as an enhancement and address it separately.

I think the problem with the sklearn approach is that it will divide by eps whenever y=0 which will cause the score to blow up.

I don't think scores that large are useful to users, especially if they have a dataset where y=0 occurs frequently. I like option 1 because I prefer letting the user know at the beginning that this objective isn't great for problems with y=0 (rather than having them comb through or impl trying to reason why their objective is ~1e14).

You bring up a good point about AutoMLSearch! We should definite file that. Currently we never allow RMSLE and MSLE as the primary objective but I like the idea of only disallowing it if the target has negative values. Similar for MAPE. That sounds like it could be a data check!

@freddyaboulton agreed! And ooh yeah definitely, perhaps that could be part of the "InvalidTargetsDataCheck"

Filed #1521 !

dsherry · 2020-12-07T19:03:42Z

evalml/objectives/standard_metrics.py

@@ -345,6 +345,17 @@ def objective_function(self, y_true, y_predicted, X=None):
        return metrics.mean_absolute_error(y_true, y_predicted)


+class MAPE(RegressionObjective):
+    """Mean absolute percentage error for time series regression."""


Should we mention that we've scaled this by 100 so that its a percentage? I've seen it defined both with or without the 100 factor in the past.

dsherry · 2020-12-07T19:14:22Z

evalml/tests/objective_tests/test_standard_metrics.py

+
+    assert obj.score(s1_actual, s1_predicted) == pytest.approx(3.5 / 6 * 100)
+    assert obj.score(s2_actual, s2_predicted) == pytest.approx(4.83333333 / 7 * 100)
+    assert obj.score(s3_actual, s3_predicted) == pytest.approx(3.0625 / 7 * 100)


@ParthivNaresh nice 👍 I was able to walk through by hand and reproduce the math for the first example you have here. Only thought here is that if you could simplify the numbers a bit, make the arrays shorter, make some of those decimals more round etc., it would help make this math easier to follow.

dsherry · 2020-12-07T19:15:24Z

evalml/objectives/standard_metrics.py

@@ -345,6 +345,17 @@ def objective_function(self, y_true, y_predicted, X=None):
        return metrics.mean_absolute_error(y_true, y_predicted)


+class MAPE(RegressionObjective):


@freddyaboulton did we decide not to define TimeseriesRegressionObjective? Or did we simply not decide yet, haha, since this is the first objective explicitly defined for timeseries? I can't remember where we left that conversation.

Previously, our objectives were only defined for a single problem type. We agreed to change this to a list of valid problem types and that our existing regression objectives should work for both regression and time series regression problems.

So long story short, we didn't define a TimeSeriesRegressionObjective because we didn't have any objectives that should only work for time series. We should make MAPE a TimeSeriesRegressionObjective in that case :) Good call.

Ah got it, thanks @freddyaboulton

So @ParthivNaresh , I think what we should do here is:

Define TimeseriesRegressionObjective which subclasses RegressionObjective but allows only ProblemTypes.TIME_SERIES_REGRESSION

Have MAPE subclass TimeseriesRegressionObjective

That sound good?

@dsherry Sounds great!

…m/alteryx/evalml into 1463-mean-absolute-percentage-error

dsherry · 2020-12-08T15:23:09Z

evalml/objectives/standard_metrics.py

+    def objective_function(self, y_true, y_predicted, X=None):
+        if (y_true == 0).any():
+            raise ValueError("Mean Absolute Percentage Error cannot be used when "
+                             "targets contain the value 0.")


dsherry · 2020-12-08T15:23:26Z

evalml/objectives/time_series_regression_objective.py

+        Set to [ProblemTypes.TIME_SERIES_REGRESSION]
+    """
+
+    problem_types = [ProblemTypes.TIME_SERIES_REGRESSION]


@ParthivNaresh looks great thanks

dsherry · 2020-12-08T15:24:54Z

evalml/tests/dependency_update_check/latest_dependency_versions.txt

@@ -5,7 +5,7 @@ colorama==0.4.4
 featuretools==0.22.0
 graphviz==0.15
 ipywidgets==7.5.1
-lightgbm==3.1.0
+lightgbm==3.1.1


Ah @ParthivNaresh for future reference, if the dependency update check fails on your PR, the dependency checker bot which runs every hour will get a separate PR up in with this update. And then once that's merged you can rebase/update your PR and you should be all set.

We keep those things separate so that we can watch the dependency update step pass/fail CI, independent from any other changes like those being made in this PR. This helps us catch breaking changes made by the latest version of our dependency libs.

ParthivNaresh added 4 commits December 6, 2020 19:45

Import MAPE

c4e1cc3

MAPE metric

d02188b

MAPE tests

eb3769d

Release notes update

ea9142d

ParthivNaresh self-assigned this Dec 7, 2020

ParthivNaresh added 2 commits December 7, 2020 09:56

Changing Regression Problem Type count from 7 to 8

fab5639

Lint

915d21b

Update release notes

35eca99

ParthivNaresh marked this pull request as ready for review December 7, 2020 16:56

Merge branch 'main' into 1463-mean-absolute-percentage-error

9446be5

ParthivNaresh requested review from freddyaboulton, angela97lin, dsherry and bchen1116 and removed request for freddyaboulton December 7, 2020 18:07

freddyaboulton approved these changes Dec 7, 2020

View reviewed changes

dsherry approved these changes Dec 7, 2020

View reviewed changes

freddyaboulton mentioned this pull request Dec 7, 2020

Allow RMSLE, MSLE, MAPE as the primary objective when the target supports it #1521

Closed

ParthivNaresh added 9 commits December 7, 2020 21:54

API reference and release notes

5142983

Add ValueError check

cd72fd3

Create a new Objective for Time Series to subclass RegressionObjective

114fb14

Update tests

22ce21a

Add MAPE to non-core metrics

369838c

Lint

98d1e0a

Merge branch 'main' into 1463-mean-absolute-percentage-error

8f3021b

API ref update

a6514c1

Merge branch '1463-mean-absolute-percentage-error' of ssh://github.co…

40f429b

…m/alteryx/evalml into 1463-mean-absolute-percentage-error

Lint and dependency

795f546

ParthivNaresh merged commit 21a313b into main Dec 8, 2020

dsherry reviewed Dec 8, 2020

View reviewed changes

dsherry mentioned this pull request Dec 29, 2020

Release v0.17.0 #1623

Merged

freddyaboulton deleted the 1463-mean-absolute-percentage-error branch May 13, 2022 15:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

1463 mean absolute percentage error #1510

1463 mean absolute percentage error #1510

ParthivNaresh commented Dec 7, 2020

CLAassistant commented Dec 7, 2020 •

edited

Loading

codecov bot commented Dec 7, 2020 •

edited

Loading

freddyaboulton left a comment

freddyaboulton Dec 7, 2020 •

edited

Loading

freddyaboulton Dec 7, 2020

freddyaboulton Dec 7, 2020

dsherry left a comment

dsherry Dec 7, 2020

ParthivNaresh Dec 7, 2020

freddyaboulton Dec 7, 2020 •

edited

Loading

dsherry Dec 7, 2020

freddyaboulton Dec 7, 2020

dsherry Dec 7, 2020

freddyaboulton Dec 7, 2020

dsherry Dec 7, 2020

dsherry Dec 7, 2020

dsherry Dec 7, 2020

freddyaboulton Dec 7, 2020

dsherry Dec 7, 2020

ParthivNaresh Dec 8, 2020

dsherry Dec 8, 2020

dsherry Dec 8, 2020

dsherry Dec 8, 2020

dsherry Dec 8, 2020

		@@ -345,6 +345,17 @@ def objective_function(self, y_true, y_predicted, X=None):
		return metrics.mean_absolute_error(y_true, y_predicted)


		class MAPE(RegressionObjective):

1463 mean absolute percentage error #1510

1463 mean absolute percentage error #1510

Conversation

ParthivNaresh commented Dec 7, 2020

CLAassistant commented Dec 7, 2020 • edited Loading

codecov bot commented Dec 7, 2020 • edited Loading

Codecov Report

freddyaboulton left a comment

Choose a reason for hiding this comment

freddyaboulton Dec 7, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freddyaboulton Dec 7, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CLAassistant commented Dec 7, 2020 •

edited

Loading

codecov bot commented Dec 7, 2020 •

edited

Loading

freddyaboulton Dec 7, 2020 •

edited

Loading

freddyaboulton Dec 7, 2020 •

edited

Loading