Adding inverse transform method to pipelines #2256

freddyaboulton · 2021-05-11T22:13:16Z

Pull Request Description

Fixes #2046

Follows the recommendation and implementation plan outlined in the design document

After creating the pull request: in order to pass the release_notes_updated check you will need to update the "Future Release" section of docs/source/release_notes.rst to include this pull request by adding :pr:123.

freddyaboulton · 2021-05-11T22:41:13Z

evalml/pipelines/time_series_regression_pipeline.py

        if self.estimator.predict_uses_y:
            y_arg = y
        predictions = self.estimator.predict(features_no_nan, y_arg).to_series()
+        predictions.index = y.index


The polynomial detrender needs the predictions to have an index of the same type as the data it was fit on.
I think this is another reason to do #1639

Is this line equivalent to using this function? It seems nit picky and trivial, but I feel like I've personally been bitten with weird behavior when I don't use pandas' established API

I think unfortunately in this case reindex_like is not avaliable for series, but even if I'm wrong it'll fill NaNs in the places where the indices don't match. I think reindex does the same thing.

freddyaboulton · 2021-05-11T23:10:23Z

evalml/pipelines/components/transformers/transformer.py

        return {}
+
+
+class TargetTransformer(Transformer):


I think we need this abstraction to make it clear which components we expect to be able to invert. It also makes the implementation of ComponentGraph.from_list much easier (imo) without having to make every component return both X and y.

codecov · 2021-05-12T00:26:53Z

Codecov Report

Merging #2256 (a67528c) into main (432f876) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main   #2256     +/-   ##
=======================================
+ Coverage   99.9%   99.9%   +0.1%     
=======================================
  Files        281     281             
  Lines      24607   24719    +112     
=======================================
+ Hits       24578   24690    +112     
  Misses        29      29

Impacted Files	Coverage Δ
evalml/pipelines/component_graph.py	`100.0% <100.0%> (ø)`
evalml/pipelines/components/component_base_meta.py	`100.0% <100.0%> (ø)`
...transformers/preprocessing/polynomial_detrender.py	`97.6% <100.0%> (ø)`
...l/pipelines/components/transformers/transformer.py	`100.0% <100.0%> (ø)`
evalml/pipelines/pipeline_base.py	`99.7% <100.0%> (+0.1%)`	⬆️
evalml/pipelines/pipeline_meta.py	`100.0% <100.0%> (ø)`
...valml/pipelines/time_series_regression_pipeline.py	`100.0% <100.0%> (ø)`
...tests/component_tests/test_polynomial_detrender.py	`100.0% <100.0%> (ø)`
...valml/tests/pipeline_tests/test_component_graph.py	`100.0% <100.0%> (ø)`
.../tests/pipeline_tests/test_time_series_pipeline.py	`100.0% <100.0%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 432f876...a67528c. Read the comment docs.

freddyaboulton · 2021-05-13T14:16:02Z

evalml/pipelines/time_series_regression_pipeline.py

            y_arg = y
        predictions = self.estimator.predict(features_no_nan, y_arg).to_series()
+        predictions.index = y.index
+        predictions = self.inverse_transform(predictions).to_series()


Right now the time series regression pipeline is the only pipeline that uses the inverse_transform method.

As we add more target transformers (target encoder as a component, #914), I expect the other pipeline classes to call the inverse_transform method.

freddyaboulton · 2021-05-13T14:17:02Z

evalml/tests/pipeline_tests/test_component_graph.py

+        return infer_feature_types(y / 2)
+
+
+class SubsetData(Transformer):


To simulate a transformer that modifies the target but is not a target transformer, e.g. a sampler

You might want to add this to the class' docstring so that if someone needs to write a new test, they might be able to reuse this.

chukarsten

Looks like great work!

chukarsten · 2021-05-16T16:49:16Z

evalml/pipelines/component_graph.py

+            sampler_index = next(iter([i for i, tup in enumerate(without_inverse_transform) if "sampler" in tup[0]]), -1)
+            component_dict[without_inverse_transform[sampler_index][0]].append(f"{with_inverse_transform[-1][0]}.y")


Hahaha, oh my dude. This is some big brain stuff.

chukarsten · 2021-05-16T16:52:32Z

evalml/pipelines/component_graph.py

+    def _get_parent_y(self, component_name):
+        """Helper for inverse_transform method."""
+        parents = self.get_parents(component_name)
+        return next(iter(p[:-2] for p in parents if '.y' in p), None)


Is the -2 so that we capture the string of the parent? I'm a little about what happens if we change all these ".y's"

Yea I think our component graph would break but it would be covered by unit tests at least. I think if a user does something crazy like .z or something, an exception about the graph not being connected should be raised. Maybe we can make that clearer in the future.

chukarsten · 2021-05-16T16:57:38Z

evalml/pipelines/components/transformers/transformer.py

        return {}
+
+
+class TargetTransformer(Transformer):


chukarsten · 2021-05-16T17:08:26Z

evalml/pipelines/time_series_regression_pipeline.py

        if self.estimator.predict_uses_y:
            y_arg = y
        predictions = self.estimator.predict(features_no_nan, y_arg).to_series()
+        predictions.index = y.index


Is this line equivalent to using this function? It seems nit picky and trivial, but I feel like I've personally been bitten with weird behavior when I don't use pandas' established API

chukarsten · 2021-05-16T17:11:57Z

evalml/tests/pipeline_tests/test_component_graph.py

+        return infer_feature_types(y / 2)
+
+
+class SubsetData(Transformer):


You might want to add this to the class' docstring so that if someone needs to write a new test, they might be able to reuse this.

chukarsten · 2021-05-16T17:12:29Z

evalml/tests/pipeline_tests/test_component_graph.py

+component_graphs = [(ComponentGraph({"Imputer": [Imputer],
+                                     "Log": [LogTransform],
+                                     "Random Forest": ["Random Forest Regressor", "Imputer.x", "Log.y"]}),
+                     lambda y: infer_feature_types(np.exp(y.to_series()))),
+                    (ComponentGraph({"Imputer": [Imputer],
+                                     "Log": [LogTransform],
+                                     "Double": [DoubleTransform, "Log.y"],
+                                     "Random Forest": ["Random Forest Regressor", "Imputer.x", "Double.y"]}),
+                     lambda y: infer_feature_types(np.exp(y.to_series() / 2))),
+                    (ComponentGraph({"Imputer": [Imputer],
+                                     "Log": [LogTransform, "Imputer.x"],
+                                     "Double": [DoubleTransform, "Log.x", "Log.y"],
+                                     "Random Forest": ["Random Forest Regressor", "Double.x", "Double.y"]}),
+                     lambda y: infer_feature_types(np.exp(y.to_series() / 2))),
+                    (ComponentGraph({"Imputer": [Imputer],
+                                     "OneHot": [OneHotEncoder, "Imputer.x"],
+                                     "DateTime": [DateTimeFeaturizer, "OneHot.x"],
+                                     "Log": [LogTransform],
+                                     "Double": [DoubleTransform, "Log.y"],
+                                     "Random Forest": ["Random Forest Regressor", "DateTime.x", "Double.y"]}),
+                     lambda y: infer_feature_types(np.exp(y.to_series() / 2))),
+                    (ComponentGraph({"Imputer": [Imputer],
+                                     "OneHot": [OneHotEncoder, "Imputer.x"],
+                                     "DateTime": [DateTimeFeaturizer, "OneHot.x"],
+                                     "Log": [LogTransform],
+                                     "Double": [DoubleTransform, "Log.y"],
+                                     "Double2": [DoubleTransform, "Double.y"],
+                                     "Random Forest": ["Random Forest Regressor", "DateTime.x", "Double2.y"]}),
+                     lambda y: infer_feature_types(np.exp(y.to_series() / 4))),
+                    (ComponentGraph({"Imputer": ['Imputer'],
+                                     "Double": [DoubleTransform],
+                                     "DateTime 1": ["DateTime Featurization Component", "Imputer"],
+                                     "ET": ["Extra Trees Regressor", "DateTime 1.x", "Double.y"],
+                                     "Double 2": [DoubleTransform],
+                                     "DateTime 2": ["DateTime Featurization Component", "Imputer"],
+                                     "Double 3": [DoubleTransform, "Double 2.y"],
+                                     "RandomForest": ["Random Forest Regressor", "DateTime 2.x", "Double 3.y"],
+                                     "DateTime 3": ["DateTime Featurization Component", "Imputer"],
+                                     "Double 4": [DoubleTransform],
+                                     "Linear": ["Linear Regressor", "DateTime 3.x", "Double 4.y"],
+                                     "Logistic Regression": ["Linear Regressor", "Linear", "RandomForest", "ET", 'Double 3.y']}),
+                     lambda y: infer_feature_types(y.to_series() / 4)),
+                    (ComponentGraph({"Imputer": [Imputer],
+                                     "OneHot": [OneHotEncoder, "Imputer.x"],
+                                     "DateTime": [DateTimeFeaturizer, "OneHot.x"],
+                                     "Log": [LogTransform],
+                                     "Double": [DoubleTransform, "Log.y"],
+                                     "Double2": [DoubleTransform, "Double.y"],
+                                     "Subset": [SubsetData, "DateTime.x", "Double2.y"],
+                                     "Random Forest": ["Random Forest Regressor", "Subset.x", "Subset.y"]}),
+                     lambda y: infer_feature_types(np.exp(y.to_series() / 4))),
+                    (ComponentGraph({"Imputer": [Imputer],
+                                     "Random Forest": ["Random Forest Regressor", "Imputer.x"]}),
+                     lambda y: y),
+                    (ComponentGraph({"Imputer": [Imputer],
+                                     "DateTime": [DateTimeFeaturizer, "Imputer.x"],
+                                     "OneHot": [OneHotEncoder, "DateTime.x"],
+                                     "Random Forest": ["Random Forest Regressor", "OneHot.x"]}),
+                     lambda y: y),
+                    (ComponentGraph({"Random Forest": ["Random Forest Regressor"]}),
+                     lambda y: y),
+                    (ComponentGraph({"Imputer": ['Imputer'],
+                                     "Double": [DoubleTransform],
+                                     "DateTime 1": ["DateTime Featurization Component", "Imputer"],
+                                     "ET": ["Extra Trees Regressor", "DateTime 1.x", "Double.y"],
+                                     "Double 2": [DoubleTransform],
+                                     "DateTime 2": ["DateTime Featurization Component", "Imputer"],
+                                     "Double 3": [DoubleTransform, "Double 2.y"],
+                                     "RandomForest": ["Random Forest Regressor", "DateTime 2.x", "Double 3.y"],
+                                     "DateTime 3": ["DateTime Featurization Component", "Imputer"],
+                                     "Double 4": [DoubleTransform],
+                                     "Linear": ["Linear Regressor", "DateTime 3.x", "Double 4.y"],
+                                     "Logistic Regression": ["Linear Regressor", "Linear", "RandomForest", "ET"]}),
+                     lambda y: y)
+                    ]


lol, my dude...nice.

holy potates

bchen1116

Left some comments and questions just for my own clarification purposes. The tests look great!

bchen1116 · 2021-05-18T16:49:20Z

evalml/pipelines/components/component_base_meta.py

                raise ComponentNotYetFittedError(f'This {klass} is not fitted yet. You must fit {klass} before calling {method.__name__}.')
            elif method.__name__ == 'inverse_transform':
-                return method(self, X, y)
+                return method(self, X)


wouldn't we return method(self, y)?

Since inverse transform only takes one argument, the y is actually "called" X in this piece of code.

wow that's trippy... Any chance you can leave that as a comment for future notes

bchen1116 · 2021-05-18T16:50:38Z

evalml/pipelines/pipeline_base.py

+        Components that implement inverse_transform are PolynomialDetrender, LabelEncoder (tbd).
+
+        Arguments:
+            y: (pd.Series, ww.DataTable): Final component features


nit pick (no colon after y):
y (pd.Series, ww.DataTable): Final component features

bchen1116 · 2021-05-18T16:50:54Z

evalml/pipelines/pipeline_meta.py

                return method(self, X)
            elif method.__name__ == 'predict':
                return method(self, X, objective)
+            elif method.__name__ == 'inverse_transform':


Same as above?

bchen1116 · 2021-05-18T16:58:22Z

evalml/tests/pipeline_tests/test_component_graph.py

+
+@pytest.mark.parametrize("component_list,answer", lists)
+def test_from_list_with_target_transformers(component_list, answer):
+    assert ComponentGraph.from_list(component_list).component_dict == answer.component_dict


These tests are 👌

freddyaboulton · 2021-05-18T17:18:01Z

docs/source/release_notes.rst

+        * Added ``inverse_transform`` method to pipelines :pr:`2256``
    * Fixes
    * Changes
+        * Updated logging information in ``AutoMLSearch.__init__`` to clarify pipeline generation :pr:`2263`


@ParthivNaresh I'm moving this from the 0.24.1 release to the future release!

@freddyaboulton Sure thing!

ParthivNaresh

Excellent job with the tests, looks great!

ParthivNaresh · 2021-05-18T17:24:28Z

docs/source/release_notes.rst

+        * Added ``inverse_transform`` method to pipelines :pr:`2256``
    * Fixes
    * Changes
+        * Updated logging information in ``AutoMLSearch.__init__`` to clarify pipeline generation :pr:`2263`


@freddyaboulton Sure thing!

ParthivNaresh · 2021-05-18T17:26:21Z

evalml/pipelines/component_graph.py

+        # If there are target transformers present, connect them together and then pass the final output
+        # to the sampler (if present) or the final estimator
+        with_inverse_transform = list(filter(lambda tup: issubclass(tup[1], TargetTransformer), component_list))
+        without_inverse_transform = list(filter(lambda tup: not issubclass(tup[1], TargetTransformer), component_list))


Nicely done

bchen1116

Nice!

angela97lin

Left some nitpicky comments but otherwise, really cool stuff!!

angela97lin · 2021-05-18T22:26:14Z

evalml/pipelines/component_graph.py

+        Components that implement inverse_transform are PolynomialDetrender, LabelEncoder (tbd).
+
+        Arguments:
+            y: (pd.Series, ww.DataTable): Final component features


Typo, DataColumn?

angela97lin · 2021-05-20T18:16:51Z

evalml/pipelines/components/transformers/preprocessing/polynomial_detrender.py

+class PolynomialDetrender(TargetTransformer):
    """Removes trends from time series by fitting a polynomial to the data."""
    name = "Polynomial Detrender"
+    _defines_inverse_transform = True


What is this used for?

I need to get rid of this!

angela97lin · 2021-05-20T18:19:42Z

evalml/pipelines/components/transformers/transformer.py

+    """A component that transforms the target."""
+
+    @abstractmethod
+    def inverse_transform(self, y):


Could be useful in docstring here to specify arguments + what we expect to return for when people override :)

angela97lin · 2021-05-20T18:20:20Z

evalml/pipelines/pipeline_base.py

+        Components that implement inverse_transform are PolynomialDetrender, LabelEncoder (tbd).
+
+        Arguments:
+            y (pd.Series, ww.DataTable): Final component features


DataColumn? :O

Good catch!

angela97lin · 2021-05-20T18:23:30Z

evalml/tests/pipeline_tests/test_component_graph.py

+component_graphs = [(ComponentGraph({"Imputer": [Imputer],
+                                     "Log": [LogTransform],
+                                     "Random Forest": ["Random Forest Regressor", "Imputer.x", "Log.y"]}),
+                     lambda y: infer_feature_types(np.exp(y.to_series()))),
+                    (ComponentGraph({"Imputer": [Imputer],
+                                     "Log": [LogTransform],
+                                     "Double": [DoubleTransform, "Log.y"],
+                                     "Random Forest": ["Random Forest Regressor", "Imputer.x", "Double.y"]}),
+                     lambda y: infer_feature_types(np.exp(y.to_series() / 2))),
+                    (ComponentGraph({"Imputer": [Imputer],
+                                     "Log": [LogTransform, "Imputer.x"],
+                                     "Double": [DoubleTransform, "Log.x", "Log.y"],
+                                     "Random Forest": ["Random Forest Regressor", "Double.x", "Double.y"]}),
+                     lambda y: infer_feature_types(np.exp(y.to_series() / 2))),
+                    (ComponentGraph({"Imputer": [Imputer],
+                                     "OneHot": [OneHotEncoder, "Imputer.x"],
+                                     "DateTime": [DateTimeFeaturizer, "OneHot.x"],
+                                     "Log": [LogTransform],
+                                     "Double": [DoubleTransform, "Log.y"],
+                                     "Random Forest": ["Random Forest Regressor", "DateTime.x", "Double.y"]}),
+                     lambda y: infer_feature_types(np.exp(y.to_series() / 2))),
+                    (ComponentGraph({"Imputer": [Imputer],
+                                     "OneHot": [OneHotEncoder, "Imputer.x"],
+                                     "DateTime": [DateTimeFeaturizer, "OneHot.x"],
+                                     "Log": [LogTransform],
+                                     "Double": [DoubleTransform, "Log.y"],
+                                     "Double2": [DoubleTransform, "Double.y"],
+                                     "Random Forest": ["Random Forest Regressor", "DateTime.x", "Double2.y"]}),
+                     lambda y: infer_feature_types(np.exp(y.to_series() / 4))),
+                    (ComponentGraph({"Imputer": ['Imputer'],
+                                     "Double": [DoubleTransform],
+                                     "DateTime 1": ["DateTime Featurization Component", "Imputer"],
+                                     "ET": ["Extra Trees Regressor", "DateTime 1.x", "Double.y"],
+                                     "Double 2": [DoubleTransform],
+                                     "DateTime 2": ["DateTime Featurization Component", "Imputer"],
+                                     "Double 3": [DoubleTransform, "Double 2.y"],
+                                     "RandomForest": ["Random Forest Regressor", "DateTime 2.x", "Double 3.y"],
+                                     "DateTime 3": ["DateTime Featurization Component", "Imputer"],
+                                     "Double 4": [DoubleTransform],
+                                     "Linear": ["Linear Regressor", "DateTime 3.x", "Double 4.y"],
+                                     "Logistic Regression": ["Linear Regressor", "Linear", "RandomForest", "ET", 'Double 3.y']}),
+                     lambda y: infer_feature_types(y.to_series() / 4)),
+                    (ComponentGraph({"Imputer": [Imputer],
+                                     "OneHot": [OneHotEncoder, "Imputer.x"],
+                                     "DateTime": [DateTimeFeaturizer, "OneHot.x"],
+                                     "Log": [LogTransform],
+                                     "Double": [DoubleTransform, "Log.y"],
+                                     "Double2": [DoubleTransform, "Double.y"],
+                                     "Subset": [SubsetData, "DateTime.x", "Double2.y"],
+                                     "Random Forest": ["Random Forest Regressor", "Subset.x", "Subset.y"]}),
+                     lambda y: infer_feature_types(np.exp(y.to_series() / 4))),
+                    (ComponentGraph({"Imputer": [Imputer],
+                                     "Random Forest": ["Random Forest Regressor", "Imputer.x"]}),
+                     lambda y: y),
+                    (ComponentGraph({"Imputer": [Imputer],
+                                     "DateTime": [DateTimeFeaturizer, "Imputer.x"],
+                                     "OneHot": [OneHotEncoder, "DateTime.x"],
+                                     "Random Forest": ["Random Forest Regressor", "OneHot.x"]}),
+                     lambda y: y),
+                    (ComponentGraph({"Random Forest": ["Random Forest Regressor"]}),
+                     lambda y: y),
+                    (ComponentGraph({"Imputer": ['Imputer'],
+                                     "Double": [DoubleTransform],
+                                     "DateTime 1": ["DateTime Featurization Component", "Imputer"],
+                                     "ET": ["Extra Trees Regressor", "DateTime 1.x", "Double.y"],
+                                     "Double 2": [DoubleTransform],
+                                     "DateTime 2": ["DateTime Featurization Component", "Imputer"],
+                                     "Double 3": [DoubleTransform, "Double 2.y"],
+                                     "RandomForest": ["Random Forest Regressor", "DateTime 2.x", "Double 3.y"],
+                                     "DateTime 3": ["DateTime Featurization Component", "Imputer"],
+                                     "Double 4": [DoubleTransform],
+                                     "Linear": ["Linear Regressor", "DateTime 3.x", "Double 4.y"],
+                                     "Logistic Regression": ["Linear Regressor", "Linear", "RandomForest", "ET"]}),
+                     lambda y: y)
+                    ]


holy potates

freddyaboulton force-pushed the 2046-inverse-transform branch from 842f409 to a795b19 Compare May 11, 2021 22:26

freddyaboulton commented May 11, 2021

View reviewed changes

freddyaboulton force-pushed the 2046-inverse-transform branch 2 times, most recently from 0c52447 to 2c640ca Compare May 13, 2021 14:09

freddyaboulton commented May 13, 2021

View reviewed changes

freddyaboulton force-pushed the 2046-inverse-transform branch from 2c640ca to e6f7068 Compare May 13, 2021 17:01

freddyaboulton marked this pull request as ready for review May 13, 2021 20:09

auto-assign bot assigned freddyaboulton May 13, 2021

freddyaboulton requested review from ParthivNaresh, angela97lin, bchen1116, chukarsten, dsherry and jeremyliweishih May 13, 2021 20:09

freddyaboulton force-pushed the 2046-inverse-transform branch 2 times, most recently from 7b67aa8 to 180b2dc Compare May 14, 2021 17:50

chukarsten approved these changes May 16, 2021

View reviewed changes

freddyaboulton force-pushed the 2046-inverse-transform branch from 180b2dc to 1dbecbf Compare May 18, 2021 15:04

bchen1116 reviewed May 18, 2021

View reviewed changes

freddyaboulton force-pushed the 2046-inverse-transform branch from ed17111 to f54bc19 Compare May 18, 2021 17:17

freddyaboulton commented May 18, 2021

View reviewed changes

ParthivNaresh approved these changes May 18, 2021

View reviewed changes

bchen1116 approved these changes May 18, 2021

View reviewed changes

freddyaboulton force-pushed the 2046-inverse-transform branch 3 times, most recently from 69dfa01 to 94b690d Compare May 19, 2021 14:32

angela97lin reviewed May 20, 2021

View reviewed changes

freddyaboulton force-pushed the 2046-inverse-transform branch 3 times, most recently from b17e826 to df40b9a Compare May 25, 2021 16:11

freddyaboulton force-pushed the 2046-inverse-transform branch 2 times, most recently from 8705adf to d9e484d Compare June 7, 2021 16:59

freddyaboulton added 15 commits June 7, 2021 16:47

Adding inverse transform

70fe872

Release notes for PR 2256

8f521c8

Inverse transform only takes y

031d1a7

Skipping polynomial detrender test

174aef9

Adding docstring for pipeline inverse transform

57e909d

Add PR 2256 to future release section

72222d9

Docstring for SubsetData test transformer

8ae5136

Fix PipelineBase.inverse_transform docstring

33ed951

Update release notes

8e7827b

Change variable names in ComponentGraph.from_list

4debf98

Add/fix docstrings

381f103

Deleting whitespace

d344945

Linting polynomial detrender

f331b0f

Fixing unit tests

f0a7802

Linting with black

a67528c

freddyaboulton force-pushed the 2046-inverse-transform branch from d9e484d to a67528c Compare June 7, 2021 20:47

freddyaboulton merged commit 9c414d3 into main Jun 7, 2021

freddyaboulton deleted the 2046-inverse-transform branch June 7, 2021 21:53

This was referenced Jun 9, 2021

Release v0.26.0 #2352

Merged

ReadTheDocs PR CI: PDFLATEX flake #2355

Closed

chukarsten mentioned this pull request Jun 22, 2021

Release v0.27.0 #2428

Merged

freddyaboulton mentioned this pull request Aug 24, 2021

Delete TargetTransformer class #2687

Closed

		return infer_feature_types(y / 2)


		class SubsetData(Transformer):

		sampler_index = next(iter([i for i, tup in enumerate(without_inverse_transform) if "sampler" in tup[0]]), -1)
		component_dict[without_inverse_transform[sampler_index][0]].append(f"{with_inverse_transform[-1][0]}.y")

Adding inverse transform method to pipelines #2256

Adding inverse transform method to pipelines #2256

Uh oh!

Conversation

freddyaboulton commented May 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Description

Uh oh!

freddyaboulton May 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

freddyaboulton May 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented May 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chukarsten left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bchen1116 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

freddyaboulton May 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ParthivNaresh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

freddyaboulton commented May 11, 2021 •

edited

Loading

freddyaboulton May 11, 2021 •

edited

Loading

freddyaboulton May 11, 2021 •

edited

Loading

codecov bot commented May 12, 2021 •

edited

Loading

freddyaboulton May 18, 2021 •

edited

Loading