Classification Pipelines for Time Series #1528

freddyaboulton · 2020-12-08T22:42:43Z

Pull Request Description

I'll add to AutoML in a separate PR since this is a big diff.

Most of the changes come from adding time series binary and time series multiclass problems types to the list of supported problem types for each existing classifier. I would ignore the diff in the estimators and estimator tests and focus on the diff in pipeline_base_meta.py, time_series_classification_pipelines.py and test_time_series_pipeline.py

After creating the pull request: in order to pass the release_notes_updated check you will need to update the "Future Release" section of docs/source/release_notes.rst to include this pull request by adding :pr:123.

codecov · 2020-12-08T22:50:53Z

Codecov Report

Merging #1528 (f9733d4) into main (a48ebfb) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@            Coverage Diff            @@
##             main    #1528     +/-   ##
=========================================
+ Coverage   100.0%   100.0%   +0.1%     
=========================================
  Files         238      239      +1     
  Lines       17136    17406    +270     
=========================================
+ Hits        17128    17398    +270     
  Misses          8        8

Impacted Files	Coverage Δ
evalml/tests/utils_tests/test_gen_utils.py	`100.0% <ø> (ø)`
evalml/utils/__init__.py	`100.0% <ø> (ø)`
...alml/objectives/binary_classification_objective.py	`100.0% <100.0%> (ø)`
.../objectives/multiclass_classification_objective.py	`100.0% <100.0%> (ø)`
evalml/pipelines/__init__.py	`100.0% <100.0%> (ø)`
evalml/pipelines/classification_pipeline.py	`100.0% <100.0%> (ø)`
...components/ensemble/stacked_ensemble_classifier.py	`100.0% <100.0%> (ø)`
...ents/estimators/classifiers/catboost_classifier.py	`100.0% <100.0%> (ø)`
...estimators/classifiers/decision_tree_classifier.py	`100.0% <100.0%> (ø)`
...ts/estimators/classifiers/elasticnet_classifier.py	`100.0% <100.0%> (ø)`
... and 22 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a48ebfb...f9733d4. Read the comment docs.

freddyaboulton · 2020-12-10T16:36:59Z

evalml/pipelines/pipeline_base_meta.py

@@ -24,5 +25,9 @@ def _check_for_fit(self, X=None, y=None):
            elif y is None:
                return method(self, X)
            else:
+                # For time series classification pipelines, predict will take X, y, objective


The problem is that TimeSeriesBinaryClassification.predict accepts X, y and objective. All other pipeline classes accept X and y (TimeSeriesRegressionPipeline) or X and objective (BinaryClassificationPipeline, MulticlassClassificationPipeline) or X (ClassificationPipeline).

I think adding this extra check for the number of arguments in predict is the lowest-touch way of handling the complexity (as opposed to adding another metaclass) especially since this class is not user-facing and we don't expect users to define their own pipeline base classes.

That being said, maybe we can refactor this metaclass definition to make it a little clearer what's going on?

I think checking # of arguments is a good idea. Maybe we can just refactor it into the usual if else logic?

if not self._is_fitted: raise PipelineNotYetFittedError(f'This {klass} is not fitted yet. You must fit {klass} before calling {method.__name__}.') elif X is None and y is None: return method(self) elif y is None: return method(self, X) # For time series classification pipelines, predict will take X, y, objective elif len(inspect.getfullargspec(method).args) == 4: return method(self, X, y, objective) else: # For other pipelines, predict will take X, y or X, objective return method(self, X, y)

freddyaboulton · 2020-12-10T17:02:41Z

evalml/pipelines/time_series_classifcation_pipelines.py

+        X = _convert_woodwork_types_wrapper(X.to_dataframe())
+        y = _convert_woodwork_types_wrapper(y.to_series())
+        self._encoder.fit(y)
+        y = self._encode_targets(y)


Throughout, we need to encode targets before computing features because we allow the user to pass in string-valued targets. If there is a DelayedFeatureTransformer in the pipeline, these string values will get added to the feature matrix and if there isn't a OneHotEncoder, the estimator will crash.

Long story short, this is a repeat of #1477 . I think it would be better if we pass this complexity to the DelayedFeatureTransformer and have it encode the target before delaying but we should not let that block this PR. Let me know what you guys think. I'm happy to file something for updating the DelayedFeatureTransformer!

Agree with you here! I like the idea of filing an issue and updating the DelayedFeatureTransformer as well as removing encoding in the pipelines.

@freddyaboulton ah agreed that it would be great to have DelayedFeatureTransformer be able to handle this.

Can you call super().fit(X, y) here instead of duplicating code with ClassificationPipeline.fit? ClassificationPipeline already sets up the encoder.

Oh, my bad, I forgot about _compute_features_during_fit

In any case, I wonder if there's a way to avoid duplicating code for the classification target encoder.

When the dust settles and we build more features, I want to see if there is a way to refactor these classes to avoid duplication between classification and time series classification pipelines.

It's not apparent right now because the "time-series-specific" code gets sprinkled in between the "non-time-series" code so it's not easy to just isolate the time series code into a helper function/mix-in yet. But maybe a new design will be clearer when this one becomes more stable and we test it!

freddyaboulton · 2020-12-10T17:07:53Z

evalml/pipelines/time_series_classifcation_pipelines.py

+        self.estimator.fit(X_t, y_shifted)
+        return self
+
+    def _predict(self, X, y, objective=None, pad=False):


I have added a pad parameter to _predict because in predict we want to return un-padded predictions or else self._decode_targets will error out. However in _compute_predictions (called by score) we want to return padded predictions (so that predictions, predicted probabilities, and the shifted target all have the same length. This makes dropping nans much easier).

freddyaboulton · 2020-12-10T17:15:37Z

evalml/pipelines/time_series_classifcation_pipelines.py

+        y_shifted = y_encoded.shift(-self.gap)
+        y_pred, y_pred_proba = self._compute_predictions(X, y, objectives)
+        non_nan_mask = _get_rows_without_nans(y_shifted, y_pred, y_pred_proba)
+        if y_pred is not None:


This is a bit ugly but depending on the objectives that are passed in, y_pred or y_pred_proba can be None so we can only drop nans if possible.

I've thought about avoiding this complexity by having _compute_predictions return un-padded predictions so that we don't have to worry about dropping anything. However, that doesn't really work because the predictions and y_shifted will have different number of nans depending on the values of gap and the number of delayed features computed in the pipeline. The least complex way of knowing what the common non-nan subset is by looking at the nans in the predictions and targets.

In short, we may be able to hide this complexity in a helper function but I don't think we can avoid it.

freddyaboulton · 2020-12-10T17:16:33Z

evalml/pipelines/time_series_classifcation_pipelines.py

+            if not objective.is_defined_for_problem_type(self.problem_type):
+                raise ValueError(f"Objective {objective.name} is not defined for time series binary classification.")
+
+        if self.threshold is None:


This looks a bit different from BinaryClassificationPipeline so that I only have to call pad_with_nans once.

freddyaboulton · 2020-12-10T17:17:27Z

evalml/tests/pipeline_tests/test_time_series_pipeline.py

+@patch('evalml.objectives.BinaryClassificationObjective.decision_function')
+@patch('evalml.pipelines.components.Estimator.predict_proba', return_value=pd.DataFrame({0: [1.]}))
+@patch('evalml.pipelines.components.Estimator.predict', return_value=pd.Series([1.]))
+def test_binary_classification_predictions_thresholded_properly(mock_predict, mock_predict_proba,


This is to achieve 100% coverage on the binary classification _predict method.

freddyaboulton · 2020-12-10T17:23:31Z

evalml/tests/pipeline_tests/test_time_series_pipeline.py

+                                                       (TimeSeriesRegressionPipeline, ['R2']),
+                                                       (TimeSeriesRegressionPipeline, ['R2', "Mean Absolute Percentage Error"])])
+@pytest.mark.parametrize("use_ww", [True, False])
+def test_score_works(pipeline_class, objectives, use_ww, X_y_binary, X_y_multi, X_y_regression):


I think it's good to have a unit test where we pass data through a minimal time series pipeline and make sure nothing errors out. There's a lot of stuff happening in these pipelines lol. I'm setting n_jobs=1 here and the test takes 2.5 seconds to run so I think the overall impact on test runtime is small.

jeremyliweishih

LGTM @freddyaboulton! Just left some questions and comments.

jeremyliweishih · 2020-12-14T21:09:34Z

evalml/pipelines/pipeline_base_meta.py

@@ -24,5 +25,9 @@ def _check_for_fit(self, X=None, y=None):
            elif y is None:
                return method(self, X)
            else:
+                # For time series classification pipelines, predict will take X, y, objective


I think checking # of arguments is a good idea. Maybe we can just refactor it into the usual if else logic?

if not self._is_fitted: raise PipelineNotYetFittedError(f'This {klass} is not fitted yet. You must fit {klass} before calling {method.__name__}.') elif X is None and y is None: return method(self) elif y is None: return method(self, X) # For time series classification pipelines, predict will take X, y, objective elif len(inspect.getfullargspec(method).args) == 4: return method(self, X, y, objective) else: # For other pipelines, predict will take X, y or X, objective return method(self, X, y)

jeremyliweishih · 2020-12-14T21:10:12Z

evalml/pipelines/time_series_classifcation_pipelines.py

+    """Pipeline base class for time series classifcation problems."""
+
+    def __init__(self, parameters, random_state=0):
+        """Machine learning pipeline for time series regression problems made out of transformers and a classifier.


classification instead of regression

Good catch!

jeremyliweishih · 2020-12-14T21:11:25Z

evalml/pipelines/time_series_classifcation_pipelines.py

+        X = _convert_woodwork_types_wrapper(X.to_dataframe())
+        y = _convert_woodwork_types_wrapper(y.to_series())
+        self._encoder.fit(y)
+        y = self._encode_targets(y)


Agree with you here! I like the idea of filing an issue and updating the DelayedFeatureTransformer as well as removing encoding in the pipelines.

jeremyliweishih · 2020-12-14T21:12:50Z

evalml/pipelines/time_series_classifcation_pipelines.py

+
+        X_t = self._compute_features_during_fit(X, y)
+        if X_t.empty:
+            raise RuntimeError("Pipeline computed empty features during call to .fit. This means "


@freddyaboulton do we want behavior here to be the same as the current regression pipelines or should it be like after the fixes for #1517? I think this is fine for now but if we want to change behavior in the future we should add that info to #1517.

Good point! I think we should get this merged and then fix #1517 for all ts pipelines at once. I don't want the pipelines to have differing behavior if we make the changes needed for #1517 here and not simultaneously for regression.

bchen1116

Great work, Freddy! The logic for when to keep track of adding padding versus not seemed challenging, but everything looks good to me! I left a couple questions and a comment, but solid implementation and test coverage!

bchen1116 · 2020-12-15T16:21:31Z

evalml/pipelines/time_series_classifcation_pipelines.py

+        return X, y
+
+    def fit(self, X, y):
+        """Fit a time series regression pipeline.


classification

bchen1116 · 2020-12-15T23:03:42Z

evalml/utils/gen_utils.py

+        else:
+            return pd_data
+
+    mask = reduce(lambda a, b: np.logical_and(_not_nan(a), _not_nan(b)), data)


This is cool! So this means that if the second Dataframe/Series in the sequence had a nan value in the last column, then the last value in the mask will be False, like below?

Might be nice to update the return to mask where each entry is True if and only if all corresponding entries in that index in data are non-nan or something of the sort. I wasn't sure what this was supposed to do

Yes exactly. Great suggestion on the docstring!

bchen1116 · 2020-12-15T23:22:42Z

evalml/tests/pipeline_tests/test_time_series_pipeline.py

+        mock_regressor_predict.side_effect = mock_predict
+    else:
+        mock_classifier_predict.side_effect = mock_predict
+
    if only_use_y:
        pl.fit(None, y)


How does this work? Looking at the code for fit(), it seems like X is necessary to pass in, since you do call estimator.fit() on the computed features of X?

Great question! When X is None it's assumed that the pipeline will create some features from the y before passing to the estimator (typically DelayedFeatureTransformer). We're getting rid of this assumption in #1517, however, because some (future) estimators will naturally support the X=None case.

Gotcha, that makes sense. Thanks!

…sification.

…of what the expected output is.

…names_of_defined_objectives

…rks_for_names_of_defined_objectives" This reverts commit 6ba060a.

dsherry

@freddyaboulton I left a few comments, got halfway through the impl so far :)

dsherry · 2020-12-17T17:58:11Z

evalml/pipelines/classification_pipeline.py

@@ -50,7 +50,7 @@ def fit(self, X, y):
    def _encode_targets(self, y):
        """Converts target values from their original values to integer values that can be processed."""
        try:
-            return pd.Series(self._encoder.transform(y))
+            return pd.Series(self._encoder.transform(y), index=y.index)


Oh neat.

Do we need to do this elsewhere?

I know LightGBM and Catboost also use encoders internally. I'll make a note to see if we need to do something similar there and if so file an issue!

Great thinking!

dsherry · 2020-12-17T18:00:01Z

evalml/pipelines/pipeline_base_meta.py

            klass = type(self).__name__
            if not self._is_fitted:
                raise PipelineNotYetFittedError(f'This {klass} is not fitted yet. You must fit {klass} before calling {method.__name__}.')
            elif X is None and y is None:
                return method(self)
            elif y is None:
                return method(self, X)
+            # For time series classification pipelines, predict will take X, y, objective
+            elif len(inspect.getfullargspec(method).args) == 4:


@freddyaboulton does args list positional args only? Or does it include named args?

Positional arguments! So it includes any arguments that don't have * or ** in front of their name.

dsherry · 2020-12-17T18:00:23Z

evalml/pipelines/pipeline_base_meta.py

+            # For time series classification pipelines, predict will take X, y, objective
+            elif len(inspect.getfullargspec(method).args) == 4:
+                return method(self, X, y, objective)
+            # For other pipelines, predict will take X, y or X, objective


Is this correct? The else below doesn't pass along the objective

Yea it's correct but confusing 😅 .

For non-timeseries classification pipelines, the arguments will be X and objective so the y in this else is actually the objective the user passed in!

I left this comment earlier but I guess it got overwritten when I pushed new changes:

The problem is that TimeSeriesBinaryClassification.predict accepts X, y and objective. All other pipeline classes accept X and y (TimeSeriesRegressionPipeline) or X and objective (BinaryClassificationPipeline, MulticlassClassificationPipeline) or X (ClassificationPipeline).

I think adding this extra check for the number of arguments in predict is the lowest-touch way of handling the complexity (as opposed to adding another metaclass) especially since this class is not user-facing and we don't expect users to define their own pipeline base classes.

That being said, maybe we can refactor this metaclass definition to make it a little clearer what's going on?

dsherry · 2020-12-17T18:06:10Z

evalml/pipelines/time_series_classifcation_pipelines.py

+        """
+        if "pipeline" not in parameters:
+            raise ValueError("gap and max_delay parameters cannot be omitted from the parameters dict. "
+                             "Please specify them as a dictionary with the key 'pipeline'.")


👍 awesome

dsherry · 2020-12-17T18:09:00Z

evalml/pipelines/time_series_classifcation_pipelines.py

+            X = pd.DataFrame()
+        X = _convert_to_woodwork_structure(X)
+        y = _convert_to_woodwork_structure(y)
+        return X, y


Yep. I wonder if this would be good to move to PipelineBase or even utils, and use it throughout the codebase. @angela97lin FYI :)

Hm... could be, but the conversion to an empty DataFrame if X is None is a time series specific thing, no? In other cases, we'd probably want to raise something if the user passes None as X?

dsherry · 2020-12-17T18:17:36Z

evalml/pipelines/time_series_classifcation_pipelines.py

+        X = _convert_woodwork_types_wrapper(X.to_dataframe())
+        y = _convert_woodwork_types_wrapper(y.to_series())
+        self._encoder.fit(y)
+        y = self._encode_targets(y)


@freddyaboulton ah agreed that it would be great to have DelayedFeatureTransformer be able to handle this.

Can you call super().fit(X, y) here instead of duplicating code with ClassificationPipeline.fit? ClassificationPipeline already sets up the encoder.

dsherry · 2020-12-17T18:32:53Z

evalml/pipelines/time_series_classifcation_pipelines.py

+        X = _convert_woodwork_types_wrapper(X.to_dataframe())
+        y = _convert_woodwork_types_wrapper(y.to_series())
+        self._encoder.fit(y)
+        y = self._encode_targets(y)


Oh, my bad, I forgot about _compute_features_during_fit

In any case, I wonder if there's a way to avoid duplicating code for the classification target encoder.

dsherry · 2020-12-17T18:33:11Z

evalml/pipelines/time_series_classifcation_pipelines.py

@@ -0,0 +1,204 @@
+import pandas as pd


@freddyaboulton typo in filename: evalml/pipelines/time_series_classifcation_pipelines.py --> evalml/pipelines/time_series_classification_pipelines.py

Great catch! 😬

dsherry · 2020-12-17T18:38:00Z

evalml/pipelines/time_series_classifcation_pipelines.py

+        X, y = self._convert_to_woodwork(X, y)
+        X = _convert_woodwork_types_wrapper(X.to_dataframe())
+        y = _convert_woodwork_types_wrapper(y.to_series())
+        n_features = max(len(y), X.shape[0])


@freddyaboulton this is the same nan padding math we use for TS regression, yes?

angela97lin

Left a few nit-picky comments but otherwise looks good! (Will come back and read more into impl later when I get the chance hehe)

angela97lin · 2020-12-17T19:10:44Z

evalml/pipelines/time_series_classifcation_pipelines.py

+            parameters (dict): Dictionary with component names as keys and dictionary of that component's parameters as values.
+                 An empty dictionary {} implies using all default values for component parameters. Pipeline-level
+                 parameters such as gap and max_delay must be specified with the "pipeline" key. For example:
+                 Pipeline(parameters={"pipeline": {"max_delay": 4, "gap": 2}}).


Great example, nice and clear :)

angela97lin · 2020-12-17T19:12:55Z

evalml/pipelines/time_series_classifcation_pipelines.py

+                 Pipeline(parameters={"pipeline": {"max_delay": 4, "gap": 2}}).
+            random_state (int, np.random.RandomState): The random seed/state. Defaults to 0.
+        """
+        if "pipeline" not in parameters:


Really nit-picky, but what if a user passes in a "pipeline" key mapping to a dictionary with "gap" or "max_delay" missing? Ex: {pipeline: {"gap":1}} I think we get a KeyError in the following lines, right?

Yep you're right!

We'll have to fix the same thing in regression pipelines so I'll just file an issue and change it for both after this is merged!

angela97lin · 2020-12-17T19:14:43Z

evalml/pipelines/time_series_classifcation_pipelines.py

+            X = pd.DataFrame()
+        X = _convert_to_woodwork_structure(X)
+        y = _convert_to_woodwork_structure(y)
+        return X, y


Hm... could be, but the conversion to an empty DataFrame if X is None is a time series specific thing, no? In other cases, we'd probably want to raise something if the user passes None as X?

angela97lin · 2020-12-17T19:32:11Z

evalml/pipelines/classification_pipeline.py

@@ -50,7 +50,7 @@ def fit(self, X, y):
    def _encode_targets(self, y):
        """Converts target values from their original values to integer values that can be processed."""
        try:
-            return pd.Series(self._encoder.transform(y))
+            return pd.Series(self._encoder.transform(y), index=y.index)


Curious, did this break / how did you catch this? If so, could be good to add a test for it if there isn't one already 😬

The time series tests caught this! We pass a y with a custom index there.

freddyaboulton force-pushed the 1474-classification-pipelines-for-ts branch 3 times, most recently from ed2d59e to 1d380ca Compare December 10, 2020 16:11

freddyaboulton commented Dec 10, 2020

View reviewed changes

freddyaboulton marked this pull request as ready for review December 10, 2020 17:25

freddyaboulton self-assigned this Dec 10, 2020

freddyaboulton requested review from dsherry, angela97lin, bchen1116, jeremyliweishih, christopherbunn, eccabay and ParthivNaresh December 10, 2020 17:25

jeremyliweishih approved these changes Dec 14, 2020

View reviewed changes

freddyaboulton force-pushed the 1474-classification-pipelines-for-ts branch 2 times, most recently from db0c862 to 6c3d569 Compare December 15, 2020 15:22

bchen1116 approved these changes Dec 15, 2020

View reviewed changes

freddyaboulton force-pushed the 1474-classification-pipelines-for-ts branch from 6c3d569 to 8c87b40 Compare December 16, 2020 15:31

freddyaboulton added the new feature Features which don't yet exist. label Dec 16, 2020

freddyaboulton added this to the December 2020 milestone Dec 16, 2020

freddyaboulton added 3 commits December 17, 2020 12:52

Adding rough draft of classification pipelines.

f948f5d

Adding test cases for binary and multiclass pipelines.

22f835a

Updating problem type tests.

69bd1a4

freddyaboulton added 16 commits December 17, 2020 12:52

Encoding labels in time series classification predict.

0835c34

Fixing unit tests associated with new classification problem types

f23acfb

Adding PR 1528 to release notes.

43b39b1

Fixing syntax in test_classification_pipeline_encodes_targets

4b57046

Adding predict_proba

e639c89

Adding test for score

867b4db

Adding coverage for predict with objectives in timeseries binary clas…

23d8dea

…sification.

Adding get_rows_without_nans to make score a bit cleaner.

396094d

Adding time series classification pipelines to api reference.

d1f26ba

Fixing lint in TimeSeriesBinaryClassificationPipeline._predict.

6592c8b

Adding X is None check in _convert_to_woodwork method

1dd1b8b

Refactoring changes in PipelineBaseMeta.

38d5c77

Reverting accidental changes to automl user guide.

6be079a

Linting

404968e

Replace 'regression' with 'classification' in docstrings.

6afd9a2

Updating the docstring of _get_rows_without_nans to be a bit clearer …

e65ddcf

…of what the expected output is.

freddyaboulton force-pushed the 1474-classification-pipelines-for-ts branch from 8c87b40 to e65ddcf Compare December 17, 2020 17:52

freddyaboulton added 2 commits December 17, 2020 13:03

Removing objectives defined in tests in test_get_objective_works_for_…

6ba060a

…names_of_defined_objectives

Revert "Removing objectives defined in tests in test_get_objective_wo…

ba85cb3

…rks_for_names_of_defined_objectives" This reverts commit 6ba060a.

dsherry reviewed Dec 17, 2020

View reviewed changes

angela97lin approved these changes Dec 17, 2020

View reviewed changes

Fixing typo in time_series_classification_pipelines filename.

f9733d4

freddyaboulton merged commit 913919c into main Dec 17, 2020

freddyaboulton deleted the 1474-classification-pipelines-for-ts branch December 17, 2020 20:46

freddyaboulton mentioned this pull request Dec 18, 2020

Update DelayFeatureTransformer to encode string targets as ints #1581

Closed

dsherry mentioned this pull request Dec 29, 2020

Release v0.17.0 #1623

Merged

Classification Pipelines for Time Series #1528

Classification Pipelines for Time Series #1528

Conversation

freddyaboulton commented Dec 8, 2020 • edited Loading

Pull Request Description

codecov bot commented Dec 8, 2020 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freddyaboulton Dec 10, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freddyaboulton Dec 10, 2020 • edited Loading

Choose a reason for hiding this comment

jeremyliweishih left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bchen1116 left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freddyaboulton Dec 16, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freddyaboulton Dec 17, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

angela97lin left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freddyaboulton Dec 17, 2020 • edited Loading

Choose a reason for hiding this comment

freddyaboulton commented Dec 8, 2020 •

edited

Loading

codecov bot commented Dec 8, 2020 •

edited

Loading

freddyaboulton Dec 10, 2020 •

edited

Loading

freddyaboulton Dec 10, 2020 •

edited

Loading

bchen1116 left a comment •

edited

Loading

freddyaboulton Dec 16, 2020 •

edited

Loading

freddyaboulton Dec 17, 2020 •

edited

Loading

angela97lin left a comment •

edited

Loading

freddyaboulton Dec 17, 2020 •

edited

Loading