Move ROC and confusion matrix to pipeline plot utils (#696)

* Impl

* Rename file

* Update test

* Forgot to add the new file

* Fix imports and lint

* Fix test

* Changelog

* Fix api docs

* Add missing markdown descr. Delete some ipynb elements which were failing validation

* Update docstrings

* Delete MSLE for codecov

* Increase test coverage for codecov

* Disallow unsupported options for normalize_confusion_matrix

* Add test coverage for get_objective w\ nonetype (for codecov)

* Update docstring

* Remove unnecessary raise in test mock

* Update test.

* Update docstring

docs/source/api_reference.rst

@@ -103,10 +103,22 @@ Pipeline Utils
     :toctree: generated
     :nosignatures:
 
+    all_pipelines
     get_pipelines
     list_model_families
 
 
+Pipeline Plot Utils
+~~~~~~~~~~~~~~~~~~~
+.. autosummary::
+    :toctree: generated
+    :nosignatures:
+
+    roc_curve
+    confusion_matrix
+    normalize_confusion_matrix
+
+
 .. currentmodule:: evalml.pipelines.components
 
 Components
@@ -262,7 +274,6 @@ Regression Objectives
     R2
     MAE
     MSE
-    MSLE
     MedianAE
     MaxError
     ExpVariance
@@ -331,20 +342,6 @@ Guardrails
     detect_id_columns
 
 
-.. currentmodule:: evalml.objectives
-
-Plot Metrics
-============
-
-.. autosummary::
-    :toctree: generated
-    :template: class.rst
-    :nosignatures:
-
-    ROC
-    ConfusionMatrix
-
-
 .. currentmodule:: evalml.utils
 
 Utils
@@ -354,6 +351,8 @@ Utils
     :toctree: generated
     :nosignatures:
 
+    import_or_raise
     convert_to_seconds
-    normalize_confusion_matrix
+    get_random_state
+    get_random_seed
 

docs/source/automl/search_results.ipynb

@@ -7,9 +7,7 @@
     "# Exploring search results\n",
     "\n",
     "After finishing a pipeline search, we can inspect the results. First, let's build a search of 10 different pipelines to explore."
-   ],
-   "execution_count": null,
-   "outputs": []
+   ]
   },
   {
    "cell_type": "code",
@@ -34,9 +32,7 @@
    "source": [
     "## View Rankings\n",
     "A summary of all the pipelines built can be returned as a pandas DataFrame. It is sorted by score. EvalML knows based on our objective function whether higher or lower is better."
-   ],
-   "execution_count": null,
-   "outputs": []
+   ]
   },
   {
    "cell_type": "code",
@@ -53,9 +49,7 @@
    "source": [
     "## Describe Pipeline\n",
     "Each pipeline is given an `id`. We can get more information about any particular pipeline using that `id`. Here, we will get more information about the pipeline with `id = 0`."
-   ],
-   "execution_count": null,
-   "outputs": []
+   ]
   },
   {
    "cell_type": "code",
@@ -72,9 +66,7 @@
    "source": [
     "## Get Pipeline\n",
     "We can get the object of any pipeline via their `id` as well:"
-   ],
-   "execution_count": null,
-   "outputs": []
+   ]
   },
   {
    "cell_type": "code",
@@ -91,9 +83,7 @@
    "source": [
     "### Get best pipeline\n",
     "If we specifically want to get the best pipeline, there is a convenient access"
-   ],
-   "execution_count": null,
-   "outputs": []
+   ]
   },
   {
    "cell_type": "code",
@@ -111,9 +101,7 @@
     "## Feature Importances\n",
     "\n",
     "We can get the feature importances of the resulting pipeline"
-   ],
-   "execution_count": null,
-   "outputs": []
+   ]
   },
   {
    "cell_type": "code",
@@ -130,9 +118,7 @@
    "metadata": {},
    "source": [
     "We can also create a bar plot of the feature importances"
-   ],
-   "execution_count": null,
-   "outputs": []
+   ]
   },
   {
    "cell_type": "code",
@@ -143,6 +129,15 @@
     "pipeline.graph_feature_importance()"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Access raw results\n",
+    "\n",
+    "You can also get access to all the underlying data, like this:"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,

docs/source/changelog.rst

@@ -40,6 +40,7 @@ Changelog
         * Changed AutoML's `describe_pipeline` to get problem type from pipeline instead :pr:`685`
         * Standardize `import_or_raise` error messages :pr:`683`
         * Renamed `pipeline.feature_importance_graph` to `pipeline.graph_feature_importances` :pr:`700`
+        * Moved ROC and confusion matrix methods to `evalml.pipelines.plot_utils` :pr:`696`
     * Documentation Changes
         * Fixed some sphinx warnings :pr:`593`
         * Fixed docstring for AutoClassificationSearch with correct command :pr:`599`
@@ -58,6 +59,7 @@ Changelog
         * Reorganize API ref, and clarify pipeline sub-titles :pr:`688`
         * Add and update preprocessing utils in API reference :pr:`687`
         * Documented which default objective AutoML optimizes for :pr:`699`
+        * Include more utils in API ref, like `import_or_raise` :pr:`696`
     * Testing Changes
         * Matched install commands of `check_latest_dependencies` test and it's GitHub action :pr:`578`
         * Added Github app to auto assign PR author as assignee :pr:`477`
@@ -73,11 +75,12 @@ Changelog
     * ``fit()`` and ``predict()`` now use an optional ``objective`` parameter, which is only used in binary classification pipelines to fit for a specific objective.
     * ``score()`` will now use a required ``objectives`` parameter that is used to determine all the objectives to score on. This differs from the previous behavior, where the pipeline's objective was scored on regardless.
     * ``score()`` will now return one dictionary of all objective scores.
-    * ``ROC`` and ``ConfusionMatrix`` plot methods via ``Auto(*).plot`` will currently fail due to :pr:`615`
+    * ``ROC`` and ``ConfusionMatrix`` plot methods via ``Auto(*).plot`` have been removed by :pr:`615` and are replaced by ``roc_curve`` and ``confusion_matrix`` in `evamlm.pipelines.plot_utils`` in :pr:`696`
+    * ``normalize_confusion_matrix`` has been moved to ``evalml.pipelines.plot_utils`` :pr:`696`
     * Pipelines ``_name`` field changed to ``custom_name``
     * Pipelines ``supported_problem_types`` field is removed because it is no longer necessary :pr:`678`
     * `pipeline.feature_importance_graph` has been renamed to `pipeline.graph_feature_importances` in :pr:`700`
-
+    * Removed unsupported ``MSLE`` objective :pr:`696`
 
 
 **v0.8.0 Apr. 1, 2020**

evalml/automl/auto_base.py

@@ -92,7 +92,6 @@ def __init__(self, problem_type, tuner, cv, objective, max_pipelines, max_time,
             self.search_spaces[p.name] = [s[0] for s in space]
         self._MAX_NAME_LEN = 40
         self._next_pipeline_class = None
-        self.plot_metrics = []
         try:
             self.plot = PipelineSearchPlots(self)
 
@@ -277,7 +276,6 @@ def _do_iteration(self, X, y, pbar, raise_errors):
 
         start = time.time()
         cv_data = []
-        plot_data = []
 
         for train, test in self.cv.split(X, y):
             if isinstance(X, pd.DataFrame):
@@ -325,8 +323,7 @@ def _do_iteration(self, X, y, pbar, raise_errors):
         self._add_result(trained_pipeline=pipeline,
                          parameters=parameters,
                          training_time=training_time,
-                         cv_data=cv_data,
-                         plot_data=plot_data)
+                         cv_data=cv_data)
 
         desc = "✔" + desc[1:]
         pbar.set_description_str(desc=desc, refresh=True)
@@ -342,7 +339,7 @@ def _propose_parameters(self, pipeline_class):
         proposal = zip(space, values)
         return list(proposal)
 
-    def _add_result(self, trained_pipeline, parameters, training_time, cv_data, plot_data):
+    def _add_result(self, trained_pipeline, parameters, training_time, cv_data):
         scores = pd.Series([fold["score"] for fold in cv_data])
         score = scores.mean()
 
@@ -368,8 +365,7 @@ def _add_result(self, trained_pipeline, parameters, training_time, cv_data, plot
             "score": score,
             "high_variance_cv": high_variance_cv,
             "training_time": training_time,
-            "cv_data": cv_data,
-            "plot_data": plot_data
+            "cv_data": cv_data
         }
 
         self.results['search_order'].append(pipeline_id)

evalml/automl/auto_classification_search.py

@@ -3,7 +3,7 @@
 
 from .auto_base import AutoBase
 
-from evalml.objectives import ROC, ConfusionMatrix, get_objective
+from evalml.objectives import get_objective
 from evalml.problem_types import ProblemTypes
 
 
@@ -112,7 +112,3 @@ def __init__(self,
             verbose=verbose,
             optimize_thresholds=optimize_thresholds
         )
-        if self.problem_type == ProblemTypes.BINARY:
-            self.plot_metrics = [ROC(), ConfusionMatrix()]
-        else:
-            self.plot_metrics = [ConfusionMatrix()]

evalml/objectives/__init__.py

@@ -24,7 +24,6 @@
     MAE,
     MedianAE,
     MSE,
-    MSLE,
     Precision,
     PrecisionMacro,
     PrecisionMicro,
@@ -34,8 +33,6 @@
     RecallMacro,
     RecallMicro,
     RecallWeighted,
-    ROC,
-    ConfusionMatrix
 )
 from .utils import get_objective, get_objectives
 from .binary_classification_objective import BinaryClassificationObjective

evalml/objectives/standard_metrics.py

@@ -1,10 +1,6 @@
-from abc import ABC, abstractmethod
-
 import numpy as np
-import pandas as pd
 from sklearn import metrics
 from sklearn.preprocessing import label_binarize
-from sklearn.utils.multiclass import unique_labels
 
 from .binary_classification_objective import BinaryClassificationObjective
 from .multiclass_classification_objective import MultiClassificationObjective
@@ -294,16 +290,6 @@ def objective_function(self, y_predicted, y_true, X=None):
         return metrics.mean_squared_error(y_true, y_predicted)
 
 
-class MSLE(RegressionObjective):
-    """Mean squared log error for regression"""
-    name = "MSLE"
-    greater_is_better = False
-    score_needs_proba = False
-
-    def objective_function(self, y_predicted, y_true, X=None):
-        return metrics.mean_squared_log_error(y_true, y_predicted)
-
-
 class MedianAE(RegressionObjective):
     """Median absolute error for regression"""
     name = "MedianAE"
@@ -334,35 +320,6 @@ def objective_function(self, y_predicted, y_true, X=None):
         return metrics.explained_variance_score(y_true, y_predicted)
 
 
-class PlotMetric(ABC):
-    name = None
-    score_needs_proba = False
-
-    @abstractmethod
-    def score(self, y_predicted, y_true):
-        raise NotImplementedError("score() is not implemented!")
-
-
-class ROC(PlotMetric):
-    """Receiver Operating Characteristic score for binary classification."""
-    name = "ROC"
-    score_needs_proba = True
-
-    def score(self, y_predicted, y_true):
-        return metrics.roc_curve(y_true, y_predicted)
-
-
-class ConfusionMatrix(PlotMetric):
-    """Confusion matrix for binary and multiclass classification problems"""
-    name = "Confusion Matrix"
-
-    def score(self, y_predicted, y_true):
-        labels = unique_labels(y_predicted, y_true)
-        conf_mat = metrics.confusion_matrix(y_true, y_predicted)
-        conf_mat = pd.DataFrame(conf_mat, columns=labels)
-        return conf_mat
-
-
 def _handle_predictions(y_true, y_pred):
     if len(np.unique(y_true)) > 2:
         classes = np.unique(y_true)

evalml/objectives/utils.py

@@ -32,7 +32,6 @@
     "r2": standard_metrics.R2(),
     "mae": standard_metrics.MAE(),
     "mse": standard_metrics.MSE(),
-    # "msle": standard_metrics.MSLE(), removed due to not taking in positive Y
     "median_ae": standard_metrics.MedianAE(),
     "max_error": standard_metrics.MaxError(),
     "exp_var": standard_metrics.ExpVariance()

evalml/pipelines/__init__.py

@@ -47,3 +47,8 @@
     get_pipelines,
     list_model_families
 )
+from .plot_utils import (
+    roc_curve,
+    confusion_matrix,
+    normalize_confusion_matrix
+)