Merge branch 'develop' into issue566

docs/api/contrib/index.rst

@@ -1,14 +1,19 @@
 .. -*- mode: rst -*-
 
-Yellowbrick Contrib
-===================
+Contrib and Third-Party Libraries
+=================================
 
-The ``yellowbrick.contrib`` package contains a variety of extra tools and experimental visualizers that are outside of core support or are still in development. Here is a listing of the contrib modules currently available:
+Yellowbrick's primary dependencies are scikit-learn and matplotlib, however the data science landscape in Python is large and there are many opportunties to use Yellowbrick with other machine learning and data analysis frameworks. The ``yellowbrick.contrib`` package contains several methods and mechanisms to support non-scikit-learn machine learning as well as extra tools and experimental visualizers that are outside of core support or are still in development.
+
+.. note:: If you're interested in using a non-scikit-learn estimator with Yellowbrick, please see the :doc:`wrapper` documentation. If the wrapper doesn't work out of the box, we welcome contributions to this module to include other libraries!
+
+The following contrib packages are currently available:
 
 .. toctree::
-   :maxdepth: 2
+   :maxdepth: 1
 
-   boundaries
+   wrapper
    statsmodels
+   boundaries
    scatter
    missing/index

docs/api/contrib/scatter.rst

@@ -11,7 +11,7 @@ A scatter visualizer simply plots two features against each other and colors the
     :context: close-figs
     :alt: ScatterVisualizer on occupancy dataset
 
-    from yellowbrick.contrib import ScatterVisualizer
+    from yellowbrick.contrib.scatter import ScatterVisualizer
     from yellowbrick.datasets import load_occupancy
 
     # Load the classification dataset

docs/api/contrib/statsmodels.rst

@@ -1,8 +1,54 @@
 .. -*- mode: rst -*-
 
-StatsModels Visualizers
+statsmodels Visualizers
 =======================
 
+`statsmodels <https://www.statsmodels.org/stable/index.html>`_ is a Python library that provides utilities for the estimation of several statistical models and includes extensive results and metrics for each estimator. In particular, statsmodels excels at generalized linear models (GLMs) which are far superior to scikit-learn's implementation of ordinary least squares.
+
+This contrib module allows statsmodels users to take advantage of Yellowbrick visualizers by creating a wrapper class that implements the scikit-learn ``BaseEstimator``. Using the wrapper class, statsmodels can be passed directly to many visualizers, customized for the scoring and metric functionality required.
+
+.. warning:: The statsmodel wrapper is currently a prototype and as such is currently a bit trivial. Many options and extra functionality such as weights are not currently handled. We are actively looking for statsmodels users to contribute to this package!
+
+Using the statsmodels wrapper:
+
+.. code:: python
+
+    import statsmodels.api as sm
+
+    from functools import partial
+    from yellowbrick.regressor import ResidualsPlot
+    from yellowbrick.contrib.statsmodels import StatsModelsWrapper
+
+    glm_gaussian_partial = partial(sm.GLM, family=sm.families.Gaussian())
+    model = StatsModelsWrapper(glm_gaussian_partial)
+
+    viz = ResidualsPlot(model)
+    viz.fit(X_train, y_train)
+    viz.score(X_test, y_test)
+    viz.show()
+
+You can also use fitted estimators with the wrapper to avoid having to pass a partial function:
+
+.. code:: python
+
+    from yellowbrick.regressor import prediction_error
+
+    # Create the OLS model
+    model = sm.OLS(y, X)
+
+    # Get the detailed results
+    results = model.fit()
+    print(results.summary())
+
+    # Visualize the prediction error
+    prediction_error(StatsModelWrapper(model), X, y, is_fitted=True)
+
+This example also shows the use of a Yellowbrick oneliner, which is often more suited to the analytical style of statsmodels.
+
+
+API Reference
+-------------
+
 .. automodule:: yellowbrick.contrib.statsmodels.base
     :members: StatsModelsWrapper
     :undoc-members:

docs/api/contrib/wrapper.rst

@@ -0,0 +1,104 @@
+.. -*- mode: rst -*-
+
+Using Third-Party Estimators
+============================
+
+Many machine learning libraries implement the scikit-learn estimator API to easily integrate alternative optimization or decision methods into a data science workflow. Because of this, it seems like it should be simple to drop in a non-scikit-learn estimator into a Yellowbrick visualizer, and in principle, it is. However, the reality is a bit more complicated.
+
+Yellowbrick visualizers often utilize more than just the method interface of estimators (e.g. ``fit()`` and ``predict()``), relying on the learned attributes (object properties with a single underscore suffix, e.g. ``coef_``). The issue is that when a third-party estimator does not expose these attributes, truly gnarly exceptions and tracebacks occur. Yellowbrick is meant to aid machine learning diagnostics reasoning, therefore instead of just allowing drop-in functionality that may cause confusion, we've created a wrapper functionality that is a bit kinder with it's messaging.
+
+But first, an example.
+
+.. code:: python
+
+    # Import the wrap function and a Yellowbrick visualizer
+    from yellowbrick.contrib.wrapper import wrap
+    from yellowbrick.model_selection import feature_importances
+
+    # Instantiate the third party estimator and wrap it, optionally fitting it
+    model = wrap(ThirdPartyEstimator())
+    model.fit(X_train, y_train)
+
+    # Use the visualizer
+    oz = feature_importances(model, X_test, y_test, is_fitted=True)
+
+The ``wrap`` function initializes the third party model as a ``ContribEstimator``, which passes through all functionality to the underlying estimator, however if an error occurs, the exception that will be raised looks like:
+
+.. code:: text
+
+    yellowbrick.exceptions.YellowbrickAttributeError: estimator is missing the 'fit'
+    attribute, which is required for this visualizer - please see the third party
+    estimators documentation.
+
+Some estimators are required to pass type checking, for example the estimator must be a classifier, regressor, clusterer, density estimator, or outlier detector. A second argument can be passed to the ``wrap`` function declaring the type of estimator:
+
+.. code:: python
+
+    from yellowbrick.classifier import precision_recall_curve
+    from yellowbrick.contrib.wrapper import wrap, CLASSIFIER
+
+    model = wrap(ThirdPartyClassifier(), CLASSIFIER)
+    precision_recall_curve(model, X, y)
+
+Or you can simply use the wrap helper functions of the specific type:
+
+.. code:: python
+
+    from yellowbrick.contrib.wrapper import regressor, classifier, clusterer
+    from yellowbrick.regressor import prediction_error
+    from yellowbrick.classifier import classification_report
+    from yellowbrick.cluster import intercluster_distance
+
+    reg = regressor(ThirdPartyRegressor())
+    prediction_error(reg, X, y)
+
+    clf = classifier(ThirdPartyClassifier())
+    classification_report(clf, X, y)
+
+    ctr = clusterer(ThirdPartyClusterer())
+    intercluster_distance(ctr, X)
+
+
+So what should you do if a required attribute is missing from your estimator? The simplest and quickest thing to do is to subclass ``ContribEstimator`` and add the required functionality.
+
+.. code:: python
+
+    from yellowbrick.contrib.wrapper import ContribEstimator, CLASSIFIER
+
+    class MyWrapper(ContribEstimator):
+
+        _estimator_type = CLASSIFIER
+
+        @property
+        def feature_importances_(self):
+            return self.estimator.tree_feature_importances()
+
+
+    model = MyWrapper(ThirdPartyEstimator()
+    feature_importances(model, X, y)
+
+
+This is certainly less than ideal - but we'd welcome a contrib PR to add more native functionality to Yellowbrick!
+
+Tested Libraries
+----------------
+
+The following libraries have been tested with the Yellowbrick wrapper.
+
+- `xgboost <https://xgboost.readthedocs.io/en/latest/index.html>`_: both the ``XGBRFRegressor`` and ``XGBRFClassifier`` have been tested with Yellowbrick both with and without the wrapper functionality.
+- `CatBoost <https://catboost.ai/>`_: the ``CatBoostClassifier`` has been tested with the ``ClassificationReport`` visualizer.
+
+The following libraries have been partially tested and will likely work without too much additional effort:
+
+- `cuML <https://github.com/rapidsai/cuml>`_: it is likely that clustering, classification, and regression cuML estimators will work with Yellowbrick visualizers. However, the cuDF datasets have not been tested with Yellowbrick.
+- `Spark MLlib <https://spark.apache.org/docs/latest/ml-guide.html>`_: The Spark DataFrame API and estimators should work with Yellowbrick visualizers in a local notebook context after collection.
+
+.. note:: If you have used a Python machine learning library not listed here with Yellowbrick, please let us know - we'd love to add it to the list! Also if you're using a library that is not wholly compatible, please open an issue so that we can explore how to integrate it with the ``yellowbrick.contrib`` module!
+
+API Reference
+-------------
+
+.. automodule:: yellowbrick.contrib.wrapper
+    :members: wrap, classifier, regressor, clusterer, ContribEstimator
+    :undoc-members:
+    :show-inheritance:

docs/changelog.rst

@@ -3,12 +3,47 @@
 Changelog
 =========
 
+Version 1.2
+-----------
+
+* Tag: v1.2_
+* Deployed Friday, October 9, 2020
+* Current Contributors: Rebecca Bilbro, Larry Gray, Vladislav Skripniuk, David Landsman, Prema Roman, @aldermartinez, Tan Tran, Benjamin Bengfort, Kellen Donohue, Kristen McIntyre, Tony Ojeda, Edwin Schmierer, Adam Morris, Nathan Danielsen
+
+Major Changes:
+   - Added Q-Q plot as side-by-side option to the ``ResidualsPlot`` visualizer.
+   - More robust handling of binary classification in ``ROCAUC`` visualization, standardizing the way that classifiers with ``predict_proba`` and ``decision_function`` methods are handling. A ``binary`` hyperparameter was added to the visualizer to ensure correct interpretation of binary ROCAUC plots.
+   - Fixes to ``ManualAlphaSelection`` to move it from prototype to prime time including documentation, tests, and quick method. This method allows users to perform alpha selection visualization on non-CV estimators.
+   - Removal of AppVeyor from the CI matrix after too many out-of-core (non-Yellowbrick) failures with setup and installation on the VisualStudio images. Yellowbrick CI currently omits Windows and Miniconda from the test matrix and we are actively looking for new solutions.
+   - Third party estimator wrapper in contrib to provide enhanced support for non-scikit-learn estimators such as those in Keras, CatBoost, and cuML.
+
+Minor Changes:
+   - Allow users to specify colors for the ``PrecisionRecallCurve``.
+   - Update ``ClassificationScoreVisualizer`` base class to have a ``class_colors_`` learned attribute instead of a ``colors`` property; additional polishing of multi-class colors in ``PrecisionRecallCurve``, ``ROCAUC``, and ``ClassPredictionError``.
+   - Update ``KElbowVisualizer`` fit method and quick method to allow passing ``sample_weight`` parameter through the visualizer.
+   - Enhancements to classification documentation to better discuss precision and recall and to diagnose with ``PrecisionRecallCurve`` and ``ClassificationReport`` visualizers.
+   - Improvements to ``CooksDistance`` visualizer documentation.
+   - Corrected ``KElbowVisualizer`` label and legend formatting.
+   - Typo fixes to ``ROCAUC`` documentation, labels, and legend. Typo fix to ``Manifold`` documentation.
+   - Use of ``tight_layout`` accessing the Visualizer figure property to finalize images and resolve discrepancies in plot directive images in documentation.
+   - Add ``get_param_names`` helper function to identify keyword-only parameters that belong to a specific method.
+   - Splits package namespace for ``yellowbrick.regressor.residuals`` to move ``PredictionError`` to its own module, ``yellowbrick.regressor.prediction_error``.
+   - Update tests to use ``SVC`` instead of ``LinearSVC`` and correct ``KMeans`` scores based on updates to scikit-learn v0.23.
+   - Continued maintenance and management of baseline images following dependency updates; removal of mpl.cbook dependency.
+   - Explicitly include license file in source distribution via ``MANIFEST.in``.
+   - Fixes to some deprecation warnings from ``sklearn.metrics``.
+   - Testing requirements depends on Pandas v1.0.4 or later.
+   - Reintegrates pytest-spec and verbose test logging, updates pytest dependency to v0.5.0 or later.
+   - Added Pandas v0.20 or later to documentation dependencies.
+
+.. _v1.2: https://github.com/DistrictDataLabs/yellowbrick/releases/tag/v1.2
+
 Version 1.1
 -----------
 
 * Tag: v1.1_
 * Deployed Wednesday, February 12, 2020
-* Contributors: Benjamin Bengfort, Rebecca Bilbro, Kristen McIntyre, Larry Gray, Prema Roman, Adam Morris, Shivendra Sharma, Michael Chestnut, Michael Garod, Naresh Bachwani, Piyush Gautam, Daniel Navarrete, Molly Morrison, Emma Kwiecinska, Sarthak Jain, Tony Ojeda, Edwin Schmier,  Nathan Danielsen
+* Contributors: Benjamin Bengfort, Rebecca Bilbro, Kristen McIntyre, Larry Gray, Prema Roman, Adam Morris, Shivendra Sharma, Michael Chestnut, Michael Garod, Naresh Bachwani, Piyush Gautam, Daniel Navarrete, Molly Morrison, Emma Kwiecinska, Sarthak Jain, Tony Ojeda, Edwin Schmierer,  Nathan Danielsen
 
 Major Changes:
    - Quick methods (aka Oneliners), which return a fully fitted finalized visualizer object in only a single line, are now implemented for all Yellowbrick Visualizers. Test coverage has been added for all quick methods. The documentation has been updated to document and demonstrate the usage of the quick methods.
@@ -47,7 +82,7 @@ Version 1.0
 
 * Tag: v1.0_
 * Deployed Wednesday, August 28, 2019
-* Contributors: Benjamin Bengfort, Rebecca Bilbro, Nathan Danielsen, Kristen McIntyre, Larry Gray, Prema Roman, Adam Morris, Tony Ojeda, Edwin Schmier, Carl Dawson, Daniel Navarrete, Francois Dion, Halee Mason, Jeff Hale, Jiayi Zhang, Jimmy Shah, John Healy, Justin Ormont, Kevin Arvai, Michael Garod, Mike Curry, Nabanita Dash, Naresh Bachwani, Nicholas A. Brown, Piyush Gautam, Pradeep Singh, Rohit Ganapathy, Ry Whittington, Sangarshanan, Sourav Singh, Thomas J Fan, Zijie (ZJ) Poh, Zonghan, Xie
+* Contributors: Benjamin Bengfort, Rebecca Bilbro, Nathan Danielsen, Kristen McIntyre, Larry Gray, Prema Roman, Adam Morris, Tony Ojeda, Edwin Schmierer, Carl Dawson, Daniel Navarrete, Francois Dion, Halee Mason, Jeff Hale, Jiayi Zhang, Jimmy Shah, John Healy, Justin Ormont, Kevin Arvai, Michael Garod, Mike Curry, Nabanita Dash, Naresh Bachwani, Nicholas A. Brown, Piyush Gautam, Pradeep Singh, Rohit Ganapathy, Ry Whittington, Sangarshanan, Sourav Singh, Thomas J Fan, Zijie (ZJ) Poh, Zonghan, Xie
 
 .. warning:: **Python 2 Deprecation**: Please note that this release deprecates Yellowbrick's support for Python 2.7. After careful consideration and following the lead of our primary dependencies (NumPy, scikit-learn, and Matplolib), we have chosen to move forward with the community and support Python 3.4 and later.
 

docs/faq.rst

@@ -130,3 +130,11 @@ How can I access the sample datasets used in the examples?
 ----------------------------------------------------------
 
 Visit the :doc:`api/datasets/index` page.
+
+
+Can I use Yellowbrick with libraries other than scikit-learn?
+-------------------------------------------------------------
+
+Potentially! Yellowbrick visualizers rely on the internal model implementing the scikit-learn API (e.g. having a ``fit()`` and ``predict()`` method), and often expect to be able to retrieve learned attributes from the model (e.g. ``coef_``). Some third-party estimators fully implement the scikit-learn API, but not all do.
+
+When using third-party libraries with Yellowbrick, we encourage you to ``wrap`` the model using the ``yellowbrick.contrib.wrapper`` module. Visit the :doc:`api/contrib/wrapper` page for all the details!

tests/__init__.py

@@ -27,7 +27,7 @@
 ## Test Constants
 ##########################################################################
 
-EXPECTED_VERSION = "1.1"
+EXPECTED_VERSION = "1.2"
 
 
 ##########################################################################

tests/requirements.txt

@@ -18,3 +18,6 @@ nltk>=3.2
 pandas>=1.0.4
 umap-learn==0.3.9 # reminder to bump periodically!
 
+# Third-Party Estimator Tests
+# xgboost==1.2.0
+# catboost==0.24.1