Remove FairlearnDashboard and replace it with matplotlib-based plots

.github/ISSUE_TEMPLATE/bug_report.md

@@ -67,7 +67,7 @@ Sample code to reproduce the problem
 <!--
 Please provide the following information:
 - OS: [e.g. Windows]
-- Browser (if you're reporting a dashboard bug in jupyter): [e.g. Edge, Firefox, Chrome, Safari]
+- Browser (if you're reporting a bug in jupyter): [e.g. Edge, Firefox, Chrome, Safari]
 - Python version: [e.g. 3.7.4]
 - Fairlearn version: [e.g. 0.4.5 or installed from main branch in editable mode]
 - version of Python packages: please run the following snippet and paste the output:

.gitignore

@@ -118,6 +118,3 @@ docbuild/
 
 # OSX directory info files
 .DS_Store
-
-# widget files
-fairlearn/widget/js/node_modules/*

CODEOWNERS.md

@@ -5,7 +5,6 @@
 /fairlearn/metrics/ @riedgar-ms @rihorn2
 /fairlearn/postprocessing/ @romanlutz
 /fairlearn/reductions/ @riedgar-ms @romanlutz @MiroDudik
-/fairlearn/widget/ @rihorn2
 /jupyter-config @rihorn2
 /notebooks @romanlutz @riedgar-ms @rihorn2 @MiroDudik
 /scripts @riedgar-ms @romanlutz

README.md

@@ -2,15 +2,15 @@
 
 # Fairlearn
 
-Fairlearn is a Python package that empowers developers of artificial intelligence (AI) systems to assess their system's fairness and mitigate any observed unfairness issues. Fairlearn contains mitigation algorithms as well as a Jupyter widget for model assessment. Besides the source code, this repository also contains Jupyter notebooks with examples of Fairlearn usage.
+Fairlearn is a Python package that empowers developers of artificial intelligence (AI) systems to assess their system's fairness and mitigate any observed unfairness issues. Fairlearn contains mitigation algorithms as well as metrics for model assessment. Besides the source code, this repository also contains Jupyter notebooks with examples of Fairlearn usage.
 
 Website: https://fairlearn.github.io/
 
 - [Current release](#current-release)
 - [What we mean by _fairness_](#what-we-mean-by-fairness)
 - [Overview of Fairlearn](#overview-of-fairlearn)
   - [Fairlearn algorithms](#fairlearn-algorithms)
-  - [Fairlearn dashboard](#fairlearn-dashboard)
+  - [Fairlearn metrics](#fairlearn-metrics)
 - [Install Fairlearn](#install-fairlearn)
 - [Usage](#usage)
 - [Contributing](#contributing)
@@ -44,17 +44,17 @@ To learn more about our definitions of fairness, please visit our
 
 The Fairlearn Python package has two components:
 
-- A _dashboard_ for assessing which groups are negatively impacted by a model, and for comparing multiple models in terms of various fairness and accuracy metrics.
+- _Metrics_ for assessing which groups are negatively impacted by a model, and for comparing multiple models in terms of various fairness and accuracy metrics.
 
 - _Algorithms_ for mitigating unfairness in a variety of AI tasks and along a variety of fairness definitions.
 
 ### Fairlearn algorithms
 
 For an overview of our algorithms please refer to our [website](https://fairlearn.github.io/main/user_guide/mitigation.html).
 
-### Fairlearn dashboard
+### Fairlearn metrics
 
-Check out our in-depth [guide on the Fairlearn dashboard](https://fairlearn.github.io/main/user_guide/assessment.html#fairlearn-dashboard).
+Check out our in-depth [guide on the Fairlearn metrics](https://fairlearn.github.io/main/user_guide/assessment.html).
 
 ## Install Fairlearn
 

devops/PR-Gate.yml

@@ -37,6 +37,4 @@ jobs:
     freezeArtifactStem: $(FreezeArtifactStem)
     freezeFileStem: $(FreezeFileStem)
 
-- template: templates/build-widget-job-template.yml
-
 - template: templates/limited-installation-job-template.yml

devops/nightly-requirements-fixed.yml

@@ -30,5 +30,3 @@ jobs:
     freezeArtifactStem: $(FreezeArtifactStem)
     freezeFileStem: $(FreezeFileStem)
     pinRequirements: True
-
-- template: templates/build-widget-job-template.yml

devops/nightly.yml

@@ -37,6 +37,4 @@ jobs:
     freezeArtifactStem: $(FreezeArtifactStem)
     freezeFileStem: $(FreezeFileStem)
 
-- template: templates/build-widget-job-template.yml
-
 - template: templates/limited-installation-job-template.yml

devops/pypi-release-new.yml

@@ -59,8 +59,6 @@ stages:
       freezeArtifactStem: $(freezeArtifactStem)
       freezeFileStem: $(freezeFileStem)
 
-  - template: templates/build-widget-job-template.yml
-
   - template: templates/limited-installation-job-template.yml
 
 # ==================================================================================================
@@ -101,9 +99,6 @@ stages:
         freezeArtifact: $(freezeArtifactName)
         freezeFile: $(freezeFile)
 
-    - script: python ./scripts/build_widget.py --yarn-path /usr/local/bin/yarn
-      displayName: 'Build widget'
-
     - script: python ./scripts/build_wheels.py --version-filename $(versionFilename) 
       displayName: 'Build wheels'
 

docs/api_reference/index.rst

@@ -11,4 +11,3 @@ API Reference
    fairlearn.postprocessing
    fairlearn.preprocessing
    fairlearn.reductions
-   fairlearn.widget

docs/contributor_guide/contributing_example_notebooks.rst

@@ -35,10 +35,6 @@ The example notebooks in Fairlearn are surfaced through the website's
 :ref:`example notebook page <examples>`.
 This allows them to be rendered properly with output from all cells.
 
-.. note:
-
-    Rendering the Fairlearn dashboard is still an outstanding issue.
-
 These notebooks are generated based on `plot_*.py` files in
 `percent format <https://jupytext.readthedocs.io/en/latest/formats.html#the-percent-format>`_
 in the
@@ -70,11 +66,6 @@ opens a *Python Interactive* tab to show the output.
 If you prefer working with Jupyter simply open the `.py` file with Jupyter.
 Changes made in Jupyter automatically show up in the `.py` file.
 
-.. note:
-
-    The Fairlearn dashboard does not render in VS Code yet.
-    Jupyter will be required for examples that use the dashboard.
-
 Note that the text portion of these notebooks should be written in
 restructured text (ReST), not markdown, so that the sphinx documentation build
 can render it nicely for the website. When downloading the `.ipynb` file through

docs/contributor_guide/development_process.rst

@@ -65,11 +65,6 @@ command
 
     pip install -e .[customplots]
 
-Note that the Fairlearn dashboard is built using nodejs and requires
-additional steps. To build the Fairlearn dashboard after making changes to it,
-`install Yarn <https://yarnpkg.com/lang/en/docs/install>`_, and then run the
-`widget build script <https://github.com/fairlearn/fairlearn/tree/main/scripts/build_widget.py>`_.
-
 The Requirements Files
 """"""""""""""""""""""
 

docs/quickstart.rst

@@ -20,8 +20,6 @@ Fairlearn is also available on
 
     conda install -c conda-forge fairlearn
 
-For checking out the latest version in our repository check out our
-:ref:`advanced_install`.
 If you are updating from a previous version of Fairlearn, please
 see :ref:`version_migration_guide`.
 
@@ -32,14 +30,14 @@ see :ref:`version_migration_guide`.
     Please use the version selector to get to the instructions for
     the appropriate version. The instructions for the :code:`main`
     branch require Fairlearn to be installed from a clone of the
-    repository. See :ref:`advanced_install` for the required steps.
+    repository.
 
 Overview of Fairlearn
 ---------------------
 
 The Fairlearn package has two components:
 
-- A *dashboard* for assessing which groups are negatively impacted by a model,
+- *Metrics* for assessing which groups are negatively impacted by a model,
   and for comparing multiple models in terms of various fairness and accuracy
   metrics.
 
@@ -134,37 +132,45 @@ their label:
     sex
     Female    0.0635...
     Male      0.2135...
-    Name: selection_rate, dtype: object   
-
-For a visual representation of the metrics try out the Fairlearn dashboard.
-While this page shows only screenshots, the actual dashboard is interactive.
+    Name: selection_rate, dtype: object
 
-.. note::
+Fairlearn also allows us to quickly plot these metrics from the
+:class:`fairlearn.metrics.MetricFrame`
 
-    The :code:`FairlearnDashboard` is no longer being developed as
-    part of Fairlearn.
-    The widget itself has been moved to
-    `the raiwidgets package <https://pypi.org/project/raiwidgets/>`_.
-    Fairlearn will provide some of the existing functionality
-    through :code:`matplotlib`-based visualizations.
+.. Below the doctest contains the code shown on the website.
+   It should be kept in sync with the plot_quickstart.py example.
 
 .. doctest:: quickstart
 
-    >>> from fairlearn.widget import FairlearnDashboard
-    >>> FairlearnDashboard(sensitive_features=sex,
-    ...                    sensitive_feature_names=['sex'],
-    ...                    y_true=y_true,
-    ...                    y_pred={"initial model": y_pred}) # doctest: +SKIP
-
-.. image:: ../img/fairlearn-dashboard-start.png
-
-.. image:: ../img/fairlearn-dashboard-sensitive-features.png
-
-.. image:: ../img/fairlearn-dashboard-performance.png
+    >>> from sklearn.metrics import precision_score, recall_score
+    >>> from fairlearn.metrics import false_positive_rate, true_positive_rate
+    >>> metrics = {
+    ...     'accuracy': accuracy_score,
+    ...     'precision': precision_score,
+    ...     'recall': recall_score,
+    ...     'false positive rate': false_positive_rate,
+    ...     'true positive rate': true_positive_rate,
+    ...     'selection rate': selection_rate,
+    ...     'count': lambda y_true, y_pred: y_true.shape[0]}
+    >>> metric_frame = MetricFrame(metrics, y_true, y_pred, sensitive_features=sex)
+    >>> metric_frame.by_group.plot.bar(
+    ...     subplots=True, layout=[3,3], legend=False, figsize=[12,8],
+    ...     title='Show all metrics')
+    array([[<matplotlib.axes._subplots.AxesSubplot object at 0x0...>,
+            <matplotlib.axes._subplots.AxesSubplot object at 0x0...>,
+            <matplotlib.axes._subplots.AxesSubplot object at 0x0...>],
+           [<matplotlib.axes._subplots.AxesSubplot object at 0x0...>,
+            <matplotlib.axes._subplots.AxesSubplot object at 0x0...>,
+            <matplotlib.axes._subplots.AxesSubplot object at 0x0...>],
+           [<matplotlib.axes._subplots.AxesSubplot object at 0x0...>,
+            <matplotlib.axes._subplots.AxesSubplot object at 0x0...>,
+            <matplotlib.axes._subplots.AxesSubplot object at 0x0...>]],
+          dtype=object)
+
+.. figure:: auto_examples/images/sphx_glr_plot_quickstart_001.png
+    :target: auto_examples/plot_quickstart.html
+    :align: center 
 
-.. image:: ../img/fairlearn-dashboard-disparity-performance.png
-
-.. image:: ../img/fairlearn-dashboard-disparity-predictions.png
 
 Mitigating disparity
 ^^^^^^^^^^^^^^^^^^^^
@@ -201,19 +207,6 @@ a vastly reduced difference in selection rate:
     Male      0.1715...
     Name: selection_rate, dtype: object
 
-Similarly, we can explore the difference between the initial model and the
-mitigated model with respect to selection rate and accuracy in the dashboard
-through a multi-model comparison:
-
-.. doctest:: quickstart
-
-    >>> FairlearnDashboard(sensitive_features=sex,
-    ...                    sensitive_feature_names=['sex'],
-    ...                    y_true=y_true,
-    ...                    y_pred={"initial model": y_pred, "mitigated model": y_pred_mitigated}) # doctest: +SKIP
-
-.. image:: ../img/fairlearn-dashboard-comparison.png
-
 
 What's next?
 ------------