Is there a way to export a report of the metrics?

[MartaMarchiori]

Hi there, thanks for your work :)

Is there a way to export a comprehensive file/report of the metrics showed in the Fairness Dashboard? Or a line of code through Fairlearn.Metrics to do so?

Many thanks

[romanlutz]

Hi @MartaMarchiori !

One option would be to take a screenshot of the dashboard. Not ideal, I know, but the dashboard is being removed from Fairlearn at the moment #766 in favor of simpler plots. #766 has a few screenshots of what that looks like, although you don't really need to wait for any PRs to get merged since the functionality already exists. With any MetricFrame you can simply plot it, e.g.,

>>> from sklearn.metrics import precision_score, recall_score, accuracy_score
>>> from fairlearn.metrics import false_positive_rate, true_positive_rate, selection_rate
>>> metrics = {
...     'accuracy': accuracy_score,
...     'precision': precision_score,
...     'recall': recall_score,
...     'false positive rate': false_positive_rate,
...     'true positive rate': true_positive_rate,
...     'selection rate': selection_rate,
...     'count': lambda y_true, y_pred: y_true.shape[0]}
>>> metric_frame = MetricFrame(metrics, y_true, y_pred, sensitive_features=sex)
>>> metric_frame.by_group.plot.bar(
...     subplots=True, layout=[3,3], legend=False, figsize=[12,8],
...     title='Show all metrics')  # doctest: +SKIP

... and matplotlib plots can be saved easily.

If you are really set on the dashboard, there's a newer version in the raiwidgets package. The _create_group_metric_set function from Fairlearn can give you all the metrics in a storeable format to pass to the FairnessDashboard later on when you want to look at them again. Or you can just use it out of the box with Azure (see this blog post) since your metrics are persisted in the cloud there and the dashboard can be viewed anytime in AzureML Studio.

I hope this helps!

I'm leaving this issue open since we should probably put this answer in the Q&A section of our website.
TODO:

• Create a Q&A section, something along the lines of: "What happened to the FairlearnDashboard?" [answer should point to the new repo and to our plotting functionality through MetricFrame]
• add a line to the newly created plotting documentation in Remove FairlearnDashboard and replace it with matplotlib-based plots #766 to show how one can save the generated matplotlib plots.

[rensoostenbach]

Regarding the first TODO, could a question be added to the FAQ, or is the FAQ meant for more general questions? The answer is basically written down at https://fairlearn.org/main/user_guide/assessment.html#fairlearn-dashboard.

Regarding the second TODO, I don't see any newly created plotting documentation in Remove FairlearnDashboard and replace it with matplotlib-based plots #766 . Is that yet to come?

[romanlutz]

Regarding the first TODO, could a question be added to the FAQ, or is the FAQ meant for more general questions? The answer is basically written down at https://fairlearn.org/main/user_guide/assessment.html#fairlearn-dashboard.

This could certainly go into the FAQ section as we've received that question many times. We can even move the text that's currently at https://fairlearn.org/main/user_guide/assessment.html#fairlearn-dashboard into the FAQ for that purpose.

Regarding the second TODO, I don't see any newly created plotting documentation in #766. Is that yet to come?

@rensoostenbach this is the newly created documentation: https://fairlearn.org/main/user_guide/assessment.html#plotting-grouped-metrics