Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add control features to metric plots #668

Open
romanlutz opened this issue Dec 31, 2020 · 3 comments
Open

Add control features to metric plots #668

romanlutz opened this issue Dec 31, 2020 · 3 comments
Labels
easy Relatively simple issues, harder than "good first issue" enhancement New feature or request help wanted

Comments

@romanlutz
Copy link
Member

romanlutz commented Dec 31, 2020

Is your feature request related to a problem? Please describe.

The metric plotting functionality mentioned in the docs does not mention control_features

Describe the solution you'd like

Add an example where control_features are used. In the same example we could use age as a control feature. However, whenever #793 is completed we should switch to the synthetic dataset. After all, it's not ideal to use a real dataset for an abstract purpose of showing this functionality.

Bonus

Ideally, a separator line between the control groups or a new line for each set of subgroups within a control group would be nice. Right now they're all treated as equal groups in the plots.

@romanlutz romanlutz added the enhancement New feature or request label Dec 31, 2020
riedgar-ms pushed a commit that referenced this issue May 19, 2021
…766)

With a slight delay (originally targeted for April) I'm finally removing the `FairlearnDashboard` since a newer version already exists in `raiwidgets`. The documentation is updated to instead use the plots @MiroDudik created with a single line directly from the `MetricFrame`. In the future we want to add more kinds of plots as already mentioned in #758 #666 and #668 . Specifically, the model comparison plots do not yet have a replacement yet.

Note that the "example" added to the `examples` directory is not shown under "Example notebooks" on the webpage, which is intentional since it's technically not a notebook.

This also makes #561 mostly redundant, which I'll close shortly. 
#667 is also directly addressed with this PR as the examples illustrate.

Signed-off-by: Roman Lutz <rolutz@microsoft.com>
@romanlutz romanlutz added this to To do in Visualizations May 27, 2021
@romanlutz romanlutz added the easy Relatively simple issues, harder than "good first issue" label Jul 8, 2021
@bram49
Copy link
Contributor

bram49 commented Aug 2, 2021

Hi @romanlutz, I would like to pick this up

@romanlutz
Copy link
Member Author

Sure thing, @bram49 ! To be quite honest, I'm not entirely sure where this will lead. You can play around with the samples we have in the documentation https://fairlearn.org/main/user_guide/assessment.html#plotting-grouped-metrics and see if there's a good way to showcase the control features.

You can basically do the same thing with control features as with sensitive features since the by_group result is just the same kind of DataFrame. The difference comes only through aggregation which isn't really shown. Instead of showing the bars for all groups next to each other I was just wondering whether we could have some kind of separation between the control groups. It's probably not doable with a single command like we have it in the docs linked above, because you'd need to call additional plotting commands. Perhaps the plotting code @alexquach is adding in #857 can be extended to add separators between the groups. In any case, we may just come out of this exploration with the decision not to do anything special (and that's perfectly fine!). If you want to do something more well-defined I'd suggest #666. I can walk you through what's been done on that before since I pair-programmed with @rishabhsamb on that. Just let me know what you think now that I've shared some of my thoughts 🙂

@bram49
Copy link
Contributor

bram49 commented Aug 4, 2021

Thanks for the fast and thorough reply!
I will dive into this when #896 is finished. Think I will start with creating the separation lines and see if this will look nice.
If it would not work out I will take a look at #666.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
easy Relatively simple issues, harder than "good first issue" enhancement New feature or request help wanted
Projects
No open projects
Visualizations
  
To do
Development

No branches or pull requests

2 participants