Moving the existing UI code to a separate repository

[romanlutz]

@adrinjalali @koaning @MBrouns @kevinrobinson @hildeweerts @chisingh @KeXu444 @chnldw
Please share your thoughts :-)
We'll bring this up in the developer call tomorrow as well.

[koaning]

My main feedback would also be to rewrite some of the demo notebooks. I tried running it and it was an-unintuitive experience. Apperantly jupyter notebook is supported but jupyter lab isn't. It's also a bummer that I can't see the results from github. The fact that I had to download the notebook to see the effect of an algorithm is, to me, a big barrier of entry.

The demo notebooks currently requires you to learn two tools at once. A new algorithm and a new visualisation tool. If the visuals were simple matplotlib charts then a new developer would experience less cognitive load while trying to learn the new algorithm.

[adrinjalali]

I'm all in favor.

[adrinjalali]

The demo notebooks currently requires you to learn two tools at once. A new algorithm and a new visualisation tool. If the visuals were simple matplotlib charts then a new developer would experience less cognitive load while trying to learn the new algorithm.

I think once we move away from having the widget code in the main repo, it'd make sense to use simple plots in the docs and have one or two examples with the widget to showcase the integration or something.

[kevinrobinson]

@romanlutz hello! I had written out comments on this, but ended up saying them in the developer call instead, and will hang back for a bit now.

If I can help move forward any work towards addressing practitioner needs (eg, Holstein et al. 2019), or help with unblocking other folks who want to work on that (whether in Azure ML or open source), please let me know!

[kevinrobinson]

If MS folks on the Azure side aren't able to collaborate or contribute in the open, and decide to create a separate microsoft/azure-fairlearn repo that does not have an open development or governance process, that seems like a missed opportunity to collaborate, but it's already the status quo so it seems fine to me to formalize that :)

One specific suggestion is to update this proposal so that it specifies how new development starts within fairlearn/fairlearn. This could build on the values expressed in Contributing example notebooks, the draft Mission & Roadmap, and ideas expressed in Brainstorming: Website for "fairness as a sociotechnical challenge" community and the Fairlearn: Theory of change hackpad. I'd be happy to try contributing to this work within fairlearn/fairlearn!

If folks are interested in collaborating on that, some tangible first steps would be:

1. Update the quick start example so that it demonstrates how to meet the same user needs without using any Azure bits (eg, with calls to matplotlib as @koaning suggests here and @adrinjalali suggested on the call). Add a link to microsoft/azure-fairlearn.
2. Ask Azure staff to move any issues in fairlearn/fairlearn that are specifically related to microsoft/azure-fairlearn into that repo.
3. Revise the fairlearn/fairlearn roadmap to be centered on user needs and project values, and update that prioritization given the level of MS staffing and commitment from open source volunteers. In that conversation, I'd vote to prioritize fixing immediate issues for new users (eg, examples don't work because published packages lag the website; core assessment features work the first time across all notebook platforms; usage guides aimed at first-time users), new work that starts by expressing how it's aligned with practitioner needs (eg, Holstein et al. 2019; Madaio et al. 2020), and new work on tools that support engaging with fairness as a sociotechnical challenge (eg, Selbst et al. 2019; candidate screening example notebook, updating the website to include sociotechnical talking points).

I'd love to see work that meets practitioner needs, and work that approaches fairness as a sociotechnical challenge as a core part of fairlearn/fairlearn. Thanks for listening! 👍

[romanlutz]

As you're writing this was the status quo as it is, so this isn't a step backwards but rather a formal manifestation of what was happening already. Contributions have never really been possible, but the steps outlined in the proposal will change that by

• specifying how/where to contribute
• specifying the development process
• explaining exactly what restrictions there are (related to localization, accessibility, styling)

That should in fact make it much easier to contribute. As mentioned earlier this is still open on GitHub with MIT license. From what I understand about typescript so far it's possible to create different kinds of wizards from there, e.g. one that AzureML uses and another one that we build based on a candidate screening scenario, and another one that we build based on a lending scenario, etc. Depending on their purpose they can use different views that can live alongside each other in the repo and perhaps also some shared ones.

If people feel like it's easier to do this outside of the Microsoft repo that's perfectly fine, too! As you can read in the updated proposal there are pretty much three ways of setting this up, and whatever works for people can be done. This isn't mutually exclusive with the original proposal, it's rather beyond the original proposal's scope (and that's why that scope is now extended).

The point you made (and @koaning in a different comment) about quickstart, notebooks, and user guides makes sense. I've explicitly added that to the proposed steps.

Regarding your proposed steps: these are all existing items that everyone is more than welcome to join in, and some of them are actively in progress (e.g. @riedgar-ms is versioning the documentation to avoid the difference in the webpage to the last published package version).

The entire point of this proposal is to allow everyone to move beyond restricting development so that UI development can actually happen. If that sounds like an interesting part of Fairlearn to contribute to (in the context of the examples we discuss on Thursdays or otherwise) we'd love to work with you, whether it's in the Microsoft repo or a repo under the Fairlearn org.

[adrinjalali]

To me this proposal is less clear than the one before.

There are a few aspects which need to be clearly defined here:

• Moving the current UI out of the repo, and back into the microsoft org, with its defined development style/governance.
• Whether or not we'd like to have a coordinated effort towards an alternative UI or set of dashboards.

The answer to the second question depends on how much time and effort and person/hour the community has to dedicate on the issue. There are two alternatives:

• Fork the microsoft/fairlearn-ui repo into fairlearn org, and continue development from that point, which will cause the two efforts diverging eventually. The plus side of this is that there's a place where everybody can collaborate. The downside is that there needs to be some sort of maintenance and a team who takes care of it.
• Let people/groups make their own repos/forks, and if any of them at some point starts being a good one (however we'd like to define good there, it's pretty vague), we could move it to the fairlearn org. The plus side is that it requires no centralized organizational support; and the down side is that it makes it harder for people to discover where they can contribute.

I get the feeling that there are enough people who really care about having an independent and dynamic UI, and hence having a separate repo within fairlearn org makes sense. Having this repo, would also mean people in microsoft who work on the azure-ml UI, could contribute back to the open repo, or take community contributions back to azure-ml; if the two code-bases remain compatible enough.

[romanlutz]

To me this proposal is less clear than the one before.

There are a few aspects which need to be clearly defined here:

• Moving the current UI out of the repo, and back into the microsoft org, with its defined development style/governance.

• Whether or not we'd like to have a coordinated effort towards an alternative UI or set of dashboards.

The answer to the second question depends on how much time and effort and person/hour the community has to dedicate on the issue. There are two alternatives:

• Fork the microsoft/fairlearn-ui repo into fairlearn org, and continue development from that point, which will cause the two efforts diverging eventually. The plus side of this is that there's a place where everybody can collaborate. The downside is that there needs to be some sort of maintenance and a team who takes care of it.

• Let people/groups make their own repos/forks, and if any of them at some point starts being a good one (however we'd like to define good there, it's pretty vague), we could move it to the fairlearn org. The plus side is that it requires no centralized organizational support; and the down side is that it makes it harder for people to discover where they can contribute.

I get the feeling that there are enough people who really care about having an independent and dynamic UI, and hence having a separate repo within fairlearn org makes sense. Having this repo, would also mean people in microsoft who work on the azure-ml UI, could contribute back to the open repo, or take community contributions back to azure-ml; if the two code-bases remain compatible enough.

@adrinjalali I can certainly rewrite it such that the proposal pretends the call didn't happen, but I felt like it's useful to add context on why there was confusion and how that affects the plans. To avoid having people comment on different underlying text I'll leave it as is for today but once people have a chance to give feedback I can adjust according to your feedback.

I'm curious what details you'd like to see about moving the UI. Happy to add more details or link to other places if it's too much for the proposal.

Figuring out the latter point is exactly what I'm trying to get at with the rewritten proposal (I suppose it's less of a proposal and more of a discussion of options now). This can only be answered if we know what commitments people are willing to make. So far Kevin was the only one who wanted to contribute, but that may very well have been due to the lack of openness. (Besides, there's an argument to be made for enabling the future of UI development and not only catering to present needs.)

Each of the outlined options has pros and cons and I'm perfectly happy with whatever people are comfortable with.

[kevinrobinson]

@romanlutz I don't think I have anything else to offer here, and I apologize if joining this conversation on the call or in GitHub has just resulted in noise and distraction. I'm sorry 😞