Use of GitHub Enterprise

[rprouse]

Around a year ago, the .NET Foundation added a large chunk of the member project’s GitHub organizations to the foundation’s GitHub Enterprise account.

This change was a mistake. It wasn’t communicated and surprised maintainers who found out by chance or read about it from other maintainers on Twitter. Rob Mensching, the maintainer of the WiX Toolset, wrote a blog post how this change made him feel.

Going forward, the Foundation will not make changes to member projects unless asked to do so.

What is a GitHub Enterprise account?

A GitHub Enterprise account acts as the billing account for one or more GitHub organizations. Those organizations get access to GitHub Action minutes and Codespaces provided by that Enterprise account. It also enables the Enterprise account owner to gain admin access to an organization via a GitHub support ticket.

Why was this done?

The primary goal was to centralize billing to give member projects access to additional GitHub services. A secondary goal was to ensure the continuity of membership projects, using requestable admin access, as just described.

How and when was this done?

The bulk of the projects were migrated around August 2020. This was achieved by filing a GitHub support ticket. This also required an admin user in the org to confirm the changes, for which the dnfadmin user account was used.

Lessons learned

This move was a mistake. The board deeply regrets that this happened.

Project maintainers should have been offered the option to join the GitHub Enterprise account, with an explanation of the benefits. There should also have been a written policy on when the .NET Foundation will request admin access via GitHub support.

The .NET Foundation violated the trust of project maintainers because they were under the impression that the dnfadmin user account would only be used in case of emergencies and for the CLA automation system.

Moving forward

Effective immediately, any project can request to be removed from the GitHub Enterprise account. The .NET Foundation projects committee will reach out to all affected maintainers, but you can also send an email.

The .NET Foundation will only make changes to infrastructure it operates with agreement from maintainers and according to documented policies.

[ghuntley]

This is the way. ✨🏅

[Perksey]

My personal take is that it should also be discussed transparently public, not only by the internal committee.

All committee meeting notes are made public in either the repo for that particular committee and/or this discussions page.

[Aaronontheweb]

The primary goal was to centralize billing to give member projects access to additional GitHub services.

Out of curiosity, was this benefit ever announced by the .NET Foundation to its member projects anywhere? I don't recall seeing one. It seems odd to me that we'd have access to these benefits in-place for over a year and no one would think to tell the projects about it. Did that happen and I just missed it? If not, why?

[rprouse]

Did that happen and I just missed it? If not, why?

This is just my impression, but I think communication was missed because the foundation needed more volunteers and too few people were trying to do too much. There were also too many avenues of communication.

[Perksey]

too few people were trying to do too much

💯 it seemed like Claire did all the project support with the occasional support from Chris on the onboarding, CLA, and code signing front. I think the projects committee should have a subset of volunteers which do support in addition to the normal onboarding go/no-go stuff if there is interest for that (I'd be more than happy to help out!)

[Aaronontheweb]

You have enough resources to move a project like WiX into the DNF's Github Enterprise in the 1 week window he had given DNFAdmin access, but not enough time over the course of 14 months to draft an email netting an easy public relations win for the Foundation by announcing that you were giving all of the member projects free access to CodeSpaces and Github Actions?

[glennawatson]

As discussed offline, if we can get #43 solved and clarified, that would be great as well. Agree with others this is an excellent move in the right direction.

[rprouse]

Posting another reply on #43

[vcsjones]

Can we also clarify on when dnfadmin should or can be used? This account feels like a "break the glass" kind of account that should only be used in extraordinary circumstances. Who has access to this account? How are credentials managed, rotated, etc?

[richlander]

Legit questions. The board is going to work through those among the next things. We're directionally aligned with the intent and flavor of your questions.

[Aaronontheweb]

I appreciate your response @richlander - I agree with @vcsjones that the admin access should be treated as a "break the glass" option and should only be used under narrowly prescribed circumstances where there is no alternative. Having what those are spelled out in writing will go a long way. I appreciate you acknowledging it.

[glennawatson]

Worth noting that there is functionality provided by github for handling succession but it has some limitations discussed in #44

[SeanKilleen]

I think this is a well communicated message that understands the impact and lays out clear potential changes. I really appreciate this stance and the effort to begin rectifying things. 👏

I'll offer an additional idea to support your aims here, in case it's useful--

Since one of the problems of usage of dnfadmin is that folks aren't aware of what it was doing, one step toward transparency might be to list all the times it is used (going forward; historically may he impractical) and their purposes. If a public log of this exists, others can weigh in if the usage goes into a gray area, and there will likely be a sense that communication should happen before it's used. It could be a useful check on the overall usage of the account (which does I think still have its benefits from a management perspective).

If we have a list of usages after the fact, this can be used to shift things to the left -- instead of "we did x" with the account, it can become "we're thinking about doing x with the account" as a request for comment.

But again, I like where this is going and I appreciate the energy that the board appears to be bringing in meeting the challenge.

[glennawatson]

Might be worth bringing that over here as well #44

[rprouse]

Excellent suggestions, let's continue them in #44

[jcmontx]

I'm not involved in OSS, but as a .NET developer, I'm enraged by the collateral brand damage. The average developer doesn't know the difference between .NET and .NET Foundation.

This sounds like the bare minimum you should already be doing, but too late. I'm disappointed.

[Aaronontheweb]

@jcmontx it's a start - there's a lot more even about the GitHub Enterprise episode that I want to know about. Specifically, did the board know about this as it was happening? What oversight mechanism broke down to a degree where no one thought to object to this on the board?

There's even more to be done still and I outlined some of those ideas here yesterday: https://aaronstannard.com/future-of-dotnet-foundation/

That the foundation is now at least speaking directly to maintainers' concerns and admitting to mistakes, a major departure from their last public apology, is a productive step forward and should be encouraged - but I agree with the spirit of what you're saying: this is a start, not a conclusion. By the sound of it the new board sees it that way too.

[leastprivilege]

Specifically, did the board know about this as it was happening? What oversight mechanism broke down to a degree where no one thought to object to this on the board?

Agreed.

[oskardudycz]

Yup, it would be good to have a transparent retrospective explaining how we ended up in this situation.

[eglasius]

I'm not involved in OSS, but as a .NET developer, I'm enraged by the collateral brand damage. The average developer doesn't know the difference between .NET and .NET Foundation.

I don't get how that makes a difference. There was a bad situation in the dotnet ecosystem, the community spoke up and things are being addressed.

Also can you clarify what's the brand damage here? Specially in a couple months when the episode finishes playing out?

[gortok]

First off,

Thank you.

It's not easy to post reputation-damaging news, and it shows a bias towards trust-building that you did so.

Secondly, this was much better than the alternative, which is not posting anything at all and not confirming what we already knew.

Now, on to the harder part:

Where do we go from here?

1. We need a post-mortem, investigation, or whatever you want to call it. It should contain these elements:

a). An impartial investigation into :
i). the genesis and execution of the idea to create the dnfadmin service account
ii). have the maintainers make that service account an owner in their projects, and
iii). the decision to use that service account to move the projects to the Foundation, and
iv). why there wasn't communication on the subject at all
b). Recommendations for the board going forward, into:
i). formulating decisions that affect maintainers
ii). communication of these decisions
iii). who can make these decisions, and under what circumstances
iv). when these decisions shouldn't be made by an outgoing board -- for instance this happened in august 2020, during a transition time from one board to another
c). Writing up the results from this investigation and the recommendations to the board for review by members of the .NET Foundation (even if their dues are waived).

2. We need more transparency into what the board does; I recommend debate of and passing this bylaw amendment.
3. The process around selecting a new Executive Director of the Foundation should be discussed on these Github Discussion forums; so that members can understand what that process is and what it entails.
4. The understanding of what it practically means for a project to join the foundation through either model needs to be publicized and discussed on the forums as well. @shawnwildermuth made a comment that is troubling and reveals a wide gulf between what board members think happens and what community members think happens when a project joins the .NET Foundation:

I think that the community and projects may have not understood what they were agreeing to when they were brought under the .NET Foundation umbrella.

Rob, I appreciate your efforts. I think this is an adequate start, but the board needs to do more. This is a very serious mishap on the part of the board and the .NET Foundation leadership, and we can't just 'call it a day' with this apology.

Why an investigation? Well, the words of a former board member come to mind:

I don't believe [moving DNF projects to the Foundation's Github Enterprise account] was ever discussed in a meeting. I only knew about it being a maintainer of a DNF project. Anytime I approached the subject of member projects I was met with less than favorable responses.

[richlander]

There is no plan to stop here and "call it a day". You will continue to see more.

[eglasius]

I think everybody can agree this is the first step along a set of actions that will be taken. That said, I don't see what there is to gain by narrowing down on who and instead all energies should go into finishing correcting what needs to be corrected including reducing/removing the possibility of similar situations repeating.

[dapalmi]

Around a year ago, ...

I don't think it is a coincidence that almost exactly one year ago this happened: https://leastprivilege.com/2020/10/01/the-future-of-identityserver/
Were the actions "around a year ago" an attempt to ensure the .NET Foundation's continuation policy for projects (especially since IdentityServer is arguably the most important OSS project for Microsoft)?
And what are the .NET Foundation's steps to ensure continuation for projects like IdentityServer?
I think transparency would be fair for both the maintainers of .NET Foundation projects but also all the other contributors, who spend their time and effort into a .NET Foundation project, with the understanding that the continuation is taken care of.

[dapalmi]

Sorry. Just noticed that my first question was already answered:

A secondary goal was to ensure the continuity of membership projects, using requestable admin access, as just described.

[rprouse]

We've worked with @dansiegel and @brianlagunas to remove the Prism project from GitHub Enterprise. The process was fairly straightforward, so we are ready to move other projects that want to make the change. Get in touch if you do.

Thanks to Dan and Brian for being patient and working with us.

https://twitter.com/DanJSiegel/status/1448749005513519114

[BillWagner]

@leastprivilege Send an email to board - at - dotnetfoundation.org, and we'll get it done.

[BillWagner]

Follow up: @leastprivilege reached out via email, and this is now completed: https://github.com/IdentityServer

[rprouse]

For projects that want their projects removed from GitHub Enterprise, the process is fairly simple, although we ask that you contact us via email (see above) CC'ing all project maintainers so that everyone is in the loop.

The process has been smooth so far. No settings, permissions, etc. are lost in the process. We have only seen one issue that you should be aware of. If your organization had a free coupon applied before, that will not be automatically reapplied afterwards. If you still have the coupon code, you might be able to apply it again, or work with GitHub support to get it re-applied.

Also, be aware that when you move out of GitHub Enterprise, you lose the extra benefits like increased build minutes. It isn't an issue for most projects, but we want you to be aware.

[JeremySkinner]

Hi @rprouse I emailed contact@dotnetfoundation.org over a week ago asking for the FluentValidation project to be removed from GH Enterprise but haven't received a response. Any chance you could chase this up? Thanks!

[rprouse]

@JeremySkinner I'll find out who is receiving the contact emails and get it done.

[pascalberger]

Thanks @rprouse for the well communicated message 👍

I can answer for the Cake project, that we prefer to stay on the GitHub Enterprise instance.

However, we are also of the opinion that the processes in this case did not work and that improvements are needed at the communication level.

We therefore would like to see:

• Better docs of when and why dnfadmin account is used
• Notice before it's used in the future.

[lostmsu]

It is great to hear we can now move out. I have a question and a concern though.

Question is: who's decision it was to silently move orgs under .NET Foundation Enterprise account? Please, publish the names, so that OSS community would know people to avoid.

If it was a decision by the board vote, we need names of people who voted 'yes'. Without this voting for the board members is uninformed choice.

If it was a decision without the board approval, board needs to implement a mechanism to prevent this happening in the future.

As for the concern. If moving organizations out of the enterprise account is as painless as you say, you should move everyone out right away and make it an opt-in (though I agree this is debatable). Or at very least give everyone a link they could simply click (as org admins) to move out without the need compose an email. Remember, there are hundreds of projects, and the .NET Foundation is the party at fault there, wasting everyone's time on avoidable bureaucracy. Foundation might also already have a script to do the deed, as evidently Foundation had one to move them other way.

The reason I am asking to "move everyone out without another consent from the owners" is that I am worried that the decision to move us in was not simply reckless, but outright corrupt: if some entity pays Foundation's Enterprise account bill and somehow running our GitHub actions would inflate that bill, then the move to get everyone under the account did 1) defraud the party, who pays the bill, perhaps to inflate the importance of the whole thing as a career step, which is borderline criminal, and 2) takes funding that could go to other purposes and sends not where the members would want to see it sent.

At the very least to continue discussion on this I'd like to know if the Enterprise account bill depends on number of organizations that belong to it right now.

[lostmsu]

By the way, if moving out requires approval by owner of the orgs, you should initiate move out right away from your side, so that owners only needed to choose to accept/decline.

[glennawatson]

@lostmsu yeah when they did it for us there was a email saying it happened but there's no approval process.

[dansiegel]

@lostmsu this really is the wrong way. What the Board is doing is absolutely the right way. First of all we don't need more tampering with member projects where there is no communication between the Foundation and the project maintainers. Second this process is IMO is well suited to the goal of driving better transparency and communication between the Foundation and the member projects.

Perhaps the Board is already taking action on this, but I will let them speak for themselves. My only suggestion here is that we ensure that the Foundation is actively reaching out to all member projects and ensuring that contact has been made with someone from each project. It is important IMO that each project should know what is going on, and that we make sure that they have the opportunity to determine where they would like their GitHub Organization. I also want to be clear here that I do mean each project because we do know that not all member projects were moved. Obviously there has been a lot of turmoil over projects being moved into GitHub Enterprise without their permission, however it is equally possible that there may be some projects that weren't moved, but given the opportunity might want to be moved to GitHub Enterprise.

[matkoniecz]

Effective immediately, any project can request to be removed from the GitHub Enterprise account.

Only after request? Why this takeover is simply not reversed?

[rprouse]

Because many projects want to be in GHE for the benefits it provides, and some projects are not being maintained so we don't want to lose them. The biggest reason however is that we want to be as open as possible and only make changes in communication with project maintainers.

[matkoniecz]

we want to be as open as possible and only make changes in communication with project maintainers.

So inactive projects which were stolen by .NET foundation using admin bot (supposedly used for maintenance) will be kept by .NET foundation, as long as their authors will not submit this requests?

[devlead]

At this point we've chosen to not do this without maintainers consent.

It's been active for several months, so projects might depend on the extra features GHE offers, so just removing them might break and block them.

We're reaching out to maintainers, but prioritizing the maintainers that actively reach out to us.

[lostmsu]

We (Python.NET aka pythonnet) asked to be removed from .NET Foundation Enterprise account almost 2 months ago now. Why are we still in it? @devlead

My comment above about moving everyone out without the bureaucracy feels almost prophetic now. 😒

[devlead]

@lostmsu must have been missed, will investigate and get back to you.

[BillWagner]

@lostmsu @devlead We missed the email when this first came in. I've started the process, and it should happen shortly. I apologize for the delay.

[rprouse]

@lostmsu, this is on me. I asked for it to be done but it got missed and I didn't check and follow up. We'll get it done this week.

[BillWagner]

I just got email confirmation that this is complete. Thanks for the reminder, and again, I apologize for the unnecessary delay.

[filmor]

No worries, thank you.