Limited admin capacity to handle 2FA account recovery requests

[LaserPhaser]

I've lost my 2FA device about month ago and filled a bug here [#632].
No one helped me to restore my account.
BTW - I've checked the history and there are no solver issues with account-recovery tag for the last 6 month.
@di could you please gently push some one who is responsible for such help ?

[ewdurbin]

These issues have thus far been handled by me, are high effort, and high latency.

To be 100% transparent it has simply been a task I have not had the energy to dump into amid the circumstances of a global pandemic, an uprising here in the states, and an upcoming consequential election.

[LaserPhaser]

@ewdurbin Got it, could I help you somehow with these issues?

[ewdurbin]

The task requires administrator access to PyPI to retrieve verifiable information or activity for users of PyPI, as well as to complete the reset.

This is pretty much "super user" access to PyPI as it involves the ability to view sensitive, personal, and private information as well as to perform destructive actions. There are currently 6 people with the ability to view admin information and 3 with the ability to perform the actual reset. It's a tough call to open the door to additional folks in this role given what granting this kind of access could lead to.

[LaserPhaser]

Ok, if you will have any ideas how I can help you with this - please reach me.
Thanks for your effort.

[richard-scott]

Would it be possible to get an option to be emailed a url to remove 2fa from our accounts, similar to the lost password one?

[ewjoachim]

Not sure but if it was the case, without more identity checks, then it wouldn't add more security than just the password itself, would it ?

[richard-scott]

What more identity checks can you make? You only know the accounts designated email address.

From what I’ve seen on closed issues, contacting this email address you have on file seems to be the way theses get resolved.

I’m thinking of a way for the system to automatically email the main account holder to say “someone asked to remove your 2fa, click here to proceed” then ask them to re enter their password would save us all lots of time 👍

[ewdurbin]

For accounts with no projects, that is precisely what we do. For accounts with projects owned/maintained we verify ownership of the project via public source code repositories, via other maintainers, or by other means available.

Enabling MultiFactor Authentication indicates that a user has made a decision to safeguard their accounts beyond the scope of their password. Utilizing the same mechanisms to disable MFA as resetting a password negates the security that MFA provides.

[LaserPhaser]

For accounts with no projects, that is precisely what we do. For accounts with projects owned/maintained we verify ownership of the project via public source code repositories, via other maintainers, or by other means available.

Enabling MultiFactor Authentication indicates that a user has made a decision to safeguard their accounts beyond the scope of their password. Utilizing the same mechanisms to disable MFA as resetting a password negates the security that MFA provides.

Actually I think we must show security backup codes any time when user enables 2FA and highlight somehow that if user will lost 2FA and security codes he will have a big troubles. It should decrease the numbers of such cases like mine.
Just because when you see the security codes and big logo that informs you about potential problems (including lost all access), you will save these codes.

For all other cases (when you lost 2FA and codes) - it still will be manual hard work for you guys.

[richard-scott]

But if we have recovery codes we won’t need help with 2FA. Maybe enforce the downloading of recovery codes for all users with 2FA enabled that haven’t already collected them.

My 2FA stopped working as my phone died before I downloaded the recovery codes hence why I need 2FA removed from my account.

[rrthomas]

This is pretty much "super user" access to PyPI…It's a tough call to open the door to additional folks in this role…

You're the Director of Infrastructure, so if you can't make this call, who can? Clearly, the system is not working at the moment. There's work that needs doing (I'm one of several waiting for a RedHat-namesquatted package name to be assigned to me, having been approved by RedHat). You can bet people are already routing around you: using other languages, abandoning PyPI etc. You can delegate, you can release control (e.g. undo namesquatting). You can't sit there indefinitely. I'm sorry things are tough for you personally right now, but things are always tougher for many of your users (I'm lucky that they aren't for me). So please, let's have some sort of commitment to how and by when this will be sorted!

[kirankotari]

@ewdurbin Any updates on the approach? @ewjoachim do we have any timelines when these issues will be resolved? Twine is working fine to due to ssh in current pc.

[LaserPhaser]

Guys is there any chance that you will solve this problems or delegate them? I personally can spend about 3-4 hrs per week on these effort. I think it will be enough to finish with all requests before the New Year.
As one of possible ways to check that account is not stolen:

1. No logins since issue on GH was submitted.
2. Check current commits to the repositories - if there is no activity on GH then we may think that account is stolen - because it's too hard to steal an account and keep committing to the projects with the same activity :) Yes for sure there are a lot of people like me - who do not contribute every day, but for part of users it will work.
3. Ask to upload anything to the absolutely new project - it would mean that user remember his login and password (it's enough to publish pypi package) but he lost his 2fa credits.

Something like that.
I really hope that we will be able to move this things forward.

[richard-scott]

I'm in group 3: I can upload to my project but when I upgraded my phone I forgot to move PyPi's 2FA over.
I still have working 2FA on my Github Account, maybe that could be used to help in identification?

[kirankotari]

Guys is there any chance that you will solve this problems or delegate them? I personally can spend about 3-4 hrs per week on these effort. I think it will be enough to finish with all requests before the New Year.
As one of possible ways to check that account is not stolen:

1. No logins since issue on GH was submitted.
2. Check current commits to the repositories - if there is no activity on GH then we may think that account is stolen - because it's too hard to steal an account and keep committing to the projects with the same activity :) Yes for sure there are a lot of people like me - who do not contribute every day, but for part of users it will work.
3. Ask to upload anything to the absolutely new project - it would mean that user remember his login and password (it's enough to publish pypi package) but he lost his 2fa credits.

Something like that.
I really hope that we will be able to move this things forward.

I am in group 3: I have recently published a library to pypi but web access lost due to 2FA.

We wanted to help and we understand it's sensible info. Create a temporary table and mask unwanted info about the user. Grant a few more people access to this temporary table alone. We will validate and we will give feedback to your team. Now your team act accordingly on the valid or verified users.

Note:- As a team we need to discuss validate process.

[LaserPhaser]

@richard-scott @kirankotari Guys I'm with you in group 3 :) , I'm just suggesting possible ways of solving this problem. And as you are I do not have any privileged access to the pypi internals.
So let's wait for a reply from some one who has an appropriate permissions.

[LaserPhaser]

@ewdurbin
Any chance that now we can go ahead with our pypi problems.

To be 100% transparent it has simply been a task I have not had the energy to dump into amid the circumstances of a global pandemic, an uprising here in the states, and an upcoming consequential election.

2 of 3 global problems seems to be almost solved:

1. Elections are ended.
2. Vaccine is invented.

[ewdurbin]

@LaserPhaser I don't appreciate you deciding for me that I'm "better" all of the sudden. But, I have been using some of my volunteer time to proceed with this backlog. I want to be very clear that your characterization didn't really help anything, and actually hurt.

[richard-scott]

@ewdurbin I'm sorry you feel hurt, I'm sure that was not @LaserPhaser intention.

Can I ask, is your synopsis on here still an accurate reflection of your employed duties? If so, what is your involvement with PyPi as a "volunteer". We do greatly appreciate the efforts you put in as a volunteer, but this synopsis makes it sound like supporting the PyPi infrastructure is a key part of your job role.

With that in mind, are you the right person for us to direct our questions towards? If this isn't an Infrastructure issue, but a Software issue then is there someone else that you can get involved in this discussion so we can feel assured that our concerns are being listened to.

[ewdurbin]

That's a fair question, @richard-scott and my responsibilities do include keeping the PyPI infrastructure online and available. I also help to facilitate funded projects that impact PyPI. My volunteerism on the project is basically anything that falls outside of that realm, such as end user support for account issues.

When I have uncommitted time outside of my other responsibilities at the PSF (rarely), I have used paid time on these kinds of tasks, but ultimately it is low priority among my overall workload.

Ultimately it's a process issue and a question for the PyPI administrator and moderator team as a whole. cc @pypa/warehouse-team.

[kirankotari]

@ewdurbin get well soon. Do you have any thoughts on my comment? i.e. create a temporary table and mask unwanted user info, grant access to temporary table alone

[LaserPhaser]

@ewdurbin Sorry if my words hurts your feelings. Maybe i'm just too straightforward.
We stuck with this problem already for a month long(maybe more but i'm counting from the moment when I submitted the issues).
We suggested possible ways to solve the problem and ourselves as volunteers.
Please give us a hint how could we move this things forward, for me personally it does not look like a big problem that should be solved for months, it's just a lack of resources invested in this problems.

[CyanBook]

Just to understand, I have to upload my new repo as soon as possible with this account.
I just opened my issue, but I'm seeing issues opened for months without any response, and I'm wondering if I will have to wait so much time just to upload a repo. Is there any way to just speed up this process?
Also why these long times? I understand we're in a pandemic, but what do you have to check to just disable 2fa on my account/send me a recovery email? I just lost my 2FA codes on the phone. Is there anything I can do?

Hope to get back to you soon

[di]

Also why these long times? I understand we're in a pandemic, but what do you have to check to just disable 2fa on my account/send me a recovery email?

Consider that just immediately disabling 2FA for a given account based on a GitHub issue sort of defeats the purpose of 2FA. We have no way to determine that your GitHub account and PyPI account are actually the same person. The best we can do is the process we already have, which requires contacting the owner of the account via email, and is time consuming.

Our top priority is to keep every PyPI account as secure as possible. This means that we need to be as careful as possible when reducing the overall security of someone's account.

[ewjoachim]

@CyanBook Consider this the other way around :)

Let's consider somone with an account secured with a password with their email as backup. They considered that it was better to step up their account security by adding 2FA, which supports recovery codes. This is them telling PyPI "I don't want you to let anyone in my account if they don't have either the 2FA or a recovery code".

Now you're in the situation of having neither 2FA nor a recovery code, so you're in the exact situation you've been requesting that we secure your account from.

Yet, we have a procedure for making a special case and allowing a last resort recovery procedure. I think it's clear that if this was quick and automated, and only relied on email, this would completely defeat the whole 2FA. So I'm not saying that the fact it relies on human intervention by some specific people who have limited time to handle those requests is a feature by itself, but without a good level of scrutiny, this could be the weak spot of our security policy. I'd much rather we take the time it takes to ensure that, when we disable 2FA on an account, in contradiction to the owner's wishes at the time they specifically enabled 2FA on that very account, we're following the owner's later decision to change their mind, rather than letting anyone else get their hands on the account :)

Given the existence of 1) not enabling 2FA if you're unsure, 2) 2FA recovery codes, 3) special warnings displayed when you're the sole maintainer of a package, my personal opinion on the matter would be that this procedure should be reserved to cases where the whole python ecosystem would be taking a hit (e.g. if a very very popular package had all their maintainers unable to push new packages, and not doing anything would require millions of people switching to a fork of the lib, and there's a security issue on the existing popular lib).

[LaserPhaser]

In many other systems you have fallback options, like additional email/sms/phone call to recover your account without 2FA, in pypi we do not have such possibillity, that's why we will have a tons of issues with disabling 2FA requests.
At least to avoid getting more and more requests maybe we could add sms verification as fallback to 2FA?

For current issues with accounts as well as mine account, I'm agree to wait , but I would like to hear some SLA for getting things done.
Yes, all we know that it's opensource and guys are putting their non working time for solving this issues, but anyway clear timeline could make this process much more smoother and understandable.

Unfortunately for me it looks like dead end right now.

[pradyunsg]

I would like to hear some SLA for getting things done.

Funding to help the PSF employ dedicated support personnel for PyPI, would go a long way toward promising an SLA. As long as PyPI is depending on volunteer availability for user support tickets, I don't think it's feasible to make any SLA-style promises about these.

As @ewdurbin mentioned above:

Ultimately it's a process issue and a question for the PyPI administrator and moderator team as a whole.

[ewjoachim]

In many other systems you have fallback options, like additional email/sms/phone call to recover your account without 2FA, in pypi we do not have such possibillity, that's why we will have a tons of issues with disabling 2FA requests.
At least to avoid getting more and more requests maybe we could add sms verification as fallback to 2FA?

Just to understand this, could you share what other services you're thinking about, especially, if there are high-profile open-source community-developed websites that do offer those 2FA fallback options ? This could be a nice inspiration :)

[di]

In many other systems you have fallback options, like additional email/sms/phone call to recover your account without 2FA, in pypi we do not have such possibillity, that's why we will have a tons of issues with disabling 2FA requests.

PyPI supports adding multiple 2FA devices, hardware keys, WebAuthn, etc. The actual issue is that most users don't (and don't keep their recovery codes around, either).

[LaserPhaser]

In many other systems you have fallback options, like additional email/sms/phone call to recover your account without 2FA, in pypi we do not have such possibillity, that's why we will have a tons of issues with disabling 2FA requests.
At least to avoid getting more and more requests maybe we could add sms verification as fallback to 2FA?

Just to understand this, could you share what other services you're thinking about, especially, if there are high-profile open-source community-developed websites that do offer those 2FA fallback options ? This could be a nice inspiration :)

I'm not talking about opensource or community-developed, for example facebook could reset pwd this way - yep sms cost money, but theoretically we could make paid restore when user pays for his sms.

Just to avoid spam in this group:
I see by closed issues that process became alive (about 20 days ago) and @ewdurbin spending his time on it.
And here is dashboard for it https://github.com/pypa/pypi-support/projects/2
Thank you for your effort!

[kirankotari]

@ewdurbin If I am not an author of an repo and I am only the maintainer of PyPI lib. How does the verification process happens in this case? for a standalone script I have contributed to make a Python Lib. by folk the project that's how I became the maintainer of a library in PyPI.

And the Org is owned by a company. They won't allow me push a random branch to the public repo. as well right.

[ewdurbin]

All outstanding requests have been responded to at this point.

Since I'm tired of being mentioned/reminded of this thread. I'm going to close this issue, lock it, and will open a new issue for @pypa/warehouse-team to discuss potential longer term options for addressing these.