Add support for Gmail's XOAUTH2

[gsborisgithub]

I really do not want to allow "less secure apps" in my gmail account.
Would K9Mail be compatible with this gmail security some time in a future?

[ocdtrekkie]

"Less secure apps", as far as I'm aware, really just means apps that don't use Google's proprietary APIs, rather than using standards like K-9 does.

You can also use an "app-specific password" created from your Google account, which is a part of their two-factor authentication system, which you should be using anyways.

On "less secure apps": https://www.google.com/settings/security/lesssecureapps
On "app-specific passwords": https://support.google.com/accounts/answer/185833

[cketti]

See https://developers.google.com/gmail/xoauth2_protocol

[philipwhiuk]

Implementing XOAuth2 for Google will require us to:

• Obtain a client ID and 'Secret' for K-9 (which will have to be in the APK so may as well be in the repo).
• Implement a screen for users to allow K-9 access to their email (or just a link to an external browser window)
• Implement a follow-on screen to accept the verification code and generate an access code and refresh code.
• Store the refresh code and action codes as their pseudo-passwords
• Use the action code to login (this is quite similar to CRAM_MD5)
• Use the refresh code to periodically refresh the access code (1 hour for Google)

Some of this is common across anyone who uses XOAuth2. But much of it is Google's implementation. It may or may not be the same for different providers. While I'd want the developer to do it in a provider neutral fashion as possible, until you have two it's impossible to know what the changes will be - they could have no concept of token refresh for example.

Whether putting the "client secret" in a public repo is acceptable to Google is a good question.

[Valodim]

I would hope so, otherwise it's impossible to use the api for email clients, which sort of defeats the purpose.

[cketti]

As a first step we probably want to limit ourselves to Google accounts set up on the device. Then we can use AccountManager.getAuthToken(…) to retrieve the tokens.

[AndyCouch]

If you limit yourselves to Google accounts set up on the device, would that allow for access to both the free consumer Google account signed into the Android device as well as additional Google for Work accounts? I have a K-9 devotee in the family that recently switched their primary email to a Google for Work account and they are being plagued with the "Sign-in attempt prevented" emails and I've been tasked with finding a solution.

[philipwhiuk]

The work-around is to use the app-specific passwords @ocdtrekkie mentioned.

As for whether the unimplemented future solution will work for additional GfW accounts, I have no idea - not being sure how they get integrated into a phone. If they get integrated as Google account on the device possibly, depending on the implementation. It's all hypothetical at this point as I've not done much more than the initial look at the protocol.

[philipwhiuk]

There's a new xoauth2 branch from work I've done so far.

Here's what works.

• You can now select to authenticate incoming using an OAuth 2.0 token.
• If you do that it will use the AccountManager to try and fetch a token for the email address you provide.
• The IMAPStore now checks for capability for authentication via XOAuth 2 and SASL-IR.
• For authentication it will get a token, and then using the token it will make a XOAUTH2 AUTHENTICATE request. It will handle the continuation response and parse the status response.

Here's what doesn't work/needs work:

• Authentication with Gmail never actually works. So currently you can't get further than account setup :( I have no idea why right now. Google's documentation is a maze and none of it quite covers the interaction between the AccountManager Android API and the GMail XOAUTH authentication. The token type may be wrong (token types are not documented on Google's site, at all - I found a reference to the "mail' token type and it looked right...) or you may need to pass in an Application API Key or you may not be able to auth like this...
• If you are going to use a token you shouldn't need to put in a password. Currently you do. Choosing token auth may well need moving to the email address page which will then need a re-design.
• We can get a list of Google Accounts. So we should do that and allow a user to select the one they want to authenticate for rather than having to type the address and match it (e.g. gmail vs googlemail). This can be part of the re-design of the email address page.
• SMTP doesn't support XOAUTH2 at all yet - I've not looked at it.
• The AccountManager currently gets passed in to the ImapConnection class - like the Connectivity Manager. We probably want a more general AuthTokenProvider.

Think of this branch as like #884 for XOAuth 2. It's not the solution, but it's at least a starting point for someone (maybe me) to get it to work.

[philipwhiuk]

Okay, updated branch with some help from cketti regarding finding correct auth type.

Of the list of not working stuff:

• Authentication with Gmail works
• Selecting it is now done in the 'advanced options'.
• When selected, the password fields are hidden and the username field turns into a drop down of the accounts registered.
• Automatic configuration of Gmail accounts now uses XOAUTH2
• SMTP now supports XOAUTH2
• There's an OAuth2TokenProvider interface implemented by the AndroidAccountOAuth2TokenStore which manages tokens

What still needs to be done:

• I need to test token expiry timeouts. I don't want requests to fail periodically, the token get invalidated then the next request works fine. We need to ensure this gets handled gracefully. This might mean tracking invalidation times and invalidating it and getting a new one when it's close.
• I need to test removing the privelege.
• I need to workout how to provide the Application Key
• It'll all need documenting in the manual

So it's working in a very rough state. I wouldn't recommend it unless you're familiar with debugging though just yet.

[philipwhiuk]

Testing blocked by

• Support different pre and post-auth capabilities. Solve ImapConnection.java reuse issues #1297

[philipwhiuk]

I've not run into the capabilities bug recently (I applied cketti's fix). So I've been getting on with testing.

• I've written code that seems to behave correctly for token refresh
• I've started testing non-Gmail accounts ( @AndyCouch ). Not run into any issues so far.

Thought I'd share some screenshots :)

In the first you can see OAuth 2.0 has been checked and the account list is now a drop-down with the list of available Google Accounts. The password field is gone and client-certificate option is greyed out.

In the second you can see the authentication screenshot that appears. Note currently K-9 is an unregistered app. I need to fix this next because I can't work out how to de-auth an unregistered app and if I can't do that I'll be signing up to lots of Gmail accounts :)

[philipwhiuk]

@cketti Can you please obtain an API key for this from Google. https://console.developers.google.com/ - if we don't already have a project, create one, search for Gmail API, fill in the consent screen and then obtain the key (you may need to get a SHA1 of the certificate

Hopefully doing one for probably actually want one for com.fsck.k9 is enough and we don't need a separate one for com.fsck.k9.debug.

(I'm asking you because it wants to be done under the same account that holds the Play store stuff I think)

[cketti]

Done.

[philipwhiuk]

Thanks. Can you either email me the key (and secret if there is one) or commit it to the repo? They get added to the request sent to Google in the app.

[philipwhiuk]

@cketti Waiting for the above to finish this off. I think it's pretty much done otherwise.

[Maetts]

I would support - if I could code. Checked already some alternative apps but there is literally nothing like k9. pEp mail is basically a clone of k9 but with oauth support.

[tbertels]

Like already said in this thread, Fairemail supports Gmail's OAuth, though only for on-device accounts and with the Google Play version.
https://github.com/M66B/FairEmail/blob/master/FAQ.md#user-content-faq111

non on-device accounts is not supported because Google requires a yearly security audit ($15,000 to $75,000) for this.

[Maetts]

Yeah thanks for the hint, I tested fairemail but didn't like it. Not like I love k9 since decades and on manymanymany phones.

@cketti is there really nothing new on this topic? Just to know if we have to arrange with another mail client.. 😔

[Tonus1]

I have almost daily alerts from Gmail that my password was used from uncommon place... and no sync if I don't go and check their box.
I will not activate 2 factor auth. Really don't know what I will do.
Would be nice to know if @cketti is working on it at all.

[DaLiV]

take a look onto
https://developers.google.com/identity/protocols/oauth2/native-app
there is link to Standart library ... possible that simple may be linked into app ...
https://github.com/openid/AppAuth-Android
that is even made without dependency on webkit by deescription

[Maetts]

Is really nothing moving on this here? 😔

[Sempro97]

I can't also access to my university email because it use oauth2 trought outlook

[simolev]

It seems Gmail is about to remove "less secure apps" access, with Google Workspace to follow:

To help keep your account secure, starting May 30, 2022, ​​Google will no longer support the use of third-party apps or devices which ask you to sign in to your Google Account using only your username and password.

[njeyaakili]

While it will be good to have OAuth2 support, the key part in that announcement "Google Account using only your username and password", is that what is being disabled is using your google (sitewide) pw for login via 3rd-party apps. K-9 will continue to work with gmail if ... you set up, and configure K-9 to use, google "app passwords":

https://support.google.com/accounts/answer/6010255#zippy=%2Cuse-an-app-password

My sense is that MS Exchange may be disabling app passwords soon (including removing the ability for administrators to control whether they are accepted or not), but it should be noted that some bits of their implementation of OAuth2 differ from what is needed for gmail.

[pinusc]

While it will be good to have OAuth2 support, the key part in that announcement "Google Account using only your username and password", is that what is being disabled is using your google (sitewide) pw for login via 3rd-party apps. K-9 will continue to work with gmail if ... you set up, and configure K-9 to use, google "app passwords":

While this is true for most personal accounts, Google does not allow the creation of app passwords for organization (school, work) accounts. I mostly use K9 with my school email address, so it looks like on may 30 I will be unable to use K9 anymore. The fact that you can't create app passwords for organization accounts is documented here:

https://support.google.com/accounts/answer/185833

Indeed, there is no button for me to create an app password.

So, unlike the responses to the many issues/forum posts panicking about this would suggest, I believe that this will in fact completely break support for many users, myself included.

[odon-pla]

Delta Chat on F-Droid handles OAuth2. It's not a practical email client, but it shows how can be implemented.

Without OAuth2 support, I'm unable to use K-9 anymore. It's not a matter of choice, it's sheer necessity.

[pinusc]

Another FOSS app that offers authentication is FairEmail. From the sign-in process it looks like they leverage logged-in accounts on the device–as I got a microG confirmation dialogue before being able to log in; I think this idea was floated up before in this thread, maybe their implementation can be a good starting point to do the same thing in K9?

[vladimir-mencl-eresearch]

While this is true for most personal accounts, Google does not allow the creation of app passwords for organization (school, work) accounts. I mostly use K9 with my school email address, so it looks like on may 30 I will be unable to use K9 anymore. The fact that you can't create app passwords for organization accounts is documented here:

https://support.google.com/accounts/answer/185833

Indeed, there is no button for me to create an app password.

Hi,

I think the meaning is slightly different: I think for organization accounts, application passwords can be centrally disabled or left enabled. I'm using application passwords on my organization-managed gmail account. So likely driven by a central decision made by the organization's administrator.

But that unfortunately still does not help those users where the administrator decided to block application passwords...

Cheers,
Vlad

[rol1]

Hi,

while K9 is an open source project I (we) cand demand someone to add any feature, including OAUTH!
The "demand" for OAUTH is around for years, first posting in this thread is from 2015. For me it would be fine to hear from the dev-team if it is pland to implement OAUTH or not. I'm not asking when! I think it's not helping anyone if we all post here "provider XYZ is switching to OAUTH", that's a fact everybody is aware (I think).

Yours
rol1

[simolev]

My experience with Google is that app passwords can be created once 2FA is enabled.
My experience with Microsoft 365 is that an admin needs to allow app passwords for the whole organization here, and also enable Multi Factor Auth for the specific user here. After these two steps, users can create app passwords here, but they won't work anyway for imap access. I have a long-standing ticket with Microsoft support but so far they've only been able to ask why on earth would I not use "modern authentication" like everyone else, while at the same time requiring me to send them loads of Outlook (!) logs and screenshots.

[simolev]

I think it's not helping anyone if we all post here "provider XYZ is switching to OAUTH", that's a fact everybody is aware (I think).

Well, not everyone gets these announcements or has the time to interpret them and evaluate their impact. The issue here is that app passwords are supposed to be our lifeline to continue using K9-Mail (which is what we all want to do), and I for one feel that lifeline becoming increasingly thin and threatened. In my experience they work flawlessly with Yahoo, come with some caveats for Google, and are completely broken on Microsoft, just to cite a handful of major providers. What do you do if your employer uses Microsoft? Hopefully you check out if anything is moving forward with your favourite email app before switching to Spark or Bluemail for good.

[rol1]

... What do you do if your employer uses Microsoft? Hopefully you check out if anything is moving forward with your favourite email app before switching to Spark or Bluemail for good.

That's the reason why I asked if there is a plan to implement OAUTH or not. If not, I would swicht, if it will be implemented my next question would be when we could expect it, but that's question no 2!

[linkp]

Can someone lock this ticket to further commenting? This is an issue tracker not a discussion forum. Some of us have notifications enabled on this issue so we can know when progress is made toward resolving it. Seeing the same theories and speculations that have been posted regularly for the seven years this issue has been open is of no use to anyone. There is an official forum that is far better suited for redundant pondering: https://forum.k9mail.app/

[odon-pla]

"Delta Lab" can log in simultaneously into multiple oauth accounts. It doesn't need Google Services and it doesn't require any android system account.

However, since it's a chat app, it's not a viable email client. It doesn't have unified inbox and, before you can send a message to someone, you must have him in your contact list.

Nevertheless, it's open source software and the way it handles oauth authentication could be of some interest to k-9 developers, in case they wanted to implement it.

https://github.com/adbenitez/deltalab-android/releases/tag/v1.30.0-deltalab

[agoeckner]

Hello. Does the future plan for K-9 Mail include OAUTH support?

After reading this thread and others, I'm unable to tell whether this is currently in the works or whether it is considered "not necessary".

I would like to know the status of this so that I can begin my migration away from K-9 Mail as soon as possible if necessary.

[ocdtrekkie]

Hello. Does the future plan for K-9 Mail include OAUTH support?

After reading this thread and others, I'm unable to tell whether this is currently in the works or whether it is considered "not necessary".

I would like to know the status of this so that I can begin my migration away from K-9 Mail as soon as possible if necessary.

This issue has been open for seven years. I've switched phone OSes twice since I first commented.

...It's not gonna happen.

[cremor]

Looks like FairEmail is dead:

All my projects have been terminated after Google falsely flagged FairEmail as spyware without a reasonable opportunity to appeal. There will be no further development and no more support.

Source: https://email.faircode.eu

We really need this in K-9 😢

[ptman]

Looks like FairEmail is dead:

FairEmail is open source https://github.com/M66B/FairEmail and available on F-Droid. A fork is pretty likely.

[cketti]

Support for this feature is planned for the next release. See https://forum.k9mail.app/t/the-plan-for-k-9-mail-6-200/4590

Please don't use this issue tracker as a discussion forum. Use the actual forum instead: https://forum.k9mail.app/

[mmhere]

Apologies in advance if this is interpreted merely as "discussion" that could be in the forum link #655 (comment).

Firstly, that post seems pretty inactive (and requires a separate login to post).

Secondly, I think this comment informs the urgency of the task (adding OAuth support, specifically for gmail) as well as highlights an important side effect of not having OAuth support when google kills 3rd party app userid/pw login on the 30th of May.

---

k9 will still be useable with gmail via the "app password" option, but the approach google is using is crummy because you are forced to enable 2FA across your entire google account to be able to use app passwords. I know this is mentioned elsewhere. Being forced to enable 2FA is deal killer for some because it adds extra steps to access your account in all cases, just so you may gain access to the "app password" functionality. I, for one, don't want to have to reach for a 2nd device when say logging into google from a desktop browser.

Also, regarding urgency -- everyone is likely aware that regular userid/pw access (not "app password", which requires the undesirable 2FA by side effect) goes away on the 30th of May.

For those who don't want to be burdened with 2FA, this effectively renders k9 disabled until the OAuth support comes online in k9.

So the lack of k9 functionality sans OAuth seems pertinent here in the face of gmail use without 2FA.

I really like using k9 because it's the only thing I've found that does instant notification [push email / IMAP IDLE] properly. I don't want to lose instant email notifications on my phone. The gmail app, for one, doesn't do it properly and is an overly invasive app (IMHO).

P.S.: As a SW dev myself, I don't like discussing schedule. I'm therefore not asking for a schedule ;-) but I am curious to inquire: "How's it going?"

[ocdtrekkie]

@mmhere Everyone should be using 2FA on nearly every email account. 😦 It's 2022.

[drzraf]

@mmhere Everyone should be using 2FA on nearly every email account. frowning It's 2022.

Most of us are fine with our passphrases and wouldn't use 2FA if it was not forced on us by GAFA for reasons which have nothing to do with our security.