Limiting access to data in your Google account

[Pectojin]

Just got the below email. I'm hoping the Mailpile team was informed/is able to resolve this.

Same thing happened to gmvault (gaubert/gmvault#335 (comment)) and they weren't able to resolve it. It's a total PITA for end users to setup a personal oauth service to use the tool.

I'm not gonna speculate but I'm getting increasingly upset with Google pulling this stuff.

---

[BjarniRunar]

Thanks for reporting this. I'll see if I can figure out WTF is going on. 😞

[h3artbl33d]

I think this is related to what Google calls "project Strobe". Your Gmail account in Mailpile, is it configured with OAuth? If so, you can fix this by reconfiguring your account using IMAP rather than OAuth.

When adding an account:

1. Make sure IMAP is enabled for your Gmail account
2. Untick the "Detect settings" box in the Basic Settings tab
3. Configure the account as if it were a regular IMAP/SMTP account
4. If you are using two factor authentication, generate an app password rather than inputting your normal password.

This should be sufficient to work around Google's ridiculous practices. If not, please reply and I'll help you wherever I can.

[BjarniRunar]

Thanks @h3artbl33d for mentioning this; this is indeed a feasible workaround for people who are motivated and in trouble.

But for everyone else, if we can't access GMail accounts without jumping through hoops, that's a pretty major setback for the project. I'm hoping I can reach someone at Google to figure out what is going on. Their docs and instructions just don't make sense to me, I fear I may have missed some critical e-mails or something.

[h3artbl33d]

In retrospect: it was a part of a larger announcement; from the title or intro that was sent out way back, it wasn't very clear what the direct impact on Gmail OAuth was going to be; only by clicking some link and reading the webpage. More in-depth information:

https://developers.google.com/terms/api-services-user-data-policy#additional-requirements-for-specific-api-scopes
https://support.google.com/cloud/answer/9110914#restricted-scopes

It pretty much boils down to "limit your API scope, because we want to keep our monopoly at harvesting user data" (apologies for the harsh phrasing).

[BjarniRunar]

As a mail client that wants to read and write e-mail on your behalf, we can't limit our scopes.

[h3artbl33d]

I totally understand that. This move that Google is making here is hurting the community. You could apply for the restricted scope verification: https://support.google.com/cloud/answer/9110914#restricted-scopes

I'd very much like to help you out any way I can. But other than researching this development, there is not much I can do. Personally, I wouldn't entrust a random stranger (which is what I am) with the credentials to request such a verification and collaborate with Google ;)

EDIT: It's even worse than I initially thought. According to the support link in the first paragraph:

What if my app is using IMAP or SMTP? Do I need to submit for verification?
Yes, because IMAP and SMTP usage require using https://mail.google.com/, you will need to submit your app for the restricted scope verification. If your usage of IMAP/SMTP is deemed to violate the minimum scope policy within the verification process, you will need to migrate to using the Gmail API by September 15, 2019.

So, if a project fails the verification process - which has vague guidelines to put it subtle - there isn't even a fallback to IMAP/SMTP possible anymore. Way to go Google!

[BjarniRunar]

I still haven't gotten any responses from Google.

[BjarniRunar]

Having pondered this briefly, I think I'm just going to abandon the existing credentials and try to create new ones this week. We missed whatever deadlines they had set, but there's nothing that says we can't register a "new" app with the perms we need. That may need a review, but hopefully this time I'll manage to navigate that...

[h3artbl33d]

Thank you kindly for your time and effort on this issue. It's bad that Google went down this road and indirectly hurting open source projects like Mailpile. If there is anything I can do to help, please ping me, I will gladly help Mailpile advance in any way I can.

[JackDca]

For what it is worth - as of today 2019-07-18, which is 3 days past the deadline advertised by Google - I am still able to sign on and download new mail from three gmail accounts (i.e. mptest????@gmail.com) that I use for testing. I also tried sending from one to another (uses SMTP). This is using the current master e86d5ba.

The receive settings are IMAP/TLS/port 993/OAuth2.

Send settings are SMTP/TLS/port 465/OAuth2.

Either the deadline has been extended, or Google rethought the impact of invalidating IMAP and SMTP access, or Bjarni's credential efforts have been rewarded.

[BjarniRunar]

All my mails were ignored, but I got an e-mail just a few minutes ago telling me that the deadline had passed and our credentials would be revoked. So, I expect things to break badly quite soon now.

I've applied for new credentials, but they say the process may take weeks and since I never got any responses to our last application, I'm not feeling much optimism this time around. Fingers crossed?

[BjarniRunar]

So, uh, good news! Our access is revoked... but:

Turns out, brave users can still click through using the "advanced" option and decide to grant access anyway. So it's ugly and scary, but we're not dead in the water.

[BjarniRunar]

Support thread, with screenshots and an explanation: https://community.mailpile.is/t/logging-on-to-gmail-accounts-this-app-isnt-verified/204

[h3artbl33d]

@BjarniRunar I don't think I can do a better job, nor do I want to say anything about the effort you went through; would you give me a try with the big, bad and mean G to get this resolved?

[BjarniRunar]

@h3artbl33d Sorry about the late response - that's not easy to do. I'd need to give you access to the Mailpile Google accounts, and I'm just not comfortable handing out those credentials at this point in time. But thank you very much for the offer.

[zencomplex]

Can anyone recommend a free email provider that would currently work with Mailpile? Google, Yahoo, etc. all don't work at this time, and Protonmail requires a subscription to use POP3/SMTP/IMAP .

[JackDca]

Hi @zencomplex.

It appears that it is still possible to use GMail. Google has made it difficult and frightening for the non-technical user to set up, but it was possible the last time I tried. I have some existing GMail accounts that I use for testing Mailpile and I was using two of them today.

You must enable IMAP and "Less secure apps".

The first is straightforward:
Sign on to GMail click on the gear icon - Settings - Forwarding and POP/IMAP - Enable IMAP .

The second takes a few more steps. From the screen where you clicked Enable IMAP, under Configure your email client, click Configuration instructions.
Near the bottom of that screen find and click I can't sign in to my email client.
Then find and click the link to allow less secure apps to access your account .
Then click If "Less secure app access" is off for your account
Then click turn it back on
Then click on the switch to set Allow less secure apps: ON
WHEW!!!

Please let me know if that works!

[cbz20]

If you are looking for a new email provider, take a look at privacytools.io. They list four free email providers. protonmail+mailfence support SMTP/POP/etc. only for paid accounts (starting at 2.50 EUR per month); tutanota might not work well with mailpile, since their nonstandard encryption might interfere with PGP. So that leaves disroot that you could try. I have been using posteo with mailpile for over a year. At 1 EUR per month, that is as good as free, and you do not have to go through any hoops as with gmail.

[JazzTp]

Can anyone recommend a free email provider that would currently work with Mailpile? Google, Yahoo, etc. all don't work at this time, and Protonmail requires a subscription to use POP3/SMTP/IMAP .

Vivaldi.net works with Mailpile

(although sometimes new emails were not showing up while they did appear in Thunderbird... any new incoming mail made them all appear in Mailpile as well immediately... I can't figure out what changes when that happens vs. when everything is just fine)

[JazzTp]

I think this is related to what Google calls "project Strobe". Your Gmail account in Mailpile, is it configured with OAuth? If so, you can fix this by reconfiguring your account using IMAP rather than OAuth.

When adding an account:

1. Make sure [IMAP is enabled](https://support.google.com/mail/answer/7126229?hl=en) for your Gmail account

2. Untick the "Detect settings" box in the _Basic Settings_ tab

3. Configure the account as if it were a regular IMAP/SMTP account

4. If you are using _two factor authentication_, generate an app password rather than inputting your normal password.

This should be sufficient to work around Google's ridiculous practices. If not, please reply and I'll help you wherever I can.

Thank you. Mailpile is working great copying from my GMail account, and deleting rubbish is much faster via Mailpile's CLI than via GMail's webmail interface :D

I only had to generate an app password, change Mailpile's settings from oauth to password and paste in that app password.

(I do have 2FA active on that account, but I wasn't required to use Google Authenticator, instead I was asked to confirm on the PC a number I was seeing on the phone).

[JackDca]

I have heard a report of Mailpile's GMail "app password" access suddenly failing (comments @JazzTp ?). It would be interesting to know if this is a permanent failure or if it can be fixed.

Today I checked a GMail account that I have use for testing Mailpile. I had enabled Google's "less secure apps" setting to permit access by Mailpile but had not accessed it with Mailpile for months. I was initially not able to access the account. It appears that Google had turned off "less secure apps" access because I had not used it. I turned on Google "less secure apps" access, ran Mailpile again, and was able to download emails from the account.

My setup described above did not use the "app password" method. So, based on @JazzTp 's post above, it appears that there are two different methods by which Mailpile can access GMail IMAP. In any case, at least the "less secure apps" method appears to still work.

This is also discussed in the Community forum:

https://community.mailpile.is/t/logging-on-to-gmail-accounts-this-app-isnt-verified/204

[DerfOh]

Less secure apps workaround breaks when you enable 2factor on your google account (can't have one on without disabling the other)

Any chance this issue could get more attention?

Fixed by changing the "0auth2" to "password" in the recieve email setting and send email setting. After allowing autoconfig to pull defaults. Then I supplied the app specific password generated in the google account settings