GMail tokens expire after a week

[dechamps]

I set up sasl-xoauth2 using the instructions for GMail. 7 days later, it suddenly stopped working with:

smtp[260368]: Verified TLS connection established to smtp.gmail.com[173.194.76.109]:587: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256) server-digest SHA256
smtp[260358]: Verified TLS connection established to smtp.gmail.com[173.194.76.109]:587: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256) server-digest SHA256
smtp[260358]: AC58811F876: SASL authentication failed; cannot authenticate to server smtp.gmail.com[173.194.76.109]: bad protocol / cancel
sasl-xoauth2[260358]: auth failed:
sasl-xoauth2[260358]:   2022-04-02 10:17:04: Client: created
sasl-xoauth2[260358]:   2022-04-02 10:17:04: Client::DoStep: called with state 0
sasl-xoauth2[260358]:   2022-04-02 10:17:04: Client::InitialStep: TriggerAuthNameCallback err=0
sasl-xoauth2[260358]:   2022-04-02 10:17:04: Client::InitialStep: TriggerPasswordCallback err=0
sasl-xoauth2[260358]:   2022-04-02 10:17:04: TokenStore::Read: file=/var/lib/postfix/sasl-xoauth2/zy<snip>tp@gmail.com/token
sasl-xoauth2[260358]:   2022-04-02 10:17:04: TokenStore::Read: refresh=1//03<snip>ZoU, access=ya29.A0A<snip>sV-w
sasl-xoauth2[260358]:   2022-04-02 10:17:04: TokenStore::GetAccessToken: token expired. refreshing.
sasl-xoauth2[260358]:   2022-04-02 10:17:04: TokenStore::Refresh: attempt 1
sasl-xoauth2[260358]:   2022-04-02 10:17:04: TokenStore::Refresh: token_endpoint: https://accounts.google.com/o/oauth2/token
sasl-xoauth2[260358]:   2022-04-02 10:17:04: TokenStore::Refresh: request: client_id=69<snip>a9.apps.googleusercontent.com&client_secret=GO<snip>kw&grant_type=refresh_token&refresh_token=1//03<snip>oU
sasl-xoauth2[260358]:   2022-04-02 10:17:04: TokenStore::Refresh: code=400, response={
                                                        "error": "invalid_grant",
                                                        "error_description": "Token has been expired or revoked."
                                                      }
sasl-xoauth2[260358]:   2022-04-02 10:17:04: TokenStore::Refresh: request failed
sasl-xoauth2[260358]:   2022-04-02 10:17:04: Client::DoStep: new state 0 and err -5
sasl-xoauth2[260358]:   2022-04-02 10:17:04: Client: destroyed
smtp[260368]: 443E6815C3: to=<zy<snip>tp@gmail.com>, relay=smtp.gmail.com[173.194.76.109]:587, delay=321, delays=321/0/0.18/0, dsn=4.7.0, status=deferred (SASL authentication failed; cannot authenticate to server smtp.gmail.com[173.194.76.109]: bad protocol / cancel)

If this Stack Overflow entry is any indication, this is not a problem with sasl-xoauth2 per se but with GMail itself. According to Google docs:

Authorizations by a test user will expire seven days from the time of consent.

So it would appear that the only way to get a GMail token that lasts for more than a week is to "publish" the app and move it to "production" status. Looking at the "verification requirements" this doesn't seem like a realistic prospect for a dummy app used solely to configure some Postfix setup.

The sasl-xoauth2 documentation doesn't mention this problem at all. This brings me to the following questions:

• Do sasl-xoauth2 devs/users know of a workaround that I missed?
• If yes, can the workaround be shared and added to the instructions?
• If not, can the instructions be updated to warn about this? It's quite the waste of time to go through the entire setup only to realize 7 days later that things stopped working and that this approach is not actually viable for any kind of permanent deployment.

[tarickb]

Have you tried configuring your app for internal use only? That's how I have mine configured and haven't come across any 7-day token-expiry issues.

[dechamps]

I can't do that:

I'm using a normal, personal @gmail.com account. Maybe you're using Google Workspace/GSuite which is why it works for you?

[tarickb]

Indeed I am using a Workspace account. What happens if you select "external", complete the rest of the setup flow, then publish the app? In a quick experiment with my own account I was warned that the app would have to be verified, but I wasn't forced into that process -- apps for personal use are exempt from verification requirements.

[dechamps]

What happens if you select "external", complete the rest of the setup flow, then publish the app?

To be clear, the "Publishing status" of my app was "Testing". My understanding is that's why the OAuth tokens are revoked after 7 days.

Unless someone knows something I don't, it looks like the only way to get around the 7-day token expiry is to publish the app to move it to "production".

If I try to publish the app, then the "Publishing status" does show "In production", but it's "pending verification". While in that state the OAuth consent screen stops working:

I've tried filling out the verification form, but it's… quite intense and really doesn't feel right for my use case. It's asking for tons of links to app website, privacy policies, and even a demo YouTube video(!) and none of it is optional - I was forced to put dummy links everywhere. The form doesn't provide any way to apply for any kind of "personal use" exception. I submitted the form anyway while explaining the problem as best I could, but now it's telling me that verification is blocked because I need to "verify ownership" of the dummy domain I provided… that really looks like a rabbit hole and I'm not sure there's much point in attempting to forcefully shoehorn my way into their verification process.

In a quick experiment with my own account I was warned that the app would have to be verified, but I wasn't forced into that process

What do you mean you "weren't forced into that process"? If you're saying that you can use the app in testing mode then that's true… but your token will only work for 7 days.

apps for personal use are exempt from verification requirements.

They might be in theory, but in practice there doesn't seem to be a way to get around the 7-day token expiry without going through said "verification requirements". I agree that the observed behaviour is inconsistent with the Google docs.

[tarickb]

Are you using the same Gmail account to both create the app and then authorize its use? As far as I know that's how the personal-use exemption works, and if you're using different accounts then yes, you'll hit the testing-mode issue.

[dechamps]

Previously the GCP account and the authorized "test users" were separate. A few days ago I had the same thought and made them the same. Now I'm waiting for the 7-day mark and we'll see what happens then...

[tarickb]

I should have looked more carefully at your screenshot. You're bumping into the recent changes that prevent use of the out-of-band redirect URI. I wrote a quick-and-very-lightly-tested script that should help with this. Set your project to "in production" (no need to "prepare for verification" or anything like that), then:

$ python3 ./scripts/get-initial-gmail-tokens.py \
  --client_id="YOUR_CLIENT_ID" \
  --client_secret="YOUR_CLIENT_SECRET" \
  --scope="https://mail.google.com/" \
  ./token.json

Once you open the URL the script generates and allow access, it should write out token.json that you can then pass onto sasl-xoauth2. Let me know if it works and I'll update the README.

[dechamps]

Interesting! Your script does work - I was able to get a token with the app in "production" (unverified) mode. It was indeed a bit odd that the OAuth flow would just stop working when the app is moved to "production". I should have dug deeper… thanks for figuring it out.

Last Saturday I set things up with an app in "Testing" mode owned by the same Google account as the GMail user.

Today I set up a separate "production" app using your script where the app owner and the GMail user are separate accounts.

I'll wait 7 days and see what happens. If none of the above work, I'll try again with a "production" app owned by the GMail user. Hopefully at least one combination will work, and at that point we'll be able to document the exact requirements.

[dechamps]

The results are in:

• With the app in "Publishing status: Testing", the OAuth token expires after 7 days even if the app is owned by the same account as the GMail user.
• With the app in "Publishing status: In production", the OAuth token still works after 7 days even if the GMail user is different from the app owner.

I will send a PR to update the README accordingly.

[dechamps]

See #34