Problem refreshing access token with Offlineimap

[pablob127]

I have a common problem: Offlineimap works well with oama and Microsoft, for a while, but once the access token expires it stops working. I looked at the other issues with this problem, but they did not seem to apply. I apologize if I missed something obvious there.

In a previous setup I had, Offlineimap has access to both the access and refresh token, and it could easily renew its access token and keep running uninterrupted for long times. I haven't found a way to make oama return the refresh token, and on its own it does not seem to be able to renew its access token. I even tried to do a manual renew by running "oama renew ",which seemed to be successful but did not seem to help the stuck Offlineimap.

Is oama supposed to do the access token renewal on its own? If so, why isn't it doing it? What am I doing wrong?

My streamlined config files are:

config.yaml:

services_file: ~/.config/oama/services.yaml                                     
                                                                                
ring_store:                                                                     
  exec: secret-tool                                                             
  args:                                                                         
    - store                                                                     
    - --label                                                                   
                                                                                
ring_lookup:                                                                    
  exec: secret-tool                                                             
  args:                                                                         
    - lookup                                                                    
    - mailctl           

services.yaml:

microsoft:                                                                      
  auth_endpoint: https://login.microsoftonline.com/common/oauth2/v2.0/authorize 
  auth_http_method: GET                                                         
  auth_params_mode: query-string                                                
  token_endpoint: https://login.microsoftonline.com/common/oauth2/v2.0/token    
  token_http_method: POST                                                       
  token_params_mode: request-body-form                                          
  redirect_uri: http://localhost:8080                                           
  auth_scope: https://outlook.office365.com/IMAP.AccessAsUser.All https://outlook.office365.com/SMTP.Send offline_access                                        
  client_id: <obscured>
  client_secret: <obscured>

.offlineimaprc [relevant extract]:

auth_mechanisms = XOAUTH2                                                       
oauth2_request_url = https://login.microsoftonline.com/common/oauth2/v2.0/token 
oauth2_client_id = <same as in services.yaml>                         
oauth2_client_secret = <same as in services.yaml>
oauth2_access_token_eval = get_oauth_access_token('<email>')

offlineimap.py:

import subprocess                                                               
                                                                                
def get_oauth_access_token(email_address):                                      
    return subprocess.run(["oama", "access", email_address], capture_output=True, text=True).stdout   

Thanks!

[pdobsan]

On Fri, May 10, 2024 at 07:54:45AM GMT, pablob127 wrote: Is oama supposed to do the access token renewal on its own?

Yes of course, that should happen automatically.

If so, why isn't it doing it? What am I doing wrong?

oama has not been officially released yet. I assume you built it from source and ran into some inconsistent state with your old setup. I have just committed the (most likely) final changes before the release. There are breaking changes, so please read the README carefully. So you need to build it again or alternatively, can download a pre-built static binary from here: https://s3.sr.ht/builds.sr.ht/artifacts/~petrus/1217865/856d484afcb3c69e/oama.tar

[pablob127]

It turns out that I was using the latest mailctl version, not oama (I was just renaming it so it would be closer to the real one once it was released). Now that oama is released, I tried to switch but I am running into the Microsoft organizational authorization issue reported in issue #44 .
Mailctl is working with the files above, except with getting a new access token once the first one expires.

[pdobsan]

Try the most recent oama it should solve your renewal problem.

[pablob127]

Try the most recent oama it should solve your renewal problem.

Unfortunately, it does not. It is version 0.10.1, I could authorize properly (I had the same issue with the redirect URI as other people had), but I have just tried and offlineimap bailed out with an "unable to authenticate" after some time.

How can we try to debug this?

[pdobsan]

How can we try to debug this?

You can start by sending the output of the `oama printenv` command. Also by asking other users, you are not the only one using oama with similar setup.

[pablob127]

Here's the result of 'oama printenv':

 google:
    auth_endpoint: https://accounts.google.com/o/oauth2/auth
    auth_http_method: POST
    auth_params_mode: QueryString
    auth_scope: https://mail.google.com/
    tenant: null
    token_endpoint: https://accounts.google.com/o/oauth2/token
    token_http_method: POST
    token_params_mode: RequestBody
  microsoft:
    auth_endpoint: https://login.microsoftonline.com/common/oauth2/v2.0/authorize
    auth_http_method: GET
    auth_params_mode: QueryString
    auth_scope: https://outlook.office365.com/IMAP.AccessAsUser.All https://outlook.office365.com/SMTP.Send
      offline_access
    tenant: common
    token_endpoint: https://login.microsoftonline.com/common/oauth2/v2.0/token
    token_http_method: POST
    token_params_mode: RequestBodyForm
######

[pdobsan]

release 0.11.1 has a new command show <email>. Use that to debug your problem and check if the refreshing happens. Most likely it does and that is an external problem.

[david-antos]

I'm trying to solve similar problem with renewals as in this ticket (i.e. authorize works fine for 50 minutes or so, renewal doesn't happen).

I'm trying to debug using the show command:

On oama version 0.11.1 (out of current Arch AUR package):

oama access CENSORED-MAIL@gmail.com returns the token (suggesting my config should be correct), but

oama show CENSORED-MAIL@gmail.com gives

email: CENSORED-MAIL@gmail.com
service: google
scope: https://mail.google.com/
oama: Maybe.fromJust: Nothing
CallStack (from HasCallStack):
  error, called at libraries/base/Data/Maybe.hs:150:21 in base:Data.Maybe
  fromJust, called at lib/OAMa/Authorization.hs:142:39 in oama-0.11.1-inplace:OAMa.Authorization

and oama renew CENSORED-MAIL@gmail.com returns

oama: renewAccessToken: Nothing as refresh token argument
CallStack (from HasCallStack):
  error, called at lib/OAMa/Authorization.hs:281:19 in oama-0.11.1-inplace:OAMa.Authorization

[pdobsan]

Please, follow the instructions in the Issues section. You did not provide the required information. You need to provide the full output of oama printenv as requested.

In your case, the best is to do that twice. Once, immediately after the authorization while it works. Then get the printenv output again when oama has failed to renew. Also, indicate what kind of client_{id,secret} you are using.

I can see why your oama fails to renew the access_token, it does not have a refresh_token. But I cannot even guess how it happened unless you send the printenv outputs. Chances are that if you carefully read and compare these outputs you can diagnose the problem yourself.

By the way, @pablob127 did not send the full printenv output either.

[pablob127]

I am doing another test with the older version (I do not want to stop to change to the new one), extracting the refresh token from the keyring. In my case, it does not look like the refresh token is missing (I see it in the keyring), but it is somehow not working. I made a script so that Offlineimap has access to the refresh token and I am trying to see what happens. Once I see what happens, I will move to the newest version, try again and post back (with various 'printenvs', complete this time).

[david-antos]

Please, follow the instructions in the Issues section. You did not provide the required information. You need to provide the full output of oama printenv as requested.

Apologies, I haven't noticed the instructions. My prinenv is completely the
same both when the token is valid as well as when it is not, and it goes as
follows:

###  Runtime environment  ###
config:
  encryption:
    tag: GRING
  redirect_port: 8080
  services:
    google:
      auth_endpoint: null
      auth_http_method: null
      auth_params_mode: null
      auth_scope: null
      client_id: CENSORED
      client_secret: CENSORED
      tenant: null
      token_endpoint: null
      token_http_method: null
      token_params_mode: null
config_file: /home/antos/.config/oama/config.yaml
data_dir: /home/antos/.local/var/oama
oama_version: 0.11.1
op_sys: |
  Linux erik 6.6.31-1-lts #1 SMP PREEMPT_DYNAMIC Fri, 17 May 2024 11:49:30 +0000 x86_64 GNU/Linux
options:
  optCommand:
    tag: PrintEnv
  optConfig: ~/.config/oama/config.yaml
  optDebug: false
services:
  google:
    auth_endpoint: https://accounts.google.com/o/oauth2/auth
    auth_http_method: POST
    auth_params_mode: QueryString
    auth_scope: https://mail.google.com/
    client_id: CENSORED
    client_secret: CENSORED
    tenant: null
    token_endpoint: https://accounts.google.com/o/oauth2/token
    token_http_method: POST
    token_params_mode: RequestBody
######

In your case, the best is to do that twice. Once, immediately after the authorization while it works. Then get the printenv output again when oama has failed to renew. Also, indicate what kind of client_{id,secret} you are using.

I have my own client_id produced following instructions https://help.sagecrm.com/on_premise/en/2022R1/Administration/Content/Administrator/EM_GetClientIdSecret-Gmail.htm

And it is crucial to note it used to work with mailctl for a long time, up to the point Arch AUR switched to oama.

I can see why your oama fails to renew the access_token, it does not have a refresh_token. But I cannot even guess how it happened unless you send the printenv outputs. Chances are that if you carefully read and compare these outputs you can diagnose the problem yourself.

Alas, as I said, they are absolutely identical.

[pdobsan]

And it is crucial to note it used to work with mailctl for a long time, up to the point Arch AUR switched to oama.

Intresting, so something went wrong during the transition in at least two similar occasions.

Have you cleaned your keyring up before switching to oama?

Here are a few things to try, in a matrix as it makes sense ... debugging is fun.

• Remove all oama, mailctl, etc entries from your keyring. seahorse is a good gui client to use so you won't miss anything.

• Rerun authorization again and immediately after check what is in your keyring. In particular analyze the output of

  secret-tool search oama you@gmail.com

Is the secret a valid JSON document? Try to format it, do you see a good looking refresh_token?

The other two things to test:

• Try the GPG encrypted backend instead of keyring.
• Use, at least temporarily, a different client_id/secret, like thunderbird's

[david-antos]

Quick check:
I hope I cleaned up everything that may be related in the key ring, anyway, I tested GPG option as well and the behaviour is completely identical.

secret obtained from secret-tools search oama XXXX@gmail.com seems to be a valid JSON, but there's a problem:

"refresh_token":null

Where do I find thunderbird's client_id/secret to isolate this dependence?

[pablob127]

After some test, it looks like the problem is not oama but rather Offlineimap. When my version of Offlineimap (8.0.0, included in Debian Bookworm) gets an access token, once it expires it seems to just fail the authentication without trying to get a new access token. This happens even if you give it both an access token and a refresh one. It works well when giving Offlineimap a refresh token only. I ended up extracting the refresh token obtained by oama from the keyring, giving it to Offilineimap and it seems to work (we'll see for how long, I'm not sure what the validity of a refresh token is).

One question: Would it make sense to add a command to oama to return the refresh token?

Since the missing refresh token for Google seems to be different, I will close this issue leaving this for reference.

Thanks a lot for your work!

[pdobsan]

it to Offilineimap and it seems to work (we'll see for how long, I'm not sure what the validity of a refresh token is).

In my experience, it must be very long.

One question: Would it make sense to add a command to oama to return the refresh token?

There is one: oama show <email>

[david-antos]

I'm afraid I don't follow this explanation. My understanding is that offlineimap (as well as mbsync/isync I'm using) just calls
oama access <email>
that returns access token for offlineimap to use, and during this operation, oama should also renew the token.

I also suppose that oama authorize <service> <email> should set up the renew token as well. Am I missing something?

I'm happy to test this issue further, the idea by @pdobsan I haven't tested yet was "Use, at least temporarily, a different client_id/secret, like thunderbird's" (if you could point where to find those).

After some test, it looks like the problem is not oama but rather Offlineimap. When my version of Offlineimap (8.0.0, included in Debian Bookworm) gets an access token, once it expires it seems to just fail the authentication without trying to get a new access token. This happens even if you give it both an access token and a refresh one. It works well when giving Offlineimap a refresh token only. I ended up extracting the refresh token obtained by oama from the keyring, giving it to Offilineimap and it seems to work (we'll see for how long, I'm not sure what the validity of a refresh token is).

[pablob127]

it to Offilineimap and it seems to work (we'll see for how long, I'm not sure what the validity of a refresh token is).
In my experience, it must be very long.
One question: Would it make sense to add a command to oama to return the refresh token?
There is one: oama show

That one would need some text processing to isolate the refresh token out of everything else. I was thinking more along the lines of a command like:

oama refresh <email>

that would print to standard output only the refresh token (just like "oama access" does for the access token).

[pablob127]

I'm afraid I don't follow this explanation. My understanding is that offlineimap (as well as mbsync/isync I'm using) just calls oama access <email> that returns access token for offlineimap to use, and during this operation, oama should also renew the token.

That's what I thought, but I found out that when I started getting authentication errors Offlineimap was not calling the access token evaluation routine.

There's probably an issue where the expiration time of the token does not match what the server believes to be the expiration time. I should go open an issue there, but I got it working with the refresh token and I don't have much time available lately.

[pdobsan]

as well as mbsync/isync I'm using) just calls ```oama access <email>``` that returns access token for offlineimap to use, and during this operation, oama should also renew the token.

Yes that's the way it is supposed to work.

I also suppose that `oama authorize <service> <email>` should set up the renew token as well. Am I missing something?

Correct.

I'm happy to test this issue further, the idea by @pdobsan I haven't tested yet was "Use, at least temporarily, a different client_id/secret, like thunderbird's"

Feel free to open a separate issue with that. My hunch is that the problem is with your client_id/secret.

(if you could point where to find those).

You need to be a bit more resourceful. ;-) (have a look at issue #43)

[david-antos]

My hunch is that the problem is with your client_id/secret.

And you are absolutely right.

With Thunderbird's client_id/secret, everything works as expected. This is a bit puzzling, as it used to work with mailctl until recently, anyway, all's well that ends well :).

Thanks for your help as well as for this great tool!

[pdobsan]

With Thunderbird's client_id/secret, everything works as expected. This is a bit puzzling, as it used to work with mailctl until recently, anyway, all's well that ends well :).

That puzzle could be incidental to your client_id/secret expiring around this time. To register a stable id/secret pair is quite complicated and time consuming. That is why I have not done it for `oama` yet.