Email with MS Oauth2: Token refresh does not work

[ChrisTG742]

Prerequisites

• Can you reproduce the problem in a fresh installation of the "develop" branch?
• Do you have any errors in the PHP error log, or javascript console?
• Did you check the osTicket forums?
• Did you perform a cursory search to see if your bug or enhancement is already reported?

For more information on how to write a good bug report

Description

The Oauth for IMAP/SMTP authentication works as long as the access token lasts, but the refresh seems to be broken. The token does not get refreshed an the timestamp in the screen does not change.

Steps to Reproduce

1. Setup up Oauth against MS Exchange/Azure
2. Wait until the token becomes invalid
3. Try to fetch mail / send mail

Expected behavior:
It should get a ned valid access token via the refresh token and continue work normally.

Actual behavior:
When the access token is expired, no new token is fetched. No error is thrown.

Versions

osTicket v1.17 (1d8b790)
Apache/2.4.41 (Ubuntu 20 LTS)
MySQL Version 10.3.34
PHP Version 8.0.24

[ChrisTG742]

Also, the diagnotics fail here as it says "Test email sent successfully", but actually no mail could be sent as the token is invalid.

[ChrisTG742]

Any hints how I can debug this? Is there a way to enable logging for the communication in that plugin?

BTW: The current developent branch does not run for me, so testing with it is impossible:
Fatal error: Uncaught Error: Class "OAuth2AuthorizationBackend" not found in phar:///var/www/osTicket/upload/include/plugins/auth-oauth2.phar/oauth2.php:373
Also tried with development branch of the plugins, same error.

TIA!

[ChrisTG742]

Digging through the code I found that "hasExpired()" from /League/OAuth2/Client/Token/AccessToken.php is never called. Shouldn't there be somthing like

if ($token->hasExpired()) {
$this->refreshToken($token);
}

in the code?

[protich]

@ChrisTG742 - You're looking at the wrong place - the core osTicket is responsible for triggering referesh token renewal, not the Plugin or the Libraries.

Here is the section of the code that does it. If yours is failing to renew in the course of normal use, then perhaps you have a temprorary referesh token that expires after X days and you'll need to reauthorize again. I know Google, for example, limits how long a token can be refereshed to a week if the App is not approved (a.k.a in Test Mode).

[ChrisTG742]

Hi @protich,

thanks for your reply.

Of course, you're right. This is not the part of code that normally checks for expiration. But for debugging I'm fiddling in the "Manage" -> "Email" area right now.

If I for example change the setting of "Email Fetching" there and save the settings, I see that function "refreshAccessToken()" from ouath2.php (line 453) is called. Inside this function, the call of "$kb->refreshAccessToken()" (line 458) leads to the exception "invalid_request" here. I have no idea why since there is no detailed error message. I'm also wondering why this exception never gets to the system logs?

Actually the access_token expires after a few hours and I'm quite sure that we set up the app registration correctly in Azure.
Are there any "official" docs yet to check my settings against? If so, I'll double-check them again.

TIA!

[dd2513]

Hi @ChrisTG742 Did you ever find a solution for this?

I have the same issue. Everything is set up correctly. The token expires every hour and 20min or so. While the Token is valid the email fetch works great, then the token expires and we have to manually go in and submit the oauth authorization again.

How can we get this token to automatically refresh? It says the Expired Access Token gets auto-refreshed on use. However this is obviously not happening.

[dhimelick]

I also have this issue. I followed the steps in https://docs.osticket.com/en/latest/OAuth2/Microsoft%20Authorization%20Guide.html to set up my Azure app registration. I've actually done that several times, tested on two separate servers, and tested with a clean 1.17 install, to make sure I wasn't missing anything, but our token never refreshes automatically. Clicking Save Changes in the email config to resubmit OAuth refreshes the token for another ~80 minutes, as @dd2513 said.

[JediKev]

@dhimelick

It seems like you maybe don’t have a cron job setup or maybe the cron is using an older version of PHP? I tested this and could not replicate the issue. I let my O365 token expire, ran the cron, the tokens were updated automatically, and emails were fetched.

If you do have cron job setup appropriately and it's working then maybe it's something like what @protich said above:

Here is the section of the code that does it. If yours is failing to renew in the course of normal use, then perhaps you have a temprorary referesh token that expires after X days and you'll need to reauthorize again. I know Google, for example, limits how long a token can be refereshed to a week if the App is not approved (a.k.a in Test Mode).

Cheers.

[dhimelick]

@JediKev I do have a cron job (or some kind of scheduled task equivalent - my hosting provider uses Plesk so I can't see the internals). I will check to see if it's running with a different version of PHP than everything else.

I don't think it has to do with a temporary refresh token, as it literally never refreshes - the very first access token I get with a fresh app registration will expire and not get refreshed after ~80 minutes. I can check with my Azure team, though.

[dd2513]

Hi @JediKev I also have a scheduled task equivalent, windows task scheduler set up through the documentation here:https://docs.osticket.com/en/latest/Getting%20Started/POP3-IMAP%20Settings.html

I verified that it is running the same version of PHP as the rest of the system. 8.1.8.

The cron job/email fetching works great while the token is valid, it only stops when the token expires, and then the token does not refresh.

If I post a response on a ticket the sending of a email will refresh the token. So it only appears to be an issue with the fetching. And I agree with @dhimelick that it does not seem to be a problem with the refresh token, as it also never refreshes from email fetching. However it does refresh if you respond to a ticket and cause the system to send an email. As a side note it also does not refresh the token or send the diagnostic test email.

[JediKev]

@dd2513

That's very odd, why does mine work appropriately then? Unless in my instance it sent out mail first (updating the token) and then fetched - which I will double check to make sure this isn't the case. What's your OAuth2 config look like? (censor any sensitive info of course) Also, did you enable the Access Tokens and ID Tokens checkboxes in the Authentication tab in the registered app?

Cheers.

[ChrisTG742]

@JediKev could you please advise us how to debug the auto-refresh via cron that obviously does not perform like it should? Is there any change to enable some debugging log or something like that?

The normal cron-task seem to work as expected for me, only the refresh does not seem to happen.
On my system there's only one PHP version installed and I'm quite sure it's the same 8.0.24 that osTicket also uses via Apache2 module.

Any help appreciated.

[dd2513]

Hi @JediKev

I wish I knew why yours worked and mine doesn't!

I did check the access tokens and ID tokens checkboxes in the authentication tab in the registered app.

[ChrisTG742]

@dd2513 settings look exactly the same here.

[JediKev]

@ChrisTG742

There are no debug logs for this sort of thing but the code is open source so you can add debug statements at any point.

@dd2513 Mine look the same as well.

Cheers.

[ChrisTG742]

@JediKev thanks for the hints.
I added several "file_put_contents" for logging to a file in the function block of "getCredentials($auth=null, $refresh=false)". One directly after line 725.

Edit: I was too impatient.

All I could find in my log is the yet known not very helpful message coming from the "elseif" on line 800:
[refresh_token] => invalid_request
So, something must be wrong with the request that is generated.

[ChrisTG742]

For us, the solution was not to use a "temporary access password" when doing the initial OAuth handshake.
Obviously the refresh does not work afterwards, if you have enabled 2FA in your company and have used an temporary password.
I our case we logged in using a supported 2FA method and it seems that the refresh now keeps working.

I was not aware of this incongruity in conjunction with registering app endpoints. However, I colleague of me found that out more or less accidently.

Maybe this works for others affected by this "bug", too.

@JediKev perhaps it's a good idead to write a short notice about not using temporary passwords in your docs.

[dd2513]

Hi @ChrisTG742 @JediKev

I apologize in advance that this is not going to be terribly helpful for probably anyone. However I also got this to work. We were not using "temporary access passwords" so unfortunately that did not help us, but I appreciate the suggestion.

The fix for us was to completely reinstall osTicket from scratch, the site, php, database, all of it! We were very early in our deployment of this so it was not a big deal for us to blow it all up and start from scratch, however I understand that this isn't a "fix" that most people can attempt.

Thanks for all your help! Have a great day everyone!

[M1K489]

Same problem here. Everything works fine while the token lasts, but it doesn't get automatically renewed.

[ChrisTG742]

@M1K489 do you have 2FA enabled on your tenant or for that account? Did you use a TAP while settings the OAuth up?

[dhimelick]

I managed to resolve this issue for my instances of osT. @JediKev nailed it: my hosting provider didn't change the PHP version that ran the cron job despite changing the PHP version for the rest of the site, so it was still on PHP 7.x. Updating that to 8.1 immediately resolved the problem.

Thanks for the suggestion, @JediKev!

[M1K489]

@ChrisTG742 2FA is not in use for that account and we didn't use TAP. Also checked that the cron job is using the same php version as the site.

[pierromanu]

@M1K489 I faced the same issue

• no 2FA
• no TAP

Fix was to get the cron job to use php 8.1

[ChrisTG742]

Fix was to get the cron job to use php 8.1

What version of PHP exactly was it before?

[pierromanu]

Fix was to get the cron job to use php 8.1

What version of PHP exactly was it before?

7.4.32

[ChrisTG742]

Fix was to get the cron job to use php 8.1

What version of PHP exactly was it before?

7.4.32

That explains your problems before. You need at least PHP8.0. Otherwise the OAuth-stuff does not work correctly.

[GuDule68]

Hi all,

I had exactly the same problem.

I ran osTicket v1.14.2 with PHP7.2.34 before update to osTicket v1.17 with PHP8.0.25, and the ticket refresh didn't work.
It needed to get refresh manually almost every hour by going to the OAuth configuration and click "Submit" button to get refreshed.

2 days ago I've changed the PHP version to 8.1.12 and the token refresh works well yet.

I hope this feedback could help some people in needs.

[M1K489]

Still no luck. I have

• updated osTicket from 1.17 to 1.17.2
• PHP from 8.0.something to 8.1.13
• given application administrator permissions to our ticket email

Everything works while the token is valid but it doesn't get renewed automatically. I'm curious how have u set up the cron job on IIS?

[GuDule68]

@M1K489 Hi,

I'm not on IIS, I use Apache2.

Have you changed the cron job to point to the new PHP version ?

[dd2513]

@M1K489

I am using IIS and I have the cron job just set in windows task scheduler to run every minute.

I hope this helps!

[M1K489]

@dd2513

Well that looks pretty much like ours. It's just one thing I've noticed. Here u can see the cron job running every 5 minutes until the token gets invalidated at 13.12 . At 13.27 I go manually renew it:

While on the IIS the task is still running every 5 minutes:

But I guess this is normal.

I've also wondered if this refresh token is always supposed to start with 0.***:

[dd2513]

@M1K489

I don't have any good suggestions since IIS is running the task. I'm by no means an expert with osTicket. However I can tell you that my token also starts with 0.**** and has the last couple refreshes. I don't know if it will every time, but it doesn't appear to be a cause for alarm.

I hope someone else here who knows more about osTicket can offer some helpful advice.

[JediKev]

@M1K489

Are you getting the token and then making some sort of change in Azure afterwards that would prevent it from refreshing?

Cheers.

[M1K489]

@JediKev No, I haven't touched any settings in Azure since setting it up according to the instructions.

[JediKev]

@M1K489

Are you certain you are using v1.17.2 and the latest build of the OAuth2 plugin from our website? If so, I have no other suggestions as I'm unable to replicate this. Can you look in the database at the ost_email_account table and see if there is a value for last_error_msg column for this particular email? Also, can you run cron manually and see if you get any errors? Lastly, can you check your logs (general server logs, webserver error logs, PHP error logs, MySQL/MariaDB error logs, osTicket System Logs, etc.) for any related errors?

Cheers.

[M1K489]

@JediKev Here's our version info:

And plugin:

I checked that ost_email_account table for last_error_msg but it's empty. Also I couldn't find any errors in our osTicket system log, webserver nor PHP error logs.

I noticed though that if I try to run the cron job manually while the token is invalid nothing seems to happen. I have a .bat file with a command

"C:\Program Files (x86)\PHP\v8.1.13\php.exe" -f "C:\inetpub\wwwroot\osTicket\api\cron.php"

And a scheduled task that runs it every 5 mins. Now I let the token get invalid and run that .bat manually with admin rights and still nothing happened. I then went to osTicket, sent a test email and it fetched a new token. So it seems the problem is with that cron job.

[M1K489]

After further digging the logs I finally found something else. PHP-log:

[12-Dec-2022 11:36:33 UTC] PHP Fatal error: Uncaught Error: Call to a member function getName() on null in phar://C:/inetpub/wwwroot/osTicket/include/plugins/audit.phar/class.audit.php:746
Stack trace:
#0 C:\inetpub\wwwroot\osTicket\include\class.signal.php(98): AuditEntry::auditObjectEvent()
#1 C:\inetpub\wwwroot\osTicket\include\class.config.php(132): Signal::send()
#2 C:\inetpub\wwwroot\osTicket\include\class.config.php(140): Config->update()
#3 C:\inetpub\wwwroot\osTicket\include\class.email.php(1344): Config->updateAll()
#4 C:\inetpub\wwwroot\osTicket\include\class.email.php(925): EmailAccountConfig->updateInfo()
#5 C:\inetpub\wwwroot\osTicket\include\class.email.php(944): EmailAccount->updateOAuth2AuthCredentials()
#6 C:\inetpub\wwwroot\osTicket\include\class.email.php(883): EmailAccount->updateCredentials()
#7 C:\inetpub\wwwroot\osTicket\include\class.email.php(761): EmailAccount->getOAuth2AuthCredentials()
#8 C:\inetpub\wwwroot\osTicket\include\class.email.php(733): EmailAccount->getCredentials()
#9 C:\inetpub\wwwroot\osTicket\include\class.email.php(721): EmailAccount->getFreshCredentials()
#10 C:\inetpub\wwwroot\osTicket\include\class.email.php(474): EmailAccount->hasCredentials()
#11 C:\inetpub\wwwroot\osTicket\include\class.mailfetch.php(234): EmailAccount->isActive()
#12 C:\inetpub\wwwroot\osTicket\include\class.cron.php(25): osTicket\Mail\Fetcher::run()
#13 C:\inetpub\wwwroot\osTicket\include\class.cron.php(110): Cron::MailFetcher()
#14 C:\inetpub\wwwroot\osTicket\include\api.cron.php(19): Cron::run()
#15 C:\inetpub\wwwroot\osTicket\include\api.cron.php(40): CronApiController->run()
#16 C:\inetpub\wwwroot\osTicket\api\cron.php(23): LocalCronApiController::call()
#17 {main}
thrown in phar://C:/inetpub/wwwroot/osTicket/include/plugins/audit.phar/class.audit.php on line 746

[M1K489]

OMG, I think it's finally working. It seems the issue has been with that !"##¤%!"!"# Help Desk Audit plugin this whole time:

I disabled it and it managed to renew the token at least once. I thought that plugin came with the osTicket itself since it has always been there. I will do more tests tomorrow but if that was the issue, u should really add a note in the official documents telling you to disable all other plugins while setting up the OAuth2.

[JediKev]

@M1K489

That plugin should not cause issues unless you didn't update it when you upgraded the core code. When upgrading osTicket you should also update any installed/enabled plugins before running the upgrader.

Cheers.

[M1K489]

@JediKev I didn't update it since I didn't know it was doing something nor that it needed updating. I though it came with default osTicket. My boss must have plugged it in when he was first setting up the ticketing system. But came to work this morning and found out the the token valid and new tickets fetched. Case closed for our part. Thanks all for the help!

[JVC-MSF-OCB]

Hey all,

We had the token refreshing automatically for a couple of months, and a week ago it stopped. We did not have a notice of this stopping and thus did not know alerts weren't being sent for a few days...

We've manually refreshed the token and it now refreshes automatically every hour.

[dhimelick]

@JVC-MSF-OCB we saw this exact same issue on starting April 3. Like you, we failed to notice it for a few days because there aren't (as far as I know) any alerts that the token failed to refresh. Once I manually refreshed the token, it continued to automatically refresh without issue.

What email provider are you using? We use Office 365, so OAuth2 via Azure.

[dd2513]

@JVC-MSF-OCB @dhimelick I also saw this same issue, the token had been refreshing great for quite a while, but then stopped doing so. Again once manually refreshed everything seems to be working again.

We are using Office 365 OAuth 2 via Azure as well.

[JVC-MSF-OCB]

Hey @dhimelick

We use Office 365, so OAuth2 via Azure.

Same!

[dhimelick]

@JVC-MSF-OCB @dd2513 This same exact issue happened for my team again - 3 months to the day exactly from when we last manually refreshed the token. I confirmed that the OAuth token itself has not expired - it was set to expire after a year. That tracks given that a single manual refresh with the same token info is enough to fix the problem, if temporarily.

Whatever the issue is, it seems that 3 months is the timer on it, so you may want to at least set yourselves a reminder for that date. Not sure how to go about troubleshooting this.

[JediKev]

@dhimelick

Did you upgrade to v1.18, install the latest build of the OAuth2 plugin, get a new token, and retest?

Cheers.

[dhimelick]

I have not upgraded to 1.18 yet, but I'm hoping to ASAP. I'll plan to create a new token at that time as well, and I will update here if the autorefresh fails again.

Thanks!

[MikeIceGR]

• SOLVED -
  After I've tried everything I found on the entire internet to prevent token from expiring, and nothing helped... finally I decided to quit WAMP or IIS and try osTicket on XAMPP.

GUYS THIS IS IT!

1. Stop your WAMP or IIS server.
2. Install XAMPP 8.0.28 / PHP 8.0.28
3. Install osTicket Core, v1.17.4 on XAMPP or Copy your osTicket files to XAMPP virtual servers folder and Import database to XAMPP MySQL.
4. Change your Cron Job paths with the new paths from XAMPP PHP and cron.php.

THATS IT! NO PROBLEMS WITH TOKEN EXPIRATION ANY MORE!

Possible explanation..

• XAMPP comes with a SINGLE PHP and not with multiples as WAMP
• Some Security Headers on WAMP or IIS may prevent token renewal.

It really works. Quit WAMP or IIS and use XAMPP, and thank me later.. 🙂

[JVC-MSF-OCB]

Hey we're on 1.18 and the issue still happens!

Anyone else? @dhimelick @dd2513 ?

[JediKev]

@JVC-MSF-OCB

This issue has already been addressed. When you did an osTicket upgrade did you install the latest plugin as well? If not, that's most likely your issue. You also need to make sure you delete the existing Token and get a new one. It should also be noted that any account changes to the email will result in the tokens being invalidated; this is a MS limitation.

P.S.
I haven’t been able to replicate this since we fixed the issue.

Cheers.

[dd2513]

@JVC-MSF-OCB I haven't had the issue in a while now. I suppose I don't know if that means it's fixed or it just hasn't been long enough, but it has been longer than 3 months. So I think it's fixed for us sometime within the newest patches.

[JVC-MSF-OCB]

@JVC-MSF-OCB

This issue has already been addressed. When you did an osTicket upgrade did you install the latest plugin as well? If not, that's most likely your issue. You also need to make sure you delete the existing Token and get a new one. It should also be noted that any account changes to the email will result in the tokens being invalidated; this is a MS limitation.

P.S. I haven’t been able to replicate this since we fixed the issue.

Cheers.

Which plugin are referring to exactly? I'll have a look, but I think 1.18 came with a bunch of plugins already.

[JediKev]

@JVC-MSF-OCB

The latest OAuth2 Plugin build for v1.18. osTicket doesn’t come with any plugins pre-installed nor pre-downloaded. You would’ve had to manually select them on the Downloader from our website.

Cheers.

[JVC-MSF-OCB]

Ah ok thanks @JediKev
Is it possible that you did not update the plugin version? It is still V0.6, which is the same version as the one held in OST v1.17.x.

[JediKev]

@JVC-MSF-OCB

Yes, we don’t change the plugin version unless something major changes.

Cheers.

[dhimelick]

Hey we're on 1.18 and the issue still happens!

Anyone else? @dhimelick @dd2513 ?

Since upgrading to 1.18 and updating the OAuth plugin, we haven't seen the token refresh issue in our environment.

[Truepc]

how i do congfure it in exchnage server 2016

[JediKev]

@Truepc

I would assume your Exchange Server 2016 doesn't support OAuth2 authentication. If that's the case you need to use Basic Authentication (username + password). You can find your IMAP and SMTP connection information by following the steps in this guide.

Cheers.

[ChrisTG742]

It depends. If you run an Exchange Server without being part of Microsoft 365, you may still be able to use legacy auth for that account. If you run it as part of M365, 2FA may already be enforced so you will then need to create an OpenID application endpoint for osTicket in the Azure Portal. There is some short how-to about this in the docs of osTicket.