add some leeway for checking JWT expiry, to account for clock skew

[BradleyHow]

Description :

I have an error cause by a clock skew of one second between my hasura server and my JWT service (Firebase) . I think that it beneficial to allow a little bit of leeway in the validation of the iat of the token to let this kind of situation.

Error message :

{name: "FormatedError", message: "Unknown error", originalError: "cannot start as connection_init failed with : Could not verify JWT: JWTIssuedAtFuture"}

Solution :

Implementers MAY provide for some small leeway, usually no more than
a few minutes, to account for clock skew. Use of this claim should be OPTIONAL.

[ecthiender]

Hi @BradleyHow

True, the spec does say that implementers can allow leeway to account for clock skew, but we haven't implemented this yet. I'm marking this as a feature, we will prioritize this based on interest from the community.

Said that, the above error happens when the machine that is running graphql-engine does not have its time synced. To fix your issue, you can make sure the machine where you are running graphql-engine have NTP installed and the time is synced.

[ecthiender]

Ref: https://tools.ietf.org/html/rfc7519#section-4.1.4

[yuvalgut]

Im also getting that issued at future error. Hope you will add that leeway :)

[ecthiender]

@yuvalgut have you tried checking NTP on your system? Syncing time on your system should fix the issue :)

[yuvalgut]

@ecthiender , just noticed it only happens on the local env. Thanks.

[evert-smit]

Ran into this in local dev env too, a reboot fixed the problem.

[benwhitman]

+1 for leeway. I've got this currently in a multi-AZ kubernetes environment. Hasura in one AZ, Keycloak in another with 20s difference. Yes i know i should fix that discrepancy but leeway would help.

[jordiup]

For anyone else: I fixed this problem by re-enabling automatic date time on my laptop.

[frederikprijck]

I added an issue/feature request with hs-jose as Hasura can't add a leeway if it's not available (and configurable) in hs-jose. frasertweedale/hs-jose#96

[frasertweedale]

hs-jose already lets you configure an allowed skew: https://github.com/frasertweedale/hs-jose/blob/master/src/Crypto/JWT.hs#L362-L378

[frederikprijck]

@frasertweedale Thanks for pointing that out, I missed that.

[georgiosd]

Any updates on implementing this? We're getting tokens from AWS Amplify and validating them with hasura and this error comes up every so often - we obviously can't control AWS' NTP config.

[cstlaurent]

This issue is happening to all our developers using windows machines with Hasura in Docker for Windows running in WSL2... Machines and WSL are properly synchronized with NTP and happens with JWT token issued from Firebase and also Auth0.

To bypass this issue we have to manually set WSL clock 1 hour in the future on every boot, which is cumbersome.

No trouble on Docker for Mac...

[ntziolis]

We have had to result to delaying the client to making a request... Would love to see this addressed

[2001aspaceodyssey]

This is a presenting a problem for the infrastructure my team is using as well. The containers and VMs running have a variance in the timing by ~1s. The auth server is currently ahead of the Hasura instance so when Hasura consumes the auth token it immediately rejects it. This affects the initial page load depending on how fast the client can make the round trip.

I'm hoping this can be increased in priority because all of the solutions to address it are undesirable:

• Delaying the load time of our application
• Manually updating the clock on the host

The first option is completely out of the question as we optimize the load times of our applications. The second is cumbersome, error-prone, and difficult to automate.

[anshubansal2000]

We are also having the same issue. Anyone was able to solve this?

[fabiankaestner]

This issue is happening to all our developers using windows machines with Hasura in Docker for Windows running in WSL2... Machines and WSL are properly synchronized with NTP and happens with JWT token issued from Firebase and also Auth0.

I'll just add that this issue is occuring for me as well with Hasura running in Docker on Windows.

[anshubansal2000]

We were able to resolve this by changing the host's time however just for one particular machine it was not working, my guess is that Network/Firewall/Solarwind was overwriting the time. We tested on so many machine and it worked expect in a particular network.

[2001aspaceodyssey]

This is still an issue for us.

[KATT]

@tirumaraiselvan your @hasura-bot isn't playing ball

This is happening in production for me - using Auth0 + hosting Hasura on render.com

[tirumaraiselvan]

Hey folks

This commit: c14bcb6 adds a new config allowed_skew in the JWT config to provide for some leeway while comparing the JWT expiry time.

This is planned to be released in v1.3.4 and you can configure clock skew according to your requirements.

NOTE: When you face this error, please do verify that the timestamp in the token and time in graphql-engine (according to logs) is suffering from clock skew and not some other issue.

[KATT]

Great, thanks for the quick response. ETA for 1.3.4 release? :)

[erikBrightly]

Do we have a production release ETA?

[tirumaraiselvan]

Do we have a production release ETA?

This is available in v2.0 which will be stable in 2 weeks time.

[osseonews]

This issue is destroying our development for all the developers on windows. Anyone know how to fix this?

[erikBrightly]

This issue is destroying our development for all the developers on windows. Anyone know how to fix this?
Which version are you running?

[osseonews]

version of what? windows or hasura?

[osseonews]

For Hasura we are on: v1.3.3

We haven't developed in 3 days. Couldn't figure out what was going on, until we finally console.logged everything and got the Could not verify JWT: JWTIssuedAtFuture error from Hasura. Then we found this thread.

[erikBrightly]

For Hasura we are on: v1.3.3

We haven't developed in 3 days. Couldn't figure out what was going on, until we finally console.logged everything and got the Could not verify JWT: JWTIssuedAtFuture error from Hasura. Then we found this thread.

We've had to mess with time settings locally and make sure the time for the Auth server matches with the Hasura computer running. Are you running in Docker?

Alternatively try to upgrade to v2 alpha or v1.3.4 Beta as well.

Basically no ideal solution that is production ready.

[osseonews]

Thanks. Yes, Docker. It's only a problem on Windows machines for us. I guess we will upgrade Hasura to 1.3.4. Can't get any work done at all. Amazingly, yesterday it started working in late afternoon for some reason. It's kind of random when the time is off.

[namoscato]

Per @tirumaraiselvan's comment, this change is shipping with v2.0.

A simple solution for development in the meantime: Just set your computer clock slightly ahead (~minute) of actual time.

[osseonews]

Thanks for that tip about the clock! Will try it now.

[TheMrZZ]

Just tried using Hasura 2.0 alpha v9, still get this error!

[kiily]

What are the cons of using allowed_skewed. We seem to need to add over 2h for our scenario to work, was just wondering what possible issues that could cause.

We are observing this in both Windows and Mac M1

[mgan59]

Have this error with a local-dockerized Hasura on Mac with authentication JWT coming from an AWS Cognito instance. Cognito issues my JWT and when used with Hasura I get the following error -- {"errors":[{"extensions":{"path":"$","code":"invalid-jwt"},"message":"Could not verify JWT: JWTIssuedAtFuture"}]}.

---

UPDATE: For those using Cognito, I adjusted the allowed_skew on my HASURA_GRAPHQL_JWT_SECRET to 86400 and that did the trick finally.

[JuHwon]

running local hasura container, with local react app in dev mode. why is this skew happening?

[RobSchilderr]

Have this error with a local-dockerized Hasura on Mac with authentication JWT coming from an AWS Cognito instance. Cognito issues my JWT and when used with Hasura I get the following error -- {"errors":[{"extensions":{"path":"$","code":"invalid-jwt"},"message":"Could not verify JWT: JWTIssuedAtFuture"}]}.

UPDATE: For those using Cognito, I adjusted the allowed_skew on my HASURA_GRAPHQL_JWT_SECRET to 86400 and that did the trick finally.

This did the trick. Thanks a lot.

[bensoutendijk]

I am still getting this error using Windows 11 + WSL2 + Docker.

Hey folks

This commit: c14bcb6 adds a new config allowed_skew in the JWT config to provide for some leeway while comparing the JWT expiry time.

This is planned to be released in v1.3.4 and you can configure clock skew according to your requirements.

NOTE: When you face this error, please do verify that the timestamp in the token and time in graphql-engine (according to logs) is suffering from clock skew and not some other issue.

How does one use this allowed_skew config option?