App Check works at first but fails to reconnect after sleep/background

[anisabboud]

Summary

Using latest Firebase v9.12 (through latest AngularFire 7.4.1) in production, with AppCheck, Firestore, and Auth.

When leaving the webapp tab open and idle, the app often runs into a 403 error in POST request to exchange AppCheck token ("App attestation failed." "PERMISSION DENIED"), followed by a warning: @firebase/app-check: AppCheck: Requests throttled due to 403 error. Attempts allowed again after 01d:00m:00s (appCheck/throttled).

---

Console Tab

As you can see, the appCheck/throttled warning shows up twice, and is preceded by a failing 403 POST request to content-firebaseappcheck.googleapis.com/...exchangeRecaptchaV3Token

There was no exponential backoff. Quoting from @hsubox76 in #6373 (comment):

403 errors are throttled for 1 day because it is likely there's something wrong with the attestation that won't be fixed with a simple retry, such as a bad API key or an attestation failure due to failing ReCAPTCHA. You can see the code for the throttling here (this also applies to 404):

firebase-js-sdk/packages/app-check/src/providers.ts

Line 286 in 4eb5adf

* 403:

After the failed token exchange & AppCheck throttling, Firestore can no longer load data, thus logging several "FirebaseError: Missing or insufficient permissions." on all active document/collection listeners, rendering the app dysfunctional.

In a second tab running the same app, a similar crash was observed, but it was preceded by a Timeout error in recaptcha__en.js and a @firebase/app-check: FirebaseError: AppCheck: ReCAPTCHA error. (appCheck/recaptcha-error).

---

Network Tab

As you can see from the network tab, the 403 POST request failed with "App attestation failed." - when is this supposed to occur?

There were no further requests to firebaseappcheck.googleapis.com, but there was a successful OPTIONS request 200 at the same second Wed, 19 Oct 2022 10:17:35 GMT:

Right before the failing 403 request, during the same second 10:17:35 GMT, there was a POST request to Firestore with query parameter TYPE=terminate - could this be related? Sounds suspicious.

There were no further requests to firebaseappcheck.googleapis.com but there were successful token requests to securetoken.googleapis.com/v1/token every 55 minutes before & after the 403 issue:

09:07:52 GMT
10:02:53 GMT
10:57:54 GMT
11:52:55 GMT
12:47:56 GMT

---

Reproduction

The issue is tricky to reproduce, as it's related to token expiration. I often leave the tab open for several days before it happens.

Today I encountered this issue during the workday, while the computer was on all the time, in two different tabs that were open in the background. The same behavior mentioned above was observed in both tabs, 4 seconds apart:
Wed, 19 Oct 2022 10:17:35 GMT
Wed, 19 Oct 2022 10:17:39 GMT

These 4 seconds could be the difference from when I initially opened or interacted with each tab, hinting at a token expiration issue timeout, possibly initiated by Firestore terminate.

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[hsubox76]

Ok, this is a lot of information but it seems like the core of it is that you get 403 attestation errors while the tab is in the background or the computer is asleep. I don't think the Firestore "terminate" thing is related.

403 attestation errors indicate that the ReCAPTCHA token sent to the AppCheck endpoint is bad, which could be for various reasons: it wasn't able to get any ReCAPTCHA token, or it got one and you got a bad ReCAPTCHA score (it thinks you are a bot or something), or the project API key doesn't match. The backoff for this type of error is set to 1 day. This is because it was assumed a ReCAPTCHA failure was a more or less long term problem that wouldn't somehow fix itself in a few seconds, unlike other errors such as a 503 (server too busy).

It might be possible to revisit this backoff policy and change it in the specific case of it being a 403 AND it being an attestation error (there's other types of 403s), to use a shorter exponential backoff, in case the ReCAPTCHA check failed due to the tab being in the background? (I'm guessing that's what happened.) This will be somewhat difficult because you can't tell on the client side what the ReCAPTCHA score was (as intended, to avoid exploitation).

Before I do that I want to do some more research into what happens to ReCAPTCHA when the tab is backgrounded, whether it's simply unable to reach the endpoint or whether it's actually giving you a failed score.

[anisabboud]

A bit more info that might help: I left two tabs open yesterday. This time they got the 403 after ~23 & ~24 hours (while in the background but the laptop was not asleep).

I filtered the network tab for firebaseappcheck, and noticed that there were no "successful" POST requests to firebaseappcheck.
I.e., the 403 is the first and only request to firebaseappcheck. How often are POST requests typically sent to firebaseappcheck?

---

Update: Retrying the experiment with two fresh tabs the next day.
This time woke the laptop from sleep after more than 24 hours (token expires after 24 hours).
Both tabs sent two (not one) POST requests to https://content-firebaseappcheck.googleapis.com/v1/..., 10 seconds apart.

• In one tab, both requests returned 200 with a new token & ttl "86400s".
• In the second tab, the first request returned 200 and the second returned 403.
  (AppCheck did log the Requests throttled due to 403 error. warning since the second request failed.)

But since the first request succeeded on both tabs, this time both of them survived.

[hsubox76]

As mentioned above, there is a 1 day backoff when the response is a 403. I will try to look into some kind of solution for your situation that lets us maintain a long backoff for the cases it was intended for.

[anisabboud]

I'm still encountering this issue with Firebase 9.14.
For instance today I encountered it while the computer was on and active, but the crashing tab was briefly in the background.

I noticed that tokens are sent with a TTL of 3600 (1 hour), so the token is refreshed every 55 minutes (as can be seen from the AuthService timestamps prior to the crash below):

The token refresh every 55 minutes is usually accompanied/followed immedately by a Firestore POST request with "Type: terminate" (I'm guessing to refresh the Firestore connection with the new token).
Normally that flow works fine. Today for instance, the token was refreshed at 10:30, 11:25 and 12:20 (55 min apart), and each token refresh was followed by a terminate request to Firestore).

However, only 2 minutes later, at 12:22, the idle tab sent another terminate request to Firestore.
Since this request wasn't preceded by a refresh token, it might have triggered the 403 error.
This request was followed by two requests to content-firebaseappcheck.googleapis.com (which is different from the token requests every 55 minutes that are sent to securetoken.googleapis).

These are the only 3 network requests at the time of the crash:

I still don't understand what triggers this sequence of events...

[maxratajczak]

I'm getting the same issue but with out leaving the tab open, I just get the error right away when trying to interact with firebase services. I'm setting up a contact form for my app and am hoping to use app check to block potential bots from submitting the form. I've set up app check with my other firebase services.

But when trying to add a document to firestore, it gets blocked due to app check (which is supposed to happen) but with the app check 403 error (which I have no clue why is showing up).

[anisabboud]

Another similar case today where the 403 was preceded by 504s:

[taishi55]

same error here! anyone found a solution??

[hsubox76]

Just to be clear, a 403 error is not a bug in itself. It means that App Check failed attestation. That should happen when the app is not allowed to use App Check. That could mean reCAPTCHA failed or the app making the request is providing the wrong app ID, or there's a typo in your reCAPTCHA site key. This is what should happen in these cases. Further requests to the App Check backend are then throttled for 24 hours, and the reasoning behind this is that if any of these things are the case, it won't be fixed by retrying (whereas a 50x error might be), so it's throttled to not overwhelm your quota.

If you're getting this when you expect access to be blocked, that's normal. If you're testing things out in development you can clear your local indexedDB cache to get it to retry again without waiting 24h. If you're expecting access not to be blocked, make sure your reCAPTCHA account is set up properly and registered in the Firebase console.

If you feel like a 24h throttle is not appropriate in some cases (which seems like it might be the case), let us know the details, we are looking into what we can do to make it more granular somehow.

[y-takebe]

Hi, everyone.

I had the same problem, but after carefully checking it out based on @hsubox76 advice, I was able to solve it. ✅
I'll share the information for your reference.

• I using ReCaptcha Enterprise
• using firebase 9.15.0

First (and I don't know if this is truly relevant), I decided to generate a non-test Enterprise Key and use that.
(Previously, I used a key with the Test Key flag enabled).

The code is as follows:

import { initializeApp } from 'firebase/app'
import { initializeAppCheck, ReCaptchaEnterpriseProvider } from 'firebase/app-check'

const firebaseConfig = {
  // hide
}

const app = initializeApp(firebaseConfig)
initializeAppCheck(app, {
  provider: new ReCaptchaEnterpriseProvider(import.meta.env.VITE_APP_RECAPTCHA_SITE_KEY),
  isTokenAutoRefreshEnabled: true,
})

Next, I reconfigured the reCAPTCHA Enterprise > reCAPTCHA Enterprise Site Key for the target application as listed in Firebase > App Check > Apps.

Finally, just to be safe, I disabled Firebase > App Check > Cloud Firestore once and re-enabled it.

That's all there is to it. Incidentally, there was no need to wait 24 hours after this.

Best wishes for success.

[kanehjeong]

I also started getting this error today, even though it's been working fine for the past year. Did not change any configuration.

[Megamisama]

I got the solution. The problem it's not you. It's the system. AppCheck to work without that error needs that the computer IP Address is the same when you register to AppCheck. My Router changed address my PC and I got a lot of errors that's the only one that remains until Changed the IP Address Back. I hope that helps. 😊