Stop background refresh of cached data for requests that result in ACL not found errors

[mkeeler]

This is a partial fix to #9588 but is limited in scope. When the cache encounters an ACL not found error it will stop any background refresh which may be happening for that request. The assumption here is that the token was deleted and is not coming back.

There is however one case where a token may have yet to be replicated in a secondary DC that will prevent the background refresh of the request from happening when it may succeed. This might make the next request take a little longer but is preferable to spamming the logs for days with ACL not found errors.

Also of note is that cache watches for the request will cause the request to continually be made anyways and keep it in an always fetching state (or in exponential retry backoff sleeps).

[hashicorp-ci]

🤔 Double check that this PR does not require a changelog entry in the .changelog directory. Reference

[banks]

I think this makes sense. In the case where there is something blocking (or using Notify internally) you mention that we'll keep restarting the watch anyway. Does that produce the same level of log spam until the thing stops watching with an old ACL?

I think it's still better though because once the thing with the old ACL goes away, the logs will stop instead of continuing for another 3 days right?

[mkeeler]

In the case where the watch is still around it will be 1 log per minute (1m is the max time between errors that the watch will invoke getWithIndex).

Before when the fetch method was invoked for the background refresh it would do so with an attempts parameter and the exponential backoff (utilizing the same backOffWait function) was handled in the internal go routine right before reissuing the next fetch in a new go routine. So basically both cases will ensure that in the event of failures we do the exact same exponential backoff so the log spam would be identical if the watch was ongoing as it would be with no watch before this fix.

The assumption here is that the token was a short lived one for a workload. In k8s maybe that workload is already gone and the token that was generated for it has been deleted or expired. In that case, the service definition should have been deregistered and all injected Envoy proxies would be gone. Therefore no watches utilizing that token should be ongoing as they would all be cancelled when the service was deregistered.

The problem previously was that even after all watches had ended we were continuing to refresh these entries for 72h and if you churn through workloads quickly enough you would accumulate many erroring background refreshes causing potentially millions of log entries for these ACL errors.

[hashicorp-ci]

🍒 If backport labels were added before merging, cherry-picking will start automatically.

To retroactively trigger a backport after merging, add backport labels and re-run https://circleci.com/gh/hashicorp/consul/325541.

[hashicorp-ci]

🍒✅ Cherry pick of commit 8dbe342 onto release/1.9.x succeeded!

[hashicorp-ci]

🍒❌ Cherry pick of commit 8dbe342 onto release/1.8.x failed! Build Log

[hashicorp-ci]

🍒❌ Cherry pick of commit 8dbe342 onto release/1.7.x failed! Build Log