FFM-11022 Flag/Group cache recovery

[erdirowlands]

What

This adds cache recovery mechanism for flags and groups in the following ways (note the recovery can't be triggered more than once per minute to avoid overloading our API):

Groups

Since a group cache lookup is determined by the flag, and if it's missing from the cache then we know the cache state is invalid, we now trigger one group cache refresh with timeout and a warning log, and attempt to evaluate one final time. If the same error occurs and we can't find the group even after a refresh, we return the default variation and log an error.

Flags

Not finding a flag in the cache doesn't necessarily mean invalid state; the user may have supplied a flag ID that doesn't exist in their project. However, if the flag can't be found we attempt one refresh of the flag cache with timeout + warning log. if it is now found, we continue with the evaluation. If it can't be found, we return the default variation.

Includes new config option SetCacheRecoveryTimeout(int timeoutMilliseconds) which defaults to 5 seconds.

Testing

• Ran through TestGrid evaluation suite
• Flags: Manual - prod 2 - supplied wrong flag name and it attempted once before returning default.
• Groups: Manual - prod 2, using a contrived example where I hardcode the group name to be something that won't exist. It attempts the refresh and returns the default variation if still can't be found.

[davejohnston]

Looks good.