site: keep expired cache when background refresh failed

[unknwon]

We have been pretty picky about freshness of the cached content (5m) and would aggressively delete the cache even if the background refresh failed.

It has the big assumption that the content host (GitHub.com) would never go down, and creates deep coupling of our docs site liveness directly with the liveness of the GitHub.com.

This PR removes that in the spirit of favoring stableness over freshness. It is better to have content got stale, than having the entire site down. Especially in the former case, the content would not be possible to be refreshed if the content host is also down in the first place.

[rafax]

LGTM, do we expose the "freshness" of the cache in some way? Would it be possible to alert on cache getting stale - before the PR, staleness would be visible in docsite status?

[rafax]

This PR looks good to me, and should address large part of the Github induced instability - one other thing we could consider is sharing the cache between replicas (so new replicas can come up during upstream outages), but I defer to you on how hard that would be.

[unknwon]

do we expose the "freshness" of the cache in some way? Would it be possible to alert on cache getting stale - before the PR, staleness would be visible in docsite status?

We can add an endpoint to expose, the staleness wasn't visible in typical sense, but by responding server error to all requests (aka. down time) 😅