bigtable: add timeout to token refresh

[brandon-j-roberts]

Problem

Solana archival nodes have been disconnecting from big table. This causes headache for node providers, since we need to restart these nodes once they disconnect. This issue also causes 30-90 minutes of downtime for the node user, since we need to restart the node to regain access to bigtable. ( This can happen multiple times a day )

Summary of Changes

Add a timeout causing auth token for bigtable to refresh.

These changes were part of an older PR that was closed due to inactivity. I cherry picked the changes from the developed patch.

Testing

Our team has built the solana client and applied this change to our ~10 archival nodes. They haven't experienced any disconnects from bigtable in the ~1 week of testing we have undergone. These nodes have been able to access all the way to block 0 in bigtable for ~1 week without any issues. ( These would have disconnected multiple times already before the patch was applied )

Closes #20336

[brandon-j-roberts]

After about 2 weeks, we just have had 1 node disconnect from bigtable. This seems to be less common than before applying this change, but still not great.
We are getting a lot of timeouts on the token refresh it seems.
WARN solana_storage_bigtable::access_token] Token refresh timeout

[CriesofCarrots]

@brandon-j-roberts , thanks for the update. I've been trying to get further testing on this done on foundation RPC nodes to make sure out test cases are seeing significant loads. So far, it seems like the disconnects are still happening, although if we can determine they are happening less often, it could still be worth the merge.
Please keep me posted if/as you are able to get more data.

[CriesofCarrots]

Foundation nodes running with this patch have gone 10 days without a bigtable disconnect, so does seem like there's notable improvement. I will kick off CI

[CriesofCarrots]

@brandon-j-roberts , I'd like to see if a rebase fixes the docs CI failure. Could you give me push perms? Or rebase on master, please?

[brandon-j-roberts]

I just rebased, I believe. Also, I am doing some more investigation on why our node disconnected. I'll let you know if I find anything interesting. @CriesofCarrots

[CriesofCarrots]

Going to merge! Feeling Pollyanna-y today, so I'm going to have this close #20336
@brandon-j-roberts, let's start a new issue if you find out more about your node's disconnection, or still see it happening occasionally.