"could not find credentials for gcloud_client_secret" - OAuth2 token refresh error

[seamusabshere]

this has come up in a persistent way in:

• aug 2020
• oct 2020
• jan 2020

at any time during a copy to/from gcloud, dbcrossbar may fail to refresh its GCS OAuth2 token and die with a mysterious error:

Error: error copying data to `aws s3` 
  caused by: could not find credentials for gcloud_client_secret in any of: 
- The environment variable GCLOUD_CLIENT_SECRET 
- The file /root/.config/dbcrossbar/gcloud_client_secret.json 
   0: failure::backtrace::internal::InternalBacktrace::new 
   1: failure::backtrace::Backtrace::new 
   2: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll 
   3: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll 
   4: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll 
   5: tokio::runtime::task::core::Core<T,S>::poll 
   6: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once 
   7: tokio::runtime::task::harness::Harness<T,S>::poll 
   8: std::thread::local::LocalKey<T>::with 
   9: tokio::runtime::basic_scheduler::BasicScheduler<P>::block_on 
  10: tokio::runtime::context::enter 
  11: tokio::runtime::handle::Handle::enter 
  12: dbcrossbarlib::tokio_glue::run_futures_with_runtime 
  13: dbcrossbar::main 
  14: std::sys_common::backtrace::__rust_begin_short_backtrace 
  15: std::rt::lang_start::{{closure}} 
  16: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once 
             at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/core/src/ops/function.rs:259:13 
      std::panicking::try::do_call 
             at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/panicking.rs:373:40 
      std::panicking::try 
             at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/panicking.rs:337:19 
      std::panic::catch_unwind 
             at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/panic.rs:379:14 
      std::rt::lang_start_internal 
             at /rustc/18bf6b4f01a6feaf7259ba7cdae58031af1b7b39/library/std/src/rt.rs:51:25 
  17: main 

given that this appears to be transient and random, i believe we need to retry from oauth token refreshes