Increase archiver robustness to transient network issues

[zaneselvans]

Looking at some of the archiver logging output, the exponential backoff on failure / timeout seemed like was going up to about a 1 minute wait time. I've sometimes experienced network connectivity issues with the federal agency sites that last for longer than that. Right now it seems like about 1 in 20 archiver runs fails due to transient network issues like connection timeouts or closed connections. But another run done 15 minutes or an hour later has no trouble.

It might make sense to back off to as much as 15 or 30 minutes when we're doing the scheduled periodic archiving runs. They'll probably run overnight and could even take a couple of hours to complete without causing any problems.

If the maximum wait time or the multiplier between steps were a parameter, we could still have the CI go quickly, while the scheduled runs can be more robust and leisurely.

Scope

Beta Give feedback

1. configure default backoff limit
2. configure dataset backoff limits

Next steps

Beta Give feedback

1. add retry-count CLI arg, and pass that through to utils.retry_async