Add backoff-retry to uploads

[tjkirch]

Issue #, if available:

Helps work around #43.

Description of changes:

Add backoff-retry delays between block upload attempts

There's more potential for error when uploading a snapshot.  For example, we've
seen 4 "connection closed" errors in a row for a block upload.  The most
pernicious is the "snapshot does not exist" error, when there are timing issues
that cause new snapshots to not be immediately available for block upload,
which can take a couple minutes to pass.

The main issue in #43 is block upload failures due to "snapshot does not exist." The EBS team confirmed that the right approach for now is to wait and retry, and they recommended a 2 minute cap. This change adds increasing delays before retry attempts, and increases the retry total to just over 2 minutes, giving us enough buffer for the (occasional) new snapshot to become available for uploads. It should handle the vast majority of errors and cause a bit less stress.

Testing done:

A bunch of before/after testing is described in #43. In short, before, I was usually seeing failures (after 3 quick retries) within 50-100 uploads. After, I've run thousands of uploads successfully. In particular, I saw a "does not exist" case succeed after 7 retries (56 seconds).

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[tjkirch]

Made #57 to address the CI failure, which is just a new clippy warning.

[zmrow]

🧷

Nice approach!

[webern]

Nice job finding a solution.

[tjkirch]

^ Rebase on develop to get the clippy fix that tripped up CI.