Fog postgres retries ii (#1352) (backport to nascent release-1.1.3 branch)

[cbeck88]

• This adds configurable retries to postgres operations

Retries are added at the level of the SqlRecoveryDb object.

We also adjust some retry logic for fog ingest controller

• Redevelop previous per Eran's comment, to more systematically retry

This adds the retries at the level of the SqlRecoveryDb object
to systematically apply retries to all postgres operations.

• fixup missing "should_retry()" calls

• fix clippy

This repository is no longer used, please rebase your PR against https://github.com/mobilecoinfoundation/mobilecoin.git.

Thanks!

[wjuan-mob]

Quick question: Why do we use the retry crate for the db but not the controller, where we use this looping logic with increasing timeout?

[cbeck88]

@wjuan-mob the retry crate is not super well designed and usually requires wrapping it to use effectively. The main thing it offers is, the duration iterator thing, and the notion of retriable and non-retriable errors. When we are retrying in an infinite loop and all errors are retried, I think the code is simpler and easier to read if we just use loop and don't try to shoe horn it into retry interface. There were review comments about this and discussion in the master version of this PR

[wjuan-mob]

@wjuan-mob the retry crate is not super well designed and usually requires wrapping it to use effectively. The main thing it offers is, the duration iterator thing, and the notion of retriable and non-retriable errors. When we are retrying in an infinite loop and all errors are retried, I think the code is simpler and easier to read if we just use loop and don't try to shoe horn it into retry interface. There were review comments about this and discussion in the master version of this PR

Oh yeah. Sorry. I actually did remember reading that thread but I had a brain fart. Thank you.