[YAML, or no] pdr bots shouldn't stop if RPC goes down briefly

[trentmc]

Background

Reported by @idiom-bytes in slack:

Hi, our web3 middleware [web3_config.py::Web3Config] that talks to the sapphire RPC is raising exceptions while trying to get blocks. The exception is raised after the # target_retries.

This is causing pdr-bots to throw an exception => causing bots to stop running => causing users to have to maintain/ops the bots. (See image below)

DoD

I think in situations where we have a known exception (subgraph/rpc that might self-heal in 1-2 mins) that we could wrap an outer-loop wait such that the bot doesn't crash, forcing botters to monitor/ops the bots.

I.e. "just set it, and forget it"

Image from Web3Config