Rework cooldown logic

[ssalinas]

We have found that Singularity's cooldown logic does not accurately capture cases like a slow failure loop, partially due to the fact that we purge out of the deploy statistics after only a minute or two. This reworks the cooldown logic to do a few things:

• save all failures in the deploy statistics. Each is just a long, so not too worried about data size here
• add a fast vs slow cooldown. In this case 3 failures in a minute or 5 failures in 10 minutes as a default

TODOs:

• fix up unit tests
• Figure out if it's still worth scaling this to instance count somehow

[pschoenfelder]

🚢

[baconmania]

🚢