Refactor retry machinery to allow the use of exponential backoff

[moskyb]

🤔 Problem: The retry util we currently use has a couple of issues:

• There aren't any tests
• Retry configuration errors happen when you try to use the retrier, rather than when you configure it
• It only supports constant backoff - exponential backoff is really handy a lot of the time.

✅ Solution: This PR replaces the existing machinery (encapsulated in the retry.Do function and the retry.Config struct), with a struct called retry.Retrier. Retrier intentionally has a similar user interface to retry.Do, but it adds:

• Configurable backoff strategies - currently constant and exponential, but we could theoretically do some freaky stuff with it if we really wanted to
• Constructor panics when coding errors are made - eg, if a user puts a negative interval in to a constant strategy, that can never be valid and the program should complain loudly if it happens
• Tests! Hooray!

💬 Things to discuss:

• the math i use in the retry.Exponential function means that i can't have a base lower than 1 second, as i convert the time.Duration into an integer number of seconds, but if i don't do that, then the math gets all wack because time.Durations are integer numbers of nanoseconds - eg, math.Pow(2*time.Second, 2) != math.Pow(2, 2) * time.Second. is there a happy middle ground here? i'm embarrassed to say that my arithmetic skills are failing me here.
• How do we feel about the functional constructor? i feel like i was overdoing it by adding it, but... i kinda like it
• I've changed the artifact batch creation to use exponential backoff, but should we change other operations to use exponential? if so, which ones? This is no longer true, i'll change things to use expo backoff in another PR.

This PR will be most easily reviewed commit-by-commit

Closes PIP-234

[pda]

Ooh nice, I like the look of it.
(I haven't fully reviewed it yet.)

[tessereth]

First of all, YAY exponential backoff and jitter. I'm all for this.

I've only really checked out the retrier.go code in detail. I like it. But there's an annoying number of pointy bits in this kind of code. Have you explored pulling in a library for this rather than us rolling our own?

[moskyb]

@tessereth

Have you explored pulling in a library for this rather than us rolling our own?

yeah, i looked into this a bit - there weren't any libraries that i really loved, and none as nice as eg. python's tenacity or ruby's retryable. i suspect that it's the kind of thing that gets reimplemented privately a lot, because the libraries out there aren't awesome - QED, that's exactly what i did 😅

[waferbaby]

Typo: 'trying'

[waferbaby]

Looks great to me!

[sj26]

Love it