Add retries on server errors

[tra0x]

Logical flow:

1. retry_on_connection_errors in provider base class now checks for RecordStore::Provider::RetriableError
2. We leave calling RetriableError up to provider implementations
3. Currently only used by Cloudflare provider
4. Request block checks for a 5xx response with response.is_a?(Net::HTTPServerError) and catches Net::HTTPFatalError and raises RetriableError; also checks for 3xx response and catches Net::HTTPRetriableError

Tests moved to focus on Cloudflare client's use of retries

1. Test success when <= max_retries
2. Test fails when > max_retries
3. Test 5xx response raises RetriableError

[sbfaulkner]

instead of using the if match, can we raise/rescue a more specific error?

[sbfaulkner]

I'm a bit surprised this retry is at the generic provider level instead of Cloudflare-specific

do other providers have their retry logic built in?

[pascal-de-ladurantaye]

I suggest creating a new error type like RetryableError and raising that from the provider itself with logic in the provider to switch between the current Error and new RetryableError based on the failure scenario

[pascal-de-ladurantaye]

Looked up the spec, it seems like only 3XX errors fit this

https://github.com/ruby/ruby/blob/master/spec/ruby/library/net-http/httpresponse/error_spec.rb#L15-L16

[tra0x]

Good catch, thank you. I've updated the PR body explaining the fix.

[pascal-de-ladurantaye]

Why are we only raising for 3XX and 5XX?

[tra0x]

fixed to use response.value

# Raises an HTTP error if the response is not 2xx (success).
def value
  error! unless self.kind_of?(Net::HTTPSuccess)
end