[Feature]: Retry on HTTP2 GOAWAY frames

[wbh1]

Description

Occasionally, we'll see errors like this when using Linode-based service discovery in Prometheus:

ts=2022-12-01T15:21:15.357Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=linode msg="Unable to refresh target groups" err="[002] http2: server sent GOAWAY and closed the connection; LastStreamID=1999, ErrCode=NO_ERR
Dec 01 10:21:15 prometheus2 prometheus[2268065]: 2022/12/01 10:21:15.357289 ERROR RESTY http2: server sent GOAWAY and closed the connection; LastStreamID=1999, ErrCode=NO_ERROR, debug="", Attempt 1

What happens is that the connection to api.linode.com gets reset because we hit the maximum number of requests for a persistent connection (e.g. nginx defaults to 1000) and the connection is closed by the server after it sends a GOAWAY. We should be able to safely retry these by opening a new connection.

There's likely more nuance to it, though. As I believe you need to validate the accompanying error with the GOAWAY frame and check on the last stream ID rather than just naively retrying every time.

Some helpful resources I've found are:

• x/net/http2: make Transport return nicer error when Amazon ALB hangs up mid-response? golang/go#18639
• Pagerduty's retry logic
• Azure AKS retry logic
• net/http2 GoAwayError type
• Stackoverflow, of course

Example Code

// retries.go
func requestGOAWAYRetryCondition(_ *resty.Response, e error) bool {
	return errors.As(e, &http2.GoAwayError{})
}