Unset DisableKeepalive for backfill HTTP client (#2137)

* Unset DisableKeepalive for backfill HTTP client

Disabling Keep-Alive, as done by the default transport setting in the
hashicorp cleanhttp package, seems to conflict with a network setting
between the public good Rekor instances and the bastion and results in
GET requests stalling or timing out after processing a few entries. This
change adds an option to the rekor client to unset the DisableKeepalive
setting and has the backfill script utilize that option. Other uses of
the rekor client will see no change.

Signed-off-by: Colleen Murphy <colleenmurphy@google.com>

* Increase rekor client retries in backfill script

Increase the RetryCount setting from the default of 3 up to 10 in order
to avoid giving up too quickly when the script hits the server rate
limit. The retryablehttp client does an exponential backoff, so
increasing the number of tries also increases the amount of time it will
wait in between each try before it eventually succeeds.

Signed-off-by: Colleen Murphy <colleenmurphy@google.com>

---------

Signed-off-by: Colleen Murphy <colleenmurphy@google.com>

cmd/backfill-index/main.go

@@ -161,7 +161,7 @@ func main() {
 		log.Fatalf("creating index client: %v", err)
 	}
 
-	rekorClient, err := client.GetRekorClient(*rekorAddress)
+	rekorClient, err := client.GetRekorClient(*rekorAddress, client.WithNoDisableKeepalive(true), client.WithRetryCount(10))
 	if err != nil {
 		log.Fatalf("creating rekor client: %v", err)
 	}

pkg/client/options.go

@@ -24,10 +24,11 @@ import (
 type Option func(*options)
 
 type options struct {
-	UserAgent   string
-	RetryCount  uint
-	InsecureTLS bool
-	Logger      interface{}
+	UserAgent           string
+	RetryCount          uint
+	InsecureTLS         bool
+	Logger              interface{}
+	NoDisableKeepalives bool
 }
 
 const (
@@ -78,6 +79,12 @@ func WithInsecureTLS(enabled bool) Option {
 	}
 }
 
+func WithNoDisableKeepalive(noDisableKeepalive bool) Option {
+	return func(o *options) {
+		o.NoDisableKeepalives = noDisableKeepalive
+	}
+}
+
 type roundTripper struct {
 	http.RoundTripper
 	UserAgent string

pkg/client/rekor_client.go

@@ -22,6 +22,7 @@ import (
 	"github.com/go-openapi/runtime"
 	httptransport "github.com/go-openapi/runtime/client"
 	"github.com/go-openapi/strfmt"
+
 	"github.com/hashicorp/go-cleanhttp"
 	retryablehttp "github.com/hashicorp/go-retryablehttp"
 	"github.com/sigstore/rekor/pkg/generated/client"
@@ -37,6 +38,9 @@ func GetRekorClient(rekorServerURL string, opts ...Option) (*client.Rekor, error
 
 	retryableClient := retryablehttp.NewClient()
 	defaultTransport := cleanhttp.DefaultTransport()
+	if o.NoDisableKeepalives {
+		defaultTransport.DisableKeepAlives = false
+	}
 	if o.InsecureTLS {
 		/* #nosec G402 */
 		defaultTransport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}