cache etcd client on readyz

[tjungblu]

Ran this with 5000 QPS in parallel with 12 threads using vegeta without issues. Also reconnects automatically when etcd goes down, which is great.

pprof also shows no TLS handshake hotpath anymore.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tjungblu
To complete the pull request process, please assign sttts after the PR has been reviewed.
You can assign the PR to them by writing /assign @sttts in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tjungblu]

TODO figure out the reconnect behaviour of GRPC: https://stackoverflow.com/questions/56067076/grpc-connection-management-in-golang

[tjungblu]

the health check also seems to reuse the client already:

cluster-etcd-operator/pkg/cmd/monitor/monitor.go

Lines 210 to 224 in 7373143

	// Create single target checks one check per target
	for _, target := range targets {
	// one client per target to eliminate lock racing
	client, err := newETCD3Client(ctx, tlsInfo, targets)
	if err != nil {
	return nil, err
	}
	// pin endpoint for check
	client.SetEndpoints(target)
	for _, healthCheckFunc := range singleTargetChecks {
	healthCheck := health.NewCheck(lg, client, []string{target})
	healthCheck.Probe = healthCheckFunc
	healthChecks = append(healthChecks, healthCheck)
	}
	}

[tjungblu]

/retest-required

[tjungblu]

/retest-required

[tjungblu]

increase in CPU usage from https://bugzilla.redhat.com/show_bug.cgi?id=2076831

[tjungblu]

/retest-required

[openshift-ci]

@tjungblu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-gcp-five-control-plane-replicas	9154786	link	false	/test e2e-gcp-five-control-plane-replicas
ci/prow/e2e-aws-single-node	9154786	link	true	/test e2e-aws-single-node

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[tjungblu]

/hold

probably worth refactoring with the changes from #801

[tjungblu]

closing this in favour of #801 where the caching is properly implemented