the k8s client hits some concurrent connections limit and it takes very long time to send all updates to the discovery manager

[krasi-georgiev]

There was an update in the golang/net/http2 package that causes the k8s to hit some connection limit and NOT takes very long time to send full targerts update to the discovery manager.

Might be related to #4518
Found while troubleshooting #4124

the http2 update in Prometheus happened in 0f37c02

The http2 commit that causes this behaviour golang/net@1c05540

To replicate:

minikube start
kubectl apply -f https://gist.githubusercontent.com/krasi-georgiev/337a0d5a5cf728bff87c102e825425af/raw/12e6df08752f82c53182e344ed3dbc879b886c68/sd-load.yaml

• add some debugging info in discovery/manager.go

                        m.updateGroup(poolKey, tgs)
			// m.recentlyUpdatedMtx.Lock()
			// m.recentlyUpdated = true
			// m.recentlyUpdatedMtx.Unlock()

			var total int
			for _, tt := range m.allGroups() {
				for _, tg := range tt {
					if tg.Source == "endpoints/default/sd-load" {
						total += len(tg.Targets)
					}
				}
			}
			fmt.Println(total)

• run Prometheus with the attached config(adjust the api_server IP and cert file paths for all jobs.).

go run cmd/prometheus/main.go --config.file=../../.local/sd-load/config.yaml

• Scale up and down the deployment and the total number of targets should be 10x current replicas which doesn't happen most of the time.

kubectl scale deployment sd-load --replicas=10
kubectl scale deployment sd-load --replicas=1

config.zip

@cofyc any pointers on where to look in our k8s implementation to increase or disable this limit?

[cofyc]

I'm looking into this.

[brancz]

@sttts is it possible to set the max number of concurrent streams through client-go?

[krasi-georgiev]

I have narrowed it down to the changes in the canTakeNewRequestLocked logic.
https://github.com/golang/net/blob/1c05540f6879653db88113bc4a2b70aec4bd491f/http2/transport.go#L621

my feeling is that the ClientConn reports that it can handle more connections, but when the caller actually calls the RoundTrip it fails somewhere.

[krasi-georgiev]

the limitation is from the server. It all works as expected. when I start minikube with

minikube start --extra-config=apiserver.http2-max-streams-per-connection=10000

So I guess this change in the http2 client package changes the workflow so that it hits different execution branch.

[simonpasquier]

FTR the root issue is the Go HTTP2 library and tracked at golang/go#27044. It might be addressed in go 1.12.

[simonpasquier]

Closing as the root issue golang/go#27044 has been closed by https://go-review.googlesource.com/c/net/+/151857/.