Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
x/net/http2 ReadIdleTimeout/PingTimeout not exposed if configured through http package #40201
What version of Go are you using (
In prometheus/common, x/net/http2 is vendored so I could easily "hack" the field in there. At least we have somewhat good confidence it resolves our Prometheus issues now. I indeed missed golang/net#74 when searching for a solution. Since at least Prometheus seems to actually vendor the code, we might be fine over there as soon as your PR is merged (they pin some very recent master commit, anyway).
Unfortunately, the h2 transport is not exposed by
So basically, it's impossible to directly manipulate the h2 transport out of net/http package.
@JensErat is right. Currently no options is provided for upgrading h1 to h2. We'd better make it available ASAP. It really happens rarely but leads to disaster if you suddenly knock into it. See #39750
We can confirm this resolves our Prometheus and Kubernetes issues (and thus likely the issues of lots of others, some might not even be aware of this).
And I pretty much agree with him. This is something most applications will mess up. Lots of admin and developer days lost in debugging a hard to trace down issue. If I have long-lasting connections, those should be robust against network failures.
Could this be considered a breaking change or just a fix? Issues with broken servers that cannot handle whatever http2 calls are sent to run the ping?
A golang debug variable to set the value might also help in either case (force-enable or force-disable).
There are several kubernetes bugs [0,1,2] involving connection problems that seem related to the Go net/http2 library, where the stream state and connection state can get out of sync. This can manifest as a kubelet issue, where the node status gets stuck in a NotReady state, but can also happen elsewhere. In newer versions of the Go libraries some issues are fixed [3,4], but the fixes are not present in k8s 1.18. This change disables http2 in kube-apiserver and webhook-apiserver. This should be sufficient to avoid the majority of the issues, as disabling on one side of the connection is enough, and apiserver is generally either the client or the server. 0: kubernetes/kubernetes#87615 1: kubernetes/kubernetes#80313 2: kubernetes/client-go#374 3: golang/go#40423 4: golang/go#40201 Change-Id: Id693a7201acffccbc4b3db8f4e4b96290fd50288
Given that golang/go#39337 and golang/go#40201 are fixed: - We now have a function to configure HTTP/2 transport, therefore we have a "ping" every 5 minutes on the line if not traffic is doing on - Dead HTTP/2 connections will be removed from the pool Signed-off-by: Julien Pivotto <firstname.lastname@example.org>