CancelRequest not implemented by *genericclioptions.CommandHeaderRoundTripper

[ash2k]

What happened:

kubectl get pods --namespace sample-app
E0813 20:59:41.056848      14 round_tripper.go:61] CancelRequest not implemented by *genericclioptions.CommandHeaderRoundTripper
E0813 20:59:41.057569      14 round_tripper.go:61] CancelRequest not implemented by *genericclioptions.CommandHeaderRoundTripper
E0813 20:59:41.057760      14 round_tripper.go:61] CancelRequest not implemented by *genericclioptions.CommandHeaderRoundTripper
E0813 20:59:41.058988      14 round_tripper.go:61] CancelRequest not implemented by *genericclioptions.CommandHeaderRoundTripper
NAME                        READY   STATUS    RESTARTS   AGE
helloweb-5c4957dcc4-9xls7   1/1     Running   0          67m

What you expected to happen:

No errors printed.

How to reproduce it (as minimally and precisely as possible):

Not sure. This is the same thing as #73791, was fixed by #71757.

Anything else we need to know?:

Happened in a CI build https://gitlab.com/jrandazzo-demo/gitlab-kubernetes-agent-integration/configuration-repository/-/jobs/1503211935.

Environment:

• Kubernetes version (use kubectl version): kubectl version 1.22.0 from bitnami/kubectl:latest image.
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release): linux
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

/sig cli

[seans3]

/triage accepted
/assign

[eddiezane]

We'll need to cherry pick this to 1.22.

[eddiezane]

@seans3 Looks like the cancellation was indeed from a timeout.

Duration: 1 minute 22 seconds
Timeout: 1h (from project)

@ash2k Sean will have a fix but do you have more insight into the timeout?

[ash2k]

@eddiezane The build I linked above is using a feature of GitLab Kubernetes Agent called CI/CD tunnel, which allows to access Kubernetes API from a CI job. An agent is proxying requests into the cluster. When I was investigating timeouts I discovered that they happen because of throttling in the agent. Throttling happened because kubectl was making way too many requests and the code was not caching the authentication results. After fixing this, I started to investigate why kubectl is making so many requests in the first place and discovered that the image that was used (bitnami/kubectl:latest) has a read-only file system and discovery client from client-go cannot persist the discovery information to disc and hence it performs discovery 3 (!) times for a simple get pods request. This results in 125 API calls. I tried to fix that in #103900 by caching the discovery client, but it doesn't help. The solution is to cache discovery info in memory because even if a single discovery client is used, it just discards the data and re-fetches it if it's not on disk. Please have a look at that PR.

[seans3]

The code that reported the error is

func (rt *cacheRoundTripper) CancelRequest(req *http.Request) {
    type canceler interface {
        CancelRequest(*http.Request)
    }
    if cr, ok := rt.rt.Transport.(canceler); ok {
        cr.CancelRequest(req)
    } else {
        klog.Errorf("CancelRequest not implemented by %T", rt.rt.Transport)
    }
}

The http.RoundTripper interface only has one function: RoundTrip(*Request) (*Response, error)

The CancelRequest as a function of a RoundTripper appears to be a convention in most (but not all) Kubernetes RoundTrippers.

[seans3]

For more context: the cacheRoundTripper is implemented in the CachedDiscoveryClient. This error message is printed when this cached discovery client encounters a timeout, and tries to cancel the request. The CommandHeaderRoundTripper is wrapped by the cacheRoundTripper, and it it unable to propagate the CancelRequest because the CommandHeaderRoundTripper does not implement CancelRequest.

[karezza]

I'm experiencing this today, seeing this error on my two master nodes.

$ k get all
E0918 22:31:22.023337 1905604 round_tripper.go:61] CancelRequest not implemented by *genericclioptions.CommandHeaderRoundTripper
E0918 22:31:22.023337 1905604 round_tripper.go:61] CancelRequest not implemented by *genericclioptions.CommandHeaderRoundTripper
E0918 22:31:22.023489 1905604 round_tripper.go:61] CancelRequest not implemented by *genericclioptions.CommandHeaderRoundTripper
E0918 22:31:22.023377 1905604 round_tripper.go:61] CancelRequest not implemented by *genericclioptions.CommandHeaderRoundTripper
E0918 22:31:22.023394 1905604 round_tripper.go:61] CancelRequest not implemented by *genericclioptions.CommandHeaderRoundTripper
E0918 22:31:22.024615 1905604 round_tripper.go:61] CancelRequest not implemented by *genericclioptions.CommandHeaderRoundTripper
E0918 22:31:22.024616 1905604 round_tripper.go:61]

The whole cluster is down. I was performing activities which were using the hard drive quite a bit. Not sure if that helps with debugging. Might be time to reboot the system hosting the node vms. But I think I'll experiment some more being that it is a dev cluster.

 k version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:45:37Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:39:34Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}

A reboot of each of the master nodes and things are coming back up. Worker nodes were fine.