CancelRequest not implemented by *cli.retryingRoundTripper and slow performance on RHEL8 VM

[Shawn1874]

Helm community members believe that K3S is reporting the messages, but the observation has only been made running helm deployments.

Performance very slow on RHEL8 compared to Windows 10 with Rancher Desktop and prints out significant number of error messages.

Can be reproduced with these commands.

helm create nginx
helm install --debug --dry-run nginx nginx

E0613 08:34:20.554142 527651 round_tripper.go:63] CancelRequest not implemented by *cli.retryingRoundTripper
E0613 08:34:20.554177 527651 round_tripper.go:63] CancelRequest not implemented by *cli.retryingRoundTripper
E0613 08:34:20.554205 527651 round_tripper.go:63] CancelRequest not implemented by *cli.retryingRoundTripper

<E0613 message repeated at least 20 more times>

It takes helm over 2 minutes to install a basic nginx pod and service even when following a basic tutorial.

The same issue occurs when deleting the installation. Ultimately it succeeds in installing and deleting but the performance has significant impact as I continue to evaluate helm and learn to use it. When I previously evaluated helm using Rancher Desktop on a Windows 10 PC, I did not encounter these messages or performance problems.
Output of helm version:
version.BuildInfo{Version:"v3.12.0", GitCommit:"c9f554d75773799f72ceef38c51210f1842a1dea", GitTreeState:"clean", GoVersion:"go1.20.3"}

Output of kubectl version:
k3s -v
k3s version v1.26.4+k3s1 (8d0255a)
go version go1.19.8

kubectl version --output=yaml
clientVersion:
buildDate: "2023-04-20T00:33:18Z"
compiler: gc
gitCommit: 8d0255a
gitTreeState: clean
gitVersion: v1.26.4+k3s1
goVersion: go1.19.8
major: "1"
minor: "26"
platform: linux/amd64
kustomizeVersion: v4.5.7
serverVersion:
buildDate: "2023-04-20T00:33:18Z"
compiler: gc
gitCommit: 8d0255a
gitTreeState: clean
gitVersion: v1.26.4+k3s1
goVersion: go1.19.8
major: "1"
minor: "26"
platform: linux/amd64

Cloud Provider/Platform (AKS, GKE, Minikube etc.):
Using the Rancher K3S installed on a single RHEL8 Linux VM to setup single node cluster.
uname -a
Linux 4.18.0-425.13.1.el8_7.x86_64 helm/helm#1 SMP Thu Feb 2 13:01:45 EST 2023 x86_64 x86_64 x86_64 GNU/Linux
VM has 16GB of memory allocated and the usage never rises above 22% while the helm chart is being deployed.

References:
helm/helm#12143
kubernetes/kubernetes#104380

[brandond]

I can't reproduce this locally on EL8 or any other distro, even with very low amounts of system memory. How many CPU cores does this VM have? Are you perhaps using disk that is exceptionally slow? Can you attach the complete k3s log from journald on the host you're testing on?

[theolivenbaum]

I'm hitting a similar issue running helm on Windows 10, when trying to deploy an upgrade:

>helm upgrade --debug [... ommited ...]

Results in:

upgrade.go:144: [debug] preparing upgrade for XXXX

[... stuck for ~ a minute ...]

E0630 10:55:10.675400   24444 round_tripper.go:63] CancelRequest not implemented by *cli.retryingRoundTripper
E0630 10:55:10.675400   24444 round_tripper.go:63] CancelRequest not implemented by *cli.retryingRoundTripper

[... same error many times ...]

E0630 10:59:54.801680   24444 round_tripper.go:63] CancelRequest not implemented by *cli.retryingRoundTripper

I'm on helm version: version.BuildInfo{Version:"v3.11.1+6.el8", GitCommit:"66bfc44f827aea6eb8e001150914170ac0d49e2d", GitTreeState:"clean", GoVersion:"go1.18.9"}

[brandond]

@theolivenbaum what version of k3s are you using, and what resources (cpu/memory) are available to the control-plane nodes?

[Nassiel]

I don't believe it is related to K3s. I'm experiencing the same issue using Helm on Kubitect (full K8s version). It seems to be more about Helm than the Kubernetes implementation, IMO.

[rmayore]

Same. Running k3s in an EC2 t3.nano instance. Trying to install the argocd helm chart in a freshly created cluster. Happens consistently even after recreating the instance & cluster.

[jan-hudec]

I'm seeing these errors with … most clusters I access, to different extent, even with different versions of helm. But colleague does not see those errors with the same clusters and helm versions. That suggests it might be affected by local networking setup.

[kamikazechaser]

Experiencing the same on Rancher running RHEL. Might be Helm related.

history.go:56: [debug] getting history for release vm
Release "vm" does not exist. Installing it now.
install.go:218: [debug] Original chart version: "0.31.0"
install.go:235: [debug] CHART PATH: /home/kamikaze/.cache/helm/repository/victoria-metrics-operator-0.31.0.tgz

E0510 02:51:43.866397  105906 request.go:1116] Unexpected error when reading response body: net/http: request canceled (Client.Timeout or context cancellation while reading body)
E0510 02:51:43.866414  105906 round_tripper.go:63] CancelRequest not implemented by *cli.retryingRoundTripper
E0510 02:52:15.867295  105906 round_tripper.go:63] CancelRequest not implemented by *cli.retryingRoundTripper
E0510 02:52:15.867797  105906 request.go:1116] Unexpected error when reading response body: net/http: request canceled (Client.Timeout or context cancellation while reading body)
Error: unable to build kubernetes objects from release manifest: error validating "": error validating data: failed to download openapi: unexpected error when reading response body. Please retry. Original error: net/http: request canceled (Client.Timeout or context cancellation while reading body)
helm.go:84: [debug] error validating "": error validating data: failed to download openapi: unexpected error when reading response body. Please retry. Original error: net/http: request canceled (Client.Timeout or context cancellation while reading body)
helm.sh/helm/v3/pkg/kube.scrubValidationError
	helm.sh/helm/v3/pkg/kube/client.go:815
helm.sh/helm/v3/pkg/kube.(*Client).Build
	helm.sh/helm/v3/pkg/kube/client.go:358
helm.sh/helm/v3/pkg/action.(*Install).RunWithContext
	helm.sh/helm/v3/pkg/action/install.go:320
main.runInstall
	helm.sh/helm/v3/cmd/helm/install.go:310
main.newUpgradeCmd.func2
	helm.sh/helm/v3/cmd/helm/upgrade.go:150
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/cobra@v1.8.0/command.go:983
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/cobra@v1.8.0/command.go:1115
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/cobra@v1.8.0/command.go:1039
main.main
	helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
	runtime/proc.go:267
runtime.goexit
	runtime/asm_amd64.s:1650
unable to build kubernetes objects from release manifest
helm.sh/helm/v3/pkg/action.(*Install).RunWithContext
	helm.sh/helm/v3/pkg/action/install.go:322
main.runInstall
	helm.sh/helm/v3/cmd/helm/install.go:310
main.newUpgradeCmd.func2
	helm.sh/helm/v3/cmd/helm/upgrade.go:150
github.com/spf13/cobra.(*Command).execute
	github.com/spf13/cobra@v1.8.0/command.go:983
github.com/spf13/cobra.(*Command).ExecuteC
	github.com/spf13/cobra@v1.8.0/command.go:1115
github.com/spf13/cobra.(*Command).Execute
	github.com/spf13/cobra@v1.8.0/command.go:1039
main.main
	helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
	runtime/proc.go:267
runtime.goexit
	runtime/asm_amd64.s:1650