Error forwarding ports: error upgrading connection #1455

Simon-Ince · 2016-10-25T09:49:02Z

I have Helm install locally and Tiller on my cluster, everything looks healthy, but running helm install stable/mysql is giving me:

Error: Error forwarding ports: error upgrading connection: dial tcp: lookup kube-4gb-lon1-02 on 8.8.8.8:53: no such host

The text was updated successfully, but these errors were encountered:

technosophos · 2016-10-25T17:00:20Z

Can you tell us more about your Kubernetes cluster (version, installation method) and which version of Helm you're running? Thanks.

Simon-Ince · 2016-10-26T08:55:28Z

@technosophos I created 5 Ubuntu servers on DigitalOcean and used Kubeadm:

kubeadm version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.0.1534+cf7301f16c0363-dirty", GitCommit:"cf7301f16c036363c4fdcb5d4d0c867720214598", GitTreeState:"dirty", BuildDate:"2016-09-27T18:10:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

to install on Ubuntu:

Distributor ID: Ubuntu
Description:    Ubuntu 16.04.1 LTS
Release:    16.04
Codename:   xenial

Then on my local machine using OSX:

System Version: OS X 10.10.5 (14F1909)
Kernel Version: Darwin 14.5.0

I installed Kubectl and connected to my cluster:

Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4+3b417cc", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"not a git tree", BuildDate:"2016-10-21T22:33:18Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"darwin/amd64"}

Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:10:32Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Then installed Helm using brew cask install helm But when I try helm install stable/mysql I get Error: Error forwarding ports: error upgrading connection: dial tcp: lookup kube-4gb-lon1-02 on 8.8.8.8:53: no such host

I think I've gone a fairly safe route, I used the latest stable release of Ubuntu and installed the recommended release of kubeadm, which installed the latest stable release of kubeclt and then connected to it from my local machine and used brew to install helm, so my set-up should be pretty typical. I'm able to use kubeclt on my local machine fine, I can use manifesting to get things set up and etc, the only thing I can't get working is Helm.

technosophos · 2016-10-27T16:10:57Z

Interesting that it is failing a DNS lookup. Is SkyDNS running in your Kube cluster?

kubectl get po -n kube-system
NAME                                READY     STATUS    RESTARTS   AGE
k8s-etcd-127.0.0.1                  1/1       Running   0          18h
k8s-master-127.0.0.1                4/4       Running   1          18h
k8s-proxy-127.0.0.1                 1/1       Running   0          18h
kube-addon-manager-127.0.0.1        2/2       Running   0          18h
kube-dns-v20-jqqts                  3/3       Running   0          18h
kubernetes-dashboard-v1.4.0-5ldke   1/1       Running   0          18h

adamreese · 2016-10-27T17:12:55Z

Can you start a port-forward using kubectl?

kubectl -n kube-system port-forward tiller-deploy-xxxxxx 44134

Simon-Ince · 2016-10-27T19:34:17Z

@technosophos Looks like DNS is there:

kube-system   etcd-kube-4gb-lon1-01                      1/1       Running   0          3d
kube-system   kube-apiserver-kube-4gb-lon1-01            1/1       Running   0          3d
kube-system   kube-controller-manager-kube-4gb-lon1-01   1/1       Running   0          3d
kube-system   kube-discovery-982812725-28b96             1/1       Running   0          3d
kube-system   kube-dns-2247936740-7lcli                  3/3       Running   0          3d
kube-system   kube-proxy-amd64-2j5gn                     1/1       Running   0          3d
kube-system   kube-proxy-amd64-9a0ob                     1/1       Running   0          3d
kube-system   kube-proxy-amd64-m6ypu                     1/1       Running   0          3d
kube-system   kube-proxy-amd64-n702h                     1/1       Running   0          3d
kube-system   kube-proxy-amd64-yjk4a                     1/1       Running   0          3d
kube-system   kube-scheduler-kube-4gb-lon1-01            1/1       Running   0          3d
kube-system   kubernetes-dashboard-1655269645-nyptv      1/1       Running   0          3d
kube-system   tiller-deploy-2434200834-8t7yt             1/1       Running   0          2d
kube-system   weave-net-09tbe                            2/2       Running   0          3d
kube-system   weave-net-f7qsx                            2/2       Running   0          3d
kube-system   weave-net-ihvuq                            2/2       Running   0          3d
kube-system   weave-net-phlva                            2/2       Running   0          3d
kube-system   weave-net-yx4wv                            2/2       Running   0          3d

Simon-Ince · 2016-10-27T19:41:34Z

@adamreese Looks like that gives back a similar error.
kubectl -n kube-system port-forward tiller-deploy-2434200834-8t7yt 44134
results in:
error: error upgrading connection: dial tcp: lookup kube-4gb-lon1-02 on 8.8.8.8:53: no such host
I've tried on both my local machine and while ssh'ed into the master node.

Simon-Ince · 2016-10-28T08:27:41Z

Just tried setting up a cluster again from scratch and still having the same issue.

When using kubeadm to set up the cluster, step "(3/4) Installing a pod network" (http://kubernetes.io/docs/getting-started-guides/kubeadm/) requiers you to pick the addon, I chose Weave Net, is Helm not compatible with this?

The options are:

Weave Net
Calico
Canal
Romana

Should I chose another one?

technosophos · 2016-10-28T16:59:20Z

I'm running Weave on a single-node Kubeadm-based install. I can open a tunnel with kubectl port-forward from the master node (which is the only node). I wonder if this has to do with networking across multiple nodes?

There is the possibility that asking about this in the #kubernetes-users slack channel or on StackOverflow might yield some answers, since this is actually a Kubernetes configuration thing, not a Helm-specific thing. I'll do some more hunting and update here if I find out anything.

mgoodness · 2016-10-28T17:52:52Z

@Simon-lush Based on the names of your apiserver, scheduler, and controller-manager pods, it seems like kubectl/helm should be trying to lookup kube-4gb-lon1-01. But your error messages show lookups for kube-4gb-lon1-02. Dueling entries in your kubeconfig?

Simon-Ince · 2016-10-31T16:57:58Z

I found help on the Slack group and now have Helm working.

@awh explained that

What you're experiencing is a known issue with k8s where for some operations it expects to be able to resolve your node names in the global DNS

And suggested a work around would be to:

Add entries to /etc/hosts on the master mapping your hostnames to their public IPs

Install dnsmasq on the master (e.g. apt install -y dnsmasq)

Kill the k8s api server container on master (kubelet will recreate it)

Then systemctl restart docker (or reboot the master) for it to pick up the /etc/resolv.conf changes

technosophos · 2016-10-31T17:26:17Z

I'm going to leave this open until I get the above put into the install FAQ. I suspect this is an issue that may crop up again.

Thanks for the help @mgoodness and the follow-up @Simon-lush .

The solution to helm#1455 is to configure dnsmasq on each of your nodes. This adds brief documentation on how to do so. Closes helm#1455

cmcconnell1 · 2016-12-01T21:14:18Z

Thanks @technosophos for your help in IRC today the helm and deis community are really helpful and responsive.
Just wanted to note in case this help others, ran into this K8s global DNS bug/issue today and did a quick hack of just adding my kube clusters IP's to my master/ectd:/etc/hosts file and was then able to proceed with installing and configuring deis workflow and helm was happy, etc (at least for now). . . ;-)

# without etcd node /etc/hosts hack entry for my cluster (its defaulting to second Nameserver in my /etc/resolv.conf where the first is our internal/forwarder and second is my ISP's
helm list
Error: Get https://deis-kube1.dev.foo.com/api/v1/namespaces/kube-system/pods?labelSelector=app%3Dhelm%2Cname%3Dtiller: dial tcp: lookup deis-kube1.dev.foo.com on 75.75.75.75:53: no such host

# and clearly records exist in our internal DNS
host deis-kube1.dev.foo.com
deis-kube1.dev.foo.com is an alias for deis-kube-elbapise-161cju3twc1uw-123456789.us-west-1.elb.amazonaws.com.
deis-kube-elbapise-161cju3twc1uw-123456789.us-west-1.elb.amazonaws.com has address 54.x.xxx.xxx
deis-kube-elbapise-161cju3twc1uw-123456789.us-west-1.elb.amazonaws.com has address 52.x.xxx.xxx

# after adding our above noted cluster IP's to kube cluster etcd nodes /etc/hosts
helm repo add deis https://charts.deis.com/workflow
"deis" has been added to your repositories

helm install deis/workflow --namespace deis
Fetched deis/workflow to workflow-v2.8.0.tgz
NAME: newbie-stingeray
LAST DEPLOYED: Thu Dec  1 12:44:01 2016
NAMESPACE: deis
STATUS: DEPLOYED

 helm list
NAME            	REVISION	UPDATED                 	STATUS  	CHART
newbie-stingeray	1       	Thu Dec  1 12:44:01 2016	DEPLOYED	workflow-v2.8.0

We may run into issues later without having completed many of the suggested work-around steps above, but for now we're moving forward. Hope this helps others.

evfurman · 2017-04-19T01:27:35Z

Seeing this error after installing helm/tiller on the tectonic stack. I can search helm and use kubectl just fine but fails upon installing new helm chart from default repo. Any idea when a fix will be merged?

[root@host ~]$ helm install stable/selenium Error: forwarding ports: error upgrading connection: error dialing backend: dial tcp: lookup ip-10-148-222-140 on 10.148.213.33:53: no such host

cmcconnell1 · 2017-04-19T16:41:20Z

'Just wanted to note (for me) that as of the previous release of helm v2.3.0 and with kubernetes clusters deployed with kube-aws v0.9.6-rc.2 I did not ever encounter this problem again. After getting this email, checked and noted that there is an updated helm version released 15 hours ago: helm latest release (I'm upgrading now. . .) Not sure about tectonics, I tried it a few months ago but it was not in a working state for us without any H/A capability, etc.. Curious what version of helm you're running?

cmcconnell1 · 2017-04-19T17:45:07Z

@evfurman, just a thought. . . (not sure about tectonics, but for others with kubernetes (kube-aws, etc.), one thing to note is that if your kube cluster is behind a load balancer (in AWS ELB, etc.), your processes for re-deploys may not manage any existing DNS records. So, if you had a previous cluster with an ELB with a CNAME for my-cluster.foo and you redeploy a new cluster with the same name, the DNS record will still point to the old ELB CNAME and you will see that error. In this case, take a look at the new clusters ELB and its FQDN and update any existing DNS CNAMEs to resolve.

shadycuz · 2017-11-26T19:16:33Z

@technosophos Any idea how to fix this when using helm inside of a pod in the k8?

I'm using gitlab-runners to build projects inside of pods. Kubectl works, its pointing to https://kubernetes.default but helm doesn't.

$ docker run --rm ${CONTAINER_TEST_IMAGE} helm ls
Error: Get http://localhost:8080/api/v1/namespaces/kube-system/pods?labelSelector=app%3Dhelm%2Cname%3Dtiller: dial tcp [::1]:8080: getsockopt: connection refused

$ docker run --rm ${CONTAINER_TEST_IMAGE} kubectl cluster-info
Kubernetes master is running at http://localhost:8080

axot · 2018-06-15T07:24:15Z

I got similar error today, any advice? Thank you.

$ helm ls
Error: forwarding ports: error upgrading connection: error dialing backend: ssh: rejected: connect failed (Connection timed out)

$ kubectl -n kube-system port-forward tiller-deploy-b774bc49f-6lthf 44134
error: error upgrading connection: error dialing backend: ssh: rejected: connect failed (Connection timed out)

axot · 2018-06-21T11:58:26Z

helm not work when the number nodes of cluster larger than 1,

Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Kubernetes: &version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.7-gke.3", GitCommit:"9b5b719c5f295c99de68ffb5b63101b0e0175376", GitTreeState:"clean", BuildDate:"2018-05-31T18:32:23Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}
Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}

tirumaraiselvan · 2018-06-21T14:00:11Z

Not sure if this is related. I am getting this error in AKS but not in GKE. I use helm from within the cluster to make releases.

Error: forwarding ports: error upgrading connection: Upgrade request required

Just to be safe (in case this is unrelated), I created a new issue: #4254

bacongobbler · 2018-06-21T16:24:21Z

@axot if you're having issues with kubectl port-forward, then that's gonna be a connection issue on Kubernetes' end, not on Helm's :) I'd file a ticket with GKE.

exu · 2018-09-27T10:04:03Z

Hi guys, I have the same issue on Minikube (on Archlinux - so most recent stable versions)

❯ helm version 
Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Error: forwarding ports: error upgrading connection: 
                                                                                                  

❯ k version 
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

it's occurs quite randomly. but most often after waking up my laptop.

bacongobbler · 2018-09-27T14:08:35Z

Try and see if you can mail it down to a port-forward issue like @axot did in #1455 (comment). It's most likely a connection issue between your laptop and the API server, so it's not anything we can take action on.

ezzoueidi · 2019-03-14T16:31:51Z

looks like this issue is still happening even when I followed all the steps in this PR (https://github.com/helm/helm/pull/1492/files).

commandodev · 2019-03-21T17:25:52Z

I've just hit this on a brand new set up of kubernetes on nixos (single machine running master and node). I've trying to follow the instruction here https://github.com/weaveworks/flux/blob/master/site/helm-get-started.md and get to

helm upgrade -i flux --set helmOperator.create=true --set helmOperator.createCRD=false --set git.url=git@github.com:boothead/flux-get-started --namespace flux weaveworks/flux
Error: forwarding ports: error upgrading connection: unable to upgrade connection: Forbidden (user=system:anonymous, verb=create, resource=nodes, subresource=proxy)

I tried this earlier today just running kind https://kind.sigs.k8s.io/ yesterday and it seems to all work fine.

bacongobbler · 2019-03-21T17:32:27Z

Is RBAC enabled? Check and see if you can kubectl port-forward to the tiller pod manually. Under the covers, helm is requesting a forwarded port from the kubernetes API server using your local Kube config. I'd start there and see if the same error comes up.

commandodev · 2019-03-21T18:05:51Z

Thank @bacongobbler - I just tried that and got the same error - I'm a bit new to k8s, could you point me at any decent resources to understand what's going on? For example I don't really know how to determine what's enabled or not, so I'm not sure if RBAC is enabled...

maximestevenot · 2019-04-26T21:21:53Z

Hi, from my side I solved the Error: forwarding ports: error upgrading connection: Upgrade request required issue with the following command:

$ export HELM_HOST=":44134"
$ tiller -listen ${HELM_HOST} -alsologtostderr >/dev/null 2>&1 &

Note: The export of the variable is useful too

Configuration:

Kubernetes Client Version: 1.14.1
Kubernetes Server Version: 1.10.3
Helm Client Version: 2.12.3
Helm Server Version: 2.12.3

prostil · 2019-07-05T11:43:25Z

helm does not work with the folowing K8S cluster

OS Ubuntu 18.04
Total of 2 VM's
1 running as a master and one as a worker
helm pod is deployed on the worker
Error: forwarding ports: error upgrading connection: unable to upgrade connection: pod does not exist

jarvisuser90 · 2019-08-01T13:28:40Z

helm does not work with the folowing K8S cluster

OS Ubuntu 18.04

Total of 2 VM's

1 running as a master and one as a worker

helm pod is deployed on the worker
Error: forwarding ports: error upgrading connection: unable to upgrade connection: pod does not exist

I'm getting the exact error too

karamsahu · 2019-08-22T07:42:57Z

@maximestevenot thanks for your solution. It worked for me. The error is gone now.

nidhi-ag · 2019-09-21T14:30:19Z

Hi, from my side I solved the Error: forwarding ports: error upgrading connection: Upgrade request required issue with the following command:
$ export HELM_HOST=":44134"
$ tiller -listen ${HELM_HOST} -alsologtostderr >/dev/null 2>&1 &
Note: The export of the variable is useful too

Configuration:

Kubernetes Client Version: 1.14.1

Kubernetes Server Version: 1.10.3

Helm Client Version: 2.12.3

Helm Server Version: 2.12.3

Hi, This worked for me. Could you please explain us your solution ?

caseyclarkjamf · 2019-10-03T17:57:45Z

@nidhi-ag It starts a local instance of tiller which connects to your remote Kubernetes cluster which works around the error related to failing to connect to the remote instance of tiller. https://helm.sh/docs/install/#running-tiller-locally

technosophos added the support label Oct 25, 2016

technosophos added a commit to technosophos/k8s-helm that referenced this issue Nov 1, 2016

docs(install_faq): document dnsmasq fix for kubeadm

1498461

The solution to helm#1455 is to configure dnsmasq on each of your nodes. This adds brief documentation on how to do so. Closes helm#1455

technosophos mentioned this issue Nov 1, 2016

docs(install_faq): document dnsmasq fix for kubeadm #1492

Merged

technosophos closed this as completed in #1492 Nov 2, 2016

adam-stokes mentioned this issue Dec 19, 2016

Consider adding conjure-up deis conjure-up/conjure-up#520

Closed

philchristensen mentioned this issue Jul 13, 2017

Helm DNS/Routing Error: forwarding ports: error upgrading connection coreos/tectonic-forum#164

Closed

veeresh1982 mentioned this issue Nov 13, 2017

Error: context deadline exceeded #2409

Closed

bacongobbler mentioned this issue Jul 3, 2019

“Error: forwarding ports: Upgrade request required” Error in helm of a kubernetes cluster #5965

Closed

ac427 mentioned this issue Aug 27, 2020

helm timeout ac427/libvirt-k8s#2

Open

Sicaine mentioned this issue May 12, 2021

rke2 requires hostnames to be resolvable per dns rancher/rke2#979

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error forwarding ports: error upgrading connection #1455

Error forwarding ports: error upgrading connection #1455

Simon-Ince commented Oct 25, 2016 •

edited

Loading

technosophos commented Oct 25, 2016

Simon-Ince commented Oct 26, 2016 •

edited

Loading

technosophos commented Oct 27, 2016

adamreese commented Oct 27, 2016

Simon-Ince commented Oct 27, 2016

Simon-Ince commented Oct 27, 2016

Simon-Ince commented Oct 28, 2016

technosophos commented Oct 28, 2016

mgoodness commented Oct 28, 2016

Simon-Ince commented Oct 31, 2016

technosophos commented Oct 31, 2016

cmcconnell1 commented Dec 1, 2016

evfurman commented Apr 19, 2017 •

edited

Loading

cmcconnell1 commented Apr 19, 2017

cmcconnell1 commented Apr 19, 2017

shadycuz commented Nov 26, 2017

axot commented Jun 15, 2018 •

edited

Loading

axot commented Jun 21, 2018

tirumaraiselvan commented Jun 21, 2018 •

edited

Loading

bacongobbler commented Jun 21, 2018 •

edited

Loading

exu commented Sep 27, 2018

bacongobbler commented Sep 27, 2018

ezzoueidi commented Mar 14, 2019

commandodev commented Mar 21, 2019

bacongobbler commented Mar 21, 2019

commandodev commented Mar 21, 2019 •

edited

Loading

maximestevenot commented Apr 26, 2019

prostil commented Jul 5, 2019

jarvisuser90 commented Aug 1, 2019

karamsahu commented Aug 22, 2019

nidhi-ag commented Sep 21, 2019 •

edited

Loading

caseyclarkjamf commented Oct 3, 2019

Error forwarding ports: error upgrading connection #1455

Error forwarding ports: error upgrading connection #1455

Comments

Simon-Ince commented Oct 25, 2016 • edited Loading

technosophos commented Oct 25, 2016

Simon-Ince commented Oct 26, 2016 • edited Loading

technosophos commented Oct 27, 2016

adamreese commented Oct 27, 2016

Simon-Ince commented Oct 27, 2016

Simon-Ince commented Oct 27, 2016

Simon-Ince commented Oct 28, 2016

technosophos commented Oct 28, 2016

mgoodness commented Oct 28, 2016

Simon-Ince commented Oct 31, 2016

technosophos commented Oct 31, 2016

cmcconnell1 commented Dec 1, 2016

evfurman commented Apr 19, 2017 • edited Loading

cmcconnell1 commented Apr 19, 2017

cmcconnell1 commented Apr 19, 2017

shadycuz commented Nov 26, 2017

axot commented Jun 15, 2018 • edited Loading

axot commented Jun 21, 2018

tirumaraiselvan commented Jun 21, 2018 • edited Loading

bacongobbler commented Jun 21, 2018 • edited Loading

exu commented Sep 27, 2018

bacongobbler commented Sep 27, 2018

ezzoueidi commented Mar 14, 2019

commandodev commented Mar 21, 2019

bacongobbler commented Mar 21, 2019

commandodev commented Mar 21, 2019 • edited Loading

maximestevenot commented Apr 26, 2019

prostil commented Jul 5, 2019

jarvisuser90 commented Aug 1, 2019

karamsahu commented Aug 22, 2019

nidhi-ag commented Sep 21, 2019 • edited Loading

caseyclarkjamf commented Oct 3, 2019

Simon-Ince commented Oct 25, 2016 •

edited

Loading

Simon-Ince commented Oct 26, 2016 •

edited

Loading

evfurman commented Apr 19, 2017 •

edited

Loading

axot commented Jun 15, 2018 •

edited

Loading

tirumaraiselvan commented Jun 21, 2018 •

edited

Loading

bacongobbler commented Jun 21, 2018 •

edited

Loading

commandodev commented Mar 21, 2019 •

edited

Loading

nidhi-ag commented Sep 21, 2019 •

edited

Loading