Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error forwarding ports: error upgrading connection #1455

Closed
Simon-Ince opened this issue Oct 25, 2016 · 32 comments
Closed

Error forwarding ports: error upgrading connection #1455

Simon-Ince opened this issue Oct 25, 2016 · 32 comments

Comments

@Simon-Ince
Copy link

@Simon-Ince Simon-Ince commented Oct 25, 2016

I have Helm install locally and Tiller on my cluster, everything looks healthy, but running helm install stable/mysql is giving me:

Error: Error forwarding ports: error upgrading connection: dial tcp: lookup kube-4gb-lon1-02 on 8.8.8.8:53: no such host
@technosophos
Copy link
Member

@technosophos technosophos commented Oct 25, 2016

Can you tell us more about your Kubernetes cluster (version, installation method) and which version of Helm you're running? Thanks.

@Simon-Ince
Copy link
Author

@Simon-Ince Simon-Ince commented Oct 26, 2016

@technosophos I created 5 Ubuntu servers on DigitalOcean and used Kubeadm:

kubeadm version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.0.1534+cf7301f16c0363-dirty", GitCommit:"cf7301f16c036363c4fdcb5d4d0c867720214598", GitTreeState:"dirty", BuildDate:"2016-09-27T18:10:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

to install on Ubuntu:

Distributor ID: Ubuntu
Description:    Ubuntu 16.04.1 LTS
Release:    16.04
Codename:   xenial

Then on my local machine using OSX:

System Version: OS X 10.10.5 (14F1909)
Kernel Version: Darwin 14.5.0

I installed Kubectl and connected to my cluster:

Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4+3b417cc", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"not a git tree", BuildDate:"2016-10-21T22:33:18Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"darwin/amd64"}

Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:10:32Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Then installed Helm using brew cask install helm But when I try helm install stable/mysql I get Error: Error forwarding ports: error upgrading connection: dial tcp: lookup kube-4gb-lon1-02 on 8.8.8.8:53: no such host

I think I've gone a fairly safe route, I used the latest stable release of Ubuntu and installed the recommended release of kubeadm, which installed the latest stable release of kubeclt and then connected to it from my local machine and used brew to install helm, so my set-up should be pretty typical. I'm able to use kubeclt on my local machine fine, I can use manifesting to get things set up and etc, the only thing I can't get working is Helm.

@technosophos
Copy link
Member

@technosophos technosophos commented Oct 27, 2016

Interesting that it is failing a DNS lookup. Is SkyDNS running in your Kube cluster?

kubectl get po -n kube-system
NAME                                READY     STATUS    RESTARTS   AGE
k8s-etcd-127.0.0.1                  1/1       Running   0          18h
k8s-master-127.0.0.1                4/4       Running   1          18h
k8s-proxy-127.0.0.1                 1/1       Running   0          18h
kube-addon-manager-127.0.0.1        2/2       Running   0          18h
kube-dns-v20-jqqts                  3/3       Running   0          18h
kubernetes-dashboard-v1.4.0-5ldke   1/1       Running   0          18h
@adamreese
Copy link
Member

@adamreese adamreese commented Oct 27, 2016

Can you start a port-forward using kubectl?

kubectl -n kube-system port-forward tiller-deploy-xxxxxx 44134

@Simon-Ince
Copy link
Author

@Simon-Ince Simon-Ince commented Oct 27, 2016

@technosophos Looks like DNS is there:

kube-system   etcd-kube-4gb-lon1-01                      1/1       Running   0          3d
kube-system   kube-apiserver-kube-4gb-lon1-01            1/1       Running   0          3d
kube-system   kube-controller-manager-kube-4gb-lon1-01   1/1       Running   0          3d
kube-system   kube-discovery-982812725-28b96             1/1       Running   0          3d
kube-system   kube-dns-2247936740-7lcli                  3/3       Running   0          3d
kube-system   kube-proxy-amd64-2j5gn                     1/1       Running   0          3d
kube-system   kube-proxy-amd64-9a0ob                     1/1       Running   0          3d
kube-system   kube-proxy-amd64-m6ypu                     1/1       Running   0          3d
kube-system   kube-proxy-amd64-n702h                     1/1       Running   0          3d
kube-system   kube-proxy-amd64-yjk4a                     1/1       Running   0          3d
kube-system   kube-scheduler-kube-4gb-lon1-01            1/1       Running   0          3d
kube-system   kubernetes-dashboard-1655269645-nyptv      1/1       Running   0          3d
kube-system   tiller-deploy-2434200834-8t7yt             1/1       Running   0          2d
kube-system   weave-net-09tbe                            2/2       Running   0          3d
kube-system   weave-net-f7qsx                            2/2       Running   0          3d
kube-system   weave-net-ihvuq                            2/2       Running   0          3d
kube-system   weave-net-phlva                            2/2       Running   0          3d
kube-system   weave-net-yx4wv                            2/2       Running   0          3d
@Simon-Ince
Copy link
Author

@Simon-Ince Simon-Ince commented Oct 27, 2016

@adamreese Looks like that gives back a similar error.
kubectl -n kube-system port-forward tiller-deploy-2434200834-8t7yt 44134
results in:
error: error upgrading connection: dial tcp: lookup kube-4gb-lon1-02 on 8.8.8.8:53: no such host
I've tried on both my local machine and while ssh'ed into the master node.

@Simon-Ince
Copy link
Author

@Simon-Ince Simon-Ince commented Oct 28, 2016

Just tried setting up a cluster again from scratch and still having the same issue.

When using kubeadm to set up the cluster, step "(3/4) Installing a pod network" (http://kubernetes.io/docs/getting-started-guides/kubeadm/) requiers you to pick the addon, I chose Weave Net, is Helm not compatible with this?

The options are:

  • Weave Net
  • Calico
  • Canal
  • Romana

Should I chose another one?

@technosophos
Copy link
Member

@technosophos technosophos commented Oct 28, 2016

I'm running Weave on a single-node Kubeadm-based install. I can open a tunnel with kubectl port-forward from the master node (which is the only node). I wonder if this has to do with networking across multiple nodes?

There is the possibility that asking about this in the #kubernetes-users slack channel or on StackOverflow might yield some answers, since this is actually a Kubernetes configuration thing, not a Helm-specific thing. I'll do some more hunting and update here if I find out anything.

@mgoodness
Copy link

@mgoodness mgoodness commented Oct 28, 2016

@Simon-lush Based on the names of your apiserver, scheduler, and controller-manager pods, it seems like kubectl/helm should be trying to lookup kube-4gb-lon1-01. But your error messages show lookups for kube-4gb-lon1-02. Dueling entries in your kubeconfig?

@Simon-Ince
Copy link
Author

@Simon-Ince Simon-Ince commented Oct 31, 2016

I found help on the Slack group and now have Helm working.

@awh explained that

What you're experiencing is a known issue with k8s where for some operations it expects to be able to resolve your node names in the global DNS

And suggested a work around would be to:

  1. Add entries to /etc/hosts on the master mapping your hostnames to their public IPs
  2. Install dnsmasq on the master (e.g. apt install -y dnsmasq)
  3. Kill the k8s api server container on master (kubelet will recreate it)
  4. Then systemctl restart docker (or reboot the master) for it to pick up the /etc/resolv.conf changes
@technosophos
Copy link
Member

@technosophos technosophos commented Oct 31, 2016

I'm going to leave this open until I get the above put into the install FAQ. I suspect this is an issue that may crop up again.

Thanks for the help @mgoodness and the follow-up @Simon-lush .

technosophos added a commit to technosophos/k8s-helm that referenced this issue Nov 1, 2016
The solution to helm#1455 is to configure dnsmasq on each of your nodes.
This adds brief documentation on how to do so.

Closes helm#1455
@cmcconnell1
Copy link

@cmcconnell1 cmcconnell1 commented Dec 1, 2016

Thanks @technosophos for your help in IRC today the helm and deis community are really helpful and responsive.
Just wanted to note in case this help others, ran into this K8s global DNS bug/issue today and did a quick hack of just adding my kube clusters IP's to my master/ectd:/etc/hosts file and was then able to proceed with installing and configuring deis workflow and helm was happy, etc (at least for now). . . ;-)

# without etcd node /etc/hosts hack entry for my cluster (its defaulting to second Nameserver in my /etc/resolv.conf where the first is our internal/forwarder and second is my ISP's
helm list
Error: Get https://deis-kube1.dev.foo.com/api/v1/namespaces/kube-system/pods?labelSelector=app%3Dhelm%2Cname%3Dtiller: dial tcp: lookup deis-kube1.dev.foo.com on 75.75.75.75:53: no such host

# and clearly records exist in our internal DNS
host deis-kube1.dev.foo.com
deis-kube1.dev.foo.com is an alias for deis-kube-elbapise-161cju3twc1uw-123456789.us-west-1.elb.amazonaws.com.
deis-kube-elbapise-161cju3twc1uw-123456789.us-west-1.elb.amazonaws.com has address 54.x.xxx.xxx
deis-kube-elbapise-161cju3twc1uw-123456789.us-west-1.elb.amazonaws.com has address 52.x.xxx.xxx

# after adding our above noted cluster IP's to kube cluster etcd nodes /etc/hosts
helm repo add deis https://charts.deis.com/workflow
"deis" has been added to your repositories

helm install deis/workflow --namespace deis
Fetched deis/workflow to workflow-v2.8.0.tgz
NAME: newbie-stingeray
LAST DEPLOYED: Thu Dec  1 12:44:01 2016
NAMESPACE: deis
STATUS: DEPLOYED

 helm list
NAME            	REVISION	UPDATED                 	STATUS  	CHART
newbie-stingeray	1       	Thu Dec  1 12:44:01 2016	DEPLOYED	workflow-v2.8.0

We may run into issues later without having completed many of the suggested work-around steps above, but for now we're moving forward. Hope this helps others.

@evfurman
Copy link

@evfurman evfurman commented Apr 19, 2017

Seeing this error after installing helm/tiller on the tectonic stack. I can search helm and use kubectl just fine but fails upon installing new helm chart from default repo. Any idea when a fix will be merged?

[root@host ~]$ helm install stable/selenium Error: forwarding ports: error upgrading connection: error dialing backend: dial tcp: lookup ip-10-148-222-140 on 10.148.213.33:53: no such host

@cmcconnell1
Copy link

@cmcconnell1 cmcconnell1 commented Apr 19, 2017

'Just wanted to note (for me) that as of the previous release of helm v2.3.0 and with kubernetes clusters deployed with kube-aws v0.9.6-rc.2 I did not ever encounter this problem again. After getting this email, checked and noted that there is an updated helm version released 15 hours ago: helm latest release (I'm upgrading now. . .) Not sure about tectonics, I tried it a few months ago but it was not in a working state for us without any H/A capability, etc.. Curious what version of helm you're running?

@cmcconnell1
Copy link

@cmcconnell1 cmcconnell1 commented Apr 19, 2017

@evfurman, just a thought. . . (not sure about tectonics, but for others with kubernetes (kube-aws, etc.), one thing to note is that if your kube cluster is behind a load balancer (in AWS ELB, etc.), your processes for re-deploys may not manage any existing DNS records. So, if you had a previous cluster with an ELB with a CNAME for my-cluster.foo and you redeploy a new cluster with the same name, the DNS record will still point to the old ELB CNAME and you will see that error. In this case, take a look at the new clusters ELB and its FQDN and update any existing DNS CNAMEs to resolve.

@shadycuz
Copy link

@shadycuz shadycuz commented Nov 26, 2017

@technosophos Any idea how to fix this when using helm inside of a pod in the k8?

I'm using gitlab-runners to build projects inside of pods. Kubectl works, its pointing to https://kubernetes.default but helm doesn't.

$ docker run --rm ${CONTAINER_TEST_IMAGE} helm ls
Error: Get http://localhost:8080/api/v1/namespaces/kube-system/pods?labelSelector=app%3Dhelm%2Cname%3Dtiller: dial tcp [::1]:8080: getsockopt: connection refused
$ docker run --rm ${CONTAINER_TEST_IMAGE} kubectl cluster-info
Kubernetes master is running at http://localhost:8080
@axot
Copy link

@axot axot commented Jun 15, 2018

I got similar error today, any advice? Thank you.

$ helm ls
Error: forwarding ports: error upgrading connection: error dialing backend: ssh: rejected: connect failed (Connection timed out)

$ kubectl -n kube-system port-forward tiller-deploy-b774bc49f-6lthf 44134
error: error upgrading connection: error dialing backend: ssh: rejected: connect failed (Connection timed out)
@axot
Copy link

@axot axot commented Jun 21, 2018

helm not work when the number nodes of cluster larger than 1,

Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Kubernetes: &version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.7-gke.3", GitCommit:"9b5b719c5f295c99de68ffb5b63101b0e0175376", GitTreeState:"clean", BuildDate:"2018-05-31T18:32:23Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}
Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
@tirumaraiselvan
Copy link

@tirumaraiselvan tirumaraiselvan commented Jun 21, 2018

Not sure if this is related. I am getting this error in AKS but not in GKE. I use helm from within the cluster to make releases.

Error: forwarding ports: error upgrading connection: Upgrade request required

Just to be safe (in case this is unrelated), I created a new issue: #4254

@bacongobbler
Copy link
Member

@bacongobbler bacongobbler commented Jun 21, 2018

@axot if you're having issues with kubectl port-forward, then that's gonna be a connection issue on Kubernetes' end, not on Helm's :) I'd file a ticket with GKE.

@exu
Copy link

@exu exu commented Sep 27, 2018

Hi guys, I have the same issue on Minikube (on Archlinux - so most recent stable versions)

❯ helm version 
Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Error: forwarding ports: error upgrading connection: 
                                                                                                  

❯ k version 
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

it's occurs quite randomly. but most often after waking up my laptop.

@bacongobbler
Copy link
Member

@bacongobbler bacongobbler commented Sep 27, 2018

Try and see if you can mail it down to a port-forward issue like @axot did in #1455 (comment). It's most likely a connection issue between your laptop and the API server, so it's not anything we can take action on.

@nzoueidi
Copy link

@nzoueidi nzoueidi commented Mar 14, 2019

looks like this issue is still happening even when I followed all the steps in this PR (https://github.com/helm/helm/pull/1492/files).

@commandodev
Copy link

@commandodev commandodev commented Mar 21, 2019

I've just hit this on a brand new set up of kubernetes on nixos (single machine running master and node). I've trying to follow the instruction here https://github.com/weaveworks/flux/blob/master/site/helm-get-started.md and get to

helm upgrade -i flux --set helmOperator.create=true --set helmOperator.createCRD=false --set git.url=git@github.com:boothead/flux-get-started --namespace flux weaveworks/flux
Error: forwarding ports: error upgrading connection: unable to upgrade connection: Forbidden (user=system:anonymous, verb=create, resource=nodes, subresource=proxy)

I tried this earlier today just running kind https://kind.sigs.k8s.io/ yesterday and it seems to all work fine.

@bacongobbler
Copy link
Member

@bacongobbler bacongobbler commented Mar 21, 2019

Is RBAC enabled? Check and see if you can kubectl port-forward to the tiller pod manually. Under the covers, helm is requesting a forwarded port from the kubernetes API server using your local Kube config. I'd start there and see if the same error comes up.

@commandodev
Copy link

@commandodev commandodev commented Mar 21, 2019

Thank @bacongobbler - I just tried that and got the same error - I'm a bit new to k8s, could you point me at any decent resources to understand what's going on? For example I don't really know how to determine what's enabled or not, so I'm not sure if RBAC is enabled...

@maximestevenot
Copy link

@maximestevenot maximestevenot commented Apr 26, 2019

Hi, from my side I solved the Error: forwarding ports: error upgrading connection: Upgrade request required issue with the following command:

$ export HELM_HOST=":44134"
$ tiller -listen ${HELM_HOST} -alsologtostderr >/dev/null 2>&1 &

Note: The export of the variable is useful too


Configuration:

  • Kubernetes Client Version: 1.14.1
  • Kubernetes Server Version: 1.10.3
  • Helm Client Version: 2.12.3
  • Helm Server Version: 2.12.3
@prostil
Copy link

@prostil prostil commented Jul 5, 2019

helm does not work with the folowing K8S cluster

  1. OS Ubuntu 18.04
  2. Total of 2 VM's
  3. 1 running as a master and one as a worker
  4. helm pod is deployed on the worker
    Error: forwarding ports: error upgrading connection: unable to upgrade connection: pod does not exist
@jarvisuser90
Copy link

@jarvisuser90 jarvisuser90 commented Aug 1, 2019

helm does not work with the folowing K8S cluster

  1. OS Ubuntu 18.04
  2. Total of 2 VM's
  3. 1 running as a master and one as a worker
  4. helm pod is deployed on the worker
    Error: forwarding ports: error upgrading connection: unable to upgrade connection: pod does not exist

I'm getting the exact error too

@karamsahu
Copy link

@karamsahu karamsahu commented Aug 22, 2019

@maximestevenot thanks for your solution. It worked for me. The error is gone now.

@nidhi-ag
Copy link

@nidhi-ag nidhi-ag commented Sep 21, 2019

Hi, from my side I solved the Error: forwarding ports: error upgrading connection: Upgrade request required issue with the following command:

$ export HELM_HOST=":44134"
$ tiller -listen ${HELM_HOST} -alsologtostderr >/dev/null 2>&1 &

Note: The export of the variable is useful too

Configuration:

  • Kubernetes Client Version: 1.14.1
  • Kubernetes Server Version: 1.10.3
  • Helm Client Version: 2.12.3
  • Helm Server Version: 2.12.3

Hi, This worked for me. Could you please explain us your solution ?

@caseyclarkjamf
Copy link

@caseyclarkjamf caseyclarkjamf commented Oct 3, 2019

@nidhi-ag It starts a local instance of tiller which connects to your remote Kubernetes cluster which works around the error related to failing to connect to the remote instance of tiller. https://helm.sh/docs/install/#running-tiller-locally

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.