Error forwarding ports: error upgrading connection

[Simon-Ince]

I have Helm install locally and Tiller on my cluster, everything looks healthy, but running helm install stable/mysql is giving me:

Error: Error forwarding ports: error upgrading connection: dial tcp: lookup kube-4gb-lon1-02 on 8.8.8.8:53: no such host

[technosophos]

Can you tell us more about your Kubernetes cluster (version, installation method) and which version of Helm you're running? Thanks.

[Simon-Ince]

@technosophos I created 5 Ubuntu servers on DigitalOcean and used Kubeadm:

kubeadm version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.0.1534+cf7301f16c0363-dirty", GitCommit:"cf7301f16c036363c4fdcb5d4d0c867720214598", GitTreeState:"dirty", BuildDate:"2016-09-27T18:10:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

to install on Ubuntu:

Distributor ID: Ubuntu
Description:    Ubuntu 16.04.1 LTS
Release:    16.04
Codename:   xenial

Then on my local machine using OSX:

System Version: OS X 10.10.5 (14F1909)
Kernel Version: Darwin 14.5.0

I installed Kubectl and connected to my cluster:

Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4+3b417cc", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"not a git tree", BuildDate:"2016-10-21T22:33:18Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"darwin/amd64"}

Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:10:32Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Then installed Helm using brew cask install helm But when I try helm install stable/mysql I get Error: Error forwarding ports: error upgrading connection: dial tcp: lookup kube-4gb-lon1-02 on 8.8.8.8:53: no such host

I think I've gone a fairly safe route, I used the latest stable release of Ubuntu and installed the recommended release of kubeadm, which installed the latest stable release of kubeclt and then connected to it from my local machine and used brew to install helm, so my set-up should be pretty typical. I'm able to use kubeclt on my local machine fine, I can use manifesting to get things set up and etc, the only thing I can't get working is Helm.

[technosophos]

Interesting that it is failing a DNS lookup. Is SkyDNS running in your Kube cluster?

kubectl get po -n kube-system
NAME                                READY     STATUS    RESTARTS   AGE
k8s-etcd-127.0.0.1                  1/1       Running   0          18h
k8s-master-127.0.0.1                4/4       Running   1          18h
k8s-proxy-127.0.0.1                 1/1       Running   0          18h
kube-addon-manager-127.0.0.1        2/2       Running   0          18h
kube-dns-v20-jqqts                  3/3       Running   0          18h
kubernetes-dashboard-v1.4.0-5ldke   1/1       Running   0          18h

[adamreese]

Can you start a port-forward using kubectl?

kubectl -n kube-system port-forward tiller-deploy-xxxxxx 44134

[Simon-Ince]

@technosophos Looks like DNS is there:

kube-system   etcd-kube-4gb-lon1-01                      1/1       Running   0          3d
kube-system   kube-apiserver-kube-4gb-lon1-01            1/1       Running   0          3d
kube-system   kube-controller-manager-kube-4gb-lon1-01   1/1       Running   0          3d
kube-system   kube-discovery-982812725-28b96             1/1       Running   0          3d
kube-system   kube-dns-2247936740-7lcli                  3/3       Running   0          3d
kube-system   kube-proxy-amd64-2j5gn                     1/1       Running   0          3d
kube-system   kube-proxy-amd64-9a0ob                     1/1       Running   0          3d
kube-system   kube-proxy-amd64-m6ypu                     1/1       Running   0          3d
kube-system   kube-proxy-amd64-n702h                     1/1       Running   0          3d
kube-system   kube-proxy-amd64-yjk4a                     1/1       Running   0          3d
kube-system   kube-scheduler-kube-4gb-lon1-01            1/1       Running   0          3d
kube-system   kubernetes-dashboard-1655269645-nyptv      1/1       Running   0          3d
kube-system   tiller-deploy-2434200834-8t7yt             1/1       Running   0          2d
kube-system   weave-net-09tbe                            2/2       Running   0          3d
kube-system   weave-net-f7qsx                            2/2       Running   0          3d
kube-system   weave-net-ihvuq                            2/2       Running   0          3d
kube-system   weave-net-phlva                            2/2       Running   0          3d
kube-system   weave-net-yx4wv                            2/2       Running   0          3d

[Simon-Ince]

@adamreese Looks like that gives back a similar error.
kubectl -n kube-system port-forward tiller-deploy-2434200834-8t7yt 44134
results in:
error: error upgrading connection: dial tcp: lookup kube-4gb-lon1-02 on 8.8.8.8:53: no such host
I've tried on both my local machine and while ssh'ed into the master node.

[Simon-Ince]

Just tried setting up a cluster again from scratch and still having the same issue.

When using kubeadm to set up the cluster, step "(3/4) Installing a pod network" (http://kubernetes.io/docs/getting-started-guides/kubeadm/) requiers you to pick the addon, I chose Weave Net, is Helm not compatible with this?

The options are:

• Weave Net
• Calico
• Canal
• Romana

Should I chose another one?

[technosophos]

I'm running Weave on a single-node Kubeadm-based install. I can open a tunnel with kubectl port-forward from the master node (which is the only node). I wonder if this has to do with networking across multiple nodes?

There is the possibility that asking about this in the #kubernetes-users slack channel or on StackOverflow might yield some answers, since this is actually a Kubernetes configuration thing, not a Helm-specific thing. I'll do some more hunting and update here if I find out anything.

[mgoodness]

@Simon-lush Based on the names of your apiserver, scheduler, and controller-manager pods, it seems like kubectl/helm should be trying to lookup kube-4gb-lon1-01. But your error messages show lookups for kube-4gb-lon1-02. Dueling entries in your kubeconfig?

[Simon-Ince]

I found help on the Slack group and now have Helm working.

@awh explained that

What you're experiencing is a known issue with k8s where for some operations it expects to be able to resolve your node names in the global DNS

And suggested a work around would be to:

1. Add entries to /etc/hosts on the master mapping your hostnames to their public IPs
2. Install dnsmasq on the master (e.g. apt install -y dnsmasq)
3. Kill the k8s api server container on master (kubelet will recreate it)
4. Then systemctl restart docker (or reboot the master) for it to pick up the /etc/resolv.conf changes

[technosophos]

I'm going to leave this open until I get the above put into the install FAQ. I suspect this is an issue that may crop up again.

Thanks for the help @mgoodness and the follow-up @Simon-lush .