Connection refused error when issuing helm install command #3460

OpusX · 2018-02-05T21:21:49Z

When running "helm list" or a "helm install .... " I am getting the following error:
Error: Get http://localhost:8080/api/v1/namespaces/kube-system/configmaps?labelSelector=OWNER%3DTILLER: dial tcp [::1]:8080: getsockopt: connection refused

This was a manual install in a lab on CentOS.

Tiller pod is up and running:
kubectl -n kube-system get pods
NAME READY STATUS RESTARTS AGE
tiller-deploy-978853713-7h871 1/1 Running 0 108d

helm version
Client: &version.Version{SemVer:"v2.6.2", GitCommit:"be3ae4ea91b2960be98c07e8f73754e67e87963c", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.6.2", GitCommit:"be3ae4ea91b2960be98c07e8f73754e67e87963c", GitTreeState:"clean"}

OpusX · 2018-02-06T13:48:13Z

Here is my k8s config:

apiVersion: v1
clusters:
- cluster:
    server: http://kubrelab01:8080
  name: default-cluster
contexts:
- context:
    cluster: default-cluster
    user: default-admin
  name: default-context
current-context: default-context
kind: Config
preferences: {}
users: []

kubectl cluster-info
Kubernetes master is running at http://kubrelab01:8080

kubectl config current-context
default-context

bacongobbler · 2018-02-09T16:55:28Z

looks like tiller is attempting to connect to the kubernetes API server using ipv6. Because this is a manual install, I'm going to immediately assume that this is due to a misconfigured cluster with ipv6 enabled and not exactly tiller's fault. I know there have been issues in the past with Kubernetes and ipv6. I'd either try to disable ipv6 from within your cluster or otherwise look into that part of your configuration.

mrene · 2018-02-23T18:30:37Z

I have the same issue on GKE (1.9.2) and RBAC on and installing tiller in a specific namespace. The pod didn't have anything mounted inside /var/run/secrets - I had to edit the deployment and add automountServiceAccountToken: true which fixed the problem. Might be a new default in k8s 1.9?

mrene · 2018-02-23T18:34:47Z

Actually, it's the service account that I created that had automountServiceAccountToken: false set by default.

Adds automountServiceAccountToken when a serviceAccount is specified. Fixes helm#3460, fixes helm#3467.

Adds automountServiceAccountToken when a serviceAccount is specified. Prior to this, tiller falls back to contacting the KUBERNETES_SERVICE on localhost:8080 rather than respecting the cluster IP in the KUBERNETES_SERVICE_{HOST,PORT} environment variables. Fixes helm#3460, fixes helm#3467.

Adds automountServiceAccountToken when a serviceAccount is specified. Prior to this, tiller falls back to contacting the KUBERNETES_SERVICE on localhost:8080 rather than respecting the cluster IP in the KUBERNETES_SERVICE_{HOST,PORT} environment variables. Fixes #3460, fixes #3467. (cherry picked from commit 1e03f1b)

bacongobbler · 2018-04-30T12:55:10Z

Re-opening this one due to having to revert the PR that addressed this issue due to breaking other parts of the codebase (e.g. helm init with no --service-account set)

geekdave · 2018-05-01T18:17:59Z

Any workarounds for this? Getting the same issue on an AWS cluster I set up with kubeadm. Saw the comment about adding automountServiceAccountToken: true but not sure where this would go?

hendrikhalkow · 2018-05-01T22:24:45Z

Getting the same on OpenShift, deployed via openshift-ansible. I want to use IPv6 and I am pretty store that I set it up correctly. Any idea where that localhost:8080 URL comes from?

bacongobbler · 2018-05-01T22:52:20Z

workaround:

helm init --service-account default

hendrikhalkow · 2018-05-01T23:09:39Z

Doesn't change a thing:

$ rm -rf ~/.helm/ ~/.kube/
$ oc login https://my.domain
Authentication required for https://my.domain.io:443 (openshift)
Username: myusername
Password: 
Login successful.

You have access to the following projects and can switch between them with 'oc project <projectname>':

* default
    glusterfs
    hello
    kube-public
    kube-service-catalog
    kube-system
    logging
    management-infra
    openshift
    openshift-ansible-service-broker
    openshift-infra
    openshift-node
    openshift-template-service-broker
    openshift-web-console

Using project "default".
Welcome! See 'oc help' to get started.
$ kubectl get pods
NAME                       READY     STATUS        RESTARTS   AGE
docker-registry-1-cp2vh    1/1       Running       1          15h
docker-registry-1-g4f46    1/1       Running       1          9h
my-hello-1-q4gnd           1/1       Running       0          21m
registry-console-1-vdqgq   1/1       Running       1          9h
router-1-6djkg             0/1       Pending       0          8h
router-1-njwbk             1/1       Running       1          15h
router-1-r58c8             0/1       Terminating   0          9h
$ helm init
$ helm init --service-account default
Creating /Users/myusername/.helm 
Creating /Users/myusername/.helm/repository 
Creating /Users/myusername/.helm/repository/cache 
Creating /Users/myusername/.helm/repository/local 
Creating /Users/myusername/.helm/plugins 
Creating /Users/myusername/.helm/starters 
Creating /Users/myusername/.helm/cache/archive 
Creating /Users/myusername/.helm/repository/repositories.yaml 
Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com 
Adding local repo with URL: http://127.0.0.1:8879/charts 
$HELM_HOME has been configured at /Users/myusername/.helm.
Warning: Tiller is already installed in the cluster.
(Use --client-only to suppress this message, or --upgrade to upgrade Tiller to the current version.)
Happy Helming!
$ helm ls
Error: Get http://localhost:8080/api/v1/namespaces/kube-system/configmaps?labelSelector=OWNER%!D(MISSING)TILLER: dial tcp 127.0.0.1:8080: connect: connection refused

bacongobbler · 2018-05-01T23:12:43Z

it won't work if you ran helm init previously. Sorry! I meant that you need to replace helm init with helm init --service-account default. Run kubectl -n kube-system delete deploy tiller-deploy and re-run helm init --service-account default again.

cookkkie · 2018-05-02T02:55:40Z

Hi, after doing kubectl -n kube-system delete deploy tiller-deploy, I'm getting Warning: Tiller is already installed in the cluster. from the helm init --service-account default. Deleting the deployment isn't enough to uninstall tiller?

EDIT: I tried helm reset, it gives me Error: configmaps is forbidden: User "system:serviceaccount:kube-system:default" cannot list configmaps in the namespace "kube-system": Unknown user "system:serviceaccount:kube-system:default". So probably just a perms issue on my side.

bacongobbler · 2018-05-02T03:56:26Z

yes. helm init provisions a deployment and a service, so in order to remove that warning you also need to remove the tiller-deploy service.

innovia · 2018-05-02T14:13:41Z

@cookkkie

The service account needs to have a clusterRole: Admin default does not have a cluster admin role

default service account is used at no RBAC clusters.

kubectl delete svc tiller-deploy -n kube-system
kubectl -n kube-system delete deploy tiller-deploy
kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
helm init --service-account tiller
helm ls # does not return an error

Proof that it's working:

helm upgrade -i --namespace kube-system autoscale -f cluster-autoscaler/my-values.yaml cluster-autoscaler

helm ls
NAME     	REVISION	UPDATED                 	STATUS  	CHART                   	NAMESPACE
autoscale	1       	Wed May  2 17:07:52 2018	DEPLOYED	cluster-autoscaler-0.6.1	kube-system

chekcout my blog post on how to setup tiller per namespace

bacongobbler · 2018-05-14T19:08:39Z

The root issue for this has been fixed in helm v2.9.1. Thanks everyone!

@mrene if there's a doc issue where we create a service account with automountServiceAccountToken set to false by default, could you please open a new ticket/PR for that? Thank you so much. :)

liijuun · 2018-08-16T08:43:08Z

@innovia
It's not working for me.

root@neo-1:~/kubernetes/prometheus-operator/helm# helm ls
Error: Get https://10.96.0.1:443/api/v1/namespaces/kube-system/configmaps?labelSelector=OWNER%!D(MISSING)TILLER: dial tcp 10.96.0.1:443: i/o timeout

root@neo-1:~/kubernetes/prometheus-operator/helm# kubectl delete svc tiller-deploy -n kube-system
service "tiller-deploy" deleted
root@neo-1:~/kubernetes/prometheus-operator/helm# kubectl -n kube-system delete deploy tiller-deploy
deployment.extensions "tiller-deploy" deleted
root@neo-1:~/kubernetes/prometheus-operator/helm# kubectl -n kube-system delete serviceaccount tiller
serviceaccount "tiller" deleted
root@neo-1:~/kubernetes/prometheus-operator/helm# kubectl -n kube-system delete clusterrolebinding tiller-cluster-rule
clusterrolebinding.rbac.authorization.k8s.io "tiller-cluster-rule" deleted



root@neo-1:~/kubernetes/prometheus-operator/helm#
root@neo-1:~/kubernetes/prometheus-operator/helm#
root@neo-1:~/kubernetes/prometheus-operator/helm# kubectl create serviceaccount --namespace kube-system tiller
serviceaccount/tiller created
root@neo-1:~/kubernetes/prometheus-operator/helm# kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
clusterrolebinding.rbac.authorization.k8s.io/tiller-cluster-rule created
root@neo-1:~/kubernetes/prometheus-operator/helm# helm init --service-account tiller
$HELM_HOME has been configured at /root/.helm.

Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation
Happy Helming!
root@neo-1:~/kubernetes/prometheus-operator/helm# helm ls
Error: Get https://10.96.0.1:443/api/v1/namespaces/kube-system/configmaps?labelSelector=OWNER%!D(MISSING)TILLER: dial tcp 10.96.0.1:443: i/o timeout



root@neo-1:~/kubernetes/prometheus-operator/helm# helm version
Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}

Adds automountServiceAccountToken when a serviceAccount is specified. Prior to this, tiller falls back to contacting the KUBERNETES_SERVICE on localhost:8080 rather than respecting the cluster IP in the KUBERNETES_SERVICE_{HOST,PORT} environment variables. Fixes helm#3460, fixes helm#3467.

summerzhangft · 2018-12-07T02:14:59Z

kubectl get pods -n kube-system
kubectl delete pod tiller-deploy-bf49955d6-tcchs
helm init --service-account tiller

Adds automountServiceAccountToken when a serviceAccount is specified. Prior to this, tiller falls back to contacting the KUBERNETES_SERVICE on localhost:8080 rather than respecting the cluster IP in the KUBERNETES_SERVICE_{HOST,PORT} environment variables. Fixes helm#3460, fixes helm#3467.

rgylan · 2019-03-24T17:25:04Z

Had the same issue: Get https://10.96.0.1:443/api/v1/namespaces/kube-system/configmaps?labelSelector=OWNER%3DTILLER: dial tcp 10.96.0.1:443: i/o timeout

Issuing "helm list" produces above.

In my case flannel is the culprit. I had kubeadm init below:
kubeadm init --apiserver-advertise-address=192.168.137.189 --pod-network-cidr=192.168.1.0/18
But somehow flannel forced the CIDR to be 10.244.0.0/16
You can check by: kubectl edit cm -n kube-system kube-flannel-cfg
So I edited using 192.168.1.0/18
Then: kubectl delete pod -n kube-system -l app=flannel to make sure you have a newly spawned flannel pod which will get the 192.168.1.0/18

Then as per above post to start new tiller pod with correct cluster role:
kubectl delete svc tiller-deploy -n kube-system
kubectl -n kube-system delete deploy tiller-deploy
kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
helm init --service-account tiller
helm ls # does not return an error

Then: helm list, no more timeout error.

Thanks to this post: https://gravitational.com/blog/kubernetes-flannel-dashboard-bomb/#fn:https-kubernetes

tunicashashi · 2019-04-15T15:21:08Z

After doing everything like above I still see this issue with my Kubernetes 1. 14 and helm 2.13:
I have --pod-network-cidr: 172.16.0.0/16
and --service-network-cidr: 172.18.0.0/16
and my master VM apiserver is on : 192.168.10.14

I had created a service account as above mentioned: tiller
My kubectl command provides as below log:
[centos@master-centos oisp-k8s]$ kubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-fb8b8dccf-85szb 1/1 Running 0 3d 172.16.0.98 master-centos.novalocal
coredns-fb8b8dccf-w5swd 1/1 Running 0 3d 172.16.0.99 master-centos.novalocal
etcd-master-centos.novalocal 1/1 Running 0 3d 192.168.10.14 master-centos.novalocal
kube-apiserver-master-centos.novalocal 1/1 Running 0 3d 192.168.10.14 master-centos.novalocal
kube-controller-manager-master-centos.novalocal 1/1 Running 0 3d 192.168.10.14 master-centos.novalocal
kube-flannel-ds-amd64-6qh4q 1/1 Running 0 3d 192.168.10.3 node03-centos.novalocal
kube-flannel-ds-amd64-g5m9c 1/1 Running 0 3d 192.168.10.14 master-centos.novalocal
kube-proxy-2g7j8 1/1 Running 0 3d 192.168.10.3 node03-centos.novalocal
kube-proxy-llqdk 1/1 Running 0 3d 192.168.10.14 master-centos.novalocal
kube-scheduler-master-centos.novalocal 1/1 Running 0 3d 192.168.10.14 master-centos.novalocal
tiller-deploy-8458f6c667-xtnv9 1/1 Running 0 3h15m 172.16.1.4 node03-centos.novalocal

$kubectl logs tiller-deploy-8458f6c667-xtnv9 -n kube-system

main] 2019/04/15 12:05:05 Starting Tiller v2.13.1 (tls=false)
[main] 2019/04/15 12:05:05 GRPC listening on :44134
[main] 2019/04/15 12:05:05 Probes listening on :44135
[main] 2019/04/15 12:05:05 Storage driver is ConfigMap
[main] 2019/04/15 12:05:05 Max history per release is 0
[storage] 2019/04/15 12:57:47 listing all releases with filter
[storage/driver] 2019/04/15 12:58:17 list: failed to list: Get https://172.18.0.1:443/api/v1/namespaces/kube-system/configmaps?labelSelector=OWNER%3DTILLER: dial tcp 172.18.0.1:443: i/o timeout
[storage] 2019/04/15 14:23:45 listing all releases with filter

damien-roche · 2019-07-04T15:07:54Z

Considering Helm (3) is moving away from Tiller is it about time we forget it exists? I've had nothing but problems and it seems like a huge security issue to jump through RBAC hoops and give it unnecessary control over the cluster.. to manage releases?

I've started using helm template .., kubectl apply and have no intention of using Tiller ever again. I'd highly recommend those above who have issues to do the same.

cforce · 2019-11-01T12:42:52Z

#3480 (comment)

rrmuchedzi · 2020-03-30T08:21:58Z

First, make sure you have Docker and Kubernetes setup, then run (Replace <YOUR_USER> with your host Username)

mkdir -p ~/.kube
ln -sf /mnt/c/users/<YOUR_USER>/.kube/config ~/.kube/config

Then run,
helm init

mrvrbabu · 2020-07-21T18:34:16Z

@rgylan Your solution worked perfectly for me and it saved my day. :)

AntonOfTheWoods · 2020-08-31T07:57:44Z

If you are running on something like microk8s or minikube and have used a VPN or something that messes with your iptables then this might be due to network funkiness. Try restarting your host network and kubernetes (e.g, microk8s.stop && microk8s.start).

pohvak · 2021-02-09T03:23:16Z

workaround:

helm init --service-account default

init is not a command for helm

bacongobbler · 2021-02-09T06:40:52Z

@pohvak it was for Helm 2 back when I posted that comment in 2018.

pohvak · 2021-02-09T06:42:08Z

@pohvak it was for Helm 2 back when I posted that comment in 2018.

@bacongobbler oh, I see now, sorry

bacongobbler added the question/support label Feb 5, 2018

jascott1 mentioned this issue Feb 7, 2018

Error: Get http://localhost:8080/api/v1/namespaces/kube-system/configmaps?labelSelector=OWNER%3DTILLER: dial tcp [::1]:8080: getsockopt: connection refused #3467

Closed

jkoleszar added a commit to jkoleszar/helm that referenced this issue Mar 29, 2018

Fix tiller deployment on RBAC clusters

06c2876

Adds automountServiceAccountToken when a serviceAccount is specified. Fixes helm#3460, fixes helm#3467.

jkoleszar added a commit to jkoleszar/helm that referenced this issue Mar 29, 2018

Fix tiller deployment on RBAC clusters

fec7c34

Adds automountServiceAccountToken when a serviceAccount is specified. Fixes helm#3460, fixes helm#3467.

jkoleszar mentioned this issue Mar 29, 2018

Fix tiller deployment on RBAC clusters #3784

Merged

thomastaylor312 closed this as completed in #3784 Apr 3, 2018

michelleN mentioned this issue Apr 4, 2018

helm list failed #3816

Closed

bacongobbler reopened this Apr 30, 2018

robbiezhang mentioned this issue May 2, 2018

Error: Get http://localhost:8080/version: dial tcp [::1]:8080: connect: connection refused when installing connector virtual-kubelet/virtual-kubelet#184

Closed

bacongobbler closed this as completed May 14, 2018

derba mentioned this issue May 20, 2019

tiller won't install k3s-io/k3s#489

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Connection refused error when issuing helm install command #3460

Connection refused error when issuing helm install command #3460

OpusX commented Feb 5, 2018

OpusX commented Feb 6, 2018 •

edited

bacongobbler commented Feb 9, 2018

mrene commented Feb 23, 2018

mrene commented Feb 23, 2018

bacongobbler commented Apr 30, 2018

geekdave commented May 1, 2018

hendrikhalkow commented May 1, 2018

bacongobbler commented May 1, 2018

hendrikhalkow commented May 1, 2018

bacongobbler commented May 1, 2018 •

edited

cookkkie commented May 2, 2018 •

edited

bacongobbler commented May 2, 2018 •

edited

innovia commented May 2, 2018

bacongobbler commented May 14, 2018

liijuun commented Aug 16, 2018

summerzhangft commented Dec 7, 2018

rgylan commented Mar 24, 2019

tunicashashi commented Apr 15, 2019

damien-roche commented Jul 4, 2019

cforce commented Nov 1, 2019

rrmuchedzi commented Mar 30, 2020 •

edited

mrvrbabu commented Jul 21, 2020

AntonOfTheWoods commented Aug 31, 2020

pohvak commented Feb 9, 2021

bacongobbler commented Feb 9, 2021

pohvak commented Feb 9, 2021

Connection refused error when issuing helm install command #3460

Connection refused error when issuing helm install command #3460

Comments

OpusX commented Feb 5, 2018

OpusX commented Feb 6, 2018 • edited

bacongobbler commented Feb 9, 2018

mrene commented Feb 23, 2018

mrene commented Feb 23, 2018

bacongobbler commented Apr 30, 2018

geekdave commented May 1, 2018

hendrikhalkow commented May 1, 2018

bacongobbler commented May 1, 2018

hendrikhalkow commented May 1, 2018

bacongobbler commented May 1, 2018 • edited

cookkkie commented May 2, 2018 • edited

bacongobbler commented May 2, 2018 • edited

innovia commented May 2, 2018

bacongobbler commented May 14, 2018

liijuun commented Aug 16, 2018

summerzhangft commented Dec 7, 2018

rgylan commented Mar 24, 2019

tunicashashi commented Apr 15, 2019

damien-roche commented Jul 4, 2019

cforce commented Nov 1, 2019

rrmuchedzi commented Mar 30, 2020 • edited

mrvrbabu commented Jul 21, 2020

AntonOfTheWoods commented Aug 31, 2020

pohvak commented Feb 9, 2021

bacongobbler commented Feb 9, 2021

pohvak commented Feb 9, 2021

OpusX commented Feb 6, 2018 •

edited

bacongobbler commented May 1, 2018 •

edited

cookkkie commented May 2, 2018 •

edited

bacongobbler commented May 2, 2018 •

edited

rrmuchedzi commented Mar 30, 2020 •

edited