New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

helm needs socat on the nodes of the k8s cluster #966

Closed
sebgoa opened this Issue Jul 20, 2016 · 21 comments

Comments

Projects
None yet
8 participants
@sebgoa
Contributor

sebgoa commented Jul 20, 2016

But cluster nodes, may not have socat installed by default.

For instance coreOS stable does not ship socat:

coreos/bugs#1114

This is a problem since helm use a port-forwarding technique to connect to tiller.

ps: there might be the same issue with nsenter

@maratoid

This comment has been minimized.

Show comment
Hide comment
@maratoid

maratoid Jul 21, 2016

Contributor

That's not so much a Helm problem, as a Kubernetes problem (https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockertools/docker_manager.go#L1167)

Personally I ended up bootstrapping all of our CoreOs nodes with socat in /opt/bin and adding /opt/bin to kubelet service PATH environment variable with a systemd drop-in.

Contributor

maratoid commented Jul 21, 2016

That's not so much a Helm problem, as a Kubernetes problem (https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockertools/docker_manager.go#L1167)

Personally I ended up bootstrapping all of our CoreOs nodes with socat in /opt/bin and adding /opt/bin to kubelet service PATH environment variable with a systemd drop-in.

@technosophos

This comment has been minimized.

Show comment
Hide comment
@technosophos

technosophos Jul 22, 2016

Member

Yeah, I think we are going to have to bump this up to the CoreOS/Kubernetes level. However, I'm leaving the issue open here while we track what's going on upstream.

Member

technosophos commented Jul 22, 2016

Yeah, I think we are going to have to bump this up to the CoreOS/Kubernetes level. However, I'm leaving the issue open here while we track what's going on upstream.

@sebgoa

This comment has been minimized.

Show comment
Hide comment
@sebgoa

sebgoa Jul 24, 2016

Contributor

Yes, I did not mean to say it was a Helm problem. But we as Helm are assuming that k8s cluster nodes will have socat...which may not be true.

Honestly I think protobuf is cool, but for exposing API outside k8s it makes things a bit tricky.

Contributor

sebgoa commented Jul 24, 2016

Yes, I did not mean to say it was a Helm problem. But we as Helm are assuming that k8s cluster nodes will have socat...which may not be true.

Honestly I think protobuf is cool, but for exposing API outside k8s it makes things a bit tricky.

@rimusz

This comment has been minimized.

Show comment
Hide comment
@rimusz

rimusz Aug 12, 2016

Member

@maratoid where did you get the socat binary? I tried the statically built from https://github.com/andrew-d/static-binaries but it almost crashes VM with a very high CPU usage when I do helm install some chart

Member

rimusz commented Aug 12, 2016

@maratoid where did you get the socat binary? I tried the statically built from https://github.com/andrew-d/static-binaries but it almost crashes VM with a very high CPU usage when I do helm install some chart

@maratoid

This comment has been minimized.

Show comment
Hide comment
Contributor

maratoid commented Aug 16, 2016

@rimusz

This comment has been minimized.

Show comment
Hide comment
@rimusz

rimusz Aug 16, 2016

Member

@maratoid this is what I was testing with helm v2, socat from there works fine only with kubectl port-forward but with helm socat goes nuts of using a lot of CPU and helm install just hangs

Member

rimusz commented Aug 16, 2016

@maratoid this is what I was testing with helm v2, socat from there works fine only with kubectl port-forward but with helm socat goes nuts of using a lot of CPU and helm install just hangs

@technosophos

This comment has been minimized.

Show comment
Hide comment
@technosophos

technosophos Aug 18, 2016

Member

FWIW, in Slack, dhilipkumars hit this issue when running Kubernetes on Ubuntu locally. The fix was to export HELM_HOST to the k8s API address/port. We need a FAQ or something to document issues like this.

Member

technosophos commented Aug 18, 2016

FWIW, in Slack, dhilipkumars hit this issue when running Kubernetes on Ubuntu locally. The fix was to export HELM_HOST to the k8s API address/port. We need a FAQ or something to document issues like this.

@philips

This comment has been minimized.

Show comment
Hide comment
@philips

philips Aug 19, 2016

Contributor

CoreOS doesn't ship socat but the container that the kubelet runs in has ./usr/bin/socat. What is the issue with that socat? How do I reproduce?

Contributor

philips commented Aug 19, 2016

CoreOS doesn't ship socat but the container that the kubelet runs in has ./usr/bin/socat. What is the issue with that socat? How do I reproduce?

@philips

This comment has been minimized.

Show comment
Hide comment
@philips
Contributor

philips commented Aug 19, 2016

@rimusz

This comment has been minimized.

Show comment
Hide comment
@rimusz

rimusz Aug 19, 2016

Member

@philips if kubernetes e.g. is run via go binaries then kubectl port-forward and helm v2 install chart do not work

Member

rimusz commented Aug 19, 2016

@philips if kubernetes e.g. is run via go binaries then kubectl port-forward and helm v2 install chart do not work

@sebgoa

This comment has been minimized.

Show comment
Hide comment
@sebgoa

sebgoa Aug 21, 2016

Contributor

@philips the issue I faced was the following. I have a coreOS k8s cluster with coreOS 899.
./helm init deploy the tiller server fine, but by default the helm client tries to setup a port-forward to the tiller pod. port-forward uses socat, hence helm fails talking to tiller because socat is not found. I personally setup a NodePort service and set my HELM_HOST to that port, but it is insecure.

Contributor

sebgoa commented Aug 21, 2016

@philips the issue I faced was the following. I have a coreOS k8s cluster with coreOS 899.
./helm init deploy the tiller server fine, but by default the helm client tries to setup a port-forward to the tiller pod. port-forward uses socat, hence helm fails talking to tiller because socat is not found. I personally setup a NodePort service and set my HELM_HOST to that port, but it is insecure.

@philips

This comment has been minimized.

Show comment
Hide comment
@philips

philips Sep 9, 2016

Contributor

@rimusz OK, CoreOS Linux won't support this setup because kubelet has a number of dependencies that aren't. Please run kubelet in a container.

@runseb I am able to successfully use helm v2.0.0-alpha.4 with CoreOS Linux and Kubernetes v1.4-beta0:

$ helm install docs/examples/alpine
Error: file "docs/examples/alpine" not found
$ cd
$ cd src/k8s.io/he
heapster/ helm/
$ cd src/k8s.io/helm/
$ helm install docs/examples/alpine
washing-squid
Last Deployed: Fri Sep  9 00:08:03 2016
Namespace: default
Status: DEPLOYED

Resources:
==> v1/Pod
NAME                      READY     STATUS              RESTARTS   AGE
washing-squid-my-alpine   0/1       ContainerCreating   0          1s

Contributor

philips commented Sep 9, 2016

@rimusz OK, CoreOS Linux won't support this setup because kubelet has a number of dependencies that aren't. Please run kubelet in a container.

@runseb I am able to successfully use helm v2.0.0-alpha.4 with CoreOS Linux and Kubernetes v1.4-beta0:

$ helm install docs/examples/alpine
Error: file "docs/examples/alpine" not found
$ cd
$ cd src/k8s.io/he
heapster/ helm/
$ cd src/k8s.io/helm/
$ helm install docs/examples/alpine
washing-squid
Last Deployed: Fri Sep  9 00:08:03 2016
Namespace: default
Status: DEPLOYED

Resources:
==> v1/Pod
NAME                      READY     STATUS              RESTARTS   AGE
washing-squid-my-alpine   0/1       ContainerCreating   0          1s

@sebgoa

This comment has been minimized.

Show comment
Hide comment
@sebgoa

sebgoa Sep 9, 2016

Contributor

@philips which version of coreOS ?

Contributor

sebgoa commented Sep 9, 2016

@philips which version of coreOS ?

@philips

This comment has been minimized.

Show comment
Hide comment
@philips

philips Sep 9, 2016

Contributor

@runseb alpha using the hyperkube kubelet container

Contributor

philips commented Sep 9, 2016

@runseb alpha using the hyperkube kubelet container

@philips

This comment has been minimized.

Show comment
Hide comment
@philips

philips Sep 9, 2016

Contributor

@runseb CoreOS 1153.0.0 to be precise

Contributor

philips commented Sep 9, 2016

@runseb CoreOS 1153.0.0 to be precise

@sebgoa

This comment has been minimized.

Show comment
Hide comment
@sebgoa

sebgoa Sep 9, 2016

Contributor

thanks, I will try that one

Contributor

sebgoa commented Sep 9, 2016

thanks, I will try that one

@sebgoa

This comment has been minimized.

Show comment
Hide comment
@sebgoa

sebgoa Sep 22, 2016

Contributor

sorry for the delay. Indeed running the kubelet with the wrapper solves this.
Check kubelet wrapper for example of unit file to run the wrapper.

tested under coreOS stable 1122 and helm from HEAD.

thx @philips

I am closing this.

Contributor

sebgoa commented Sep 22, 2016

sorry for the delay. Indeed running the kubelet with the wrapper solves this.
Check kubelet wrapper for example of unit file to run the wrapper.

tested under coreOS stable 1122 and helm from HEAD.

thx @philips

I am closing this.

@sstarcher

This comment has been minimized.

Show comment
Hide comment
@sstarcher

sstarcher Oct 24, 2016

Contributor

The documentation for the kubelet makes it seem like running the kubelet in a docker container is not a good idea for production.

--containerized                                  Experimental support for running kubelet in a container.  Intended for testing. [default=false]

http://kubernetes.io/docs/admin/kubelet/

Is the kubelet production ready?

Contributor

sstarcher commented Oct 24, 2016

The documentation for the kubelet makes it seem like running the kubelet in a docker container is not a good idea for production.

--containerized                                  Experimental support for running kubelet in a container.  Intended for testing. [default=false]

http://kubernetes.io/docs/admin/kubelet/

Is the kubelet production ready?

@mattthias

This comment has been minimized.

Show comment
Hide comment
@mattthias

mattthias Nov 4, 2016

Contributor

If you don't want to run kubelet in a container you may install socat in /opt/bin/ and add it to the PATH of the kubelet systemd.unit

Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin"

Contributor

mattthias commented Nov 4, 2016

If you don't want to run kubelet in a container you may install socat in /opt/bin/ and add it to the PATH of the kubelet systemd.unit

Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin"

@rimusz

This comment has been minimized.

Show comment
Hide comment
@rimusz

rimusz Nov 4, 2016

Member

I use @mattthias approach too

Member

rimusz commented Nov 4, 2016

I use @mattthias approach too

@amolsh

This comment has been minimized.

Show comment
Hide comment
@amolsh

amolsh Nov 21, 2016

I was testing helm on ubuntu VM and faced same issue and resolved after installing socat and nsenter
For helm to properly work, you need socat and nsenter installed on all nodes

for socat install use:
sudo apt-get update && sudo apt-get install socat

for nsenter install use :
cd /tmp; curl https://www.kernel.org/pub/linux/utils/util-linux/v2.25/util-linux-2.25.tar.gz | tar -zxf-; cd util-linux-2.25;
sudo apt-get install autopoint autoconf libtool automake
./configure --without-python --disable-all-programs --enable-nsenter --without-ncurses
make nsenter; cp nsenter /usr/local/bin

amolsh commented Nov 21, 2016

I was testing helm on ubuntu VM and faced same issue and resolved after installing socat and nsenter
For helm to properly work, you need socat and nsenter installed on all nodes

for socat install use:
sudo apt-get update && sudo apt-get install socat

for nsenter install use :
cd /tmp; curl https://www.kernel.org/pub/linux/utils/util-linux/v2.25/util-linux-2.25.tar.gz | tar -zxf-; cd util-linux-2.25;
sudo apt-get install autopoint autoconf libtool automake
./configure --without-python --disable-all-programs --enable-nsenter --without-ncurses
make nsenter; cp nsenter /usr/local/bin

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment