New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

First class support for Docker containers #175

Closed
drslump opened this Issue Jun 12, 2017 · 11 comments

Comments

Projects
None yet
2 participants
@drslump
Contributor

drslump commented Jun 12, 2017

Related to the discussion on https://gitter.im/datawire/telepresence?at=593eaed0c59bd9c4640b6f8d

Preface

Given the current maturity of local Docker solutions, for many projects, developing against a local container offers more benefits than drawbacks. Not only can a developer benefit from having an execution environment that closely matches the production one, it also helps aligning it with the environment on CI which is a common source of pain specially when tuning a build procedure.

Moreover, with the current trend of having many small projects to map the micro services architectures running on production, it becomes very important to improve the on-boarding experience for new hires or existing company personal, allowing to isolate arbitrarily complex build schemes from the peculiarities of personal computers.

Audience

Developers that work on different projects, probably based on different technologies, and that see containerization as a means to simplify their development setup effort but won't give up on having a quick feedback loop while developing.

Problem

Currently telepresence cannot directly proxy a Docker container which forces the developer down an uncomfortable path of modifying their containers so they run telepresence from inside Docker. Even if doable the UX is not great.

Subjectively it's hard to grasp why a tool specifically designed to simplify the development with container orchestrators doesn't offer great support for their atomic units. It certainly came as a surprise to me since like many others I've meet, we started working with containers locally before stepping into tools like Kubernetes.

Hypothesis

@itamarst proposed a clever approach to solve the issue at the lower level, yet without a great UX but something easy enough to mechanise if technically viable.

Building on top of Docker's support for sharing a network namespace among different containers (similar to Kubenertes' Pod concept), it becomes possible to isolate the telepresence VPN setup from the developer's container. This is a great solution since it not only hides the magic via UX, it actually offers guaranties that the proxying setup won't interfere with your own program.

Proof of concept

After putting together a quick test against a kubernetes cluster on AWS and minikube, the approach seems to work reliably and even solves some of the caveats with running the proxying directly on the host machine:

  • .local DNS resolution works properly since we can control the resolv configuration on the proxying container
  • minikube is also fully supported and shows no problems with DNS resolution
  • Cloud resources seem to be properly routed without requiring --also-proxy (haven't tested externalName yet but I don't see why it wouldn't work)
  • Multiple proxying containers can be launched on a single host
  • Proxying container requires some extra capabilities but not full --privileged mode
  • And obviously, the proxying doesn't interfere with other software running on the host :)

The solution is really simple and elegant, just requires a proper UX to make it useable at the same level as the other approaches.

Proposal

Some random thoughts about how this could be implemented:

  • Define a new --method named container since the caveats of vpn-tcp do not apply, so it'll be less confusing in the docs. Also it can come handy in the future if we can find a better vpn software (i.e. UDP support).
  • --run-docker [docker flags] <image> [program flags] seems a sensible UI being an alias for docker run *args. It would complain if --method is defined but it is not container. Also paves the way for a --run-rkt command.
  • Proxying container is launched (random name suffix?), cluster is updated, then the developer's container with the shared network and kube's config mount.
  • publish an optimized image with all the requirements for the proxying container (telepresence, kubectl, oc?)
  • telepresence will need to mount the $KUBECONFIG, taking into account that it can point to external files (i.e certs). Maybe we can use kubectl on the host to generate a materialized dump of a specific context?
  • workaround for core tools (i.e. ping) should not be used. They seem to work properly.
  • How to sync the life cycle of the proxying container and the developer's container? it's important to clean up properly.
  • This is more general but setting up and tearing down the VPN and volumes takes a bit of time. It'll be great if there is some trick to speed it up for repeated executions.

Note that while the proposal mechanizes the proof of concept approach, perhaps a more sensible solution would be to do all the kubectl interaction from the host, splitting the vpn setup code so it can be run on its own inside the container.

@itamarst

This comment has been minimized.

Contributor

itamarst commented Jun 13, 2017

Notes on what gets proxied:

  • Cloud names might work by accident, unrelated to Docker usage - telepresence guesses the IP ranges it should route to remote cluster, and if cloud resource happens to be in that range it'll work. But it won't always be.
  • I'm surprised ping works, too. Will investigate.
  • I'm worried that minikube will work some cases but not others, there's a whole bunch of different ways Docker networking can be setup, at a minimum Mac vs. Linux.

Notes on UX:

  • --method=container is good point... but then there's the argument coupling with --docker-run, where it's verbose and makes reading help harder. Given less caveats being its own method does seem to make sense.
  • Definitely want to publish an image, yes.

Notes on implementation:

  • Really only sshuttle needs to run inside the container, and it just needs access to the ssh tunnel. Getting the host's Docker-facing IP is annoying, but certainly possible (route | awk '/^default/ { print $2 }' inside container is best suggestion I've found so far, but need to test on Mac). That suggests container can just run sshuttle and no need to do kubectl inside the container.
@drslump

This comment has been minimized.

Contributor

drslump commented Jun 13, 2017

About the UX perhaps it should follow other tools that modify a program behaviour like xargs or nohup, taking all the arguments since the first non option and launching them, which should be backwards compatible with the current behaviour:

telepresence -n test   # runs default shell since there is no arguments given
telepresence -n test program  # runs program
telepresence -n test program --foo  # runs program --foo
telepresence -n test docker run -it busybox /bin/sh # runs docker -it busybox /bin/sh (warn?)
telepresence -n test -m container docker run -it busybox /bin/sh  # runs docker run --network container:xxx -it busybox /bin/sh

The container case is a bit weird since it'll have to modify the arguments to inject some stuff for docker but it removes the coupling between --method and --run which is indeed an issue.

Regarding what to run inside the container I think sshuttle, sshfs/fuse and ssh should run inside the container, this removes almost all requirements from the host system (perhaps even working on plain windows?) which is one of the core benefits of this approach.

@itamarst

This comment has been minimized.

Contributor

itamarst commented Jun 14, 2017

Oh, right, sshfs. That's more issues:

  • fuse is a kernel module which may not be available in the VM(?) that Docker uses... so sshfs may still not work.
  • After further thought: filesystems are not shared across containers. Having the telepresence container run sshfs doesn't help other containers that share network namespace... the only way to work is to do sshfs on parent and mount that in.

Plus, there's environment variables, which in fact was reason why I had telepresence run the other docker container.

That means we have to have Telepresence run the user container, because it needs to do environment variables and volumes and whatnot.

Current design plan, then:

  1. kubectl, sshfs and ssh run on host.
  2. Telepresence container runs sshuttle.
  3. telepresence starts the user's container for them, adding a number of options.
@itamarst

This comment has been minimized.

Contributor

itamarst commented Jun 14, 2017

UX options, where docker run arguments are -i -t alpine /bin/sh:

# Explicit, but end up with "--docker-run" that can only be used with "-m docker":
$ telepresence -m docker -n test --docker-run -i -t alpine /bin/sh

# Re-use --run, but means --run has inconsistent meaning:
$ telepresence -m docker -n test --run -i -t alpine /bin/sh

# Inconsistent with other means that require a --method:
$ telepresence -n test --docker-run -i -t alpine /bin/sh

# Positional args, but then positional args only apply to "-m docker":
$ telepresence -m docker -n test -- -i -t alpine /bin/sh

# New command allows for new UX:
$ telepresence-docker -n test -- -i -t alpine /bin/sh
$ telepresence-docker -n test --run -i -t alpine /bin/sh

@itamarst

This comment has been minimized.

Contributor

itamarst commented Jun 14, 2017

I guess I'll go with:

# Works:
$ telepresence -n test --docker-run -i -t alpine /bin/sh

# Works:
$ telepresence -m container -n test --docker-run -i -t alpine /bin/sh

# Complains that --docker-run requires "-m container":
$ telepresence -m vpn-tcp -n test --docker-run -i -t alpine /bin/sh
@itamarst

This comment has been minimized.

Contributor

itamarst commented Jun 14, 2017

Technical risks: mostly that sshuttle doesn't work with Docker somehow. Since @drslump verified this it's probably going to work, but will try it myself too to get and sense, and also see if I can reproduce some of the lack of limitations he described.

Also: exposing service on host to a container, since routing that depends on different ways Docker runs on different platforms.

@itamarst

This comment has been minimized.

Contributor

itamarst commented Jun 14, 2017

Testing for risks:

  1. Testing with setup @drslump posted on gitter. ping doesn't work, as expected. It probably worked for him because he has a host on his local network with same IP as the Service. In general, seems to work.
  2. On Linux I've verified this works. On Mac https://forums.docker.com/t/access-host-not-vm-from-inside-container/11747/22 suggests it's possible with something like sudo ifconfig lo0 alias 172.16.1.1. Other alternatives include listening on public IP (which would require extra work for security), or doing yet another ssh tunnel!

Update: looks like the mac workaround is actually in docs, so it's legit - https://docs.docker.com/docker-for-mac/networking/#per-container-ip-addressing-is-not-possible

@drslump

This comment has been minimized.

Contributor

drslump commented Jun 14, 2017

Some testing on my side:

  • Host: Mac OSX Yosemite
  • Docker For Mac 17.06.0-rc2-ce-mac14
  • Kubectl 1.6.4
  • Minikube 0.19.1 (kubernetes 1.6.4)

With this telepresence image:

# TODO: Base the image on Alpine so we generate a leaner image
FROM ubuntu:xenial

# Install some required base software
RUN apt-get -qq update  && apt-get -qq install -y curl sudo iptables

# Install kubectl
RUN curl -sLO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl && \
    chmod +x kubectl && \
    mv ./kubectl /usr/local/bin/kubectl

# Install telepresence
RUN curl -s https://packagecloud.io/install/repositories/datawireio/telepresence/script.deb.sh | bash && \
    apt-get -qq install -y --no-install-recommends telepresence

# HACK: allow to run against minikube
RUN apt-get -qq install sed && sed -ri 's/minikube/patched-minikube/' /usr/bin/telepresence

# Clean up a bit
RUN rm -rf /var/lib/apt/lists/*

ENTRYPOINT ["telepresence", "--method", "vpn-tcp"]

and launching with:

#!/bin/bash

kubecfg=${KUBECONFIG:-~/.kube/config}
eval kubecfg=$kubecfg  # expand home directory

minihome=${MINIKUBE_HOME:-~/.minikube}
eval minihome=$minihome

docker build -t telepresence-docker:latest .

docker run 													\
  --cap-add NET_ADMIN 				`# required to setup the VPN` 					\
  --cap-add SYS_ADMIN --device /dev/fuse  	`# required for sshfs mounts` 					\
  --security-opt seccomp=unconfined             `# only needed on Docker < 1.12`				\
  --rm --name telepresence-vpn 											\
  -v "$kubecfg:/root/.kube/config"   		`# expose the kubect configuration to the container` 		\
  -v "$minihome:$minihome"                	`# expose the minikube config to the container (if any)`	\
  -it  														\
  telepresence-docker:latest 											\
    --new-deployment test-telepresence-docker 									\
    --expose 8080 												\
    --run-shell

In the launched shell:

@minikube|bash-4.3# apt update && apt install inetutils-ping
...
@minikube|bash-4.3# /bin/ping google.com
PING google.com (216.58.210.142): 56 data bytes
64 bytes from 216.58.210.142: icmp_seq=0 ttl=37 time=0.238 ms
...
@minikube|bash-4.3# /bin/ping hello-minikube.default.svc.cluster.local
PING hello-minikube.default.svc.cluster.local (10.0.0.2): 56 data bytes
64 bytes from 10.0.0.2: icmp_seq=0 ttl=37 time=0.143 ms

Same for a remote kubernetes cluster on AWS. Also dig, traceroute, host and nslookup seem to work properly.


Regarding sshfs support, indeed the Docker For Mac virtual machine already includes /dev/fuse so sshfs works out of the box inside the container (with custom capabilities). I guess docker uses fuse for something internally. My thinking was to mount with sshfs inside the container and then use appropriate docker -v params to mount the telepresence container sshfs mounts to the host and then to the user container, something like [k8s] <-(sshfs)-> [telepresence] <-(docker)-> [host] <-(docker)-> [container].

It's a bit complex and unless it simplifies mounting on Windows it's probably better to just require sshfs on the host across all platforms and avoid bridging the mounts between both containers. Installing fuse+sshfs on Mac is simple enough nowadays.

@itamarst itamarst moved this from Next to In progress in Telepresence Jun 14, 2017

@itamarst

This comment has been minimized.

Contributor

itamarst commented Jun 15, 2017

Re ping:

It's possible things like ping would work on Mac, yes - sshuttle uses a different mechanism there than Linux where I tested. But it's also possible you're pinging 10.0.0.2 on your local network, rather than the Kubernetes network.

If you look at sshuttle docs you'll see it only captures TCP and DNS packets on OS X, though perhaps those docs are out of date: http://sshuttle.readthedocs.io/en/stable/requirements.html#client-side-requirements

So what may be happening is that for TCP it gets routed via sshuttle to Kubernetes, and for ICMP (ping) it gets routed to 10.0.0.2 on your local network. Just guessing, of course.

Re sshfs:

Even if it's there, I'm hesitant to rely on Docker VM continuing to provide fuse, since it's not part of their public API. So will stick to plan of doing it on host, for now.

@itamarst

This comment has been minimized.

Contributor

itamarst commented Jun 15, 2017

Starting work in docker-support branch.

@itamarst

This comment has been minimized.

Contributor

itamarst commented Jun 15, 2017

Tasks:

  • Build local Docker image
  • tag and push local Docker image as part of release
  • On mac, set IP on lo0 😢
  • Test for connecting to cluster
  • Test for env variables
  • Test for connection from cluster
  • Test for volumes
  • --method container complains when used with --run and --run-shell
  • --docker-run complains when used with --method that isn't docker
  • Environment variables passed to --docker-run via envfile
  • --docker-run uses network namespace of proxy container
  • --docker-run gets volumes via volume mount; make sure TELEPRESENCE_ROOT adjusted
  • figure out why unmounts fail (race condition?), fix it
  • Containers shut down on exit, notice when they exit (attach?)
  • Changelog
  • Howto for Docker
  • Remove docker mention from generic limitations
  • Describe Docker method in proxying methods, or at least mention
  • Optional: --method container can be omitted when --docker-run is used

@itamarst itamarst referenced this issue Jun 19, 2017

Merged

Docker support #181

itamarst added a commit that referenced this issue Jun 21, 2017

@itamarst itamarst moved this from In progress to Done in Telepresence Jun 21, 2017

@itamarst itamarst removed this from Done in Telepresence Jun 22, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment