New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VPN based communication method as alternative to LD_PRELOAD #128

Closed
itamarst opened this Issue May 1, 2017 · 11 comments

Comments

Projects
None yet
1 participant
@itamarst
Contributor

itamarst commented May 1, 2017

As an alternative to our current method of proxying we could also use a VPN of some sort.

Positive:

  • Works with all programs and all languages.
  • UDP support

Negative:

  • On macOS (lacking Linux network namespaces) at least this would impact the whole operating system, as far as I can tell, and can't be isolated to a particular process.
  • Typically requires root on both ends. Root on Kubernetes side means this won't work with OpenShift.

Probably want this as an option, not a replacement for current method.

Please upvote or comment if this interests you.

@itamarst itamarst added the enhancement label May 1, 2017

@itamarst itamarst added this to Next in Telepresence May 3, 2017

@itamarst

This comment has been minimized.

Contributor

itamarst commented May 5, 2017

Some VPN options:

  1. OpenVPN. Popular, but annoying to setup.
  2. wireguard. Not clear it works on OS X.
  3. ssh-based vpn, e.g. using https://github.com/halhen/pvpn
  4. http://sshuttle.readthedocs.io, which seems popular and possibly even easy to use. sshuttle would still not support UDP on OS X, but may be still be better than LD_PRELOAD. So this may be a half-way solution.

Some of these have DNS support.

I will start by trying sshuttle.

@itamarst

This comment has been minimized.

Contributor

itamarst commented May 8, 2017

I am having issues with sshuttle. In particular, when testing against minikube:

  1. Method tproxy doesn't work at all (this may be bug in Ubuntu 16.04 kernel, based on sshuttle issue tracker).
  2. Method nat proxies IP connections (I can do curl https://10.0.0.1), and allows me to do host lookups using host kubernetes.default.svc.cluster.local, but curl to DNS hostname (curl https://kubernetes.default.svc.cluster.local).

@itamarst itamarst self-assigned this May 8, 2017

@itamarst

This comment has been minimized.

Contributor

itamarst commented May 8, 2017

So far failed to get pvpn to work at all. So might return to sshuttle and see if I can get DNS working one way or another.

@itamarst

This comment has been minimized.

Contributor

itamarst commented May 10, 2017

Made some progress with sshuttle.

In NAT mode, sshuttle makes DNS work by capturing packets to port 53 that are directed to nameservers listed in /etc/resolv.conf, and forwarding them via SSH tunnel.

If you use host or nslookup for "kubernetes.default.svc.cluster.local" this works just fine.

If you use curl, though, you're going through glibc's NSS subsystem.On Ubuntu nsswitch.conf has mdns configured, and that captures .local domain lookups because .local is only supposed to be for mDNS and k8s is abusing that.

Disabling mdns allows DNS through sshuttle, with the remaining caveat that kubernetes.default.svc.cluster.local works but kubernetes doesn't because of search path issues. But those can be solved with e.g. custom resolv.conf.

@itamarst

This comment has been minimized.

Contributor

itamarst commented May 10, 2017

Solution, on Linux:

  1. unshare --mount.
  2. Inside resulting mount namespace, sudo bind mount a new nsswitch.conf over /etc/nsswitch.conf, run subprocess there.

Unclear if this problem happens on OS X, will try tomorrow, but it probably does. And it's not clear how and if you can disable mdns on OS X at all.

@itamarst

This comment has been minimized.

Contributor

itamarst commented May 10, 2017

We might need custom DNS server for sshuttle (or VPN approaches) running inside the remote pod in order for this approach to work on OS X, since overriding DNS resolution will be tricky otherwise.

@itamarst

This comment has been minimized.

Contributor

itamarst commented May 11, 2017

Getting sshuttle working requires:

  • Implement custom DNS server that does gethostbyname for A records and forwards all others.
  • Run it on port > 1024.
  • Modify sshuttle (fork) to connect to custom host, port in sshuttle.server.DNSProxy. Submit PR upstream in hopes they include it.
  • Including (forked) sshuttle in our packages, probably in a private path.

@itamarst itamarst moved this from Next to In progress in Telepresence May 12, 2017

@itamarst

This comment has been minimized.

Contributor

itamarst commented May 16, 2017

sshuttle had reliability issues, unfortunately. And third parties have indicated this is actually a common problem.

I've tried another approach that looks workable: SSH VPN tunnel using tun devices. Resources:

On Linux enabled PermitTunnel=yes in sshd.

Then on OS X:

mac$ sudo ssh -o Tunnel=point-to-point -w 0:0 root@linux

and then

root@mac$ ifconfig tun0 inet 192.168.254.2 192.168.254.1 netmask 255.255.255.0
root@mac$ networksetup -setdnsservers Wi-Fi 192.168.254.1  # really should be DNS server I wrote for sshuttle, see above
root@mac$ route delete default
root@mac$ route add default 192.168.254.1

On Linux after the above:

root@linux$ sysctl -w net.ipv4.ip_forward=1
root@linux$ iptables -t nat -A POSTROUTING -s 192.168.254.0/8 -o wlp4s0 -j MASQUERADE
root@linux$ ifconfig tun0 192.168.254.1 pointopoint 192.168.254.2 netmask 255.255.255.0

And then IP was routed via Linux, and DNS queries sent to Linux server.

@itamarst

This comment has been minimized.

Contributor

itamarst commented May 16, 2017

For Linux we can override /etc/resolv.conf using a bind mount inside a mount namespace (created via unshare).

So we'd need code to implement the above (and cleanup) on Linux and OS X clients, a remote Docker image capable of being the proxy, and running the DNS server originally written for sshuttle variant. And basically what we'd have is real VPN into k8s: all traffic would be routed there.

Risks:

  • Traffic back to client. Can iptables actually work in Docker container? Probably given correct capabilities.
  • Telepresence crashes, breaks system. Reboots won't always help, e.g. networksetup -setdnsservers does not get reset on reboot!

@itamarst itamarst moved this from In progress to Next in Telepresence May 23, 2017

@itamarst

This comment has been minimized.

Contributor

itamarst commented May 30, 2017

Swithcing back to sshuttle for a while, just to see if I can figure out what its problems are.

@itamarst

This comment has been minimized.

Contributor

itamarst commented May 30, 2017

sshuttle does actually work, mostly, so continuing with it.

Current known caveats to sshuttle, which need to be in docs:

  1. Can't use minikube (or minishift, or other ways of running k8s on same machine) due to infinite DNS looping for non-k8s domains. Perhaps should be checked by code?
  2. Full .local domains don't work on Linux (and probably OS X), because of mDNS. Can be fixed on Linux, perhaps as follow-up?
  3. Should only one run one at a time. Perhaps should be checked by code?

Help messages and interactive messages should also note limitations on startup.

@itamarst itamarst moved this from Next to In progress in Telepresence Jun 1, 2017

@itamarst itamarst closed this in 36d7f64 Jun 7, 2017

@itamarst itamarst moved this from In progress to Done in Telepresence Jun 7, 2017

@itamarst itamarst removed this from Done in Telepresence Jun 22, 2017

@plombardi89 plombardi89 added this to Bugs in Roadmap Feb 21, 2018

@plombardi89 plombardi89 removed this from Bugs in Roadmap Feb 21, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment