Sshuttle not working over OpenVPN Tunnel

[craigwatson]

My scenario: I'm running OSX El Capitan, and sometimes work remotely and need to connect to my company's OpenVPN server in order to hook into ACLs and firewalls set up by our customers - this connection is set up to route all traffic over the VPN. I also need to use sshuttle to connect to remote networks behind bastion hosts in our customers' infrastructure.

When using sshuttle and OpenVPN individually, everything works as intended, however when using sshuttle after connecting to an OpenVPN tunnel, the firewall proxy rules don't take effect.

I'm aware that this could be due to OpenVPN or sshuttle, however colleagues running Linux can connect with no problem, so I suspect this is due to sshuttle using pf rather than iptables on Mac.

Can anyone help resolve this? Happy to provide more detail if necessary.

[vieira]

Hello @craigwatson,

I have an idea about what might be causing this issue, but I don't have OpenVPN to test right now on OSX. When you say that the rules don't take effect are you referring to the rules installed by OpenVPN or by sshuttle? Does the VPN connection still work after running sshuttle, i.e., are you able to reach, e.g, the bastion hosts?

[craigwatson]

Hi @vieira,

The VPN connection works, and I can SSH into the bastion hosts manually, and sshuttle also reports connected, but traffic that should be forwarded via sshuttle isn't.

When I disconnect from the OpenVPN and run sshuttle to a network directly (e.g. a bastion that already has my public IP address whitelisted), traffic is routed correctly via the remote SSH host.

Hope that's clear enough :)

[vieira]

Ah I see, thanks. That was not what I was expecting. I will take a look.

On 23 Jun 2016, at 16:03, Craig Watson notifications@github.com wrote:

Hi @vieira,

The VPN connection works, and I can SSH into the bastion hosts manually, and sshuttle also reports connected, but traffic that should be forwarded via sshuttle isn't.

When I disconnect from the OpenVPN and run sshuttle to a network directly, traffic is routed correctly via the remote SSH host.

Hope that's clear enough :)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[vieira]

Hi again @craigwatson,

Could you please post the output of (translation rules and filter rules is enough, I think) sudo pfctl -sa with only OpenVPN running and then with both OpenVPN and sshuttle running?

Thanks in advance.

[craigwatson]

No problem.

Only OpenVPN:

craig@zeus:~$ sudo pfctl -sa
Password:
No ALTQ support in kernel
ALTQ related functions disabled
TRANSLATION RULES:
rdr-anchor "sshuttle" all

FILTER RULES:
pass on lo all flags S/SA keep state
anchor "sshuttle" all

INFO:
Status: Disabled for 0 days 01:32:48          Debug: Urgent

State Table                          Total             Rate
  current entries                        0
  searches                         1462101          262.6/s
  inserts                             1333            0.2/s
  removals                            1333            0.2/s
Counters
  match                            1062202          190.8/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                            111            0.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                      1331            0.2/s
  state-insert                          34            0.0/s
  state-limit                            0            0.0/s
  src-limit                              0            0.0/s
  synproxy                               0            0.0/s
  dummynet                               0            0.0/s

TIMEOUTS:
tcp.first                   120s
tcp.opening                  30s
tcp.established           86400s
tcp.closing                 900s
tcp.finwait                  45s
tcp.closed                   90s
tcp.tsdiff                   30s
udp.first                    60s
udp.single                   30s
udp.multiple                 60s
icmp.first                   20s
icmp.error                   10s
grev1.first                 120s
grev1.initiating             30s
grev1.estblished           1800s
esp.first                   120s
esp.estblished              900s
other.first                  60s
other.single                 30s
other.multiple               60s
frag                         30s
interval                     10s
adaptive.start             6000 states
adaptive.end              12000 states
src.track                     0s

LIMITS:
states        hard limit    10000
app-states    hard limit    10000
src-nodes     hard limit    10000
frags         hard limit     5000
tables        hard limit     1000
table-entries hard limit   200000

OS FINGERPRINTS:
696 fingerprints loaded

With OpenVPN and sshuttle:

craig@zeus:~$ sudo pfctl -sa
No ALTQ support in kernel
ALTQ related functions disabled
TRANSLATION RULES:
rdr-anchor "sshuttle" all

FILTER RULES:
pass on lo all flags S/SA keep state
anchor "sshuttle" all

INFO:
Status: Enabled for 0 days 00:00:03           Debug: Urgent

State Table                          Total             Rate
  current entries                        0
  searches                         1462539       487513.0/s
  inserts                             1333          444.3/s
  removals                            1333          444.3/s
Counters
  match                            1062402       354134.0/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                            111           37.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                      1331          443.7/s
  state-insert                          34           11.3/s
  state-limit                            0            0.0/s
  src-limit                              0            0.0/s
  synproxy                               0            0.0/s
  dummynet                               0            0.0/s

TIMEOUTS:
tcp.first                   120s
tcp.opening                  30s
tcp.established           86400s
tcp.closing                 900s
tcp.finwait                  45s
tcp.closed                   90s
tcp.tsdiff                   30s
udp.first                    60s
udp.single                   30s
udp.multiple                 60s
icmp.first                   20s
icmp.error                   10s
grev1.first                 120s
grev1.initiating             30s
grev1.estblished           1800s
esp.first                   120s
esp.estblished              900s
other.first                  60s
other.single                 30s
other.multiple               60s
frag                         30s
interval                     10s
adaptive.start             6000 states
adaptive.end              12000 states
src.track                     0s

LIMITS:
states        hard limit    10000
app-states    hard limit    10000
src-nodes     hard limit    10000
frags         hard limit     5000
tables        hard limit     1000
table-entries hard limit   200000

TABLES:

OS FINGERPRINTS:
696 fingerprints loaded

[vieira]

@craigwatson Thanks a lot. I thought some skip lo might have ended up on the chain, but that doesn't seem to be the case. 😞
With OpenVPN running, could you please run sshuttle with -vvv and then execute pfctl -a sshuttle -sr and share both outputs, just to be sure our rules are there?

[vieira]

Another thing I would like to check is if the range that is going through OpenVPN contains the subnets that you are trying to rdr with sshuttle. If that's the case it might explain the problem. netstat -nr (with the VPN running) should show your routing tables, including the ranges that are being routed through OpenVPN's tun.

[craigwatson]

Hi @vieira,

Here we go - all commands run with the OpenVPN connection active.

The IP address range routed via sshuttle is 10.10.216.0/22, and the OpenVPN tunnel should be routing all traffic.

craig@zeus:~$ sshuttle -vvv -r craig@x.x.x.x 10.10.216.0/22
Starting sshuttle proxy.
firewall manager: Starting firewall with Python version 2.7.10
firewall manager: ready method name pf.
IPv6 enabled: False
UDP enabled: False
DNS enabled: False
Binding redirector: 12300
TCP redirector listening on ('127.0.0.1', 12300).
TCP redirector listening with <socket._socketobject object at 0x1083a0130>.
Starting client with Python version 2.7.10
c : connecting to server...
c : executing: ['ssh', 'craig@x.x.x.x', '--', 'exec /bin/sh -c \'P=python3.5; $P -V 2>/dev/null || P=python; exec "$P" -c \'"\'"\'import sys, os; verbosity=3; sys.stdin = os.fdopen(0, "rb"); exec(compile(sys.stdin.read(958), "assembler.py", "exec"))\'"\'"\'\'']
c :  > channel=0 cmd=PING len=7 (fullness=0)
server: assembling u'sshuttle' (8 bytes)
server: assembling u'sshuttle.cmdline_options' (27 bytes)
server: assembling u'sshuttle.helpers' (950 bytes)
server: assembling u'sshuttle.ssnet' (5540 bytes)
server: assembling u'sshuttle.hostwatch' (2364 bytes)
server: assembling u'sshuttle.server' (3091 bytes)
Starting server with Python version 2.6.6
 s: latency control setting = True
 s: available routes:
 s:   2/10.10.217.160/28
 s:   2/10.10.217.160/28
 s:  > channel=0 cmd=PING len=7 (fullness=0)
 s:  > channel=0 cmd=ROUTES len=38 (fullness=7)
 s: Waiting: 1 r=[4] w=[5] x=[] (fullness=45/0)
 s:   Ready: 1 r=[] w=[5] x=[]
 s: mux wrote: 15/15
 s: Waiting: 1 r=[4] w=[5] x=[] (fullness=45/0)
 s:   Ready: 1 r=[] w=[5] x=[]
 s: mux wrote: 46/46
 s: Waiting: 1 r=[4] w=[] x=[] (fullness=45/0)
c : Connected.
c : Waiting: 2 r=[4, 8] w=[8] x=[] (fullness=7/0)
c :   Ready: 2 r=[8] w=[8] x=[]
c : <  channel=0 cmd=PING len=7
c :  > channel=0 cmd=PONG len=7 (fullness=7)
c : <  channel=0 cmd=ROUTES len=38
firewall manager: Got subnets: [(2, 22, False, '10.10.216.0'), (2, 8, True, '127.0.0.1')]
firewall manager: Got nslist: []
firewall manager: Got ports: 0,12300,0,0
firewall manager: Got udp: False
firewall manager: setting up.
firewall manager: setting up IPv4.
>> pfctl -f /dev/stdin
>> pfctl -s all
rules:
---> table <forward_subnets> {10.10.216.0/22,!127.0.0.1/8}
---> rdr pass on lo0 proto tcp to <forward_subnets> -> 127.0.0.1 port 12300
---> pass out route-to lo0 inet proto tcp to <forward_subnets> keep state
>> pfctl -a sshuttle -f /dev/stdin
>> pfctl -E
c : mux wrote: 15/15
c : mux wrote: 15/15
c : Waiting: 2 r=[4, 8] w=[] x=[] (fullness=14/0)
 s:   Ready: 1 r=[4] w=[] x=[]
 s: <  channel=0 cmd=PING len=7
 s:  > channel=0 cmd=PONG len=7 (fullness=45)
 s: <  channel=0 cmd=PONG len=7
 s: received PING response
c :   Ready: 2 r=[8] w=[] x=[]
c : <  channel=0 cmd=PONG len=7
c : received PING response
 s: mux wrote: 15/15
 s: Waiting: 1 r=[4] w=[] x=[] (fullness=0/0)
c : Waiting: 2 r=[4, 8] w=[] x=[] (fullness=0/0)

craig@zeus:~$ sudo pfctl -a sshuttle -sr
No ALTQ support in kernel
ALTQ related functions disabled
pass out route-to lo0 inet proto tcp from any to <forward_subnets> flags S/SA keep state

craig@zeus:~$ sudo netstat -nr
Routing tables

Internet:
Destination        Gateway            Flags        Refs      Use   Netif Expire
0/1                10.200.200.93      UGSc           15        0   utun3
default            192.168.1.254      UGSc           16        0     en0
10.200.200.1/32    10.200.200.93      UGSc            2        0   utun3
10.200.200.93      10.200.200.94      UHr            34       18   utun3
x.x.x.x/32         192.168.1.254      UGSc            3        0     en0
127                127.0.0.1          UCS             2        0     lo0
127.0.0.1          127.0.0.1          UH             15    88359     lo0
127.94.0.1         127.94.0.1         UH              2        0     lo0
127.94.0.2         127.94.0.2         UH              2        0     lo0
128.0/1            10.200.200.93      UGSc           16        0   utun3
169.254            link#5             UCS             2        0     en0
192.168.1          link#5             UCS             6        0     en0
192.168.1.74/32    link#5             UCS             3        0     en0
192.168.1.74       6c:40:8:b0:f5:58   UHLWIi          2        8     lo0
192.168.1.117      70:56:81:f1:22:73  UHLWIi          5     1098     en0    856
192.168.1.122      64:9a:be:8f:65:8a  UHLWIi          2      300     en0    651
192.168.1.212      50:7a:55:9c:52:c6  UHLWIi          2      244     en0    101
192.168.1.254/32   link#5             UCS             3        0     en0
192.168.1.254      5c:dc:96:37:e5:46  UHLWIir         7     1264     en0   1196
192.168.1.255      link#5             UHLWbI          2      280     en0
192.168.192/22     10.200.200.93      UGSc            2        0   utun3
192.168.196/22     10.200.200.93      UGSc            2        0   utun3
224.0.0            link#5             UmCS            3        0     en0
224.0.0.251        1:0:5e:0:0:fb      UHmLWI          2        0     en0
255.255.255.255/32 link#5             UCS             3        0     en0
255.255.255.255    link#5             UHLWbI          2      189     en0

For comparison, here is output from sshuttle outside of OpenVPN:

craig@zeus:~$ sshuttle -vvv -r craig@bastion.example.com 172.20.8.0/22
Starting sshuttle proxy.
firewall manager: Starting firewall with Python version 2.7.10
firewall manager: ready method name pf.
IPv6 enabled: False
UDP enabled: False
DNS enabled: False
Binding redirector: 12300
TCP redirector listening on ('127.0.0.1', 12300).
TCP redirector listening with <socket._socketobject object at 0x10aec4130>.
Starting client with Python version 2.7.10
c : connecting to server...
c : executing: ['ssh', 'craig@bastion.example.com', '--', 'exec /bin/sh -c \'P=python3.5; $P -V 2>/dev/null || P=python; exec "$P" -c \'"\'"\'import sys, os; verbosity=3; sys.stdin = os.fdopen(0, "rb"); exec(compile(sys.stdin.read(958), "assembler.py", "exec"))\'"\'"\'\'']
c :  > channel=0 cmd=PING len=7 (fullness=0)
server: assembling u'sshuttle' (8 bytes)
server: assembling u'sshuttle.cmdline_options' (27 bytes)
server: assembling u'sshuttle.helpers' (950 bytes)
server: assembling u'sshuttle.ssnet' (5540 bytes)
server: assembling u'sshuttle.hostwatch' (2364 bytes)
server: assembling u'sshuttle.server' (3091 bytes)
Starting server with Python version 2.7.5
 s: latency control setting = True
 s: available routes:
 s:   2/172.20.8.0/25
 s:  > channel=0 cmd=PING len=7 (fullness=0)
 s:  > channel=0 cmd=ROUTES len=16 (fullness=7)
 s: Waiting: 1 r=[4] w=[5] x=[] (fullness=23/0)
 s:   Ready: 1 r=[] w=[5] x=[]
 s: mux wrote: 15/15
 s: Waiting: 1 r=[4] w=[5] x=[] (fullness=23/0)
 s:   Ready: 1 r=[] w=[5] x=[]
 s: mux wrote: 24/24
 s: Waiting: 1 r=[4] w=[] x=[] (fullness=23/0)
c : Connected.
c : Waiting: 2 r=[4, 8] w=[8] x=[] (fullness=7/0)
c :   Ready: 2 r=[8] w=[8] x=[]
c : <  channel=0 cmd=PING len=7
c :  > channel=0 cmd=PONG len=7 (fullness=7)
c : <  channel=0 cmd=ROUTES len=16
firewall manager: Got subnets: [(2, 22, False, '172.20.8.0'), (2, 8, True, '127.0.0.1')]
firewall manager: Got nslist: []
firewall manager: Got ports: 0,12300,0,0
firewall manager: Got udp: False
firewall manager: setting up.
firewall manager: setting up IPv4.
>> pfctl -f /dev/stdin
>> pfctl -s all
rules:
---> table <forward_subnets> {172.20.8.0/22,!127.0.0.1/8}
---> rdr pass on lo0 proto tcp to <forward_subnets> -> 127.0.0.1 port 12300
---> pass out route-to lo0 inet proto tcp to <forward_subnets> keep state
>> pfctl -a sshuttle -f /dev/stdin
>> pfctl -E
c : mux wrote: 15/15
c : mux wrote: 15/15
c : Waiting: 2 r=[4, 8] w=[] x=[] (fullness=14/0)
 s:   Ready: 1 r=[4] w=[] x=[]
 s: <  channel=0 cmd=PING len=7
 s:  > channel=0 cmd=PONG len=7 (fullness=23)
 s: mux wrote: 15/15
 s: Waiting: 1 r=[4] w=[] x=[] (fullness=30/0)
 s:   Ready: 1 r=[4] w=[] x=[]
 s: <  channel=0 cmd=PONG len=7
 s: received PING response
 s: Waiting: 1 r=[4] w=[] x=[] (fullness=0/0)
c :   Ready: 2 r=[8] w=[] x=[]
c : <  channel=0 cmd=PONG len=7
c : received PING response
c : Waiting: 2 r=[4, 8] w=[] x=[] (fullness=0/0)

[vieira]

Hi @craigwatson

0/1 together with 128.0/1 will route all your traffic through 10.200.200.93/utun3 unless there is a more specific route (OpenVPN installs these to effectively set your default route without having to explicitly change/override it).

For 10.10.216.0/22 it seems to me there isn't, so the traffic is getting routed through the OpenVPN tunnel and the pf rule that would route it to lo0 does not seem to be executed, although I don't understand why yet.

Can you try to run OpenVPN, then run route add -host 10.10.216.0/22 -interface lo0 and only after launch sshuttle? In the mean time I will set up OpenVPN on some OSX machine to have a look.

Thanks!

[craigwatson]

Perfect, running sudo route add -net 10.10.216.0/22 -interface lo0 (rather than -host) works!

Unsure what the next steps are - would it be beyond the scope of sshuttle to add lo0 routing for the remote subnets on OSX only?

[vieira]

Good to know. 👍 I installed OpenVPN on OpenBSD and configured all traffic to go through it, then launched sshuttle and it worked without problems, so it is possible that there is some bug or weird behavior in darwin's implementation of pf.

I will test on FreeBSD to check if it behaves the same as OSX.

If I am able to get a better understanding of what is actually going on maybe I can slightly tweak the rules to avoid it. If it turns out that the only way is adding routes (and if we add them we will also have to write code to remove on exit)... I don't know if that's something we should do, we will have to ask @brianmay.

[vieira]

Working as expected on FreeBSD. This problem seems to be specific do darwin's implementation of pf.

[craigwatson]

Interesting, thanks for checking this out. Have you been able to reproduce on OSX?

[vieira]

Hi,

Yes, I was able to reproduce the problem exactly as reported by you on OSX using both Tunnelblick and Viscosity OpenVPN clients. The problem is probably on darwin's implementation of pf.

On 26 Jun 2016, at 18:26, Craig Watson notifications@github.com wrote:

Interesting, thanks for checking this out. Have you been able to reproduce on OSX?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[vieira]

So, pf route-to should bypass the routing table and route the packet through the interface specified in the rule. FreeBSD and OpenBSD behave as expected but in OSX the routing table is used and the route-to rule is ignored.

Opened bug 27061884 with Apple.

Can be reproduced without sshuttle or openvpn with:

# Add a route for a subnet <A> through some interface (other than lo0)
sudo route add -net <A> -interface en0
# add the following pf rules, where <B> is a subnet in <A>
sudo pfctl -e
sudo pfctl -f /dev/stdin
table <forward_subnets> {<B>,!127.0.0.1/8}
rdr pass on lo0 proto tcp to <forward_subnets> -> 127.0.0.1 port 12300
pass out route-to lo0 inet proto tcp to <forward_subnets> keep state

nc -l 12300
curl -I <B> -vvv
# nothing gets routed to nc