add --without-masquerade option to weave expose

[aaronhurt]

Add the ability to skip creation of MASQUERADE rules via iptables when executing the weave expose command.

[bboreham]

Looks great, thanks!

Couple of comments below.

[aaronhurt]

Updated to address comments. Let me know if there is anything else.

[brb]

@leprechau Thanks for your contribution! Mind explaining why someone would want to skip the creation of the NAT rules?

Also, noting that we already have a similar flag, please see: #3298.

[aaronhurt]

We’re using weave in a standard Docker environment without k8s and want external systems (not part of the weave network) to see the container IP and not the IP of the host.

In our specific case we are using a helper written in-house (https://github.com/myENA/aardvark) to publish container routes and update the default route within the container when it starts.

[brb]

@leprechau Wouldn't --no-masq-local from #3298 be enough for your case?

[aaronhurt]

@brb Trying now ... I see the flag in prog/weaver/main.go but didn't see it listed in the actual weave help usage. If it works I'll send a much smaller PR adding the option to the usage docs.

[aaronhurt]

So, the --no-masq-local does not seem to work for my particular case.

What I did ...

1. Reset weave via weave reset on all test nodes
2. Restarted weave

weave launch --password xxxxxx --no-detect-tls --ipalloc-range 10.156.8.0/22 --no-dns --without-dns --proxy=false --no-masq-local

3. Created a new docker network using the weave driver

docker network create -d weave --subnet=10.156.8.0/22 --gateway 10.156.8.1 app

4. Started a test container on the app network

docker run -it --net app alpine

5. Checking inside the container

/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
148: ethwe0@if149: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1376 qdisc noqueue state UP 
    link/ether 8e:ce:c7:e4:90:7a brd ff:ff:ff:ff:ff:ff
    inet 10.156.8.2/22 scope global ethwe0
       valid_lft forever preferred_lft forever
    inet6 fe80::8cce:c7ff:fee4:907a/64 scope link 
       valid_lft forever preferred_lft forever
150: eth1@if151: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.2/16 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe12:2/64 scope link 
       valid_lft forever preferred_lft forever
/ # ip route
default via 10.156.8.183 dev ethwe0 
10.156.8.0/22 dev ethwe0 scope link  src 10.156.8.2 
172.18.0.0/16 dev eth1 scope link  src 172.18.0.2 
/ # 

6. Host WEAVE rules

[root@dkr10.dev ~]# iptables-save|grep WEAVE
:WEAVE-IPSEC-IN - [0:0]
:WEAVE-IPSEC-IN-MARK - [0:0]
:WEAVE-IPSEC-OUT - [0:0]
:WEAVE-IPSEC-OUT-MARK - [0:0]
-A INPUT -j WEAVE-IPSEC-IN
-A OUTPUT -j WEAVE-IPSEC-OUT
-A WEAVE-IPSEC-IN -s 10.156.0.127/32 -d 10.156.0.183/32 -p esp -m esp --espspi 612773799 -j WEAVE-IPSEC-IN-MARK
-A WEAVE-IPSEC-IN -s 10.156.0.136/32 -d 10.156.0.183/32 -p esp -m esp --espspi 3375555300 -j WEAVE-IPSEC-IN-MARK
-A WEAVE-IPSEC-IN-MARK -j MARK --set-xmark 0x20000/0x20000
-A WEAVE-IPSEC-OUT -s 10.156.0.183/32 -d 10.156.0.127/32 -p udp -m udp --dport 6784 -j WEAVE-IPSEC-OUT-MARK
-A WEAVE-IPSEC-OUT -s 10.156.0.183/32 -d 10.156.0.136/32 -p udp -m udp --dport 6784 -j WEAVE-IPSEC-OUT-MARK
-A WEAVE-IPSEC-OUT-MARK -j MARK --set-xmark 0x20000/0x20000
:WEAVE - [0:0]
-A POSTROUTING -j WEAVE
-A WEAVE -m set --match-set weaver-no-masq-local dst -m comment --comment "Prevent SNAT to locally running containers" -j RETURN
-A WEAVE -s 10.156.8.0/22 -d 224.0.0.0/4 -j RETURN
-A WEAVE ! -s 10.156.8.0/22 -d 10.156.8.0/22 -j MASQUERADE
-A WEAVE -s 10.156.8.0/22 ! -d 10.156.8.0/22 -j MASQUERADE
:WEAVE-EXPOSE - [0:0]
:WEAVE-IPSEC-IN - [0:0]
-A INPUT -j WEAVE-IPSEC-IN
-A FORWARD -o weave -j WEAVE-EXPOSE
-A WEAVE-EXPOSE -d 10.156.8.0/22 -j ACCEPT
-A WEAVE-IPSEC-IN -s 10.156.0.127/32 -d 10.156.0.183/32 -p udp -m udp --dport 6784 -m mark ! --mark 0x20000/0x20000 -j DROP
-A WEAVE-IPSEC-IN -s 10.156.0.136/32 -d 10.156.0.183/32 -p udp -m udp --dport 6784 -m mark ! --mark 0x20000/0x20000 -j DROP

Result ... traffic leaving the container 10.156.8.2 still shows up outside of the weave network as being NAT'd through the host 10.156.0.183

If I remove the MASQ rules via ...

for rule in $(iptables -t nat -L WEAVE --line-numbers | awk '/MASQUERADE /{print $1}' | sort -rn); do
  iptables -t nat -D WEAVE ${rule}
done

The traffic shows on external hosts as coming from the container and not from the host.

[aaronhurt]

@brb Am I missing something in how the --no-masq-local option is supposed to work? The default route in my container is the weave bridge but the traffic is still NAT'd through the host unless I completely remove the MASQ rules.

[brb]

@leprechau

Result ... traffic leaving the container 10.156.8.2 still shows up outside of the weave network as being NAT'd through the host 10.156.0.183

OK, I misread your requirement - I thought that you want external systems to access containers w/o NAT, this is what the --no-masql-local flag for. Unfortunately, "traffic leaving the container" (= "traffic originating from the container") will be NAT'ed.

[aaronhurt]

@brb yes, I'm wanting hosts outside of the weave network to see traffic that originates from a container on the weave network to see the weave assigned IP address of the container.

[bboreham]

Thanks!

[aaronhurt]

@bboreham @brb Awesome, thank you both as well!