Help, vpn peer usage in nat environment. #13

huapox · 2019-08-13T09:41:43Z

I've follow README.MD, docs/vpn.md for the following settings.

vpn-sam.yml and the node's routes

apiVersion: kilo.squat.ai/v1alpha1
kind: Peer
metadata:
  name: sam
spec:
  allowedIPs:
  - 10.5.0.1/32 # Example IP address on the peer's interface.
  publicKey: FLS------hzpNFbJ/JUiN4He8pTxLmFC5ZtQLK5Oc0A= #- replace 6 char
  persistentKeepalive: 10

[root@ali-vm1 ~]# route -n |grep kilo
7.0.1.0         10.4.0.2        255.255.255.0   UG    0      0        0 kilo0
10.4.0.0        0.0.0.0         255.255.0.0     U     0      0        0 kilo0
10.5.0.1        0.0.0.0         255.255.255.255 UH    0      0        0 kilo0
192.168.0.105   10.4.0.2        255.255.255.255 UGH   0      0        0 kilo0

[root@hw-vm1 ~]# route -n |grep kilo
7.0.0.0         10.4.0.1        255.255.255.0   UG    0      0        0 kilo0
10.4.0.0        0.0.0.0         255.255.0.0     U     0      0        0 kilo0
10.5.0.1        0.0.0.0         255.255.255.255 UH    0      0        0 kilo0
172.16.168.255  10.4.0.1        255.255.255.255 UGH   0      0        0 kilo0

my nat working-vm: (debian 10)

root@deb10:/home/sam# lsmod |grep wire
wireguard             221184  0
ip6_udp_tunnel         16384  2 wireguard,vxlan
udp_tunnel             16384  2 wireguard,vxlan
root@deb10:/home/sam# ip a |grep wg
4455: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 10.5.0.1/32 scope global wg0

root@deb10:/home/sam# route -n |grep wg
7.0.1.0         0.0.0.0         255.255.255.0   U     0      0        0 wg0
10.4.0.1        0.0.0.0         255.255.255.255 UH    0      0        0 wg0
10.4.0.2        0.0.0.0         255.255.255.255 UH    0      0        0 wg0
172.16.168.255  0.0.0.0         255.255.255.255 UH    0      0        0 wg0
192.168.0.105   0.0.0.0         255.255.255.255 UH    0      0        0 wg0

dev wg0:  (ListenPort = 5555, nofirewall run in deb10)
root@deb10:/home/sam# cat /etc/wireguard/wg0.conf 
[Interface]
Address = 10.5.0.1/32
PrivateKey = +Dsm------FVL3e83lTIVC9dI1rYwjEI7ljI9wbyFWk=  #replace 6 char
ListenPort = 5555

peer.ini:
[root@(⎈ |default:default) ~]$ kgctl showconf peer sam
[Peer]
AllowedIPs = 7.0.0.0/24, 172.16.168.255/32, 10.4.0.1/32
Endpoint = 47.98.xxx.xxx:51820
PersistentKeepalive = 0
PublicKey = nOW------dKxE0NDuCxN1GnXXz+0UiseSOYOrq14Nz4=

[Peer]
AllowedIPs = 7.0.1.0/24, 192.168.0.105/32, 10.4.0.2/32
Endpoint = 139.159.xxx.xxx:51820
PersistentKeepalive = 0
PublicKey = fQz------H70oWHUWzSGiMZJdH9wzq9eKGogDO9fWmc=

IFACE=wg0
wg-quick up $IFACE
wg setconf $IFACE peer.ini

ip route add 10.4.0.2/32 dev wg0
...

The text was updated successfully, but these errors were encountered:

huapox · 2019-08-13T09:43:00Z

test conn: fails both side

[root@ali-vm1 ~]# ping 10.5.0.1
PING 10.5.0.1 (10.5.0.1) 56(84) bytes of data.
From 10.4.0.1 icmp_seq=1 Destination Host Unreachable
[root@hw-vm1 ~]# ping 10.5.0.1
PING 10.5.0.1 (10.5.0.1) 56(84) bytes of data.
From 10.4.0.2 icmp_seq=1 Destination Host Unreachable
From 10.4.0.2 icmp_seq=1 Destination Host Unreachable

root@deb10:/home/sam# ping 10.4.0.2
PING 10.4.0.2 (10.4.0.2) 56(84) bytes of data.
^C
--- 10.4.0.2 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 38ms

root@deb10:/home/sam# ping 10.4.0.1
PING 10.4.0.1 (10.4.0.1) 56(84) bytes of data.
^C
--- 10.4.0.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 54ms

root@deb10:/home/sam# ping 192.168.0.105
PING 192.168.0.105 (192.168.0.105) 56(84) bytes of data.
^C
--- 192.168.0.105 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 27ms

root@deb10:/home/sam# ping 172.16.168.255
PING 172.16.168.255 (172.16.168.255) 56(84) bytes of data.
^C
--- 172.16.168.255 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 7ms

root@deb10:/home/sam# ping 7.0.1.16
PING 7.0.1.16 (7.0.1.16) 56(84) bytes of data.
^C
--- 7.0.1.16 ping statistics ---
14 packets transmitted, 0 received, 100% packet loss, time 311ms

huapox · 2019-08-13T12:50:43Z

this time in my home debian9, it works: (ip: 10.5.0.2/16)

kgctl showconf peer xxx (in cluster console)
wg-quick up/down wg0 (set /etc/wireguard/wg0.conf first)

root @ debian in /etc/wireguard |20:45:59  
$ cat wg0.conf
[interface]
Address = 10.5.0.2/16
PrivateKey = EPGB/----------YsHf1LbR14+3Pu6jKxggc+85yQGE=
ListenPort = 51820

[Peer]
AllowedIPs = 7.0.0.0/24, 172.16.168.255/32, 10.4.0.1/32
Endpoint = 47.98.xxx.xxx:51820
PersistentKeepalive = 0
PublicKey = nOW------------NDuCxN1GnXXz+0UiseSOYOrq14Nz4=

[Peer]
AllowedIPs = 7.0.1.0/24, 192.168.0.105/32, 10.4.0.2/32
Endpoint = 139.159.xxx.xxx:51820
PersistentKeepalive = 0
PublicKey = fQzI------HUWzSGiMZJdH9wzq9eKGogDO9fWmc=

[root@hw-vm1 ~]# wg show
interface: kilo0
  public key: fQzIcE5-----------GiMZJdH9wzq9eKGogDO9fWmc=
  private key: (hidden)
  listening port: 51820

peer: wk+0COy--------s5jSGF50rLDnK/HQ=
  endpoint: 112.96.xxx.xxx:57027
  allowed ips: 10.5.0.2/32
  latest handshake: 30 seconds ago
  transfer: 1.01 KiB received, 2.55 KiB sent
  persistent keepalive: every 10 seconds

peer: nOWTsg------------1GnXXz+0UiseSOYOrq14Nz4=
  endpoint: 47.98.xxx.xxx:51820
  allowed ips: 7.0.0.0/24, 172.16.168.255/32, 10.4.0.1/32
  latest handshake: 1 minute, 8 seconds ago
  transfer: 3.39 MiB received, 3.09 MiB sent

peer: FLSYbngn-------------8pTxLmFC5ZtQLK5Oc0A=
  allowed ips: 10.5.0.1/32
  persistent keepalive: every 10 seconds

huapox · 2019-08-13T12:58:48Z

With some items left:

company's vpn peer vm-deb10 conns fail. why? TODO
home's vpn peer host-deb9 conns ok. (But at first ping frequently no response, then I try ping on both sides, then it works.)
the ali-vm1, hw-vm1's node/pod routes auto added to host-deb9's routes table. why? (needn't by hand)
the svc of k3s cluster, in order to use this seems need to install kube-proxy in my vpn peer. can it be set by the wireguard-vpn? thinking...

squat · 2019-08-13T13:17:30Z

@huapox that's great progress! Regarding the bullet points:

company's vpn peer vm-deb10 conns fail. why? TODO

I need to take a closer look, I will get back to you

home's vpn peer host-deb9 conns ok. (But at first ping frequently no response, then I try ping on both sides, then it works.)

This is because the home vpn peer doesn't specify an external IP address for the WireGuard endpoint. This means that the peer must first try to connect the k8s cluster so that the cluster can record the public IP. Once the peer has pinged the cluster, the cluster can ping the peer.

the ali-vm1, hw-vm1's node/pod routes auto added to host-deb9's routes table. why? (needn't by hand)

This is because of some extra magic that wg-quick does for you. Take a look at https://github.com/WireGuard/WireGuard/blob/HEAD/src/tools/wg-quick/linux.bash#L161-L173 for details.

the svc of k3s cluster, in order to use this seems need to install kube-proxy in my vpn peer. can it be set by the wireguard-vpn? thinking...

Luckily there is an easy workaround rather than running kube-proxy on the peer! You are on the right track. We need to pick one of the cluster nodes to be the gateway to the service CIDR. Then, we add the service CIDR as one of the allowed IPs for that node in the home vpn peer.ini. Once we add the route for the service CIDR (or let wg-quick do it for us), we can connect to any service IP.

huapox · 2019-08-13T14:46:43Z

Great!

1st, 2nd: I'll try later for more stable usable improvment.
3rd, 4th: thx for the patient guild, I'll use it and explose the dev-to-cluster mode for more convenient, like the dns usage that we can use svc's name direct. and try to spread the usage to the team.

huapox · 2019-08-14T03:29:54Z

update:
for the 1st item, after i reboot vm-deb10 in company, then it works:
ali-vm1 : 10.4.0.1
vm-deb10: 10.5.0.1

to ali-vm1

root@deb10:/etc/wireguard# wg-quick up wg0
[#] ip link add wg0 type wireguard
[#] wg setconf wg0 /dev/fd/63
[#] ip -4 address add 10.5.0.1/32 dev wg0
[#] ip link set mtu 1420 up dev wg0
[#] ip -4 route add 172.16.168.255/32 dev wg0
[#] ip -4 route add 10.4.0.1/32 dev wg0
[#] ip -4 route add 7.0.0.0/24 dev wg0
root@deb10:/etc/wireguard# ip a |grep wg
21: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 10.5.0.1/32 scope global wg0
root@deb10:/etc/wireguard# route -n |grep wg
7.0.0.0         0.0.0.0         255.255.255.0   U     0      0        0 wg0
10.4.0.1        0.0.0.0         255.255.255.255 UH    0      0        0 wg0
172.16.168.255  0.0.0.0         255.255.255.255 UH    0      0        0 wg0
root@deb10:/etc/wireguard# ping 10.4.0.1
PING 10.4.0.1 (10.4.0.1) 56(84) bytes of data.
64 bytes from 10.4.0.1: icmp_seq=1 ttl=64 time=55.1 ms
64 bytes from 10.4.0.1: icmp_seq=2 ttl=64 time=27.2 ms
64 bytes from 10.4.0.1: icmp_seq=3 ttl=64 time=29.1 ms

ali-vm1 to vm-deb10

[root@ali-vm1 opt]# ping 10.5.0.1
PING 10.5.0.1 (10.5.0.1) 56(84) bytes of data.
64 bytes from 10.5.0.1: icmp_seq=1 ttl=64 time=45.4 ms
64 bytes from 10.5.0.1: icmp_seq=2 ttl=64 time=27.8 ms

huapox · 2019-08-15T06:32:46Z

update 2: (test dns)

sam@deb10:~$ cat /etc/resolv.conf
domain lan
search lan
nameserver 6.7.8.10
nameserver 10.1.1.7
nameserver 114.114.114.114
sam@deb10:~$ 
sam@deb10:~$ dig @6.7.8.10 +short gitea-svc.default.svc.t1.k3s
6.7.8.144
sam@deb10:~$ 
sam@deb10:~$ curl -s gitea-svc.default.svc.t1.k3s:3000 |head -5
<!DOCTYPE html>
<html lang="en-US">
<head data-suburl="">
	<meta charset="utf-8">
	<meta name="viewport" content="width=device-width, initial-scale=1">

squat · 2019-08-15T07:02:54Z

@huapox, great work! You bring up a great point about making kubernetes services discoverable to vpn peers via dns. I will add this setup to the vpn docs!

huapox closed this as completed Aug 15, 2019

huapox mentioned this issue Sep 9, 2019

Encrypted Networking Support k3s-io/k3s#50

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Help, vpn peer usage in nat environment. #13

Help, vpn peer usage in nat environment. #13

huapox commented Aug 13, 2019 •

edited

huapox commented Aug 13, 2019

huapox commented Aug 13, 2019 •

edited

huapox commented Aug 13, 2019

squat commented Aug 13, 2019 •

edited

huapox commented Aug 13, 2019

huapox commented Aug 14, 2019

huapox commented Aug 15, 2019

squat commented Aug 15, 2019

Help, vpn peer usage in nat environment. #13

Help, vpn peer usage in nat environment. #13

Comments

huapox commented Aug 13, 2019 • edited

huapox commented Aug 13, 2019

huapox commented Aug 13, 2019 • edited

huapox commented Aug 13, 2019

squat commented Aug 13, 2019 • edited

huapox commented Aug 13, 2019

huapox commented Aug 14, 2019

huapox commented Aug 15, 2019

squat commented Aug 15, 2019

huapox commented Aug 13, 2019 •

edited

huapox commented Aug 13, 2019 •

edited

squat commented Aug 13, 2019 •

edited