opnsense-wireguard will not stop or restart properly #866

belmarca · 2018-09-23T05:31:54Z

Versions:

opnsense-wireguard plugin patch: 303a17d
OPNsense 18.7.3-amd64
FreeBSD 11.1-RELEASE-p14
LibreSSL 2.6.5

Example output:

root@OPNsense:~ # service opnsense-wireguard stop
stopping wireguard
wg-quick: `wg0' is not a WireGuard interface
ifconfig: SIOCIFDESTROY: Device not configured

Problem description:

I have set up a server using NAT with two endpoints (which should actually be called "peers"). One for my computer and one for my Android phone. The service did not seem to work so I enabled only one of the peers and rebooted the firewall. After rebooting, the service is working properly and I can ping it from my computer. The tunnel is working and the wg0 interface on the firewall has the following configuration:

root@OPNsense:~ # ifconfig wg0
wg0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1420
	options=80000<LINKSTATE>
	inet 192.168.4.1 --> 192.168.4.1  netmask 0xffffff00 
	inet6 fe80::20d:b9ff:fe4c:50c%wg0 prefixlen 64 scopeid 0x9 
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
	groups: tun wg 
	Opened by PID 77351

The "List Configuration" and "Handshakes" tabs work properly and show the correct information.

Upon clicking the "Save" button in any page (with or without making changes), the service stops working. The new configuration for the wg0 interface is the same with the exception that the groups line is now groups: wg (tun is missing):

root@OPNsense:~ # ifconfig wg0
wg0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1420
	options=80000<LINKSTATE>
	inet 192.168.4.1 --> 192.168.4.1  netmask 0xffffff00 
	inet6 fe80::20d:b9ff:fe4c:50c%wg0 prefixlen 64 scopeid 0x9 
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
	groups: wg 
	Opened by PID 77351

To fix this I tried to restart the service through ssh, but the following error occurs:

root@OPNsense:~ # service opnsense-wireguard stop
stopping wireguard
wg-quick: `wg0' is not a WireGuard interface
ifconfig: SIOCIFDESTROY: Device not configured

root@OPNsense:~ # clog /var/log/system.log
...
Sep 23 01:24:02 OPNsense kernel: ifa_maintain_loopback_route: deletion failed for interface wg0: 3
Sep 23 01:24:02 OPNsense kernel: wg0: link state changed to DOWN

which I can fix by manually deleting the interface and restarting the service:

root@OPNsense:~ # ifconfig wg0 inet6 fe80::20d:b9ff:fe4c:50c%wg0 delete
root@OPNsense:~ # service opnsense-wireguard start
starting wireguard
[#] wireguard-go wg0
WARNING WARNING WARNING WARNING WARNING WARNING WARNING
W                                                     G
W   This is alpha software. It will very likely not   G
W   do what it is supposed to do, and things may go   G
W   horribly wrong. You have been warned. Proceed     G
W   at your own risk.                                 G
W                                                     G
WARNING WARNING WARNING WARNING WARNING WARNING WARNING
INFO: (wg0) 2018/09/23 01:25:41 Starting wireguard-go version 0.0.20180613
[#] wg setconf wg0 /tmp/tmp.9bQVjyMz/sh-np.PYZzir
[#] ifconfig wg0 inet 192.168.4.1/24 192.168.4.1 alias
[#] ifconfig wg0 mtu 1420
[#] ifconfig wg0 up
[#] route -q -n add -inet 192.168.4.5/32 -interface wg0
[+] Backgrounding route monitor

root@OPNsense:~ # clog /var/log/system.log
...
Sep 23 01:25:41 OPNsense kernel: tun0: link state changed to UP
Sep 23 01:25:41 OPNsense kernel: tun0: changing name to 'wg0'

Now everything is back to normal, except that the saved changes haven't been committed.

The text was updated successfully, but these errors were encountered:

mimugmail · 2018-09-23T06:38:11Z

Can you use the latest master code? I had this too some time ago but wasn't able to reproduce again.

opnsense-code plugins
cd /usr/plugins/net/wireguard
make upgrade

Seems you run an old version since with 0.4 you can't use Wireguard with multiple endpoints :)

belmarca · 2018-09-23T14:43:11Z

Actually I was using os-wireguard-devel-0.5.

root@OPNsense:/usr/plugins/net/wireguard # opnsense-code plugins
Fetching origin
Already up to date.

I get the same problem. It still does not work after a pkg remove followed by opnsense-code plugins and make upgrade of the package.

mimugmail · 2018-09-23T14:47:31Z

Can you post your wg.conf please?

belmarca · 2018-09-23T16:10:34Z

[Interface]
Address = 192.168.4.1/24
ListenPort = 51820
PrivateKey = A
[Peer]
PublicKey = B
AllowedIPs = 192.168.4.2
[Peer]
PublicKey = C
AllowedIPs = 192.168.4.5

Peers B and C can successfully ping A.

mimugmail · 2018-09-24T05:50:48Z

So, after you fixed it, you can start the service via CLI and it works until you hit save via UI, then you again have only group wg in ifconfig, correct?

belmarca · 2018-09-24T15:02:26Z

No, after I delete the interface and restart the service, I get this:

root@OPNsense:~ # ifconfig wg0
wg0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1420
	options=80000<LINKSTATE>
	inet 192.168.4.1 --> 192.168.4.1  netmask 0xffffffff 
	inet6 fe80::20d:b9ff:fe4c:50c%wg0 prefixlen 64 scopeid 0x9 
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
	groups: tun wg 
	Opened by PID 84321

which works.

The problem seems to be with what happens once the "Save" button is pressed. I don't know enough about how OPNsense works to really diagnose the issue.

mimugmail · 2018-10-03T07:34:46Z

Would you mind letting me again on your system with the previous password? I'd need to do a bit more testing.

belmarca · 2018-10-04T17:41:47Z

Would you mind letting me again on your system with the previous password? I'd need to do a bit more testing.

Excuse me? I'm not sure what you mean, or what you imply.

mimugmail · 2018-10-04T21:33:05Z

Sorry, meant you were the guy offering me access

ChromoX · 2018-10-08T02:38:33Z

I have this same issue. Basically anytime I push "Save" the service dies(but looks like doesn't report that properly), and does not restart at all.

How can I help debug this?

mimugmail · 2018-10-08T03:55:07Z

I'd need access to your system if possible

ChromoX · 2018-10-08T04:58:56Z

That's not possible, but if you tell me where I need to look or what information you need I can do my best.

mimugmail · 2018-10-08T05:39:57Z

Check after save via ifconfig if the groups are only wireguard and not tun. If yes and you only can fix it with a reboot then restart the machine, and do via cli service opnsense-wireguard stop and post the output like @belmarca

ChromoX · 2018-10-08T15:46:57Z

When WireGuard is working and I've done the instructions above to delete the interface and restart the service I get this:

wg0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1420
	options=80000<LINKSTATE>
	inet 192.168.51.1 --> 192.168.51.1  netmask 0xffffff00 
	inet6 fe80::ae1f:6bff:fe43:b6d8%wg0 prefixlen 64 scopeid 0x19 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: tun wg 
	Opened by PID 35525

If I just click the "Save" on the General Tab no restart happens and therefore the above still stands.
If I click "Save" on the Server tab the WireGuard service shuts down, but never restarts. I can manually run service opnsense-wireguard start and it starts up no problem.
If I click "Save" on the Endpoints tab the WireGuard service shuts down and restarts, but has the following configuration:

wg0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1420
	options=80000<LINKSTATE>
	inet 192.168.51.1 --> 192.168.51.1  netmask 0xffffff00 
	inet6 fe80::ae1f:6bff:fe43:b6d8%wg0 prefixlen 64 scopeid 0x19 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: wg 
	Opened by PID 69856

I can fix the above by simply running service opnsense-wireguard stop and then service opnsense-wireguard start.

Let me know if I can provide any other information.

mimugmail · 2018-10-08T15:55:06Z

I think I found it, @fichtner when is deadline for .5?

fichtner · 2018-10-08T16:06:33Z

Friday, probably 18.7.5 on Monday next week.

mimugmail · 2018-10-08T19:27:42Z

@ChromoX you are at latest version of wireguard plugin?

ChromoX · 2018-10-08T20:15:38Z

os-wireguard-devel (installed) | 0.5 | 41.7KiB | WireGuard VPN service

mimugmail · 2018-10-09T06:04:36Z

Can you post a screenshot of the configured server and endpoints and how it looks in wgX.conf? I can't reproduce here. On every tab I hit save and check via ifconfig, the PID opened the process changes ..

ChromoX · 2018-10-09T16:09:22Z

root@firewall:/usr/local/etc/wireguard # cat wg0.conf 

[Interface]
Address = 192.168.52.1/24
ListenPort = 52180
PrivateKey = <priv key>
[Peer]
PublicKey = <pub key>
AllowedIPs = 192.168.52.2/32
[Peer]
PublicKey = <pub key>
AllowedIPs = 192.168.52.3/32

Changing just a name on an Endpoint and pushing save ended up with an interface that looks like this:

wg0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1420
	options=80000<LINKSTATE>
	inet 192.168.52.1 --> 192.168.52.1  netmask 0xffffff00 
	inet6 fe80::ae1f:6bff:aa43:b6d8%wg0 prefixlen 64 scopeid 0x19 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: wg 
	Opened by PID 31347

Which doesn't work because of the lack of tun from groups I think.

After deleting the interface manually and then running stop, start service I can get WireGuard back up again.

wg0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1420
	options=80000<LINKSTATE>
	inet 192.168.52.1 --> 192.168.52.1  netmask 0xffffff00 
	inet6 fe80::ae1f:6bff:aa43:b6d8%wg0 prefixlen 64 scopeid 0x19 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: tun wg 
	Opened by PID 44512

mimugmail · 2018-10-09T16:14:31Z

@fichtner reconfigure API call does a stop, template reload and start, correct?

ChromoX · 2018-10-19T17:52:52Z

This is still an issue for me even with 0.7_1. I attempted to add an endpoint. After clicking save...

This is what the interface looks like when it's not working. Not working = No connections, No list configuration output, No handshakes output.

wg0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1420
	options=80000<LINKSTATE>
	inet 192.168.54.1 --> 192.168.54.1  netmask 0xffffff00 
	inet6 fe80::ae1f:8aee:fe43:b6d8%wg0 prefixlen 64 scopeid 0x19 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: wireguard 
	Opened by PID 82884

The only way I can fix this is by stopping the service, removing the interface, and then restarting.

wg0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1420
	options=80000<LINKSTATE>
	inet 192.168.54.1 --> 192.168.54.1  netmask 0xffffff00 
	inet6 fe80::ae1f:8aee:fe43:b6d8%wg0 prefixlen 64 scopeid 0x19 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	groups: tun wireguard 
	Opened by PID 38805

That is what it looks like after restarting the service.

I noticed that when I run service opnsense-wireguard stop it never ever removes the wireguard interface. This seems to be the problem in my case. Does this make any sense to you @mimugmail?

LATER:
Trying to get at the heart of the issue. I decided to remove all my endpoints and re-add them. After re-adding the configs are not written with any endpoints what so ever.

mimugmail · 2018-10-19T18:26:42Z

This is a known problem of Wireguard itself, will be fixed in a few days hopefully. We will have to wait.

ChromoX · 2018-10-19T18:37:30Z

@mimugmail Have you found anyway to make things work as is?

mimugmail · 2018-10-19T19:00:36Z

No, the developer of Wireguard has to fix this. The plugin itself works fine ..

ChromoX · 2018-10-19T23:40:26Z

@mimugmail Is my problem of the endpoints not being written to the wg0.conf file the same Wireguard problem?

I noticed there was a new snapshot as well on the 18th. It included a fix "tun: only call .Fd() once" which sounds like it could describe what's been going on with at least the not restarting properly.

mimugmail · 2018-10-20T05:26:04Z

Yes, the Fd call is the issue, but I cant find it anymore in Git :/

ChromoX · 2018-10-20T05:55:58Z

@mimugmail https://git.zx2c4.com/wireguard-go/commit/?id=2e772194cf7cd7c37d24364a9f9d407dc96a25e8

mimugmail · 2018-10-20T06:10:20Z

Ah, yes checked my IRC backlog, it was in wireguard-go, thanks :)
When this one is in we are good to go stable with this plugin ...

fichtner assigned mimugmail Sep 23, 2018

fichtner added the bug Production bug label Sep 23, 2018

mimugmail mentioned this issue Oct 12, 2018

net/wireguard: fix rc stuff #911

Merged

fichtner closed this as completed in #911 Oct 12, 2018

vizv mentioned this issue Oct 23, 2021

OpenConnect Startup - deletion failed for interface #1359

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

opnsense-wireguard will not stop or restart properly #866

opnsense-wireguard will not stop or restart properly #866

belmarca commented Sep 23, 2018

mimugmail commented Sep 23, 2018

belmarca commented Sep 23, 2018

mimugmail commented Sep 23, 2018

belmarca commented Sep 23, 2018

mimugmail commented Sep 24, 2018

belmarca commented Sep 24, 2018 •

edited

Loading

mimugmail commented Oct 3, 2018

belmarca commented Oct 4, 2018

mimugmail commented Oct 4, 2018

ChromoX commented Oct 8, 2018

mimugmail commented Oct 8, 2018

ChromoX commented Oct 8, 2018

mimugmail commented Oct 8, 2018

ChromoX commented Oct 8, 2018 •

edited

Loading

mimugmail commented Oct 8, 2018

fichtner commented Oct 8, 2018

mimugmail commented Oct 8, 2018

ChromoX commented Oct 8, 2018

mimugmail commented Oct 9, 2018

ChromoX commented Oct 9, 2018

mimugmail commented Oct 9, 2018

ChromoX commented Oct 19, 2018 •

edited

Loading

mimugmail commented Oct 19, 2018

ChromoX commented Oct 19, 2018

mimugmail commented Oct 19, 2018

ChromoX commented Oct 19, 2018 •

edited

Loading

mimugmail commented Oct 20, 2018

ChromoX commented Oct 20, 2018

mimugmail commented Oct 20, 2018

opnsense-wireguard will not stop or restart properly #866

opnsense-wireguard will not stop or restart properly #866

Comments

belmarca commented Sep 23, 2018

mimugmail commented Sep 23, 2018

belmarca commented Sep 23, 2018

mimugmail commented Sep 23, 2018

belmarca commented Sep 23, 2018

mimugmail commented Sep 24, 2018

belmarca commented Sep 24, 2018 • edited Loading

mimugmail commented Oct 3, 2018

belmarca commented Oct 4, 2018

mimugmail commented Oct 4, 2018

ChromoX commented Oct 8, 2018

mimugmail commented Oct 8, 2018

ChromoX commented Oct 8, 2018

mimugmail commented Oct 8, 2018

ChromoX commented Oct 8, 2018 • edited Loading

mimugmail commented Oct 8, 2018

fichtner commented Oct 8, 2018

mimugmail commented Oct 8, 2018

ChromoX commented Oct 8, 2018

mimugmail commented Oct 9, 2018

ChromoX commented Oct 9, 2018

mimugmail commented Oct 9, 2018

ChromoX commented Oct 19, 2018 • edited Loading

mimugmail commented Oct 19, 2018

ChromoX commented Oct 19, 2018

mimugmail commented Oct 19, 2018

ChromoX commented Oct 19, 2018 • edited Loading

mimugmail commented Oct 20, 2018

ChromoX commented Oct 20, 2018

mimugmail commented Oct 20, 2018

belmarca commented Sep 24, 2018 •

edited

Loading

ChromoX commented Oct 8, 2018 •

edited

Loading

ChromoX commented Oct 19, 2018 •

edited

Loading

ChromoX commented Oct 19, 2018 •

edited

Loading