OpenVPN Client (As a Gateway) Connection Issues 18.7.10 -> 19.1.x

[DanMc85]

RE: https://forum.opnsense.org/index.php?topic=12281.0

There are issues with utilizing an OpenVPN client in a multi-gateway setup (not redirecting all traffic) on any 19.1.x build of OPNSense.

I have tried both a clean reinstall/rebuild and the usual upgrade with existing configuration with same result. There is a bug somewhere.

So here is my basic setup...
I have a VLAN 100 on my LAN... any device in this subnet goes out a Private Internet Access VPN Client GATEWAY that is running on OPNSense as a client. Others do this with a simple Alias for specific devices, regardless the principal setup is the same.

So from what I can tell on any build of 19.1.x (tried them all) and currently 19.1.4 this setup stops working.
Here is what I can see so far:

• OpenVPN client connects perfectly
• OpenVPN client obtains DHCP IP Address from VPN Server (Private Internet Access) and assigns an IP address to the OPNSense Firewall.
• There is an active interface on the firewall (OVPNC1) which then activates a DYNAMIC IPv4 Gateway for this connection... Monitor IP is set to Private Internet Access DNS Server: 209.222.18.218
• There are firewall rules for OpenVPN to allow Any Any
• There are firewall rules for the VLAN 100 interface to allow any traffic out Private Internet Access VPN Gateway.
• There are manual Outbound NAT Rules created

Somehow something is broken somewhere. If I go to ping interface diagnostics, chose the VLAN 100 or Private Internet Access Interfaces. Ping any address. It fails.

On the home screen dashboard, dpinger shows the gateway as down/offline. VPN connection is up perfectly.

• Makes no sense.

I feel this is an outbound NAT issue, but I am not sure where to dig deeper for troubleshooting other than modifying NAT rules, firewall rules, etc... which I have already played around with.

Reference Topics:

https://forum.opnsense.org/index.php?topic=4979.msg52493#msg52493

https://forum.opnsense.org/index.php?topic=11843.msg53785#msg53785

https://blog.networkprofile.org/pia-vpn-on-pfsense-2-4-4/

[DanMc85]

19.1.4:

18.7.10:

Config:

[DanMc85]

[DanMc85]

Ignore some of the VPN IP Addresses, these screenshots were taken at different time periods.

[DanMc85]

Here are the Firewall Advanced Settings on 18.7.10

[DanMc85]

I did notice that the VPN Server tries to push these options on to OPNSense Client:

openvpn[22479]: PUSH: Received control message: 'PUSH_REPLY,redirect-gateway def1,dhcp-option DNS 209.222.18.222,dhcp-option DNS 209.222.18.218,ping 10,comp-lzo no,route 10.43.1.1,topology net30,ifconfig 10.43.1.10 10.43.1.9,auth-token'

Apr 2 18:49:52	openvpn[22479]: Options error: option 'route' cannot be used in this context ([PUSH-OPTIONS])
Apr 2 18:49:52	openvpn[22479]: Options error: option 'dhcp-option' cannot be used in this context ([PUSH-OPTIONS])
Apr 2 18:49:52	openvpn[22479]: Options error: option 'dhcp-option' cannot be used in this context ([PUSH-OPTIONS])
Apr 2 18:49:52	openvpn[22479]: Options error: option 'redirect-gateway' cannot be used in this context ([PUSH-OPTIONS])

[DanMc85]

Per mimugmail:

• Screenshot of NAT rules
• tcpdump on ovpnc interface while pinging your monitor IP
• Routing table showing your open vpn routes"

I messed with these outbound nat rules quite a bit during troubleshooting... it is longer that I normally would have it.

I took a comparison of routing table on a 18.7.10 config vs 19.1.4:

18.7.10:

19.1.4:

Then I did the TCPDump... I think I can kind of see what is happening. It looks like the firewall itself is trying to submit the pings on 19.1.4. Since the hostname of the firewall is shown on the dump.
On 18.7.10 the OpenVPN Private Internet Access Client IP is doing the pings which is what it should be.

18.7.10:

19.1.4:

[mimugmail]

Please consider closing the the issue here and go back to forums. IMHO this is not a bug, even more a constellation which should also not work with 18.7. Your first 2 NAT rules doesn't make sense as you want to NAT LAN connections to your PIA IF IP .. but the first NAT your PIA IP to OpenVPN IP which will break if you configure a second client config or a server config.

[DanMc85]

Please consider closing the the issue here and go back to forums. IMHO this is not a bug, even more a constellation which should also not work with 18.7. Your first 2 NAT rules doesn't make sense as you want to NAT LAN connections to your PIA IF IP .. but the first NAT your PIA IP to OpenVPN IP which will break if you configure a second client config or a server config.

I am aware. That was me testing and messing around as I noted. No matter what was in outbound or the order, it wasn't working. I never had the OpenVPN one in there previously. It isn't in there now but was when I took the screenshot. I even switched the entire thing from hybrid to manual since that post and manually made all the normal WAN rules.

I made copies of my VMWare instance for testing, to not modify my "production" firewall.

[DanMc85]

This is what my active/production FW manual outbound NAT rules look like on 18.7.10
It works perfectly

I imported this entire config into 19.1.4 and the VPN gateway could not connect anywhere. (WAN Gateway worked perfectly)... Which is why I really think there is a bug somewhere. I have been using this OpenVPN Client setup since I started using OPNSense 2 years ago, all of a sudden it is an issue.

My Firewall has:

• 1 OpenVPN Client Config
• 3 OpenVPN Server Configs
• Note: all outbound traffic is not going over VPN Client. Only traffic from VLAN 100 is going over the VPN Client - Private Internet Access - Default Gateway.

[DanMc85]

New 19.1.5_1 Update:

TCPDump... it looks like the connection is working now in 19.1.5, but there is another issue present. The OpenVPN Client connection keeps reloading every 10-15 seconds. You can see by the IP address changing for the source in the TCPDump. Now I just need to get it stable and to stay connected.

OpenVPN Client Log:

Apr 5 19:05:35	openvpn[8480]: Initialization Sequence Completed
Apr 5 19:05:35	openvpn[8480]: /usr/local/etc/inc/plugins.inc.d/openvpn/ovpn-linkup ovpnc2 1500 1572 10.38.1.6 10.38.1.5 init
Apr 5 19:05:35	openvpn[8480]: /sbin/ifconfig ovpnc2 10.38.1.6 10.38.1.5 mtu 1500 netmask 255.255.255.255 up
Apr 5 19:05:35	openvpn[8480]: TUN/TAP device /dev/tun2 opened
Apr 5 19:05:35	openvpn[8480]: TUN/TAP device ovpnc2 exists previously, keep at program end
Apr 5 19:05:33	openvpn[8480]: /usr/local/etc/inc/plugins.inc.d/openvpn/ovpn-linkdown ovpnc2 1500 1572 10.36.1.6 10.36.1.5 init
Apr 5 19:05:33	openvpn[8480]: Closing TUN/TAP interface
Apr 5 19:05:33	openvpn[8480]: NOTE: Pulled options changed on restart, will need to close and reopen TUN/TAP device.
Apr 5 19:05:33	openvpn[8480]: Preserving previous TUN/TAP instance: ovpnc2
Apr 5 19:05:33	openvpn[8480]: Incoming Data Channel: Using 256 bit message hash 'SHA256' for HMAC authentication
Apr 5 19:05:33	openvpn[8480]: Incoming Data Channel: Cipher 'AES-256-CBC' initialized with 256 bit key
Apr 5 19:05:33	openvpn[8480]: Outgoing Data Channel: Using 256 bit message hash 'SHA256' for HMAC authentication
Apr 5 19:05:33	openvpn[8480]: Outgoing Data Channel: Cipher 'AES-256-CBC' initialized with 256 bit key
Apr 5 19:05:33	openvpn[8480]: OPTIONS IMPORT: --ifconfig/up options modified
Apr 5 19:05:33	openvpn[8480]: OPTIONS IMPORT: compression parms modified
Apr 5 19:05:33	openvpn[8480]: OPTIONS IMPORT: timers and/or timeouts modified
Apr 5 19:05:33	openvpn[8480]: Pushed option removed by filter: 'auth-token gcQ4JAnfydk15X+saPzz8SLNVy9c5keHHth+Q0z9sP0='
Apr 5 19:05:33	openvpn[8480]: Options error: option 'route' cannot be used in this context ([PUSH-OPTIONS])
Apr 5 19:05:33	openvpn[8480]: Pushed option removed by filter: 'dhcp-option DNS 209.222.18.218'
Apr 5 19:05:33	openvpn[8480]: Pushed option removed by filter: 'dhcp-option DNS 209.222.18.222'
Apr 5 19:05:33	openvpn[8480]: Options error: option 'redirect-gateway' cannot be used in this context ([PUSH-OPTIONS])
Apr 5 19:05:33	openvpn[8480]: PUSH: Received control message: 'PUSH_REPLY,redirect-gateway def1,dhcp-option DNS 209.222.18.222,dhcp-option DNS 209.222.18.218,ping 10,comp-lzo no,route 10.38.1.1,topology net30,ifconfig 10.38.1.6 10.38.1.5,auth-token'
Apr 5 19:05:33	openvpn[8480]: SENT CONTROL [2ac510320d4bc5d41adb89b71be0b736]: 'PUSH_REQUEST' (status=1)
Apr 5 19:05:32	openvpn[8480]: [2ac510320d4bc5d41adb89b71be0b736] Peer Connection Initiated with [AF_INET]209.95.50.27:501
Apr 5 19:05:32	openvpn[8480]: Control Channel: TLSv1.2, cipher TLSv1/SSLv3 DHE-RSA-AES256-GCM-SHA384, 4096 bit RSA
Apr 5 19:05:32	openvpn[8480]: VERIFY OK: depth=0, C=US, ST=CA, L=LosAngeles, O=Private Internet Access, OU=Private Internet Access, CN=2ac510320d4bc5d41adb89b71be0b736, name=2ac510320d4bc5d41adb89b71be0b736
Apr 5 19:05:32	openvpn[8480]: VERIFY OK: depth=1, C=US, ST=CA, L=LosAngeles, O=Private Internet Access, OU=Private Internet Access, CN=Private Internet Access, name=Private Internet Access, emailAddress=secure@privateinternetaccess.com
Apr 5 19:05:32	openvpn[8480]: TLS: Initial packet from [AF_INET]209.95.50.27:501, sid=0d1561ad 93c9a03d
Apr 5 19:05:32	openvpn[8480]: TCPv4_CLIENT link remote: [AF_INET]209.95.50.27:501
Apr 5 19:05:32	openvpn[8480]: TCPv4_CLIENT link local (bound): [AF_INET]IP:PORT
Apr 5 19:05:32	openvpn[8480]: TCP connection established with [AF_INET]209.95.50.27:501
Apr 5 19:05:30	openvpn[8480]: Attempting to establish TCP connection with [AF_INET]209.95.50.27:501 [nonblock]
Apr 5 19:05:30	openvpn[8480]: Socket Buffers: R=[131072->131072] S=[131072->131072]
Apr 5 19:05:30	openvpn[8480]: TCP/UDP: Preserving recently used remote address: [AF_INET]209.95.50.27:501
Apr 5 19:05:30	openvpn[8480]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts

[DanMc85]

System Log:

Apr 5 19:13:14 kernel: ovpnc2: link state changed to DOWN
Apr 5 19:13:10 opnsense: /usr/local/etc/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use VPN_PRIVATEINTERNETACCESS_VPNV4.
Apr 5 19:13:08 opnsense: /usr/local/etc/rc.newwanip: Adding static route for monitor 208.67.222.222 via 73.218.86.1
Apr 5 19:13:08 opnsense: /usr/local/etc/rc.newwanip: Removing static route for monitor 208.67.222.222 via 73.218.86.1
Apr 5 19:13:08 opnsense: /usr/local/etc/rc.newwanip: Adding static route for monitor 209.222.18.222 via 10.42.1.9
Apr 5 19:13:08 opnsense: /usr/local/etc/rc.newwanip: Removing static route for monitor 209.222.18.222 via 10.42.1.9
Apr 5 19:13:07 opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv6 default route
Apr 5 19:13:07 opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv4 default route
Apr 5 19:13:07 opnsense: /usr/local/etc/rc.newwanip: ROUTING: no IPv6 default gateway set, assuming wan
Apr 5 19:13:07 opnsense: /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan
Apr 5 19:13:07 opnsense: /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'opt3'
Apr 5 19:13:07 opnsense: /usr/local/etc/rc.newwanip: On (IP address: 10.42.1.10) (interface: VPN_PRIVATEINTERNETACCESS[opt3]) (real interface: ovpnc2).
Apr 5 19:13:07 opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'ovpnc2'
Apr 5 19:13:07 kernel: ovpnc2: link state changed to UP

Apr 5 19:13:06 kernel: ovpnc2: link state changed to DOWN
Apr 5 19:13:01 opnsense: /usr/local/etc/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use VPN_PRIVATEINTERNETACCESS_VPNV4.
Apr 5 19:12:59 opnsense: /usr/local/etc/rc.newwanip: Adding static route for monitor 208.67.222.222 via 73.218.86.1
Apr 5 19:12:59 opnsense: /usr/local/etc/rc.newwanip: Removing static route for monitor 208.67.222.222 via 73.218.86.1
Apr 5 19:12:59 opnsense: /usr/local/etc/rc.newwanip: Adding static route for monitor 209.222.18.222 via 10.40.1.9
Apr 5 19:12:59 opnsense: /usr/local/etc/rc.newwanip: Removing static route for monitor 209.222.18.222 via 10.40.1.9
Apr 5 19:12:59 opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv6 default route
Apr 5 19:12:59 opnsense: /usr/local/etc/rc.newwanip: ROUTING: skipping IPv4 default route
Apr 5 19:12:59 opnsense: /usr/local/etc/rc.newwanip: ROUTING: no IPv6 default gateway set, assuming wan
Apr 5 19:12:59 opnsense: /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to wan
Apr 5 19:12:59 opnsense: /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'opt3'
Apr 5 19:12:58 opnsense: /usr/local/etc/rc.newwanip: On (IP address: 10.40.1.10) (interface: VPN_PRIVATEINTERNETACCESS[opt3]) (real interface: ovpnc2).
Apr 5 19:12:58 opnsense: /usr/local/etc/rc.newwanip: IP renewal is starting on 'ovpnc2'
Apr 5 19:12:58 kernel: ovpnc2: link state changed to UP

Apr 5 19:12:57 kernel: ovpnc2: link state changed to DOWN
Apr 5 19:12:51 opnsense: /usr/local/etc/rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use VPN_PRIVATEINTERNETACCESS_VPNV4.
Apr 5 19:12:49 opnsense: /usr/local/etc/rc.newwanip: Adding static route for monitor 208.67.222.222 via 73.218.86.1
Apr 5 19:12:49 opnsense: /usr/local/etc/rc.newwanip: Removing static route for monitor 208.67.222.222 via 73.218.86.1
Apr 5 19:12:49 opnsense: /usr/local/etc/rc.newwanip: Adding static route for monitor 209.222.18.222 via 10.38.1.13
Apr 5 19:12:49 opnsense: /usr/local/etc/rc.newwanip: Removing static route for monitor 209.222.18.222 via 10.38.1.13

[mimugmail]

Second device using same credentials?

[DanMc85]

Second device using same credentials?

Yes, and what's odd. It is like it is still an outbound issue.
If I delete all the WAN outbound rules. Just having the Private Internet Access outbound rules.
The Private Internet Access Gateway goes online, pings start flowing both directions, and the internet works all going through the VPN. Connection stays up.... extremely odd.

However, having both in there isn't working in 19.x vs 18.7. Which is why I still think there is a bug somewhere to begin with.

[smokydragon]

Will this be fixed in some future release?

[fichtner]

As of now this is a support case. We don't know what to "fix".

[DanMc85]

As of now this is a support case. We don't know what to "fix".

I think it may be a firewall rule issue with the pf firewall sending traffic over the proper gateway. It only works if all the outbound rules are deleted for regular WAN connections and only the VPN ones are left. The issue is probably something that is behind the scenes and probably hidden with another issue.

I am starting to think I need to try manual policy based routing rules, if there are any suggestions.

Just makes no sense at all that a OpenVPN config from when I started with OPNsense in 2017, suddenly breaks in 19.1 and worked perfect in every other OPNSense version since then.

[fichtner]

Just makes no sense at all that a OpenVPN config from when I started with OPNsense in 2017, suddenly breaks in 19.1 and worked perfect in every other OPNSense version since then.

I do not think this is necessarily true. We have more upcoming changes in 19.7 for behavioural changes that will require manual intervention -- ease of use features that have complicated the integration in an unnecessary way that is error prone now or in the future.

We are happy to add any of these into migration notes but that requires widespread use of the "broken idiom" for deploying OpenVPN which I'm not aware we have documented on docs.opnsense.org or may be a side effect of multi WAN which easily eludes the scope of "OpenVPN broke" because it did in fact not.

You will find that I am personally weary of "it doesn't work anymore" battle cry for community support with hundreds of occasions where the status quo is a lot better off now than it was 6 months ago, 1 year ago, 2 years ago, or even 4 years ago. :)

Cheers,
Franco

[smokydragon]

I made a try with the new version 19.7. By the way, good job! The update went without problems!
Unfortunately it didn't work, but I experimented a bit with the settings and tried to find a clue.

In the old version, an outbound NAT rule was created. The rule looks like this:
Interface: "OVPN Interface", Source: "specific IP", NAT Address: "Interface Address".
In other words: If the source address on the OVPN interface is a specific internal IP address, then "Internet Traffic" should be used as the NAT address for the OVPN interface address.

What I noticed while experimenting:
It doesn't work with the NAT rule mentioned above. If I change this outbound rule, the interface from "OVPN" to "WAN", it works. The traffic from the specific internal IP address that goes to the Internet has the public IP address in the Internet that comes from the OVPN. (Verified with the website https://ifconfig.co)

Therefore my question: Is it possible that Opnsense detects the traffic that should go over the VPN not as traffic from the OVPN interface, but as traffic from the WAN interface? (The VPN connection is established via the WAN interface...)

P.S.:
A little bit about my setup, because maybe this is not too exactly the same as with the other posts?
The Opnsense Firewall is an OVPN client. The OVPN client connection was configured as a separate interface. I want certain IP addresses to be connected to the Internet via the OVPN connection (and not via the normal WAN).

Cheers.

[mimugmail]

This should be fixed in 19.7.3, If all fails you can always use manual nat

[smokydragon]

I am already using version 19.7.3 (Release Type Production) an there are no pending updates listed. And I use Manual outbound NAT rule generation (no automatic rules are being generated).

[mimugmail]

@smokydragon can you open a thread in forums, this is a usual support question and should be easy to handle with some screenshots.

[mimugmail]

@DanMc85 19.7.3 should mostly eliminate all of your issues, can you check with latest version (making a backup before) if this issue can be closed?

[smokydragon]

@mimugmail thanks for your answer, I'll create a thread in forums.

[fichtner]

Closing as per https://github.com/opnsense/core/blob/master/CONTRIBUTING.md

[eliaspizarro]

uncheck this: