openvpn: multiple remote server support (client and client export)

[gitdevmod]

It could be useful to add directly from the GUI multiple remote servers (ip and port) and an option to enable random server (--remote-random).
Thanks

[mimugmail]

Any intentions to push this one? :)
I have a project with one HQ (2 lines) and a Branch (2 lines). Would perfectly fit for this 👍

[fichtner]

sure, just let me know what to do :D

[mimugmail]

For the beginning multiple remote entries for OPN as a client would just be fine:

https://community.openvpn.net/openvpn/wiki/GettingStartedwithOVPN

You can list multiple --remote options in the configuration file, and OpenVPN will try all of them until it gets a connection.

You can also set different port numbers and protocols for each --remote, like this:

   remote myvpn.example.com 1194 udp
   remote myvpn.example.com 443 tcp-client

[mimugmail]

@fichtner No chance for a quick addition with "same protocol only"?
Out last discussion was that it's hard to implement multiple peers when they use different protocols/ports. But since noone is asking .. just to have multiple remotes would be cool :)

[fichtner]

separate hosts by comma maybe?

[mimugmail]

If it ends like this in config export:

proto udp
port 1194
remote 1.2.3.4
remote 4.3.2.1

YES :)

[fichtner]

yep

To be able to move further than this we would really need to join the port and protocol and host in a new type of model format

so let's try the easier one first :)

[fichtner]

@mimugmail Now I wanted to do this but I ended up confused... how do we export configs for clients?! This is about the OpenVPN client, at least the original ticket was.

[mimugmail]

You're right, it's just for OPN as Client. I had a case at Sophos with similar stuff and mixed it up.

[fichtner]

Alright, you can add several servers to the same field now separated by whitespace or comma. Validation works and normalises the input for the config into a simple CSV so it also works with the original entries (or looks like it).

# opnsense-patch 64fb9ac

What I don't like is that we should probably make the port flexible as well and support a sane resolv-retry and random-remote directives...

Implementing a load-balancing/failover configuration

Client

The OpenVPN client configuration can refer to multiple servers for load balancing and failover. For example:

remote server1.mydomain
remote server2.mydomain
remote server3.mydomain
will direct the OpenVPN client to attempt a connection with server1, server2, and server3 in that order. If an existing connection is broken, the OpenVPN client will retry the most recently connected server, and if that fails, will move on to the next server in the list. You can also direct the OpenVPN client to randomize its server list on startup, so that the client load will be probabilistically spread across the server pool.

remote-random
If you would also like DNS resolution failures to cause the OpenVPN client to move to the next server in the list, add the following:

resolv-retry 60
The 60 parameter tells the OpenVPN client to try resolving each remote DNS name for 60 seconds before moving on to the next server in the list.

The server list can also refer to multiple OpenVPN server daemons running on the same machine, each listening for connections on a different port, for example:

remote smp-server1.mydomain 8000
remote smp-server1.mydomain 8001
remote smp-server2.mydomain 8000
remote smp-server2.mydomain 8001
If your servers are multi-processor machines, running multiple OpenVPN daemons on each server can be advantageous from a performance standpoint.

OpenVPN also supports the remote directive referring to a DNS name which has multiple A records in the zone configuration for the domain. In this case, the OpenVPN client will randomly choose one of the A records every time the domain is resolved.

[fichtner]

This will be worth at least one beer.

[fichtner]

@mimugmail Only because I really like you and @AdSchellevis would beat me for doing stupid stuff like the previous commit: 58061809d48

[fichtner]

Now we support:

o Multiple client server/port combinations
o remote-random
o resolve-retry 60 if infinite resolve is not checked
o regen-sec on the client page
o descriptions for server and client are now at the top below the disable button
o client overview page shows all servers
o and while there i have removed spurious statements from the help text of the advanced settings on both server and client pages

# opnsense-patch 5806180

(requires also the former patch)

[mimugmail]

Whohooo :) I'll setup a testbed here as soon as I have finished my client stuff! Really appreciated 🥇

[mimugmail]

Good news .. in principal it's working!
Two problems right now:

1. Client takes 2-3 minutes until it checks the other remote, perhaps there's some tweaking available
2. If client connects to WAN2 and WAN1 (default) is available, it'll reply with WAN1 IP/IF. Only when it's really down it works. This would mean in a OPN to OPN scenario random remote would run into problems if remote are two WANs of one machine. (confusing enough?)

[mimugmail]

Seems my cron hack to rc.newwanip breaks when using with openvpn, but seems only relevant for my own setup

[fichtner]

Are you using DNS? "resolve-retry 60" may cause this. We can lower to 10 seconds or remove it again. It doesn't matter for multi-remote. But I don't know what the default is.

[fichtner]

Regarding 2. -- server binds to "any"? that's really inconvenient. it goes out the default route or current gateway rule.

[fichtner]

der default is 5 sekunden, ich nehm die 60 wieder raus

[fichtner]

# opnsense-patch 7ced5ac

[mimugmail]

No, I'm using just IP addresses. Perhaps lowering keepalive value help.
I'll digg into it on Monday. Thanks for all your efforts this week <3

[mimugmail]

Tested again, failover took 140sec. :)

[fichtner]

maybe the connection timeout / keepalive then?

[mimugmail]

Oh, forget it .. I reenabled the primary WAN and then this happens:

23:18:03.787115 IP 81.24.66.162.22614 > 91.137.72.214.1194: UDP, length 144
23:18:03.800058 IP 217.5.204.212.1194 > 81.24.66.162.22614: UDP, length 144
23:18:04.828174 IP 81.24.66.162.22614 > 91.137.72.214.1194: UDP, length 144
23:18:04.843491 IP 217.5.204.212.1194 > 81.24.66.162.22614: UDP, length 144
23:18:05.830065 IP 81.24.66.162.22614 > 91.137.72.214.1194: UDP, length 144
23:18:05.843891 IP 217.5.204.212.1194 > 81.24.66.162.22614: UDP, length 144
23:18:06.867143 IP 81.24.66.162.22614 > 91.137.72.214.1194: UDP, length 144
23:18:06.881287 IP 217.5.204.212.1194 > 81.24.66.162.22614: UDP, length 144
23:18:07.881104 IP 81.24.66.162.22614 > 91.137.72.214.1194: UDP, length 144
23:18:07.895039 IP 217.5.204.212.1194 > 81.24.66.162.22614: UDP, length 144
23:18:08.927706 IP 81.24.66.162.22614 > 91.137.72.214.1194: UDP, length 144
23:18:08.942429 IP 217.5.204.212.1194 > 81.24.66.162.22614: UDP, length 144
23:18:09.976368 IP 81.24.66.162.22614 > 91.137.72.214.1194: UDP, length 144
23:18:09.993554 IP 217.5.204.212.1194 > 81.24.66.162.22614: UDP, length 144
23:18:11.024586 IP 81.24.66.162.22614 > 91.137.72.214.1194: UDP, length 144
23:18:11.038588 IP 217.5.204.212.1194 > 81.24.66.162.22614: UDP, length 144

[mimugmail]

So, when WAN1 disabled, took 140s to take over to WAN2 and then ping was working, after reenabling WAN1 replys are sent over WAN1 (also with source IP of WAN1)

[fichtner]

--inactive n [bytes]
Causes OpenVPN to exit after n seconds of inactivity on the TUN/TAP device. The time length of inactivity is measured since the last incoming or outgoing tunnel packet. The default value is 0 seconds, which disables this feature.
If the optional bytes parameter is included, exit if less than bytes of combined in/out traffic are produced on the tun/tap device in n seconds.

In any case, OpenVPN's internal ping packets (which are just keepalives) and TLS control packets are not considered "activity", nor are they counted as traffic, as they are used internally by OpenVPN and are not an indication of actual user activity.

--ping n
Ping remote over the TCP/UDP control channel if no packets have been sent for at least n seconds (specify --ping on both peers to cause ping packets to be sent in both directions since OpenVPN ping packets are not echoed like IP ping packets). When used in one of OpenVPN's secure modes (where --secret, --tls-server, or --tls-client is specified), the ping packet will be cryptographically secure.
This option has two intended uses:

(1) Compatibility with stateful firewalls. The periodic ping will ensure that a stateful firewall rule which allows OpenVPN UDP packets to pass will not time out.

(2) To provide a basis for the remote to test the existence of its peer using the --ping-exit option.

--ping-exit n
Causes OpenVPN to exit after n seconds pass without reception of a ping or other packet from remote. This option can be combined with --inactive, --ping, and --ping-exit to create a two-tiered inactivity disconnect.
For example,

openvpn [options...] --inactive 3600 --ping 10 --ping-exit 60

when used on both peers will cause OpenVPN to exit within 60 seconds if its peer disconnects, but will exit after one hour if no actual tunnel data is exchanged.

--ping-restart n
Similar to --ping-exit, but trigger a SIGUSR1 restart after n seconds pass without reception of a ping or other packet from remote.
This option is useful in cases where the remote peer has a dynamic IP address and a low-TTL DNS name is used to track the IP address using a service such as http://dyndns.org/ + a dynamic DNS client such as ddclient.

If the peer cannot be reached, a restart will be triggered, causing the hostname used with --remote to be re-resolved (if --resolv-retry is also specified).

In server mode, --ping-restart, --inactive, or any other type of internally generated signal will always be applied to individual client instance objects, never to whole server itself. Note also in server mode that any internally generated signal which would normally cause a restart, will cause the deletion of the client instance object instead.

In client mode, the --ping-restart parameter is set to 120 seconds by default. This default will hold until the client pulls a replacement value from the server, based on the --keepalive setting in the server configuration. To disable the 120 second default, set --ping-restart 0 on the client.

See the signals section below for more information on SIGUSR1.

Note that the behavior of SIGUSR1 can be modified by the --persist-tun, --persist-key, --persist-local-ip, and --persist-remote-ip options.

Also note that --ping-exit and --ping-restart are mutually exclusive and cannot be used together.

--keepalive interval timeout
A helper directive designed to simplify the expression of --ping and --ping-restart.
This option can be used on both client and server side, but it is in enough to add this on the server side as it will push appropriate --ping and --ping-restart options to the client. If used on both server and client, the values pushed from server will override the client local values.

The timeout argument will be twice as long on the server side. This ensures that a timeout is detected on client side before the server side drops the connection.

For example, --keepalive 10 60 expands as follows:

 if mode server:
   ping 10                    # Argument: interval
   ping-restart 120           # Argument: timeout*2
   push "ping 10"             # Argument: interval
   push "ping-restart 60"     # Argument: timeout
 else
   ping 10                    # Argument: interval
   ping-restart 60            # Argument: timeout
--ping-timer-rem
Run the --ping-exit / --ping-restart timer only if we have a remote address. Use this option if you are starting the daemon in listen mode (i.e. without an explicit --remote peer), and you don't want to start clocking timeouts until a remote peer connects.

[fichtner]

is that ping weirdness with or without default gw switching?

[mimugmail]

Yes, gw switching enabled, no rules with GWGROUPS active.

[mimugmail]

This was down and switchover at the client

Sep 25 23:13:52 OPNsense openvpn[59031]: Inactivity timeout (--ping-restart), restarting
Sep 25 23:13:52 OPNsense openvpn[59031]: SIGUSR1[soft,ping-restart] received, process restarting
Sep 25 23:13:57 OPNsense openvpn[59031]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Sep 25 23:13:57 OPNsense openvpn[59031]: Re-using pre-shared static key
Sep 25 23:13:57 OPNsense openvpn[59031]: Preserving previous TUN/TAP instance: ovpnc1
Sep 25 23:13:57 OPNsense openvpn[59031]: TCP/UDP: Preserving recently used remote address: [AF_INET]217.5.204.212:1194
Sep 25 23:13:57 OPNsense openvpn[59031]: UDP link local (bound): [AF_INET]81.24.66.162:0
Sep 25 23:13:57 OPNsense openvpn[59031]: UDP link remote: [AF_INET]217.5.204.212:1194
Sep 25 23:14:57 OPNsense openvpn[59031]: Inactivity timeout (--ping-restart), restarting
Sep 25 23:14:57 OPNsense openvpn[59031]: SIGUSR1[soft,ping-restart] received, process restarting
Sep 25 23:15:02 OPNsense openvpn[59031]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Sep 25 23:15:02 OPNsense openvpn[59031]: Re-using pre-shared static key
Sep 25 23:15:02 OPNsense openvpn[59031]: Preserving previous TUN/TAP instance: ovpnc1
Sep 25 23:15:02 OPNsense openvpn[59031]: TCP/UDP: Preserving recently used remote address: [AF_INET]91.137.72.214:1194
Sep 25 23:15:02 OPNsense openvpn[59031]: UDP link local (bound): [AF_INET]81.24.66.162:0
Sep 25 23:15:02 OPNsense openvpn[59031]: UDP link remote: [AF_INET]91.137.72.214:1194
Sep 25 23:15:03 OPNsense openvpn[59031]: Peer Connection Initiated with [AF_INET]91.137.72.214:1194
Sep 25 23:15:04 OPNsense openvpn[59031]: Initialization Sequence Completed

And when reenabling WAN:

Sep 25 23:18:35 OPNsense openvpn[59031]: Inactivity timeout (--ping-restart), restarting
Sep 25 23:18:35 OPNsense openvpn[59031]: SIGUSR1[soft,ping-restart] received, process restarting
Sep 25 23:18:40 OPNsense openvpn[59031]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Sep 25 23:18:40 OPNsense openvpn[59031]: Re-using pre-shared static key
Sep 25 23:18:40 OPNsense openvpn[59031]: Preserving previous TUN/TAP instance: ovpnc1
Sep 25 23:18:40 OPNsense openvpn[59031]: TCP/UDP: Preserving recently used remote address: [AF_INET]91.137.72.214:1194
Sep 25 23:18:40 OPNsense openvpn[59031]: UDP link local (bound): [AF_INET]81.24.66.162:0
Sep 25 23:18:40 OPNsense openvpn[59031]: UDP link remote: [AF_INET]91.137.72.214:1194
Sep 25 23:19:40 OPNsense openvpn[59031]: Inactivity timeout (--ping-restart), restarting
Sep 25 23:19:40 OPNsense openvpn[59031]: SIGUSR1[soft,ping-restart] received, process restarting
Sep 25 23:19:45 OPNsense openvpn[59031]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Sep 25 23:19:45 OPNsense openvpn[59031]: Re-using pre-shared static key
Sep 25 23:19:45 OPNsense openvpn[59031]: Preserving previous TUN/TAP instance: ovpnc1
Sep 25 23:19:45 OPNsense openvpn[59031]: TCP/UDP: Preserving recently used remote address: [AF_INET]217.5.204.212:1194
Sep 25 23:19:45 OPNsense openvpn[59031]: UDP link local (bound): [AF_INET]81.24.66.162:0
Sep 25 23:19:45 OPNsense openvpn[59031]: UDP link remote: [AF_INET]217.5.204.212:1194
Sep 25 23:19:50 OPNsense openvpn[59031]: Peer Connection Initiated with [AF_INET]217.5.204.212:1194
Sep 25 23:19:52 OPNsense openvpn[59031]: Initialization Sequence Completed

[mimugmail]

Oh, after 2 minutes it's working again as WAN2 wasnt alive since replys came from WAN1 IP .. so a double error fixed it :D

[mimugmail]

Funny thing, today I stopped WAN2 (static) where OpenVPN was conntected to, switched to WAN1 (DHCP) and everything work. But at 9:00 my half hour cron with "rc.newwanip WAN1" kicked in and did this:

Sep 26 09:00:00 dialin opnsense: /usr/local/etc/rc.newwanip: Resyncing OpenVPN instances for interface WAN_LWL.
Sep 26 09:00:00 dialin opnsense: /usr/local/etc/rc.newwanip: Switching default gateway to Default_WAN (217.5.204.209)
Sep 26 09:00:00 dialin opnsense: /usr/local/etc/rc.newwanip: The command '/sbin/route add -'inet' default '217.5.204.209'' returned exit code '1', the output was 'route: writing to routing socket: Network is unreachable add net default: gateway 217.5.204.209 fib 0: Network is unreachable'

WAN_LWL get's resynced and there seems to be a process to switch the default gateway to other if no matter what the status is, because the switchport of Default_WAN (217.5.204.209) is down. Now there's the situation where where both interfaces aren't reachable.

[mimugmail]

@fichtner
Oh, doesn't this script run to the end cause of this:

Sep 26 09:00:00 dialin opnsense: /usr/local/etc/rc.newwanip: The command '/sbin/route add -'inet' default '217.5.204.209'' returned exit code '1', the output was 'route: writing to routing socket: Network is unreachable add net default: gateway 217.5.204.209 fib 0: Network is unreachable'

Perhaps this stop some things which results in this fuzzy multi wan errors

[mimugmail]

And when WAN2 (static) comes back, it sets default route to WAN2 again and WAN1 is reachable again? :)
I think I have to study rc.linkup and rc.newwanip

[mimugmail]

https://github.com/opnsense/core/blob/master/src/etc/rc.newwanip#L164

This returns the exit 1 core cause it doesn't respect/exclude interfaces which are in false/error state

[mimugmail]

Summary of yesterdays checks:

• Changing protocol to TCP works as expected
• Setting a gateway in WAN rules in Firewall doesn't work

Suspicion was that WAN1 was DHCP and doesn't have a fixed gateway in interfaces. Changed to static and set a gateway doesn't fix this behavior also. So with UDP a packet entering WAN2 gets replied on WAN1, also with source address of WAN1, BUT: only when the way between is broken and the interface is UP. BUT2: access to TCP services work as expected on WAN1 and WAN2.

Next tests are with different services relying on UDP to check if it's multiwan or OpenVPN

[mimugmail]

@fichtner any chance to get the multiremote support in 17.7.6? Since the open topics are not related to OpenVPN itself it would be nice to have this on stable

[fichtner]

I think so :)

[fichtner]

I'm closing this. No apparent need to support multi-remote client export at this point.