openfortivpn 1.12 / DNS configuration problems

[slartibart70]

Since using openfortivpn 1.12 i do see issues with the DNS resolution, the tunnel is up and running, but no name resolution is possible (see log).
My current workaround is to stay with version 1.11 which runs properly
(running on fedora31, latest updates)

Using ip addresses instead of names is ok, querying the (remote/tunneled) nameserver directly with 'dig @a.b.c.d hostname' is ok, but a 'cat /etc/resolv.conf' only gives me my local router address, there are no remote dns server entries at all. (resolv.conf is managed by networkmanager on my system)

see error line: 'Failed to set DNS configuration: Could not activate remote peer.'

Logs:
INFO: Connected to gateway.
INFO: Authenticated.
INFO: Remote gateway has allocated a VPN.
Using interface ppp0
Connect: ppp0 <--> /dev/pts/3
INFO: Got addresses: [10.147.x.z], ns [10.62.z.zz, 10.62.z.zz]
INFO: negotiation complete
INFO: negotiation complete
local IP address 10.147.x.x
remote IP address 192.0.x.x
INFO: Interface ppp0 is UP.
INFO: Setting new routes...
WARN: Route to gateway exists already.
INFO: Adding VPN nameservers...
Failed to set DNS configuration: Could not activate remote peer.
INFO: Tunnel is up and running.
^CINFO: Cancelling threads...
INFO: Setting ppp interface down.
INFO: Restoring routes...
INFO: Removing VPN nameservers...
Failed to revert interface configuration: Could not activate remote peer.
Hangup (SIGHUP)
Modem hangup
Connect time 12.2 minutes.
Sent 113414 bytes, received 1988486 bytes.
Connection terminated.
INFO: pppd: The link was terminated by the modem hanging up.
INFO: Terminated pppd.
INFO: Closed connection to gateway.
INFO: Logged out.

Any help with this?

[DimitriPapadopoulos]

The only relevant change is:

• [+] use resolvconf if available to update resolv.conf file

This is interesting:

Failed to set DNS configuration: Could not activate remote peer.

as this error message is associated to resolvconf. See #514.

Yet resolvconf is not available on Fedora by default as far as I know. Let's see:

• What is the output of rpm -qa | grep resolvconf?
• What is the output of which resolvconf and locate resolvconf?

Finally which exact options do you pass to openfortivpn? Of particular interest are:

• the default options --set-dns=1 --pppd-use-peerdns=0 where openfortivpn calls resolvconf if available or modifies /etc/resolv.confdirectly otherwise,
• the alternative options --set-dns=0 --pppd-use-peerdns=1 where openfortivpn lets pppd handle DNS settings.

[DimitriPapadopoulos]

I'd follow the piece of advice cited in #514 (comment):

It looks like systemd-resolved might not be running. Does it help
if you run 'sudo systemctl enable --now systemd-resolved' ?

[slartibart70]

I'll start by answering your questions first, then i'll check with the new version/new cmdline params.

1. resolvconf is available:

# which resolvconf
/usr/sbin/resolvconf

# locate resolvconf
/usr/sbin/resolvconf
/usr/share/bash-completion/completions/resolvconf
/usr/share/man/man1/resolvconf.1.gz

BUT, there is no rpm package of resolvconf, because (see resolvconf -h) of this:

# resolvconf -h
resolvconf -a INTERFACE < FILE
resolvconf -d INTERFACE

Register DNS server and domain configuration with systemd-resolved.

  -h --help     Show this help
     --version  Show package version
  -a            Register per-interface DNS server and domain data
  -d            Unregister per-interface DNS server and domain data
  -f            Ignore if specified interface does not exist
  -x            Send DNS traffic preferably over this interface

This is a compatibility alias for the resolvectl(1) tool, providing native
command line compatibility with the resolvconf(8) tool of various Linux
distributions and BSD systems. Some options supported by other implementations
are not supported and are ignored: -m, -p. Various options supported by other
implementations are not supported and will cause the invocation to fail: -u,
-I, -i, -l, -R, -r, -v, -V, --enable-updates, --disable-updates,
--updates-are-enabled.

See the resolvectl(1) man page for details.

Params to openfortivpn are:

openfortivpn remote.com:443 --username="${user}" --password=${pwd} --user-cert=/home/c/.ssh/cert.pem --user-key=/home/c/.ssh/key.pem --ca-file=/home/c/.ssh/combined.cer

and i start it using a script (the /etc/openfortivpn/config file is empty).... yes, i know, password on commandline is bad :-)

2. systemd-resolved
   is indeed not started

further tests will follow later...

[DimitriPapadopoulos]

As a workaround you may instruct openfortivpn to hand over DNS settings to pppd which might not depend on systemd-resolved running:

--set-dns=0 --pppd-use-peerdns=1

That said it would probably be better to start systemd-resolved. Not sure whether it is supposed to be started and why it has not started on your Fedora 31 system.

[slartibart70]

i tried several configs for now:

1. enable systemd-resolved
   run openfortivpn .... pretty much the same, /etc/resolv.conf is not modified in any way, dns resolution not working.

2. use additional prams --set-dns=0 --pppd-use-peerdns=1
   run openfortivpn .... same, /etc/resolv.conf is not modified in any way, dns resolution not working.

3. downgrade to version 1.11:
   All fine again.

yes, if systemd-resolved is started, the above error messages disappear (Failed to set DNS configuration: Could not activate remote peer. / Failed to revert interface configuration: Could not activate remote peer.) but /etc/resolv.conf is not updated nor is the dns query possible for remote/tunneled destinations.

And, for fedora31, the default config seems not to use systemd-resolved, so /etc/resolv.conf is not symlinked to any systemd files but a genuine file

[DimitriPapadopoulos]

This looks like more of a resolvconf problem:

• It doesn't work if systemd-resolved is not enabled.
• It still doesn't work properly if systemd-resolved is enabled on your Fedora system.

If resovlconf is indeed that fragile we probably should have a way to disable its use by openfortivpn - which is what happens when you downgrade to 1.11.0.

But then I notice that --set-dns=0 --pppd-use-peerdns=1 doesn't work either which means that pppd is unable to handle DNS settings too. This cannot be a coincidence, perhaps something's wrong with the DNS settings on your system. Is this a vanilla Fedora system? Have you set DNS in any special way?

[slartibart70]

no, i happened to setup this fedora31 system completely fresh some weeks ago. Nothing special yet, and even another fedora31 system (upgraded from previous releases) shows the exact same behaviour. So i assume this to be isolated to openfortivpn only (sorry).
Wouldn't it be a good idea to make the openforti switchable between how it is done in 1.11 and the new approach in 1.12?
As far as i have read in the meantime, you can of course use systemd-resolved to handle the dns queries in fc31, but the default way of fedora is not using it (relying on networkmanager, i suppose)

btw, using 1.11 i did not use the additional params you indicated (set-dns and ...peerdns)

[DimitriPapadopoulos]

It is not isolated to openfortivpn because pppd suffers the same problem.

[slartibart70]

ok, ok :-) i already said i'm sorry!!!
This was indeed only to point to the fact of it being independent of the underlying fedora31 system

[DimitriPapadopoulos]

As far as I can understand from one of your first comments Fedora provides a resolvconf executable which is not the usual resolvconf found on Debian, Ubuntu or other systems. Instead it is an alias for resolvectl which is specifically written to work with systemd-resolved.

There are multiple methods of setting DNS servers:

• directly modify /etc/resolv.conf,
• run and let pppd handle DNS settings,
• run resolvconf,
• run resolvectl,
• use alternatives such as openresolv,
• ...

It looks like some of the above executables are made available on some systems but are not configured by default - by design. Tools such as resolvconf that are made available on some systems to help with this task are not available, or worse are available but simply don't work on other systems. This leaves us with the difficult task to find a portable way to modify DNS settings. Either we find a way to automate DNS settings in a portable way across systems, or we provide options (in addition to --set-dns and --pppd-use-peerdns) to manually switch between available methods. Third option is we revert to 1.11.0 and the FortiClient way of messing directly with /etc/resolv.conf which if I recall correctly doesn't work so well.

I think it's worth taking the time to read about the first option, perhaps have a look at OpenVPN. However I fear we will have to provide options to switch manually...

[mrbaseman]

Oh, resolvconf being installed and not working out of the box is the worst choice that Fedora could make.
If you compile openfortivpn from source you could abuse the configure option for a dirty hack:
./configure --with-resolvconf=/nonexistent
Then, it does not look for the resolvconf executable in the right place, compiles in this path, and does not find this executable at runtime, so it falls back to the old behavior.
Perhaps we should add a command line option to switch between resolvconf and the old behavior (even when resolvconf is available, but unfortunately not working)

[DimitriPapadopoulos]

@mrbaseman Yes, it looks like a command line option might be needed. What a pity, Fedora unilaterally broke resolvconf.

I fear the DNS options are already a bit confusing:

--set-dns
--no-dns
--pppd-use-peerdns
--pppd-no-peerdns

I'm wondering whether we shouldn't change to something like this, keeping old options for compatibility of course:

--set-dns=openfortivpn
--set-dns=pppd / --set-dns=ppp
--set-dns=resolvconf

We could also have a look at OpenVPN for inspiration - but they probably just rely on external scripts.

[mrbaseman]

I think we should add a new option but we should not change the existing ones.

My concern is: the more we change, the more difficulties come to light. The GUIs (NM plugin / openfortigui) also rely on these command line options and we have already broken them by accident a couple of times.

[DimitriPapadopoulos]

@mrbaseman What name for the new option?

Should we use resolvconf by default or not? Should we detect Fedora / Red Hat / CentOS or detect the broken resolvconf they ship and disable resolvconf automatically?

On Ubuntu 16.04 or 18.04:

$ resolvconf -h
resolvconf: Error: Command not recognized
Usage: resolvconf (-d IFACE|-a IFACE|-u|--enable-updates|--disable-updates|--updates-are-enabled)
$ 
$ echo $?
99
$ 

On Fedora / Red Hat / CentOS the output looks different:

$ resolvconf -h
resolvconf -a INTERFACE < FILE
resolvconf -d INTERFACE
[...]
$ 

@slartibart70 Also what is the output of echo $? just after resolvconf? I'm trying to find the exit code of either versions of resolvconf.

[slartibart70]

Fedora31

$ resolvconf
Expected either -a or -d on the command line.

exitcode:
$ echo $?
1

with the -h parameter:
$ resolvconf -h
resolvconf -a INTERFACE < FILE
resolvconf -d INTERFACE
[...]

$ echo $?
0

But,
wouldn't it be better to use
lsb_release -a
to get the info you're looking for?

[DimitriPapadopoulos]

With the real resolvconf the exit code is 99 in case of unknown/missing command line option:

$ resolvconf ; echo $?
resolvconf: Error: Command not recognized
Usage: resolvconf (-d IFACE|-a IFACE|-u|--enable-updates|--disable-updates|--updates-are-enabled)
99
dp165978@is223528 $ resolvconf -h ; echo $?
resolvconf: Error: Command not recognized
Usage: resolvconf (-d IFACE|-a IFACE|-u|--enable-updates|--disable-updates|--updates-are-enabled)
99
$ 

On Fedora is there a difference with/without option -h?

$ resolvconf -h ; echo $?

[DimitriPapadopoulos]

Unfortunately LSB packages are often not available (not installed by default?) so I wouldn't rely on lsb_ commands.

[slartibart70]

well, we still have
/etc/os-release being part of the systemd package or
/etc/*release for older systems
(see: http://linuxmafia.com/faq/Admin/release-files.html)

[DimitriPapadopoulos]

Sure, but in the end what matters is the version of resolvconf, not the OS.

[mrbaseman]

@mrbaseman What name for the new option?

good question. Maybe --use-resolvconf=<bool> similar to --pppd-use-peerdns

Should we use resolvconf by default or not?

Honestly, I don't know. On the one hand it seemed to be the right way to use resolvconf and stop messing around directly in the resolv.conf file, but now it turns out that resolvconf is sometimes broken by default (at least at the moment on Fedora).

Should we detect Fedora / Red Hat / CentOS or detect the broken resolvconf they ship and disable resolvconf automatically?

I don't feel comfortable with the idea putting a lot of effort into a mechanism to detect if something is broken on a specific platform. We already have a configure option (maybe it can be improved, e.g. RESOLVCONF_PATH=DISABLED means the runtime check for the availability of a binary called DISABLED is skipped, or maybe we add another configure option USE_RESOLVCONF=<1|0> to define the default for the new --use-resolvconf command line option). Maybe that's already too complicated. I don't know.

But essentially, if resolvconf shall be used by default, I'd say that's a decision that the packager should make. Based on the knowledge that resolvconf is broken on Fedora, I'd add a configure option in the spec file to disable it on that platform. If someone wants to configure resolvconv and wants to use it together with openfortivpn, he can still compile it from source with his own

Does that make sense?

[DimitriPapadopoulos]

@mrbaseman I was thinking of detecting the broken version of resolvconf at run-time - seems feasible. But I hear you, it is maybe too much effort - and more importantly fragile.

I like the idea of the configure option. Not sure about the effect of this option though:

• Should it disable resolvconf support altogether?
• Should it change the default from --use-resolvconf=1 to --use-resolvconf=0 so that end-users can still decide to call resolvconf?

Also how would this configure option be triggered?

• Should it be tiggered manually by Fedora / CentOS / EPEL packagers?
• Should we let configure automatically detect the version of resolvconf and decide whether to support it or not (of course we should also add a configure option to force either option manually)?

[DimitriPapadopoulos]

I would recommend a version 1.12.1 with just the change to fix this issue, #554 and #547.

While I'm confident my changes won't break anything because they are minor, it may be worth having patch versions 1.x.1 to fix issues newly introduced in minor versions 1.x.0.

[slartibart70]

just my opinion: using resolvconf seems to be the best way to handle the situation. Maybe we can add some checks (is the resolv.conf file really updated) and add some more explanatory log statements.
The end user will easily see if the dns is working or not, with the help of the log statements (hinting to the commandline options) they should figure out what to do next to make the vpn client work on their system
Too much automatism is likely to fail anyway

[DimitriPapadopoulos]

@slartibart70 Yes, I agree too much automation might fail.

On the other hand I believe resolvconf is not the way to go on Fedora / CentOS / EPEL. I agree with disabling it at compile-time.

Also detecting if resolv.conf has changed does not seem robust because:

• depending on the command line options, openfortivpn might trigger different update mechanisms (competing ones if multiple mechanisms are enabled which is a bad thing but still possible),
• in corner cases resolvconf might succeed but not change /etc/resolv.conf.

[slartibart70]

agreed, the check mechanism shoud not rely on the file itself.
But in case of logs - in fedora we have (see first comment here)
"Failed to set DNS configuration: Could not activate remote peer."
This could be hinting to the commandline options to (manually) prevent using resolvconf.

[mrbaseman]

@DimitriPapadopoulos I have been thinking about your questions:

I like the idea of the configure option. Not sure about the effect of this option though:

* Should it disable `resolvconf` support altogether?

* Should it change the default from `--use-resolvconf=1` to `--use-resolvconf=0` so that end-users can still decide to call `resolvconf`?

Both would be possible:

• ./configure --with-resolvconf=DISABLED would disable resolvconf support altogether. We could just compare RESOLVCONF_PATH to that string at runtime and fall back to the old behavior without checking for the existence of any binary. String comparison does not exist in C macros, so if we would aim for an #if ... #else ... #endif distinction we would have to do the comparison with autoconf macros with a...

AS_IF([test ...], [
        AC_DEFINE(...)
        AC_MSG_NOTICE([...])
],[
        AC_DEFINE(...)
        AC_MSG_NOTICE([...])
])

...along the lines of HAVE_PROC_NET_ROUTE

• Quite similarly, we could implement a --enable-resolvconf configure option that feeds in a default value which then can be overridden by a config file setting or the the --use-resolvconf command line option.

Also how would this configure option be triggered?

* Should it be tiggered manually by _Fedora_ / _CentOS_ / _EPEL_ packagers?

* Should we let _configure_ automatically detect the version of `resolvconf` and decide whether to support it or not (of course we should also add a configure option to force either option manually)?

we can implement an AC_TRY_RUN autoconf statement to detect if resolvconf is working or not, and decide a sane default based on that. In addition packagers can use the configure options discussed above to override this default (e.g. for cross-compiling).

[mrbaseman]

I have started to work on this in a new branch, in a first step a cleaner implementation of the workaround that I have suggested above a while ago (the first part of my previous comment).

Next would be the --enable-resolvconf configure option and the --use-resolvconf command line option and the according config file option.

Finally, we can think of an automatism to detect sane defaults at configure time and maybe a better solution for this initial commit, which avoids the string comparison at runtime.

[mrbaseman]

I have crated a first draft for a pull request

[DimitriPapadopoulos]

Meanwhile I have opened an issue on the Fedora bug tracker:
https://bugzilla.redhat.com/show_bug.cgi?id=1815605

Hopefully we can understand why resolvconf doesn't work on Fedora and perhaps find suggestions how to modify /etc/resolv.conf.

[DimitriPapadopoulos]

As a workaround until a new version of openfortivpn is released, you might move resolvconf out of the way so that openfortivpn cannot use it and falls back on the previous mechanism:

mv /usr/sbin/resolvconf /usr/sbin/resolvconf.bak

My guess is that it won't matter since resolvconf is broken on Fedora anyway.

[DimitriPapadopoulos]

Fixed in version 1.13.1.

Fedora / CentOS packages are on the way.

[DimitriPapadopoulos]

openfortivpn 1.13.1 has been packaged for Fedora:
https://koji.fedoraproject.org/koji/packageinfo?packageID=20983
and packages are in testing right now:
https://apps.fedoraproject.org/packages/openfortivpn

Do not hesitate to install and test them. On Fedora 31 for example:

sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-331db7bdf2

If it works for you and you have a Fedora account do not hesitate to add a karma point so that packages are released rapidly.

[angystardust]

Thanks @DimitriPapadopoulos I can confirm that the testing version works as expected on a F31 running openfortivpn from the cli. The dns entries are injected at the top of the /etc/resolv.conf
I didn't tested using NetworkManager-fortisslvpn yet

[ctsuzuki]

Hi, could you create a package to FC30, please ??
Iḿ having the same problem ....

[DimitriPapadopoulos]

Fedora 30 is not maintained anymore so I doubt there will be new packages:
https://fedoraproject.org/wiki/End_of_life