Coturn may start before network is fully configured

[ruedigerkupper]

This is a boot-time race condition appearing with automatic discovery of listener addresses.

When coturn is starts up without any explicit listener addresses configured, it automatically detects them on startup. By default, it will listen to all available non-local addresses. If it cannot discover any non-local address, startup will fail. (Error log: "0: main: Cannot configure any meaningful IP listener address")

Many installation instructions recommend configuring coturn without specific listener addresses set (e.g. this one, which brought me here: https://nextcloud-talk.readthedocs.io/en/latest/TURN/)

Analysis

The systemd service file for coturn has "After=network.target" in it. This only means that the network is generally up, but it does not require non-local addresses configured.

This is usually no problem if the user starts the service manually, because the network is completely configured at this time. In contrast, during boot up systemd may well launch coturn before IP addresses are fully configured, causing start up to fail.

Solutions

There are two possible solutions:

1. Configure explicit listener addresses in turnserver.conf.
   Doing this I see from the logs that my coturn server keeps trying to bind to these addresses, which in my case takes 27 attempts, but then works. And there's another problem I can't get around this way (unless you have static IPs): I cannot be sure about my machine's IPv6 address, as this may be changed by my provider at any time. So I cannot specify my IPv6 listener address in turnserver.conf.
2. Have the service file include the "network-online" target, instead of the "network" target. The user can achieve this by calling "systemctl edit coturn" and putting the following into the file:

 [Unit]
 After=network-online.target
 Wants=network-online.target

This will make systemd delay the start of coturn, until all network interfaces have their IPs configured.
See https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ for a discussion.

The second option works fine for me, but there may be disadvantages of using "network-online" instead of "network". In particular it may(?) slow down the boot process unnecessarily in case of static IPs.

Suggestion

Change target "network" to "network-online" in the systemd service file. This will make coturn start up correctly for all users. If this is not desirable, at least warn users who use coturn without explicit listener addresses configured that startup upon boot is likely to fail.

[ruedigerkupper]

Another possibility would be to give the user the choice at package install time. Like "Will this coturn instance run on a machine with static IP configured?" and then change the target accordigly.

[Neustradamus]

@misi: Can you look?

[TinfoilSubmarine]

I have also ran into this issue. I did the same thing that was mentioned in the OP (specifying network-online.target instead of just network.target). However, in my case, even this does not bind to all addresses; it only binds to my local IPv4 address and the IPv4 and IPv6 loopback addresses. In contrast, when I restart the service later, it binds to all of my SLAAC IPv6 addresses as well as the local IPv4 and loopback addresses.

[wferi]

As the already referenced https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ explains, the best general solution is binding the server to the always available loopback or wildcard addresses instead of specific addresses, because that way the server can adapt to a dynamically changing network environment. The other solutions are either substantially more complicated and non-portable (like explicitly acting on network change events or using IP_FREEBIND) or less flexible (like configuring specific addresses and starting/stopping the daemon as they come and go).

[ruedigerkupper]

Can you please elaborate on that or point me to some explaining ressources? I do not know how I would operate a TURN server bound to localhost? It is the the whole point it is reachable from outside? How do I manage that if it listens on localhost?
I'm sure I'm missing something, sorry for my stupid question and thanks for your help.

[ruedigerkupper]

See also #438

[wferi]

No, you're right. My point (quoted from the NetworkTarget documentation) was that the default bind address should be one which is always available, that is either loopback or wildcard. Of course running a TURN server bound to the loopback address isn't a useful service in general. The NetworkTarget documentation explains the reasoning behind this principle in great detail, I can't really add anything to that. I don't even know offhand whether coturn specifically supports binding to the wildcard address, but in my opinion it really should, since that's a more general and more resilient solution than using the blunt network-online.target. Those who want something more fancy can still stitch that together, but the default should just work (preferably in a simple way) in my opinion.

[Neustradamus]

Guys, have you tested: https://github.com/processone/eturnal?
Good or bad?

[ruedigerkupper]

@wferi: Just tried and you were right: Coturn will happily bind to the wildcard address if you explicitly ask it to do so. Just put listening-ip=0.0.0.0 into the conf file. It's working fine and there is no need for using network-online.target.
Of course, this is only acceptable if it is really safe for your server to listen on all addresses. But in my case (home server behind a cable router acting as firewall) that's fine. Thank you!

Note: Perhaps this should be the default behavior for coturn, instead the auto-discovery of addresses which usually fails at boot time?

[wferi]

I also support the wildcard default, because that works with dynamic address changes as well. Preferably with IPv6 wildcard (::) even.

[misi]

@ruedigerkupper Please accept my apologies for my late reply.
@wferi Thanks for your great and valuable help!
I think changing default listener behavior is a breaking change, so we need to be careful.
If I understand correctly the network-online target will only delay little bit the start.

I see in actual systemd unit file the following.

After=network.target
After=network-online.target
After=remote-fs.target
Wants=network-online.target

@wferi Can you help I am not sure, am I right/wrong that we need to remove After=network.target from this list?

[misi]

I added new option --systemd. To allow better integration with systemd. Notify systemd about coturn status, and start in systemd coturn not as forked type service but notified type service.
See commit ef79168

[misi]

It is closed by mistake

[paulmenzel]

systemd notify integration is a very nice. Thank you for that.

How does Coturn handle removed network interfaces? I think the optimal solution would be, that Coturn would discover the network interfaces dynamically, so it would be able to start without any interfaces set up yet, and then gets notified about new interfaces or detects them itself.

[misi]

Unfortunately AFAIK coturn does not handle dynamic interface removal or add :(
If config not specify listener/relay ips, then it discovers during start the actual interfaces and ip addresses, and it will listen on these addresses..

[wferi]

I think changing default listener behavior is a breaking change, so we need to be careful.

Strictly speaking, you needn't change the default (enumeration and use of currently available addresses) behaviour, it's enough to add -L :: to the default ExecStart directive. That isn't very elegant, though; are there real use cases that the behaviour change would break?

If I understand correctly the network-online target will only delay little bit the start.

That "little bit" may be arbitrarily long, but the main problems are that

1. the network-online.target is rather weakly defined (for example it might race with IPv6 SLAAC) ; and
2. later network events may also change the set of usable addresses, but you can't adapt to those with a single startup-time enumeration.

I see in actual systemd unit file the following.

After=network.target
After=network-online.target
After=remote-fs.target
Wants=network-online.target

@wferi Can you help I am not sure, am I right/wrong that we need to remove After=network.target from this list?

After=network.target basically means that your unit will be stopped before the network is torn down. It makes sense if you want to cleanly terminate application-level network protocols, but can also work against you if you'd rather not break UDP forwarding for a network restart (just an example, I didn't check if that's actually viable with coturn).

I'd also recommend looking into systemd socket activation, which is somewhat more involved than notify support, but nicely sidesteps the question of ordering.

[eTaurus]

So how do I work around this before 4.5.3 milestone? Is it by using listening-ip=0.0.0.0?
I have this problem since upgrading from Debian Buster to Bullseye. Was working well before.

[ruedigerkupper]

Yes, this should work. But be sure that it is safe to do so in your network.