olad not waiting for network to init

[iFreilicht]

When olad starts on my Raspberry Pi, it is quicker than the network module, so InterfacePicker fails to find any interfaces:

~ sudo service olad status
● olad.service - LSB: OLA daemon
   Loaded: loaded (/etc/init.d/olad; generated; vendor preset: enabled)
   Active: active (running) since Mon 2019-03-11 21:25:05 UTC; 14min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 540 ExecStart=/etc/init.d/olad start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/olad.service
           └─554 /usr/bin/olad --syslog --log-level 3 --config-dir /etc/ola

Mar 11 21:25:06 raspberrypi olad[554]: olad/PluginManager.cpp:73: Skipping StageProfi because it was disabled
Mar 11 21:25:06 raspberrypi olad[554]: olad/PluginManager.cpp:73: Skipping Serial USB because it was disabled
Mar 11 21:25:06 raspberrypi olad[554]: olad/PluginManager.cpp:73: Skipping USB because it was disabled
Mar 11 21:25:06 raspberrypi olad[554]: olad/PluginManager.cpp:73: Skipping Pathport because it was disabled
Mar 11 21:25:06 raspberrypi olad[554]: olad/PluginManager.cpp:73: Skipping FTDI USB DMX because it was disabled
Mar 11 21:25:06 raspberrypi olad[554]: olad/PluginManager.cpp:73: Skipping UART native DMX because it was disabled
Mar 11 21:25:06 raspberrypi olad[554]: olad/PluginManager.cpp:195: Trying to start ArtNet
Mar 11 21:25:06 raspberrypi olad[554]: common/network/InterfacePicker.cpp:59: No interfaces found
Mar 11 21:25:06 raspberrypi olad[554]: plugins/artnet/ArtNetDevice.cpp:98: Failed to find an interface
Mar 11 21:25:06 raspberrypi olad[554]: olad/PluginManager.cpp:198: Failed to start ArtNet

Restart the service and everything works fine:

~ sudo service olad stop
~ sudo service olad start
~ sudo service olad status
● olad.service - LSB: OLA daemon
   Loaded: loaded (/etc/init.d/olad; generated; vendor preset: enabled)
   Active: active (running) since Mon 2019-03-11 21:39:54 UTC; 5s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1159 ExecStop=/etc/init.d/olad stop (code=exited, status=0/SUCCESS)
  Process: 1221 ExecStart=/etc/init.d/olad start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/olad.service
           └─1227 /usr/bin/olad --syslog --log-level 3 --config-dir /etc/ola

Mar 11 21:39:54 raspberrypi olad[1227]: olad/PluginManager.cpp:73: Skipping Serial USB because it was disabled
Mar 11 21:39:54 raspberrypi olad[1227]: olad/PluginManager.cpp:73: Skipping USB because it was disabled
Mar 11 21:39:54 raspberrypi olad[1227]: olad/PluginManager.cpp:73: Skipping Pathport because it was disabled
Mar 11 21:39:54 raspberrypi olad[1227]: olad/PluginManager.cpp:73: Skipping FTDI USB DMX because it was disabled
Mar 11 21:39:54 raspberrypi olad[1227]: olad/PluginManager.cpp:73: Skipping UART native DMX because it was disabled
Mar 11 21:39:54 raspberrypi olad[1227]: olad/PluginManager.cpp:195: Trying to start ArtNet
Mar 11 21:39:55 raspberrypi olad[1227]: olad/plugin_api/DeviceManager.cpp:105: Installed device: ArtNet [192.168.178.38]:2-1
Mar 11 21:39:55 raspberrypi olad[1227]: olad/plugin_api/Universe.cpp:522: Full RDM discovery triggered for universe 1
Mar 11 21:39:55 raspberrypi olad[1227]: olad/plugin_api/PortManager.cpp:119: Patched 2-1-I-0 to universe 1
Mar 11 21:39:55 raspberrypi olad[1227]: olad/PluginManager.cpp:200: Started ArtNet

This would be ok if olad correctly reported that it didn't start correctly, systemd would then restart it automatically, but it reports success, so nothing will happen until I restart manually.

[peternewman]

Hi @iFreilicht ,

Briefly initially, but more in the near future, this question is asked a lot, have a look here and on our forum ( https://groups.google.com/forum/#!forum/open-lighting ). I think you can configure systemd to wait for networking. A lack of networking isn't inherently a failure for OLA (e.g. standalone RDM testing, DMX to SPI, DMX playback etc), so we wouldn't want systemd to keep restarting it in these cases.

I believe from past discussion we should ideally just handle interfaces better as they come up/go down. In the short term, SIGHUP reloads our plugins, so firing this on a network change should also have the desired result.

There's also a systemd pull request I need to find some time to re-review:
#1444

[iFreilicht]

Thanks a lot for the pointer. I found this thread that describes the same problem, but the solution doesn't seem to work for me.

I'll make a new post there, hopefully someone can help me out.

[gobo-ws]

This workaround works:

Edit the file: /etc/rc.local
Add the following lines before the line exit 0 at the end, then save the file and exit.

echo "Sleep 15 seconds before starting olad"  
sleep 15  
echo "Starting olad"  
sudo -u oladuser olad -f --syslog

[iFreilicht]

@gobo-ws Thanks! Still, I'd like to get to the bottom of this.

Right now it seems like there is a problem with the network-online.target in systemd on Raspbian, meaning that it completes even before any interfaces are actually initialised. I'll work on fixing this properly and get back to you.

[shenghaoyang]

@peternewman can the art-net plugin listen on the wildcard address? From what's recommended on:
https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ that is unconditionally available.

To restrict access and block connections, one can use the IP accounting facilities of systemd (since we're knee deep into using using it, anyway):
http://0pointer.net/blog/ip-accounting-and-access-lists-with-systemd.html

They also suggest a linux specific flag: IP_FREEBIND, which sounds promising. Maybe the user can statically configure an IP address and we listen on those without discovering interfaces?

(Not sure how that will affect Art-Net operation)

Shipping a "wait-for-network" in the unit file might not be a good idea, actually. What constitutes as the network being up would likely be different for each user and shipping a general unit file that works for one group may lead to the service not starting for others.

[iFreilicht]

I think what would also help in this case is to have an option for olad to exit with an error code if one of the plugins fails to load. That way systemd would know if an error occurred and one could just use Restart=on-failure.

@shenghaoyang Interestingly, the ola http interface already lists that as "Primary IP" when it fails to find an interface during startup, but that's probably just a standard value.

And yeah, wait-for-network is really hard to define. I thought having an IP and Hostname would be enough, but even after that olad starting without an additional delay couldn't find an interface.

[peternewman]

@peternewman can the art-net plugin listen on the wildcard address? From what's recommended on:
https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ that is unconditionally available.

I believe we do currently, but it was on the roadmap to move away from it to provide multiple interfaces if necessary, or do various other curious things:
https://wiki.openlighting.org/index.php/OLA_ArtNet_Roadmap#Multiple_ArtNet_Devices

[peternewman]

I think what would also help in this case is to have an option for olad to exit with an error code if one of the plugins fails to load. That way systemd would know if an error occurred and one could just use Restart=on-failure.

Do you want to open a new issue for that as a possible feature request @iFreilicht ? I think it would need some fine tuning, like if you want any failure to generate, a failure, or only none working, and pre-defining what to expect, but there's no harm in having the idea and someone may implement it.

[peternewman]

Shipping a "wait-for-network" in the unit file might not be a good idea, actually. What constitutes as the network being up would likely be different for each user and shipping a general unit file that works for one group may lead to the service not starting for others.

I'd agree with that. I think it wants to be generally safe. Unless I've missed something, firing off a nohup when an interface changes ought to work. Personally I'd rather move towards making it more dynamic, so e.g. plugging in a USB NIC just appears while OLA is running, in the same way as some of the USB dongles do. I assume it would just appear in the OS already, so it ought to show up for us too. I think this would need the rtnetlink stuff mentioned here http://man7.org/linux/man-pages/man7/rtnetlink.7.html which we already do a little of for default route detection.

@shenghaoyang Interestingly, the ola http interface already lists that as "Primary IP" when it fails to find an interface during startup, but that's probably just a standard value.

Just for the avoidance of doubt @iFreilicht , what IP does it list as the Primary IP?

[peternewman]

They also suggest a linux specific flag: IP_FREEBIND, which sounds promising. Maybe the user can statically configure an IP address and we listen on those without discovering interfaces?

I wonder if the IP_FREEBIND stuff will benefit or just cause confusion. If the plugin starts, but doesn't actually work as the interface isn't up, is that more clear? If we ask the user to pre-enter the IP they want manually, that seems like it could just cause even more problems than it solves.

[iFreilicht]

I think what would also help in this case is to have an option for olad to exit with an error code if one of the plugins fails to load. That way systemd would know if an error occurred and one could just use Restart=on-failure.

Do you want to open a new issue for that as a possible feature request @iFreilicht ? I think it would need some fine tuning, like if you want any failure to generate, a failure, or only none working, and pre-defining what to expect, but there's no harm in having the idea and someone may implement it.

Sure, I'll get onto that ASAP. Maybe I'll even find the time to implement it myself.

Just for the avoidance of doubt @iFreilicht , what IP does it list as the Primary IP?

0.0.0.0