Retry start of salt-minion under systemd

[wchao]

I have noticed that salt-minion sometimes fails to start on a server (especially an LXC container or VM where things may be a little slow during startup) when using systemd. If I merely wait 30 seconds and run systemctl start salt-minion, things work fine.

Can you add something to the systemd service file (/lib/systemd/system/salt-minion.service) so that the service attempts to restart five times, separated by 15 seconds each time? I think that would make salt-minion more robust during boot.

[eliasp]

@wchao Simply trying to start until it succeeds is a rather non-deterministic approach and just random shooting at things.
We should instead identify the reason why it isn't starting.

Could you provide the journal of the salt-minion service?: journalctl -b -u salt-minion

It might help to increase the log level to debug by doing this before the next reboot:

• Create the file /etc/systemd/systemd/salt-minion.service.d/log-level.conf
• Add the following content to this file:

[Service]
ExecStart=/usr/bin/salt-minion -l debug

[wchao]

Here is what I see:

-- Logs begin at Mon 2015-01-05 22:08:28 EST, end at Tue 2015-01-06 07:52:11 EST. --
Jan 05 22:08:46 swan.microoffice.com systemd[1]: Starting The Salt Minion...
Jan 05 22:08:46 swan.microoffice.com systemd[1]: Started The Salt Minion.
Jan 05 22:10:05 swan.microoffice.com salt-minion[2216]: [ERROR   ] Attempt to authenticate with the salt master failed
Jan 05 23:22:00 swan.microoffice.com systemd[1]: Starting The Salt Minion...
Jan 05 23:22:00 swan.microoffice.com systemd[1]: Started The Salt Minion.

As you point out, it is a bit nondeterministic, so I can't reproduce it every time. It happens to some hosts when a bunch of them start up at once, especially on the same VM/container host.

Where do I create log-level.conf? I don't see an /etc/systemd/systemd directory, although I do see an /etc/systemd/system directory, but there is no salt-minion.service.d subdirectory. This is what I see in /etc/systemd/system:

[root@swan ~]# ls -l /etc/systemd/system
total 8
drwxr-xr-x. 2 root root   54 Sep 23 11:03 basic.target.wants
lrwxrwxrwx. 1 root root   41 Sep 23 11:02 dbus-org.fedoraproject.FirewallD1.service -> /usr/lib/systemd/system/firewalld.service
lrwxrwxrwx. 1 root root   44 Sep 23 11:00 dbus-org.freedesktop.Avahi.service -> /usr/lib/systemd/system/avahi-daemon.service
lrwxrwxrwx. 1 root root   37 Sep 23 11:05 default.target -> /lib/systemd/system/multi-user.target
drwxr-xr-x. 2 root root   85 Sep 23 11:00 default.target.wants
drwxr-xr-x. 2 root root   31 Sep 23 11:00 getty.target.wants
drwxr-xr-x. 2 root root 4096 Dec 14 21:26 multi-user.target.wants
lrwxrwxrwx  1 root root    9 Dec 14 21:43 NetworkManager.service -> /dev/null
drwxr-xr-x. 2 root root   29 Sep 23 11:00 nfs.target.wants
drwxr-xr-x. 2 root root 4096 Sep 23 11:00 sockets.target.wants
drwxr-xr-x. 2 root root   85 Sep 23 11:01 sysinit.target.wants
drwxr-xr-x. 2 root root   43 Sep 23 11:00 system-update.target.wants

Did you mean /lib/systemd/system? I still don't see any "*.d" subdirectories in that directory, either.

[eliasp]

Here is what I see: […]

Not really much which could point out, why salt-minion isn't starting… I hope increasing the log level will help here.

Where do I create log-level.conf? I don't see an /etc/systemd/systemd directory, although I do see an /etc/systemd/system directory, but there is no salt-minion.service.d subdirectory

Just create the directory.
mkdir -p /etc/systemd/system/salt-minion.service.d/

[jfindlay]

@wchao, thanks for the report. As @eliasp indicates, the debug log information from when this happens would be useful; also please include the output from salt --versions-report and salt 'minion' test.versions_report, thanks.

[stromnet]

Hi,

not sure if this is related but I've had problems with startup on FreeBSD 10.1. It turned out to be the minion shutting down if it could not reach the master within 60 seconds, which it couldn't due to connectivity delay on boot (VPNs).

With debug logging, I can see this on boot:

2015-02-21 12:11:58,736 [salt             ][INFO    ] Setting up the Salt Minion "myhostname"
2015-02-21 12:11:58,754 [salt.utils.process][DEBUG   ] Created pidfile: /var/run/salt-minion.pid
2015-02-21 12:11:58,756 [salt.config      ][DEBUG   ] Reading configuration from /usr/local/etc/salt/minion
2015-02-21 12:12:00,448 [salt.minion                              ][DEBUG   ] Attempting to authenticate with the Salt Master at 172.28.1.31
2015-02-21 12:12:00,465 [salt.crypt                               ][DEBUG   ] Loaded minion key: /usr/local/etc/salt/pki/minion/minion.pem
2015-02-21 12:13:00,607 [salt                                     ][ERROR   ] Attempt to authenticate with the salt master failed

After that, the process seems to have quit. In my case, the master isn't reachable due to a VPN+routing protocol session not being up within the 60 seconds. Increasing the auth_retries from 1 proved to be successful:

# Number of consecutive SaltReqTimeoutError that are acceptable when trying to
# authenticate.
#auth_tries: 1
# If authentication fails due to SaltReqTimeoutError, continue without stopping the
# minion.
#auth_safemode: True

However, I did not expect it to shutdown? My interpretation of the auth_safemode default would be that it does NOT shutdown if it cannot reach it. Also, a default of 1 retry seems a bit low?

# salt --versions-report
       Salt: 2014.7.2
     Python: 2.7.9 (default, Feb 16 2015, 07:03:30)
     Jinja2: 2.7.3
   M2Crypto: 0.22
msgpack-python: 0.4.2
msgpack-pure: Not Installed
   pycrypto: 2.6.1
    libnacl: Not Installed
     PyYAML: 3.11
      ioflo: Not Installed
      PyZMQ: 14.3.1
       RAET: Not Installed
        ZMQ: 4.0.5
       Mako: Not Installed