New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SysV init script can lead to multiple processes running #78

Closed
jessecarbon opened this Issue Nov 2, 2016 · 2 comments

Comments

Projects
None yet
2 participants
@jessecarbon
Contributor

jessecarbon commented Nov 2, 2016

Problem description

I noticed this when using Graylog Collector Sidecar which uses this library to generate init scripts (see here).

The init script created for SysV systems (CentOS/RedHat 6) doesn't create a consistent PID file, which can lead to multiple processes running, especially if you try restarting the process after it's already been started on boot (very common).

The template here creates a PID file using the name of the running script. The problem is that SysV scripts are symlinked from their runlevel to the main script, and the filenames are different. So the script name called on boot is different from the script called from the service command.

For example when you boot a system into a runlevel, it will look for that level's startup scripts, in this case /etc/rc.d/rc3.d/S99collector-sidecar which is actually a symlink to the main startup script:

# runlevel
N 3

# ls -lh /etc/rc.d/rc3.d/S99collector-sidecar
lrwxrwxrwx. 1 root root 27 Nov  2 15:56 /etc/rc.d/rc3.d/S99collector-sidecar -> ../init.d/collector-sidecar

This means the PID file from the init on boot is named /var/run/S99collector-sidecar.pid. Now if you try to restart the service using the service command it's going to call /etc/rc.d/init.d/collector-sidecar (NOT S99collector-sidecar), and create a PID file at /var/run/collector-sidecar.pid which is different. The result is the is_running() function doesn't find the original PID and now there's two processes running:

# service collector-sidecar restart
Not running
Starting collector-sidecar

# ps -ef | grep -i graylog
root      2819     1  0 15:58 ?        00:00:00 /usr/bin/graylog-collector-sidecar
root      2896  2819  0 15:58 ?        00:00:00 /usr/bin/filebeat -c /etc/graylog/collector-sidecar/generated/filebeat.yml
root      3070     1  0 16:00 pts/0    00:00:00 /usr/bin/graylog-collector-sidecar
root      3104  3070  0 16:00 pts/0    00:00:00 /usr/bin/filebeat -c /etc/graylog/collector-sidecar/generated/filebeat.yml

Proposed solutions

I would suggest setting the name variable to the symlink destination by using the readlink command, so this:

name=$(basename $0)

would become:

name=$(basename $(readlink -f $0))

Now both scripts result in the same filename, whether they were executed via symlink or not:

# basename $(readlink -f /etc/rc.d/init.d/collector-sidecar)
collector-sidecar

# basename $(readlink -f /etc/rc.d/rc3.d/S99collector-sidecar)
collector-sidecar

Optionally (or in addition to this), you could make the name variable itself be configurable in the template so that it can be set statically. Not sure if the template language supports optional parameters.

@jessecarbon

This comment has been minimized.

Contributor

jessecarbon commented Nov 2, 2016

Forgot to mention, I'd be happy to submit a PR for the readlink solution, if it's acceptable.

@kardianos

This comment has been minimized.

Owner

kardianos commented Nov 2, 2016

That would be totally acceptable. I'd love to tackle this, but I'm a bit
busy this week.

On Wed, Nov 2, 2016 at 1:46 PM Jesse Jarzynka notifications@github.com
wrote:

Forgot to mention, I'd be happy to submit a PR for the readlink solution,
if it's acceptable.


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#78 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAuFsY3dzt6kC-bbGE-EzDsKQqV2fJBbks5q6PargaJpZM4KnumI
.

jessecarbon added a commit to jessecarbon/service that referenced this issue Nov 3, 2016

Fix SysV pid_file to be consistent when ran from a differently named …
…symlink

SysV init scripts are ran on boot from symlinks in `/etc/rc.d/rc.X`, where
X is the runlevel. These symlinks are usually named something like
`S99foobar`, which is a link to the real script `foobar`. This leads to
differences in the pid_file filename and can lead to multiple processes
running. By using readlink to get the actual destination filename, the
pid_file name should always be consistent.

Resolves: kardianos#78

@kardianos kardianos closed this in #79 Nov 4, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment