Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SysV init script can lead to multiple processes running #78

jessecarbon opened this issue Nov 2, 2016 · 2 comments

SysV init script can lead to multiple processes running #78

jessecarbon opened this issue Nov 2, 2016 · 2 comments


Copy link

@jessecarbon jessecarbon commented Nov 2, 2016

Problem description

I noticed this when using Graylog Collector Sidecar which uses this library to generate init scripts (see here).

The init script created for SysV systems (CentOS/RedHat 6) doesn't create a consistent PID file, which can lead to multiple processes running, especially if you try restarting the process after it's already been started on boot (very common).

The template here creates a PID file using the name of the running script. The problem is that SysV scripts are symlinked from their runlevel to the main script, and the filenames are different. So the script name called on boot is different from the script called from the service command.

For example when you boot a system into a runlevel, it will look for that level's startup scripts, in this case /etc/rc.d/rc3.d/S99collector-sidecar which is actually a symlink to the main startup script:

# runlevel
N 3

# ls -lh /etc/rc.d/rc3.d/S99collector-sidecar
lrwxrwxrwx. 1 root root 27 Nov  2 15:56 /etc/rc.d/rc3.d/S99collector-sidecar -> ../init.d/collector-sidecar

This means the PID file from the init on boot is named /var/run/ Now if you try to restart the service using the service command it's going to call /etc/rc.d/init.d/collector-sidecar (NOT S99collector-sidecar), and create a PID file at /var/run/ which is different. The result is the is_running() function doesn't find the original PID and now there's two processes running:

# service collector-sidecar restart
Not running
Starting collector-sidecar

# ps -ef | grep -i graylog
root      2819     1  0 15:58 ?        00:00:00 /usr/bin/graylog-collector-sidecar
root      2896  2819  0 15:58 ?        00:00:00 /usr/bin/filebeat -c /etc/graylog/collector-sidecar/generated/filebeat.yml
root      3070     1  0 16:00 pts/0    00:00:00 /usr/bin/graylog-collector-sidecar
root      3104  3070  0 16:00 pts/0    00:00:00 /usr/bin/filebeat -c /etc/graylog/collector-sidecar/generated/filebeat.yml

Proposed solutions

I would suggest setting the name variable to the symlink destination by using the readlink command, so this:

name=$(basename $0)

would become:

name=$(basename $(readlink -f $0))

Now both scripts result in the same filename, whether they were executed via symlink or not:

# basename $(readlink -f /etc/rc.d/init.d/collector-sidecar)

# basename $(readlink -f /etc/rc.d/rc3.d/S99collector-sidecar)

Optionally (or in addition to this), you could make the name variable itself be configurable in the template so that it can be set statically. Not sure if the template language supports optional parameters.

Copy link
Contributor Author

@jessecarbon jessecarbon commented Nov 2, 2016

Forgot to mention, I'd be happy to submit a PR for the readlink solution, if it's acceptable.

Copy link

@kardianos kardianos commented Nov 2, 2016

That would be totally acceptable. I'd love to tackle this, but I'm a bit
busy this week.

On Wed, Nov 2, 2016 at 1:46 PM Jesse Jarzynka

Forgot to mention, I'd be happy to submit a PR for the readlink solution,
if it's acceptable.

You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#78 (comment),
or mute the thread

jessecarbon added a commit to jessecarbon/service that referenced this issue Nov 3, 2016

SysV init scripts are ran on boot from symlinks in `/etc/rc.d/rc.X`, where
X is the runlevel. These symlinks are usually named something like
`S99foobar`, which is a link to the real script `foobar`. This leads to
differences in the pid_file filename and can lead to multiple processes
running. By using readlink to get the actual destination filename, the
pid_file name should always be consistent.

Resolves: kardianos#78
@kardianos kardianos closed this in #79 Nov 4, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants