Remove manual logstash process management #2
Conversation
|
We already have init suport for logstash and elasticsearch since the last version. We use v2 resources for this the v1 have been deprecated since version 0.2.3. POST /dmon/v2/overlord/core/es The init scripts used can be found here. |
|
I should also mention that under heavy load issuing restart or stop to logstash is not deterministic and calling kill -9 is the only way to effectively stop it. |
tadeboro
force-pushed
the
fix/remove-logstash-kill
branch
3 times, most recently
from
aaff7ce
to
871b823
Compare
tadeboro
force-pushed
the
fix/remove-logstash-kill
branch
2 times, most recently
from
ccf1051
to
99f709a
Compare
tadeboro
force-pushed
the
fix/remove-logstash-kill
branch
from
99f709a
to
a7fb00e
Compare
tadeboro
force-pushed
the
fix/remove-logstash-kill
branch
from
a7fb00e
to
46e3feb
Compare
Logstash process is managed by system service, which makes it dangerous to call kill -9 on it manually. Instead of manual intervention, we let the system tools take care of the process management. Manually killing the logstash process was required since SystemV init scripts cannot handle processes that do not terminate voluntarily. By using upstart to manage the logstash process, we bypass this problem completely, since any process that does not terminate voluntarily withing five seconds is sent SIGKILL that forcefully terminates it.
tadeboro
force-pushed
the
fix/remove-logstash-kill
branch
from
46e3feb
to
0546ee0
Compare
|
Updated patch now allows controlling the logstash heap size environment variable. This makes it possible to set the heap size at runtime. As for logstash not terminating voluntarily, this is a non-issue with newer init systems, since they forcefully terminate the process after some grace period. In case of upstart that we are using, process is killed after 5 seconds. |
Logstash process management has been delegated to system's init system and manually killing them is not an option anymore, since this can create some really nasty race conditions. This pull requests removes offending code and redirects all of the process management to init helpers.
Note that because of this change, dmon now needs to run as root user. It would be better to have dmon call sudo for any operations that need elevated privileges, but this change would affect large parts of the DMon's code, so it was not done in this pull request,