Do not kill process on service shutdown

[tlrx]

When installed as a service with a DEB or RPM package, we should gently wait for elasticsearch to stop (flushing indices on closing can take some time) and never kill the process.

Closes #11248

[tlrx]

@costin the Windows service manager accepts a ES_STOP_TIMEOUT variable which is set to 0. I don't really find any documentation about this. I guess procrun is used but I'm not sure of how the process is terminated on Windows. Do you have any clue or pointer? Thanks.

[costin]

From the (commons-daemon docs](
http://commons.apache.org/proper/commons-daemon/procrun.html):

--StopTimeoutNo TimeoutDefines the timeout in seconds that procrun waits
for service to exit gracefully.

On Thu, Jul 16, 2015 at 10:48 PM, Tanguy Leroux notifications@github.com
wrote:

@costin https://github.com/costin the Windows service manager accepts a
ES_STOP_TIMEOUT variable which is set to 0. I don't really find any
documentation about this. I guess procrun is used but I'm not sure of how
the process is terminated on Windows. Do you have any clue or pointer?
Thanks.

—
Reply to this email directly or view it on GitHub
#12298 (comment)
.

[tlrx]

@costin I supposed - wrongly - that 0 had a specific signification compared with no timeout at all. So it means that nothing need to be done in windows scripts.

Thanks for your quick response.

[nik9000]

Complains about no newline at end of file.

[nik9000]

So I'm right there with you that kill -9 is going to cause trouble. But maybe we should have some more documentation on what to do if the process doesn't die? When you hit a bug that eats all the memory nothing is going to stop elasticsearch but kill -9. Maybe a --force option or something. Its just that I've relied on this kill -9 behavior in the past when some nodes have filled their heap.

[clintongormley]

I'm OK with this going in without a force option. What does any sysadmin do if any process won't stop? They kill -9 it. I don't think we need to document everything here. I'd rather get the fix in.

[tlrx]

@nik9000 I tend to agree with @clintongormley on this. I rebased the code, can I push this?

[nik9000]

Sure
On Aug 7, 2015 4:37 AM, "Tanguy Leroux" notifications@github.com wrote:

@nik9000 https://github.com/nik9000 I tend to agree with @clintongormley
https://github.com/clintongormley on this. I rebased the code, can I
push this?

—
Reply to this email directly or view it on GitHub
#12298 (comment)
.

[tlrx]

@nik9000 thanks!

[sathieu]

I had a problem with this setting blocking a server shutdown. Maybe change this to 10 minutes or so ?

(NB: the problem was coming from a snapshot directory on an unavailable NAS)

[tlrx]

Maybe change this to 10 minutes or so ?

We can't do that: the scripts must wait for Elasticsearch to stop nicely and it can sometimes takes more than 10 min. The good thing is that it let you the time to investigate the issue and take the appropriate action (here, kill the node manually?). Doing this automatically in the scripts would have obfuscate the underlying issue.

Also, if you have logs and a reproducing scenario it might worth it to create a new issue for the snapshot vs unavailable NAS.

[sathieu]

The problem is that it will prevent the shutdown, but I can't investigate anything as sshd is stopped and even I can't have a console (ttys are already closed).