[FEATURE] Add option to write a pid file

[hron84]

It would be good if logstash can write a pid file if needed. Maybe it would be a startup argument.

[IngaFeick]

+1

[jordansissel]

What would you use it for?

On Wednesday, July 8, 2015, Gabor Garami notifications@github.com wrote:

It would be good if logstash can write a pid file if needed. Maybe it
would be a startup argument.

—
Reply to this email directly or view it on GitHub
#3577.

[hron84]

Integrate it with Monit, as Monit can track daemons via pid number that read from pid file. Regex matching is also an option, but could be more difficult than a simple pid file.

[IngaFeick]

Same here. Monitoring / restarting.

[jordansissel]

With modern operating systems (Windows, major Linux sisters, Solaris,
Illumos, OSX, etc) the service managers (systemd , upstart, smf, launchd,
etc) all provide a way to query the process state (alive, pid, etc). I
don't really feel we need pid files anymore. I am open to discussion,
though.

On Wednesday, December 23, 2015, IngaFeick notifications@github.com wrote:

Same here. Monitoring / restarting.

—
Reply to this email directly or view it on GitHub
#3577 (comment).

[hron84]

With really modern operating systems (Linux, BSD, OS X) you can change how do you start your services and how do you use them. Maybe the shipped starting method is works for you, maybe not. And, if you choose to not use the recommended way just because it does not fits to your requirements or you have a thing that blocks you from using that way, you will be left alone if you still want to track your service status.

Guys, seriously, we talking about writing a single file with only one number to a configurable place at startup. It is not a big-big feature like exposing an XMLRPC interface for queries, just writing a file as a part of the bootstrap process. I am not a full-time developer but I know bits and bolts about the development and this is not a very hard thing to implement (I just do not know Java and LogStash internals enough to make a PR for this).

And for being precise: service managers are not all providing way to query the state of the process and even if they do, they sometimes doing it wrong, provide invalid or old data about the state. It's because not all software written to fits to all service managers requirements and some managers are not adaptive enough. PID files are helping us to validate the process state against the real process list independently from what service managers, bosses or other things say. That's all we want to achieve, nothing more.

[IngaFeick]

We need it for monit. We'll find a way to work around this, but having a pid file would be really nice and the cleanest approach.

[mre]

+1

[AlexanderThaller]

You can't expect that everybody moves to a Linux distribution which already has a modern init system (especially in enterprise environments). So having an easy way to get the PID would make life so much easier.

[jordansissel]

@AlexanderThaller We have an API in Logstash now that could allow this, would that help? I'm not really comfortable adding a pidfile when the best contract we can offer is that "maybe the file contains the correct pid" (because if logstash is destroyed before shutting down, the pidfile will contain a lie)

[jordansissel]

I confess that "write a pidfile" is a super super simple feature and easy to implement. However, It's more complex than it seems, right? I don't like that whenever Logstash isn't running, that pidfile is a lie, and that permission issues would cause it to fail to write the pidfile (and what should Logstash do if that fails?), and if the directory doesn't exist, or if the filesystem is full, or if someone runs two Logstash instances with the same pidfile setting, etc.

If everyone agrees that the concerns in my previous paragraph are not meaningful concerns, then I am happy to add this pidfile option and document the cases in which it will fail or produce lies. At this point, I am willing to implement it, I just don't' want to be on the hook for the known cases where the pidfile concept produces errors or lies. Let me know :)

[hron84]

@jordansissel Unexpectedly stopped processes leave behind pid files that are a lie, this is how pid files work, everyone knows this issue with them. This is not a problem. All things that uses pid files also ensures the process is exists or not. Process IDs in PID files aren't handled as a statement of the process state, they're more like just a pointer how and where you can find the process of the service. It is super important in cases of Java-based services, because too much Java process can run, and the executable name does not contain any info about the process itself. So pidof, killall and other utilities has no chance to find the process based on the executable name. Imagine a server with Logstash and ElasticSearch on it.

The namespace of process IDs is huge enough to not fear about process ID collision.The chance of other process "steals" the ID of the dead Logstash process is very low on a production system (until someone releases a fork bomb, but that is not a normal case). I hope it helps you to understand the underlying idea.

[jordansissel]

everyone knows this issue with them

I do not believe everyone knows this. We don't really need to debate this, though.

It is super important in cases of Java-based services

The jps tool is quite nice for finding java processes (it only shows java processes).

killall and other utilities

Check out pgrep and its friend pkill - they are lovely :)

The namespace of process IDs is huge enough to not fear about process ID collision

I have experienced issues in production with collisions.

---

Downplaying the risks isn't something I want to really discuss, because the risks are real and everyone may have different experiences :)

What I'd like to know is -- for folks interested in this feature -- are you accepting the risks that:

• Process IDs are reused
• pidfiles are left behind after process/machine death and can be lies
• Not all scenarios can allow for safe use of pidfiles (file locks via flock(2))
• A pidfile might not even be written because of file system issues (permissions, invalid directory, out of space, etc).

If you accept these risks, please do a +1 on this comment using Github's new comment reactions. If there are a few +1s, I'll agree to implement the following (I am open to changing the behavior proposal below, let me know):

• Add a new flag --pid-file that lets you specify the location (directory or file) to write a pidfile
• The file will contain a single number that is the process id of the main logstash process.
• If a pid file cannot be written for any reason, Logstash will log this, but will not consider this a fatal error.
• Logstash will not read this file, only write to it.
• Logstash will write this file even if the file already exists.
• Logstash will try to flock(2) the file before writing it and keep the file open (and the lock).