Provide APIs to manage pipeline

[suyograo]

Currently, whenever Logstash configuration needs to be modified, you have to edit a configuration file and restart the Logstash process. This process requires the user to either temporarily halt the pipeline or accept the fact that an interruption in processing will occur. While file-based configuration works well (especially with Puppet/Chef), it becomes increasing difficult to deploy Logstash instances. We plan to add a REST based Logstash management API to make it easier to interact with Logstash. This will allow dynamic configuration updates without the need to restart the process. To be clear, at this time we are not looking to change the existing configuration syntax. Also, file-based configuration will continue to be supported.

Important manageability functions for this end-point are:

• reload - restarts a paused pipeline instance with new config and instantiates all the plugins
• add - Add specific parts of the pipeline dynamically. For example, we may want to add a file input to follow a new file.
• destroy - sends shutdown signal to all plugins and eventually stops
• status - include pipeline state, list of plugins, the config itself
• pause - pauses the pipeline
• unpause - resumes the paused pipeline

When interacting with the manageability APIs, care should be taken to not drop in-flight messages.

[Vineeth-Mohan]

Looking forward to this feature.

[sutyrin]

+1 to live-reload. How not to lose any log events when you need to change just parsing rules?

[ph]

@SpaceDiver the implementation of the live reload will shutdown gracefully the pipeline (we have updated all the plugins in 2.0 to support that) and will restart the pipeline with the new configuration.

The shutdown process is the following,

Shutdown inputs -> shutdown filter -> shutdown output

So we will stop consuming any new events, when the pipeline restart it will consume the new events again.Depending on your configuration and the inputs used you may of may not have data loss, this is really configuration specific.

[spuder]

@SpaceDiver The github issues isn't the place for questions. You will find lots of helpful people ready to answer questions at https://discuss.elastic.co

[elvarb]

Will community plugins have to be upgraded to support this? If so what is the behavior if a plugins does not support it?

[suyograo]

@elvarb during the shutdown semantics work for 2.0 (#3963), we've updated almost all plugins to properly implement the new calls and reviewed their shutdown logic. This should help. There may still be some plugins which may not gracefully support these semantics (shutdown, reload, start) and we aim to tackle it one by one.

[driverpt]

@suyograo Will it work like nginx?

e.g.:

/etc/init.d/nginx reload

[suyograo]

@driverpt yes we just added reload in #5088

[suyograo]

Will reopen a new issue with updated details. Config reloading has shipped in 2.3

[ashishpok]

Was a new issue ever opened or has there been a concrete date for this?

[MorrieAtElastic]

A customer is inquiring about such an API; has this enhancement been scheduled?

[Allaoua9]

Any update on this ?

[kdHub]

Was this feature ever released or new issue generated to track progress?