[Centralized Config Management] Ability to turn on / off pipeline

[elasticmachine]

Original comment by @ycombinator:

Originally filed as #16759.

Feature Request

It will be really helpful to have the ability to turn ON / OFF pipelines directly from the pipeline management UI in Kibana. We recently see the apparition of a new "clone" button (6.2.0), I think a button ON / OFF (switching between green and red) next to the "clone" button might be a must have.

Sometimes, we need to stop Logstash to send data in Elasticsearch (for maintenance purpose for example). Having the ability to do so directly from Kibana would be great. It can also be useful in a development context : stop some pipelines and start others to deal with the few resources we have in a dev environment. That's where this feature request is coming from.

I think this feature also involve some evolution on Logstash side, except if you decide to write two indices in Elasticsearch, one with the "ON" pipelines and one with the "OFF" pipelines. That way it should be transparent for Logstash. But my guess is that is not "clean". Should I post a feature request in Logstash too ?

Kuaaaly

[elasticmachine]

Original comment by @ycombinator:

If I'm understanding this request correctly, they're essentially asking for the ability for centralized pipeline management to keep track of pipelines (in the .logstash index) but not have logstash executing them.

I think this might be pretty easy to do by adding a status field to the index, with two (for now) possible values: active and inactive (or enabled and disabled, etc.). Logstash nodes would fetch pipelines like they do today but instead of immediately executing them, they'd check the status field first. If the value is active, they could restart or continue running the pipeline. If the value is inactive they could stop the pipeline unless it is stopped already.

Thoughts @acchen97 @andrewvc?

[guylando]

Its inefficient performance wise to disable it on the index level because logstash will keep sending traffic to elasticsearch. Would be more efficient to use Logstash api to access it from Kibana and perform the status change on Logstash itself for it to stop sending data in the pipeline. Might require to add this functionality to the Logstash api.