Add the ability to require an ingest pipeline

[jasontedor]

This commit adds the ability to require an ingest pipeline on an index. Today we can have a default pipeline, but that could be overridden by a request pipeline parameter. This commit introduces a new index setting index.required_pipeline that acts similarly to index.default_pipeline, except that it can not be overridden by a request pipeline parameter. Additionally, a default pipeline and a request pipeline can not both be set. The required pipeline can be set to _none to ensure that no pipeline ever runs for index requests on that index.

[elasticmachine]

Pinging @elastic/es-core-features

[jakelandis]

@jasontedor could you add a near clone of https://github.com/elastic/elasticsearch/blob/master/modules/ingest-common/src/test/resources/rest-api-spec/test/ingest/200_default_pipeline.yml to help test required pipelines from the various ways to issue index request.

[martijnvg]

Looks good. I left a few comments.

[jasontedor]

@elasticmachine run elasticsearch-ci/2

[jakelandis]

LGTM - did some manual testing with multi-node forwarding and all works as expected.

[martijnvg]

LGTM

[cwurm]

Being able to force data to run through a pipeline before indexing is really useful. However, I worry about making the required pipeline the only one that executes when set.

As a concrete example, we've been talking about using the ingest timestamp instead of the event timestamp for running detection rules (searches/queries, but also ML jobs) in SIEM. This would make sure that we always consider all newly arrived data and are never fooled, e.g. by an attacker manipulating system time to send data pretending to be from the distant past or future.

For that, a required ingest pipeline with a set processor setting event.ingested to {{_ingest.timestamp}} would work. We could have all Beats setup such a pipeline on all the Beats indices. However, we would also still need the ability to have another pipeline execute before the required one, e.g. the data-parsing pipelines of any enabled Filebeat modules.

The required_pipeline could always execute last, but allow any other pipeline (default_pipeline or specified in the request) to run before it. What do you think?

/cc @tsg @MikePaquette @randomuserid

[randomuserid]

I am not sure I follow - is this a trade off between regular field parsing vs. having an ingest timestamp and using sort of alternative parsing scheme? For signal purposes, if we are going to use ingest timestamps, we would need to follow the general convention of giving each event both timestamps - ingest time and event time (the latter being the timestamp in the original event being ingested.) We would also need the events to be parsed into ECS fields as they are now in order for signals to evaluate them.

Independent of the ingest time idea, we always need the original event timestamps in order to create forensic timelines as analysts work on cases. If we had to make a Sophie’s choice, the original timestamp is more critical, even if it means running long and expensive searches in order to ensure solving for correctness.

[cwurm]

@randomuserid In my example, the ingest timestamp would not replace the event timestamp - the event timestamp would still be in @timestamp, the ingest timestamp in event.ingested (field name tbd). Let's continue the discussion about this concrete use case off this PR.

[randomuserid]

OK what are are applications for the ingest pipeline - maybe the ability to make a signal on an event tagged as important or special by an endpoint agent without consideration of time? As an alternative method of ensuring important events with weird timestamps are made into signals?

[cwurm]

@randomuserid It could be that, yeah. More straightforward maybe is adding an ingestion timestamp, or dropping a field (e.g. for privacy reasons), and probably many other things. Today, this kind of central control is often exercised in centrally managed Logstash pipelines, but required_pipeline would allow doing it in Elasticsearch as well.