Expose API key name to the ingest pipeline

[bytebilly]

Description

We have the Set Security User Processor, that is able to add information about the authenticated user that performs the request.

We could do something similar when an API key is used to authenticate instead of a regular user.

Proposal

We can create a new processor (or extend the Set Security User Processor) to make API key information available to the ingest pipeline.

We should expose the name field. We can also expose other fields if they could be relevant.

[elasticmachine]

Pinging @elastic/es-security (:Security/Authentication)

[albertzaharovits]

Authentication with API Keys or tokens is thought to appear as if coming from the user itself, i.e. be transparent. Hence the API key name or id are not exposed in the user metadata. There is a lapse here, because the realm that did the authentication in case of API Key authentication is _es_api_key , but the realm name is not exposed in Set Security User Processor.

If we were to expose this information to an ingest processor then I would be inclined to add a new processor.

@bytebilly can you please detail a little more the use case for this?

[tvernum]

@albertzaharovits ES 7.5 will support required pipelines. This allows an admin to force that documents include information about how they were ingested.
For models where ingest tools (e.g. beats) are authenticating with API keys, knowing which API key performed the ingestion is necessary in order to know which client ingested the document.

Because we have not been able to enforce a pipeline until now, we haven't really kept Set Security User Processor up to date because it was so easy to by-pass.
Since that has changed, we ought to expand that processor with additional fields (e.g. authentication type, realm name, and API key id/name).

[albertzaharovits]

Thank you for the context @tvernum . I can now clearly see the need for it. Upon further reflection I think that the API Key id/name should be part of the User object we attach to the ThreadContext during authentication, and afterwards it should be simple for the Set Security User Processor to pull that into a doc field. Please let me know if this doesn't jibe with your idea for it.

[tvernum]

It probably makes sense for it to be on the User object for other reasons, but some of that information (original realm, key-id) is already in the Authentication object's metadata (and the name would be easy to add), so I think it's already fairly simple to add support in the Set Security User Processor.

I prefer adding to the existing processor (as new optional fields) because then it becomes possible to have a single object in the ingested doc that contains full information about which user ingested it, and how they authenticated.

[bytebilly]

I agree on improving the Set Security User Processor, this will make easier for users to build their pipelines since they don't have to know in advance it authentication comes from users or keys.