Add generic AuditTrail service

[joshdover]

In order to support #52125, we need to have a core service that supports two key features for auditable events:

• auditTrail.registerAuditor for receiving audit events
  • This is the API that a plugin could use to get all events, enrich with additional data, and forward to a logger.
• auditTrail.add / addAuditEvent / someOtherName for adding audit events
  • This is the API that Core, OSS plugins, and commercial plugins would use to add domain-events for user actions (eg. Copy to Space). These events are forwarded to any auditors registered with registerAuditor.

Optional feature:

• auditTrail.openScope
  • Allows a audit events to be associated to a wider "domain-event". For example, we may want to associate specific Elasticsearch API calls to a more descriptive "scope" such as copy_to_space. This allows tracing of all commands that were executed in order to fulfill a user request / action.

The AuditTrail service does not actually manage where these events eventually get logged or configuring any logging mechanisms. It simply collects all auditable events and forwards them to any interested auditors.

Should this be in Core or an OSS plugin?

I think this is still up for debate. For instance, we could have an OSS plugin that registers itself with Core services through a registerAuditTrail API exposed on services like ElasticsearchService or SavedObjectsService.

This would help keep the Core API smaller, but may not have many other benefits. It also makes the audit API a bit less discoverable, which may not be preferrable.

Plan

The initial implementation and integration done in #69278
Follow ups:

• If we want to have an ability to track unscoped client calls we might have to add a Continuation-Local Storage library or use built-in node.js Async Hooks
• If we want to track API calls outside of RequestContextHandler we need to consider X-Opaque-Id implementation for FakeRequests as well. @joshdover Should it be done at Scope-able elasticsearch clients #39430 or Pipe X-Opaque-Id header to AuditTrail logs and Elasticsearch API calls #62018?
• Integrate other core services (SO, UiSettings, etc.) by request from Security team.

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[mshustov]

The first iteration of the Audit Trail service is in progress #74640.
We will be track new requirements in dedicated issues, so I'm going to close this one.