Remove index audit output type (#37707)

This commit removes the Index Audit Output type, following its deprecation
in 6.7 by 8765a31. It also adds the migration notice (settings notice).

In general, the problem with the index audit output is that event indexing
can be slower than the rate with which audit events are generated,
especially during the daily rollovers or the rolling cluster upgrades.
In this situation audit events will be lost which is a terrible failure situation
for an audit system.
Besides of the settings under the `xpack.security.audit.index` namespace, the
`xpack.security.audit.outputs` setting has also been deprecated and will be
removed in 7. Although explicitly configuring the logfile output does not touch
any deprecation bits, this setting is made redundant in 7 so this PR deprecates
it as well.

Relates #29881

docs/reference/migration/migrate_7_0/settings.asciidoc

@@ -154,3 +154,18 @@ node's keystore, i.e., they are not to be specified via the cluster settings API
 `xpack.notification.pagerduty.account.<id>.secure_service_api_key`
 - `xpack.notification.slack.account.<id>.url`, instead use
 `xpack.notification.slack.account.<id>.secure_url`
+
+[float]
+[[remove-audit-index-output]]
+==== Audit index output type removed
+
+All the settings under the `xpack.security.audit.index` namespace have been
+removed. In addition, the `xpack.security.audit.outputs` setting has been
+removed as well.
+
+These settings enabled and configured the audit index output type. This output
+type has been removed because it was unreliable in certain scenarios and this
+could have lead to dropping audit events while the operations on the system
+were allowed to continue as usual. The recommended replacement is the
+use of the `logfile` audit output type and using other components from the
+Elastic Stack to handle the indexing part.

docs/reference/settings/audit-settings.asciidoc

@@ -14,38 +14,19 @@ file. For more information, see
 
 `xpack.security.audit.enabled`::
 Set to `true` to enable auditing on the node. The default value is `false`.
-
-`xpack.security.audit.outputs`::
-Specifies where audit logs are output. For example: `[ index, logfile ]`. The
-default value is `logfile`, which puts the auditing events in a dedicated
-file named `<clustername>_audit.log` on each node.
-You can also specify `index`, which puts the auditing events in an {es} index
-that is prefixed with `.security_audit_log`. The index can reside on the same
-cluster or a separate cluster.
-+
-For backwards compatibility reasons, if you use the logfile output type, a
-`<clustername>_access.log` file is also created. It contains the same
-information, but it uses the older (pre-6.5.0) formatting style.
-If the backwards compatible format is not required, it should be disabled.
-To do that, change its logger level to `off` in the `log4j2.properties` file.
-For more information, see <<configuring-logging-levels>>.
-+
---
-TIP: If the index is unavailable, it is possible for auditing events to
-be lost. The `index` output type should therefore be used in conjunction with
-the `logfile` output type and the latter should be the official record of events.
-
---
+This puts the auditing events in a dedicated file named `<clustername>_audit.log`
+on each node. For more information, see <<configuring-logging-levels>>.
 
 [[event-audit-settings]]
 ==== Audited Event Settings
 
-The events and some other information about what gets logged can be
-controlled by using the following settings:
+The events and some other information about what gets logged can be controlled
+by using the following settings:
 
 `xpack.security.audit.logfile.events.include`::
 Specifies which events to include in the auditing output. The default value is:
-`access_denied, access_granted, anonymous_access_denied, authentication_failed, connection_denied, tampered_request, run_as_denied, run_as_granted`.
+`access_denied, access_granted, anonymous_access_denied, authentication_failed,
+connection_denied, tampered_request, run_as_denied, run_as_granted`.
 
 `xpack.security.audit.logfile.events.exclude`::
 Excludes the specified events from the output. By default, no events are
@@ -113,98 +94,3 @@ A list of index names or wildcards. The specified policy will
 not print audit events when all the indices in the event match
 these values. If the event concerns several indices, some of which are
 *not* covered by the policy, the policy will *not* cover this event.
-
-[[index-audit-settings]]
-==== Audit Log Indexing Configuration Settings
-
-`xpack.security.audit.index.bulk_size`::
-Controls how many audit events are batched into a single write. The default
-value is `1000`.
-
-`xpack.security.audit.index.flush_interval`::
-Controls how often buffered events are flushed to the index. The default value
-is `1s`.
-
-`xpack.security.audit.index.rollover`::
-Controls how often to roll over to a new index: `hourly`, `daily`, `weekly`, or
-`monthly`. The default value is `daily`.
-
-`xpack.security.audit.index.events.include`::
-Specifies the audit events to be indexed. The default value is
-`anonymous_access_denied, authentication_failed, realm_authentication_failed, access_granted, access_denied, tampered_request, connection_granted, connection_denied, run_as_granted, run_as_denied`.
-See {xpack-ref}/audit-event-types.html[Audit Entry Types] for the
-complete list.
-
-`xpack.security.audit.index.events.exclude`::
-Excludes the specified auditing events from indexing. By default, no events are
-excluded.
-
-`xpack.security.audit.index.events.emit_request_body`::
-Specifies whether to include the request body from REST requests on certain
-event types such as `authentication_failed`. The default value is `false`.
-
-`xpack.security.audit.index.settings`::
-Specifies settings for the indices that the events are stored in. For example,
-the following configuration sets the number of shards and replicas to 1 for the
-audit indices:
-+
---
-[source,yaml]
-----------------------------
-xpack.security.audit.index.settings:
-  index:
-    number_of_shards: 1
-    number_of_replicas: 1
-----------------------------
---
-+
---
-NOTE: These settings apply to the local audit indices, as well as to the
-<<remote-audit-settings, remote audit indices>>, but only if the remote cluster
-does *not* have {security-features} enabled or the {es} versions are different.
-If the remote cluster has {security-features} enabled and the versions coincide,
-the settings for the audit indices there will take precedence,
-even if they are unspecified (i.e. left to defaults).
---
-
-[[remote-audit-settings]]
-==== Remote Audit Log Indexing Configuration Settings
-
-To index audit events to a remote {es} cluster, you configure the following
-`xpack.security.audit.index.client` settings:
-
-`xpack.security.audit.index.client.hosts`::
-Specifies a comma-separated list of `host:port` pairs. These hosts should be
-nodes in the remote cluster. If you are using default values for the 
-<<common-network-settings,`transport.port`>> setting, you can omit the
-`port` value. Otherwise, it must match the `transport.port` setting.
-
-`xpack.security.audit.index.client.cluster.name`::
-Specifies the name of the remote cluster.
-
-`xpack.security.audit.index.client.xpack.security.user`::
-Specifies the `username:password` pair that is used to authenticate with the
-remote cluster. This user must have authority to create the `.security-audit` 
-index on the remote cluster. 
-
-If the remote {es} cluster has Transport Layer Security (TLS/SSL) enabled, you 
-must set the following setting to `true`:
-
-`xpack.security.audit.index.client.xpack.security.transport.ssl.enabled`::
-Used to enable or disable TLS/SSL for the transport client that forwards audit 
-logs to the remote cluster. The default is `false`. 
-
-You must also specify the information necessary to access certificates. See 
-<<auditing-tls-ssl-settings>>. 
-
-You can pass additional settings to the remote client by specifying them in the
-`xpack.security.audit.index.client` namespace. For example, you can add 
-<<modules-transport,transport settings>> and 
-<<tcp-settings,advanced TCP settings>> in that namespace. To allow the remote
-client to discover all of the nodes in the remote cluster you can specify the
-`client.transport.sniff` setting:
-
-[source,yaml]
-----------------------------
-xpack.security.audit.index.client.transport.sniff: true
-----------------------------

docs/reference/settings/security-settings.asciidoc

@@ -1303,16 +1303,6 @@ transport profile, use the prefix `transport.profiles.$PROFILE.xpack.security.`
 append the portion of the setting after `xpack.security.transport.`. For the key
 setting, this would be `transport.profiles.$PROFILE.xpack.security.ssl.key`.
 
-[[auditing-tls-ssl-settings]]
-:ssl-prefix:             xpack.security.audit.index.client.xpack.security.transport
-:component:              Auditing
-:client-auth-default!:
-:server!:
-
-include::ssl-settings.asciidoc[]
-
-See also <<remote-audit-settings>>. 
-
 [float]
 [[ip-filtering-settings]]
 ==== IP filtering settings

x-pack/docs/en/security/auditing/auditing-search-queries.asciidoc

@@ -13,26 +13,13 @@ Search queries are contained inside HTTP request bodies, however, and some
 audit events that are generated by the REST layer can be toggled to output
 the request body to the audit log.
 
-To make certain audit events include the request body, edit the following 
-settings in the `elasticsearch.yml` file:
+To make certain audit events include the request body, edit the following
+setting in the `elasticsearch.yml` file:
 
-* For the `logfile` audit output:
-+  
---
 [source,yaml]
 ----------------------------
 xpack.security.audit.logfile.events.emit_request_body: true
 ----------------------------
---
-
-* For the `index` output:
-+
---
-[source,yaml]
-----------------------------
-xpack.security.audit.index.events.emit_request_body: true
-----------------------------
---
 
 IMPORTANT: No filtering is performed when auditing, so sensitive data might be
 audited in plain text when audit events include the request body. Also, the
@@ -44,10 +31,8 @@ generated in the REST layer and can access the request body. Most of them are no
 included by default.
 
 A good practical piece of advice is to add `authentication_success` to the event
-types that are audited. Add it to the list in the 
-`xpack.security.audit.logfile.events.include` or 
-`xpack.security.audit.index.events.include` settings. This type is not audited
-by default.
+types that are audited (add it to the list in the `xpack.security.audit.logfile.events.include`),
+as this event type is not audited by default.
 
 NOTE: Typically, the include list contains other event types as well, such as
 `access_granted` or `access_denied`.

x-pack/docs/en/security/auditing/event-types.asciidoc

@@ -210,8 +210,7 @@ that have been previously described:
 === Audit event attributes for the deprecated formats
 
 The following table shows the common attributes that can be associated with
-every event, when it is output to the `<clustername>_access.log` file or to the
-<<audit-index, index>>.
+every event, when it is output to the `<clustername>_access.log` file.
 
 .Common attributes
 [cols="2,7",options="header"]
@@ -229,10 +228,8 @@ every event, when it is output to the `<clustername>_access.log` file or to the
                         `run_as_denied`, `run_as_granted`.
 |======
 
-For an event in the <<audit-log-output,audit log file output>>, these are
-positional attributes, which are printed at the beginning of each log line and
-are not adjoined by the attribute name. As a matter of course, the names are
-present for each attribute when the event is forwarded to the <<audit-index, index audit output>>.
+These are positional attributes, which are printed at the beginning of each log line and
+are not adjoined by the attribute name.
 
 The attribute `origin_address` is also common to every audit event. It is always
 named, that is, it is not positional. It denotes the source IP address of the

x-pack/docs/en/security/auditing/index.asciidoc

@@ -8,11 +8,5 @@ include::event-types.asciidoc[]
 :edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/x-pack/docs/en/security/auditing/output-logfile.asciidoc
 include::output-logfile.asciidoc[]
 
-:edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/x-pack/docs/en/security/auditing/output-index.asciidoc
-include::output-index.asciidoc[]
-
 :edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/x-pack/docs/en/security/auditing/auditing-search-queries.asciidoc
 include::auditing-search-queries.asciidoc[]
-
-:edit_url: https://github.com/elastic/elasticsearch/edit/{branch}/x-pack/docs/en/security/auditing/forwarding-logs.asciidoc
-include::forwarding-logs.asciidoc[]