Merge pull request elastic#436 from elastic/main

🤖 ESQL: Merge upstream

docs/changelog/91413.yaml

@@ -0,0 +1,5 @@
+pr: 91413
+summary: "[Fleet] Add files and files data index templates and ILM policies"
+area: Infra/Plugins
+type: feature
+issues: []

docs/changelog/92074.yaml

@@ -0,0 +1,5 @@
+pr: 92074
+summary: Improve performance for role mapping with DNs
+area: Authentication
+type: bug
+issues: []

docs/changelog/92081.yaml

@@ -0,0 +1,5 @@
+pr: 92081
+summary: Improve robustness of `JwkSet` reloading
+area: Authentication
+type: bug
+issues: []

docs/changelog/92087.yaml

@@ -0,0 +1,5 @@
+pr: 92087
+summary: Use all profiling events on startup
+area: Search
+type: bug
+issues: []

docs/changelog/92100.yaml

@@ -0,0 +1,12 @@
+pr: 92100
+summary: Ensure balance threshold is at least 1
+area: Allocation
+type: deprecation
+issues: []
+deprecation:
+  title: Ensure balance threshold is at least 1
+  area: Cluster and node setting
+  details: Values for `cluster.routing.allocation.balance.threshold` smaller than
+    `1` are now ignored. Support for values less than `1` for this setting is deprecated
+    and will be forbidden in a future version.
+  impact: Set `cluster.routing.allocation.balance.threshold` to be at least `1`.

docs/changelog/92101.yaml

@@ -0,0 +1,5 @@
+pr: 92101
+summary: Add primary term supplier to Engine.IndexCommitListener
+area: Engine
+type: enhancement
+issues: []

docs/changelog/92104.yaml

@@ -0,0 +1,5 @@
+pr: 92104
+summary: Upgrading tika to 2.6.0
+area: Ingest Node
+type: upgrade
+issues: []

docs/java-rest/low-level/usage.asciidoc

@@ -203,7 +203,7 @@ include-tagged::{doc-tests}/RestClientDocumentation.java[rest-client-init-reques
 --------------------------------------------------
 <1> Set a callback that allows to modify the default request configuration
 (e.g. request timeouts, authentication, or anything that the
-https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/client/config/RequestConfig.Builder.html[`org.apache.http.client.config.RequestConfig.Builder`]
+https://hc.apache.org/httpcomponents-client-5.2.x/current/httpclient5/apidocs/org/apache/hc/client5/http/config/RequestConfig.Builder.html[`org.apache.hc.client5.http.config.RequestConfig.Builder`]
  allows to set)
 
 ["source","java",subs="attributes,callouts,macros"]
@@ -212,7 +212,7 @@ include-tagged::{doc-tests}/RestClientDocumentation.java[rest-client-init-client
 --------------------------------------------------
 <1> Set a callback that allows to modify the http client configuration
 (e.g. encrypted communication over ssl, or anything that the
-https://hc.apache.org/httpcomponents-asyncclient-dev/httpasyncclient/apidocs/org/apache/http/impl/nio/client/HttpAsyncClientBuilder.html[`org.apache.http.impl.nio.client.HttpAsyncClientBuilder`]
+https://hc.apache.org/httpcomponents-asyncclient-4.1.x/current/httpasyncclient/apidocs/org/apache/http/impl/nio/client/HttpAsyncClientBuilder.html[`org.apache.http.impl.nio.client.HttpAsyncClientBuilder`]
  allows to set)
 
 
@@ -339,9 +339,9 @@ translate to the execution of that request being cancelled, which needs to be
 specifically implemented in the API itself.
 
 The use of the `Cancellable` instance is optional and you can safely ignore this
-if you don't need it. A typical usecase for this would be using this together with 
+if you don't need it. A typical usecase for this would be using this together with
 frameworks like Rx Java or the Kotlin's `suspendCancellableCoRoutine`. Cancelling
-no longer needed requests is a good way to avoid putting unnecessary 
+no longer needed requests is a good way to avoid putting unnecessary
 load on Elasticsearch.
 
 ["source","java",subs="attributes,callouts,macros"]

docs/reference/modules/cluster/misc.asciidoc

@@ -1,8 +1,9 @@
 [[misc-cluster-settings]]
-==== Miscellaneous cluster settings
+=== Miscellaneous cluster settings
 
+[discrete]
 [[cluster-read-only]]
-===== Metadata
+==== Metadata
 
 An entire cluster may be set to read-only with the following setting:
 
@@ -21,9 +22,9 @@ WARNING: Don't rely on this setting to prevent changes to your cluster. Any
 user with access to the <<cluster-update-settings,cluster-update-settings>>
 API can make the cluster read-write again.
 
-
+[discrete]
 [[cluster-shard-limit]]
-===== Cluster shard limit
+==== Cluster shard limit
 
 There is a soft limit on the number of shards in a cluster, based on the number
 of nodes in the cluster. This is intended to prevent operations which may
@@ -101,8 +102,9 @@ number of shards for each node, use the
 setting.
 --
 
+[discrete]
 [[user-defined-data]]
-===== User-defined cluster metadata
+==== User-defined cluster metadata
 
 User-defined metadata can be stored and retrieved using the Cluster Settings API.
 This can be used to store arbitrary, infrequently-changing data about the cluster
@@ -127,8 +129,9 @@ metadata will be viewable by anyone with access to the
 <<cluster-get-settings,Cluster Get Settings>> API, and is recorded in the
 {es} logs.
 
+[discrete]
 [[cluster-max-tombstones]]
-===== Index tombstones
+==== Index tombstones
 
 The cluster state maintains index tombstones to explicitly denote indices that
 have been deleted. The number of tombstones maintained in the cluster state is
@@ -148,8 +151,9 @@ include::{es-repo-dir}/indices/dangling-indices-list.asciidoc[tag=dangling-index
 You can use the <<dangling-indices-api,Dangling indices API>> to manage
 this situation.
 
+[discrete]
 [[cluster-logger]]
-===== Logger
+==== Logger
 
 The settings which control logging can be updated <<dynamic-cluster-setting,dynamically>> with the
 `logger.` prefix. For instance, to increase the logging level of the
@@ -165,9 +169,9 @@ PUT /_cluster/settings
 }
 -------------------------------
 
-
+[discrete]
 [[persistent-tasks-allocation]]
-===== Persistent tasks allocation
+==== Persistent tasks allocation
 
 Plugins can create a kind of tasks called persistent tasks. Those tasks are
 usually long-lived tasks and are stored in the cluster state, allowing the

docs/reference/modules/cluster/shards_allocation.asciidoc

@@ -57,17 +57,18 @@ one of the active allocation ids in the cluster state.
 [[shards-rebalancing-settings]]
 ==== Shard rebalancing settings
 
-A cluster is _balanced_ when it has an equal number of shards on each node
-without having a concentration of shards from any index on any node. {es} runs
-an automatic process called _rebalancing_ which moves shards between the nodes
-in your cluster to improve its balance. Rebalancing obeys all other shard
-allocation rules such as <<cluster-shard-allocation-filtering,allocation
-filtering>> and <<forced-awareness,forced awareness>> which may prevent it from
-completely balancing the cluster. In that case, rebalancing strives to achieve
-the most balanced cluster possible within the rules you have configured. If you
-are using <<data-tiers,data tiers>> then {es} automatically applies allocation
-filtering rules to place each shard within the appropriate tier. These rules
-mean that the balancer works independently within each tier.
+A cluster is _balanced_ when it has an equal number of shards on each node, with
+all nodes needing equal resources, without having a concentration of shards from
+any index on any node. {es} runs an automatic process called _rebalancing_ which
+moves shards between the nodes in your cluster to improve its balance.
+Rebalancing obeys all other shard allocation rules such as
+<<cluster-shard-allocation-filtering,allocation filtering>> and
+<<forced-awareness,forced awareness>> which may prevent it from completely
+balancing the cluster. In that case, rebalancing strives to achieve the most
+balanced cluster possible within the rules you have configured. If you are using
+<<data-tiers,data tiers>> then {es} automatically applies allocation filtering
+rules to place each shard within the appropriate tier. These rules mean that the
+balancer works independently within each tier.
 
 You can use the following settings to control the rebalancing of shards across
 the cluster:
@@ -84,7 +85,6 @@ Enable or disable rebalancing for specific kinds of shards:
 * `none` -        No shard balancing of any kind are allowed for any indices.
 --
 
-
 `cluster.routing.allocation.allow_rebalance`::
 +
 --
@@ -98,13 +98,32 @@ Specify when shard rebalancing is allowed:
 --
 
 `cluster.routing.allocation.cluster_concurrent_rebalance`::
-      (<<dynamic-cluster-setting,Dynamic>>)
-      Allow to control how many concurrent shard rebalances are
-      allowed cluster wide. Defaults to `2`. Note that this setting
-      only controls the number of concurrent shard relocations due
-      to imbalances in the cluster. This setting does not limit shard
-      relocations due to <<cluster-shard-allocation-filtering,allocation
-      filtering>> or <<forced-awareness,forced awareness>>.
+(<<dynamic-cluster-setting,Dynamic>>)
+Defines the number of concurrent shard rebalances are allowed across the whole
+cluster. Defaults to `2`. Note that this setting only controls the number of
+concurrent shard relocations due to imbalances in the cluster. This setting does
+not limit shard relocations due to
+<<cluster-shard-allocation-filtering,allocation filtering>> or
+<<forced-awareness,forced awareness>>.
+
+`cluster.routing.allocation.type`::
++
+--
+Selects the algorithm used for computing the cluster balance. Defaults to
+`desired_balance` which selects the _desired balance allocator_. This allocator
+runs a background task which computes the desired balance of shards in the
+cluster. Once this background task completes, {es} moves shards to their
+desired locations.
+
+May also be set to `balanced` to select the legacy _balanced allocator_. This
+allocator was the default allocator in versions of {es} before 8.6.0. It runs
+in the foreground, preventing the master from doing other work in parallel. It
+works by selecting a small number of shard movements which immediately improve
+the balance of the cluster, and when those shard movements complete it runs
+again and selects another few shards to move. Since this allocator makes its
+decisions based only on the current state of the cluster, it will sometimes
+move a shard several times while balancing the cluster.
+--
 
 [[shards-rebalancing-heuristics]]
 ==== Shard balancing heuristics settings
@@ -114,28 +133,55 @@ of shards, and then moving shards between nodes to reduce the weight of the
 heavier nodes and increase the weight of the lighter ones. The cluster is
 balanced when there is no possible shard movement that can bring the weight of
 any node closer to the weight of any other node by more than a configurable
-threshold. The following settings allow you to control the details of these
-calculations.
+threshold.
+
+The weight of a node depends on the number of shards it holds and on the total
+estimated resource usage of those shards expressed in terms of the size of the
+shard on disk and the number of threads needed to support write traffic to the
+shard. {es} estimates the resource usage of shards belonging to data streams
+when they are created by a rollover. The estimated disk size of the new shard
+is the mean size of the other shards in the data stream. The estimated write
+load of the new shard is a weighted average of the actual write loads of recent
+shards in the data stream. Shards that do not belong to the write index of a
+data stream have an estimated write load of zero.
+
+The following settings control how {es} combines these values into an overall
+measure of each node's weight.
 
 `cluster.routing.allocation.balance.shard`::
-     (<<dynamic-cluster-setting,Dynamic>>)
-     Defines the weight factor for the total number of shards allocated on a node
-     (float). Defaults to `0.45f`. Raising this raises the tendency to
-     equalize the number of shards across all nodes in the cluster.
+(float, <<dynamic-cluster-setting,Dynamic>>)
+Defines the weight factor for the total number of shards allocated to each node.
+Defaults to `0.45f`. Raising this value increases the tendency of {es} to
+equalize the total number of shards across nodes ahead of the other balancing
+variables.
 
 `cluster.routing.allocation.balance.index`::
-     (<<dynamic-cluster-setting,Dynamic>>)
-     Defines the weight factor for the number of shards per index allocated
-      on a specific node (float). Defaults to `0.55f`. Raising this raises the
-      tendency to equalize the number of shards per index across all nodes in
-      the cluster.
+(float, <<dynamic-cluster-setting,Dynamic>>)
+Defines the weight factor for the number of shards per index allocated to each
+node. Defaults to `0.55f`. Raising this value increases the tendency of {es} to
+equalize the number of shards of each index across nodes ahead of the other
+balancing variables.
+
+`cluster.routing.allocation.balance.disk_usage`::
+(float, <<dynamic-cluster-setting,Dynamic>>)
+Defines the weight factor for balancing shards according to their predicted disk
+size in bytes. Defaults to `2e-11f`. Raising this value increases the tendency
+of {es} to equalize the total disk usage across nodes ahead of the other
+balancing variables.
+
+`cluster.routing.allocation.balance.write_load`::
+(float, <<dynamic-cluster-setting,Dynamic>>)
+Defines the weight factor for the write load of each shard, in terms of the
+estimated number of indexing threads needed by the shard. Defaults to `10.0f`.
+Raising this value increases the tendency of {es} to equalize the total write
+load across nodes ahead of the other balancing variables.
 
 `cluster.routing.allocation.balance.threshold`::
-     (<<dynamic-cluster-setting,Dynamic>>)
-     Minimal optimization value of operations that should be performed (non
-      negative float). Defaults to `1.0f`. Raising this will cause the cluster
-      to be less aggressive about optimizing the shard balance.
-
+(float, <<dynamic-cluster-setting,Dynamic>>)
+The minimum improvement in weight which triggers a rebalancing shard movement.
+Defaults to `1.0f`. Raising this value will cause {es} to stop rebalancing
+shards sooner, leaving the cluster in a more unbalanced state.
 
 NOTE: Regardless of the result of the balancing algorithm, rebalancing might
-not be allowed due to forced awareness or allocation filtering.
+not be allowed due to allocation rules such as forced awareness and allocation
+filtering.

docs/reference/text-structure/apis/find-structure.asciidoc

@@ -538,7 +538,8 @@ If the request does not encounter errors, you receive the following result:
   }
 }
 ----
-// TESTRESPONSE[s/"sample_start" : ".*",/"sample_start" : "$body.sample_start",/]
+// TESTRESPONSE[skip:"AwaitsFix https://github.com/elastic/elasticsearch/issues/92141"]
+// original (put back after unmuting)[s/"sample_start" : ".*",/"sample_start" : "$body.sample_start",/]
 // The substitution is because the text is pre-processed by the test harness,
 // so the fields may get reordered in the JSON the endpoint sees