Merge pull request #171 from elastic/master

🤖 ESQL: Merge upstream

build-tools-internal/version.properties

@@ -1,5 +1,5 @@
 elasticsearch     = 8.4.0
-lucene            = 9.3.0-snapshot-f3025a56e31
+lucene            = 9.3.0-snapshot-2d05f5c623e
 
 bundled_jdk_vendor = openjdk
 bundled_jdk = 18.0.1.1+2@65ae32619e2f40f3a9af3af1851d6e19

docs/changelog/87868.yaml

@@ -0,0 +1,5 @@
+pr: 87868
+summary: Add setting for `tcp_keepalive` for oidc back-channel
+area: Security
+type: enhancement
+issues: []

docs/changelog/88273.yaml

@@ -0,0 +1,6 @@
+pr: 88273
+summary: Fix potential circuit breaker leak on `InternalGeoGrid`
+area: Geo
+type: bug
+issues:
+ - 88261

docs/reference/modules/discovery/discovery.asciidoc

@@ -1,4 +1,4 @@
-[[modules-discovery-hosts-providers]]
+[[discovery-hosts-providers]]
 === Discovery
 
 Discovery is the process by which the cluster formation module finds other
@@ -33,91 +33,8 @@ it is deliberately stopped. It may also stop acting as the master if
 faulty. When a node stops being the elected master, it begins the discovery
 process again.
 
-[[modules-discovery-troubleshooting]]
-==== Troubleshooting discovery
-
-In most cases, the discovery and election process completes quickly, and the
-master node remains elected for a long period of time.
-
-If your cluster doesn't have a stable master, many of its features won't work
-correctly and {es} will report errors to clients and in its logs. You must fix
-the master node's instability before addressing these other issues. It will not
-be possible to solve any other issues while there is no elected master node or
-the elected master node is unstable.
-
-If your cluster has a stable master but some nodes can't discover or join it,
-these nodes will report errors to clients and in their logs. You must address
-the obstacles preventing these nodes from joining the cluster before addressing
-other issues. It will not be possible to solve any other issues reported by
-these nodes while they are unable to join the cluster.
-
-If the cluster has no elected master node for more than a few seconds, the
-master is unstable, or some nodes are unable to discover or join a stable
-master, then {es} will record information in its logs explaining why. If the
-problems persist for more than a few minutes, {es} will record additional
-information in its logs. To properly troubleshoot discovery and election
-problems, collect and analyse logs covering at least five minutes from all
-nodes.
-
-The following sections describe some common discovery and election problems.
-
-===== No master is elected
-
-When a node wins the master election, it logs a message containing
-`elected-as-master` and all nodes log a message containing
-`master node changed` identifying the new elected master node.
-
-If there is no elected master node and no node can win an election, all
-nodes will repeatedly log messages about the problem using a logger called
-`org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper`. By
-default, this happens every 10 seconds.
-
-Master elections only involve master-eligible nodes, so focus on the logs from
-master-eligible nodes in this situation. These nodes' logs will indicate the
-requirements for a master election, such as the discovery of a certain set of
-nodes.
-
-If the logs indicate that {es} can't discover enough nodes to form a quorum,
-you must address the reasons preventing {es} from discovering the missing
-nodes. The missing nodes are needed to reconstruct the cluster metadata.
-Without the cluster metadata, the data in your cluster is meaningless. The
-cluster metadata is stored on a subset of the master-eligible nodes in the
-cluster. If a quorum can't be discovered, the missing nodes were the ones
-holding the cluster metadata.
-
-Ensure there are enough nodes running to form a quorum and that every node can
-communicate with every other node over the network. {es} will report additional
-details about network connectivity if the election problems persist for more
-than a few minutes. If you can't start enough nodes to form a quorum, start a
-new cluster and restore data from a recent snapshot. Refer to
-<<modules-discovery-quorums>> for more information.
-
-If the logs indicate that {es} _has_ discovered a possible quorum of nodes, the
-typical reason that the cluster can't elect a master is that one of the other
-nodes can't discover a quorum. Inspect the logs on the other master-eligible
-nodes and ensure that they have all discovered enough nodes to form a quorum.
-
-===== Master is elected but unstable
-
-When a node wins the master election, it logs a message containing
-`elected-as-master`. If this happens repeatedly, the elected master node is
-unstable. In this situation, focus on the logs from the master-eligible nodes
-to understand why the election winner stops being the master and triggers
-another election.
-
-===== Node cannot discover or join stable master
-
-If there is a stable elected master but a node can't discover or join its
-cluster, it will repeatedly log messages about the problem using the
-`ClusterFormationFailureHelper` logger. Other log messages on the affected node
-and the elected master may provide additional information about the problem.
-
-===== Node joins cluster and leaves again
-
-If a node joins the cluster but {es} determines it to be faulty then it will be
-removed from the cluster again. See <<cluster-fault-detection-troubleshooting>>
-for more information.
-
+Refer to <<discovery-troubleshooting,Troubleshooting discovery>> for 
+troubleshooting issues with discovery.
 
 [[built-in-hosts-providers]]
 ==== Seed hosts providers

docs/reference/redirects.asciidoc

@@ -3,6 +3,16 @@
 
 The following pages have moved or been deleted.
 
+[role="exclude",id="modules-discovery-hosts-providers"]
+=== Node discovery
+
+Refer to <<discovery-hosts-providers,Discovery>>.
+
+[role="exclude",id="modules-discovery-troubleshooting"]
+==== Troubleshooting node discovery
+
+Refer to <<discovery-troubleshooting,Troubleshooting discovery>>.
+
 [role="exclude",id="jwt-realm"]
 === JWT realm
 

docs/reference/settings/security-settings.asciidoc

@@ -1858,6 +1858,13 @@ connections allowed per endpoint.
 Defaults to `200`.
 // end::oidc-http-max-endpoint-connections-tag[]
 
+// tag::oidc-http-tcp-keepalive-tag[]
+`http.tcp.keep_alive` {ess-icon}::
+(<<static-cluster-setting,Static>>)
+Whether to enable TCP keepalives on HTTP connections used for back-channel communication
+to the OpenID Connect Provider endpoints. Defaults to `true`.
+// end::oidc-http-tcp-keepalive-tag[]
+
 // tag::oidc-http-connection-pool-ttl-tag[]
 `http.connection_pool_ttl` {ess-icon}::
 (<<static-cluster-setting,Static>>)

docs/reference/troubleshooting.asciidoc

@@ -50,6 +50,8 @@ include::troubleshooting/data/add-tier.asciidoc[]
 
 include::troubleshooting/data/diagnose-unassigned-shards.asciidoc[]
 
+include::troubleshooting/discovery-issues.asciidoc[]
+
 include::troubleshooting/data/increase-tier-capacity.asciidoc[]
 
 include::troubleshooting/data/start-ilm.asciidoc[]

docs/reference/troubleshooting/discovery-issues.asciidoc

@@ -0,0 +1,92 @@
+[[discovery-troubleshooting]]
+== Troubleshooting discovery
+
+In most cases, the discovery and election process completes quickly, and the
+master node remains elected for a long period of time.
+
+If your cluster doesn't have a stable master, many of its features won't work
+correctly and {es} will report errors to clients and in its logs. You must fix
+the master node's instability before addressing these other issues. It will not
+be possible to solve any other issues while there is no elected master node or
+the elected master node is unstable.
+
+If your cluster has a stable master but some nodes can't discover or join it,
+these nodes will report errors to clients and in their logs. You must address
+the obstacles preventing these nodes from joining the cluster before addressing
+other issues. It will not be possible to solve any other issues reported by
+these nodes while they are unable to join the cluster.
+
+If the cluster has no elected master node for more than a few seconds, the
+master is unstable, or some nodes are unable to discover or join a stable
+master, then {es} will record information in its logs explaining why. If the
+problems persist for more than a few minutes, {es} will record additional
+information in its logs. To properly troubleshoot discovery and election
+problems, collect and analyse logs covering at least five minutes from all
+nodes.
+
+The following sections describe some common discovery and election problems.
+
+[discrete]
+[[discovery-no-master]]
+=== No master is elected
+
+When a node wins the master election, it logs a message containing
+`elected-as-master` and all nodes log a message containing
+`master node changed` identifying the new elected master node.
+
+If there is no elected master node and no node can win an election, all
+nodes will repeatedly log messages about the problem using a logger called
+`org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper`. By
+default, this happens every 10 seconds.
+
+Master elections only involve master-eligible nodes, so focus on the logs from
+master-eligible nodes in this situation. These nodes' logs will indicate the
+requirements for a master election, such as the discovery of a certain set of
+nodes.
+
+If the logs indicate that {es} can't discover enough nodes to form a quorum,
+you must address the reasons preventing {es} from discovering the missing
+nodes. The missing nodes are needed to reconstruct the cluster metadata.
+Without the cluster metadata, the data in your cluster is meaningless. The
+cluster metadata is stored on a subset of the master-eligible nodes in the
+cluster. If a quorum can't be discovered, the missing nodes were the ones
+holding the cluster metadata.
+
+Ensure there are enough nodes running to form a quorum and that every node can
+communicate with every other node over the network. {es} will report additional
+details about network connectivity if the election problems persist for more
+than a few minutes. If you can't start enough nodes to form a quorum, start a
+new cluster and restore data from a recent snapshot. Refer to
+<<modules-discovery-quorums>> for more information.
+
+If the logs indicate that {es} _has_ discovered a possible quorum of nodes, the
+typical reason that the cluster can't elect a master is that one of the other
+nodes can't discover a quorum. Inspect the logs on the other master-eligible
+nodes and ensure that they have all discovered enough nodes to form a quorum.
+
+[discrete]
+[[discovery-master-unstable]]
+=== Master is elected but unstable
+
+When a node wins the master election, it logs a message containing
+`elected-as-master`. If this happens repeatedly, the elected master node is
+unstable. In this situation, focus on the logs from the master-eligible nodes
+to understand why the election winner stops being the master and triggers
+another election.
+
+[discrete]
+[[discovery-cannot-join-master]]
+=== Node cannot discover or join stable master
+
+If there is a stable elected master but a node can't discover or join its
+cluster, it will repeatedly log messages about the problem using the
+`ClusterFormationFailureHelper` logger. Other log messages on the affected node
+and the elected master may provide additional information about the problem.
+
+[discrete]
+[[discovery-node-leaves]]
+=== Node joins cluster and leaves again
+
+If a node joins the cluster but {es} determines it to be faulty then it will be
+removed from the cluster again. See <<cluster-fault-detection-troubleshooting>>
+for more information.

modules/lang-expression/licenses/lucene-codecs-9.3.0-snapshot-2d05f5c623e.jar.sha1

@@ -0,0 +1 @@
+a2b0a4e1d763c5575f289ef7b4d8f109e7ee895f

modules/lang-expression/licenses/lucene-expressions-9.3.0-snapshot-2d05f5c623e.jar.sha1

@@ -0,0 +1 @@
+37603372705778c2c755d055ac72f835af0ebe51

modules/legacy-geo/licenses/lucene-spatial-extras-9.3.0-snapshot-2d05f5c623e.jar.sha1

@@ -0,0 +1 @@
+899069409ff5a448c641700a0371ce792a0635b1

modules/legacy-geo/licenses/lucene-spatial3d-9.3.0-snapshot-2d05f5c623e.jar.sha1

@@ -0,0 +1 @@
+3416a5a2becfd987049453c939afa2624c3db6c7

plugins/analysis-icu/licenses/lucene-analysis-icu-9.3.0-snapshot-2d05f5c623e.jar.sha1

@@ -0,0 +1 @@
+87772b1d9907c63287d7bc79cf8fa3e9a46c9c77

plugins/analysis-kuromoji/licenses/lucene-analysis-kuromoji-9.3.0-snapshot-2d05f5c623e.jar.sha1

@@ -0,0 +1 @@
+a58272a5eb0137dbcb030a7f2cd7925dd717c3e8

plugins/analysis-nori/licenses/lucene-analysis-nori-9.3.0-snapshot-2d05f5c623e.jar.sha1

@@ -0,0 +1 @@
+e15673b088608c7b7e2b5ab491665f9ea92941b2

plugins/analysis-phonetic/licenses/lucene-analysis-phonetic-9.3.0-snapshot-2d05f5c623e.jar.sha1

@@ -0,0 +1 @@
+f997bf7bdd07a5684e017294764b0a2ca398f948

plugins/analysis-smartcn/licenses/lucene-analysis-smartcn-9.3.0-snapshot-2d05f5c623e.jar.sha1

@@ -0,0 +1 @@
+7cdabbc7114cb7f2925407dd7e18fe6e36fbfc02

plugins/analysis-stempel/licenses/lucene-analysis-stempel-9.3.0-snapshot-2d05f5c623e.jar.sha1

@@ -0,0 +1 @@
+56cd7a7ae2f9db55fc4150f60745f900a1018e35

plugins/analysis-ukrainian/licenses/lucene-analysis-morfologik-9.3.0-snapshot-2d05f5c623e.jar.sha1

@@ -0,0 +1 @@
+d8a80d858ca656735e3f3e0e3803f9130a504f58

qa/os/src/test/java/org/elasticsearch/packaging/test/DebMetadataTests.java

@@ -27,7 +27,6 @@ public static void filterDistros() {
         assumeTrue("only deb", distribution.packaging == Distribution.Packaging.DEB);
     }
 
-    @AwaitsFix(bugUrl = "https://github.com/elastic/elasticsearch/issues/88090")
     public void test05CheckLintian() {
         String extraArgs = "";
         final String helpText = sh.run("lintian --help").stdout();

server/licenses/lucene-analysis-common-9.3.0-snapshot-2d05f5c623e.jar.sha1

@@ -0,0 +1 @@
+ef6b338e156eacb5ae63daa9638899b4b0c44c15

server/licenses/lucene-backward-codecs-9.3.0-snapshot-2d05f5c623e.jar.sha1

@@ -0,0 +1 @@
+25886039976045fb8f9859e12f525eedbb64860a

server/licenses/lucene-core-9.3.0-snapshot-2d05f5c623e.jar.sha1

@@ -0,0 +1 @@
+98b0f69f9d285dcf822bcd90974e8aa4972187b9

server/licenses/lucene-grouping-9.3.0-snapshot-2d05f5c623e.jar.sha1

@@ -0,0 +1 @@
+6f6126445ac9b86ef93c6ab26907a69931ada790

server/licenses/lucene-highlighter-9.3.0-snapshot-2d05f5c623e.jar.sha1

@@ -0,0 +1 @@
+dd6fafd8701b01a3f6279447f7d7c410b1230c13

server/licenses/lucene-join-9.3.0-snapshot-2d05f5c623e.jar.sha1

@@ -0,0 +1 @@
+2bc4615335ba485b2152cd10d21ca0cb768b651e

server/licenses/lucene-memory-9.3.0-snapshot-2d05f5c623e.jar.sha1

@@ -0,0 +1 @@
+3972925a8a1e304e803491881e67c12cbcda4617

server/licenses/lucene-misc-9.3.0-snapshot-2d05f5c623e.jar.sha1

@@ -0,0 +1 @@
+18c2adc5e45d08d65e64d31402219b67d7bc5c61

server/licenses/lucene-queries-9.3.0-snapshot-2d05f5c623e.jar.sha1

@@ -0,0 +1 @@
+bc9e4af7d35799ba941993be0242101f79719cc6

server/licenses/lucene-queryparser-9.3.0-snapshot-2d05f5c623e.jar.sha1

@@ -0,0 +1 @@
+f62b8f566e37fd6cf150f13d866fcc646c210e35

server/licenses/lucene-sandbox-9.3.0-snapshot-2d05f5c623e.jar.sha1

@@ -0,0 +1 @@
+081ff7330c13bfcf0110e7140badfc8be3df910c

server/licenses/lucene-suggest-9.3.0-snapshot-2d05f5c623e.jar.sha1

@@ -0,0 +1 @@
+e3240b6ab35e61a947d94978697816d102b1bd13