Merge remote-tracking branch 'origin/main' into treat-newly-unassigne…

…d-replicas-as-special-snowflakes

distribution/tools/server-cli/src/main/java/org/elasticsearch/server/cli/ServerCli.java

@@ -237,6 +237,11 @@ public void close() {
         }
     }
 
+    // allow subclasses to access the started process
+    protected ServerProcess getServer() {
+        return server;
+    }
+
     // protected to allow tests to override
     protected Command loadTool(String toolname, String libs) {
         return CliToolProvider.load(toolname, libs).create();

docs/changelog/107081.yaml

@@ -0,0 +1,5 @@
+pr: 107081
+summary: Implement synthetic source support for range fields
+area: Mapping
+type: feature
+issues: []

docs/changelog/107435.yaml

@@ -0,0 +1,6 @@
+pr: 107435
+summary: '`NoSuchRemoteClusterException` should not be thrown when a remote is configured'
+area: Network
+type: bug
+issues:
+ - 107381

docs/changelog/107577.yaml

@@ -0,0 +1,6 @@
+pr: 107577
+summary: "ESQL: Fix MV_DEDUPE when using data from an index"
+area: ES|QL
+type: bug
+issues:
+ - 104745

docs/changelog/107678.yaml

@@ -0,0 +1,6 @@
+pr: 107678
+summary: Validate stats formatting in standard `InternalStats` constructor
+area: Aggregations
+type: bug
+issues:
+ - 107671

docs/changelog/107764.yaml

@@ -0,0 +1,5 @@
+pr: 107764
+summary: Increase size of big arrays only when there is an actual value in the aggregators
+area: Aggregations
+type: enhancement
+issues: []

docs/changelog/107785.yaml

@@ -0,0 +1,5 @@
+pr: 107785
+summary: Fix `minimized_round_trips` in lookup runtime fields
+area: Search
+type: bug
+issues: []

docs/internal/DistributedArchitectureGuide.md

@@ -79,10 +79,48 @@ caller timeouts.
 
 ### REST Layer
 
-(including how REST and Transport layers are bound together through the ActionModule)
+The REST and Transport layers are bound together through the `ActionModule`. `ActionModule#initRestHandlers` registers all the
+rest actions with a `RestController` that matches incoming requests to particular REST actions. `RestController#registerHandler`
+uses each `Rest*Action`'s `#routes()` implementation to match HTTP requests to that particular `Rest*Action`. Typically, REST
+actions follow the class naming convention `Rest*Action`, which makes them easier to find, but not always; the `#routes()`
+definition can also be helpful in finding a REST action. `RestController#dispatchRequest` eventually calls `#handleRequest` on a
+`RestHandler` implementation. `RestHandler` is the base class for `BaseRestHandler`, which most `Rest*Action` instances extend to
+implement a particular REST action.
+
+`BaseRestHandler#handleRequest` calls into `BaseRestHandler#prepareRequest`, which children `Rest*Action` classes extend to
+define the behavior for a particular action. `RestController#dispatchRequest` passes a `RestChannel` to the `Rest*Action` via
+`RestHandler#handleRequest`: `Rest*Action#prepareRequest` implementations return a `RestChannelConsumer` defining how to execute
+the action and reply on the channel (usually in the form of completing an ActionListener wrapper). `Rest*Action#prepareRequest`
+implementations are responsible for parsing the incoming request, and verifying that the structure of the request is valid.
+`BaseRestHandler#handleRequest` will then check that all the request parameters have been consumed: unexpected request parameters
+result in an error.
+
+### How REST Actions Connect to Transport Actions
+
+The Rest layer uses an implementation of `AbstractClient`. `BaseRestHandler#prepareRequest` takes a `NodeClient`: this client
+knows how to connect to a specified TransportAction. A `Rest*Action` implementation will return a `RestChannelConsumer` that
+most often invokes a call into a method on the `NodeClient` to pass through to the TransportAction. Along the way from
+`BaseRestHandler#prepareRequest` through the `AbstractClient` and `NodeClient` code, `NodeClient#executeLocally` is called: this
+method calls into `TaskManager#registerAndExecute`, registering the operation with the `TaskManager` so it can be found in Task
+API requests, before moving on to execute the specified TransportAction.
+
+`NodeClient` has a `NodeClient#actions` map from `ActionType` to `TransportAction`. `ActionModule#setupActions` registers all the
+core TransportActions, as well as those defined in any plugins that are being used: plugins can override `Plugin#getActions()` to
+define additional TransportActions. Note that not all TransportActions will be mapped back to a REST action: many TransportActions
+are only used for internode operations/communications.
 
 ### Transport Layer
 
+(Managed by the TransportService, TransportActions must be registered there, too)
+
+(Executing a TransportAction (either locally via NodeClient or remotely via TransportService) is where most of the authorization & other security logic runs)
+
+(What actions, and why, are registered in TransportService but not NodeClient?)
+
+### Direct Node to Node Transport Layer
+
+(TransportService maps incoming requests to TransportActions)
+
 ### Chunk Encoding
 
 #### XContent

docs/reference/connector/apis/cancel-connector-sync-job-api.asciidoc

@@ -15,7 +15,7 @@ Cancels a connector sync job.
 [[cancel-connector-sync-job-api-prereqs]]
 ==== {api-prereq-title}
 
-* To sync data using connectors, it's essential to have the Elastic connectors service running.
+* To sync data using self-managed connectors, you need to deploy the {enterprise-search-ref}/build-connector.html[Elastic connector service] on your own infrastructure. This service runs automatically on Elastic Cloud for native connectors.
 * The `connector_sync_job_id` parameter should reference an existing connector sync job.
 
 [[cancel-connector-sync-job-api-desc]]
@@ -49,4 +49,3 @@ The following example cancels the connector sync job with ID `my-connector-sync-
 PUT _connector/_sync_job/my-connector-sync-job-id/_cancel
 ----
 // TEST[skip:there's no way to clean up after creating a connector sync job, as we don't know the id ahead of time. Therefore, skip this test.]
-

docs/reference/connector/apis/check-in-connector-api.asciidoc

@@ -16,7 +16,7 @@ Updates the `last_seen` field of a connector with current timestamp.
 [[check-in-connector-api-prereq]]
 ==== {api-prereq-title}
 
-* To sync data using connectors, it's essential to have the Elastic connectors service running.
+* To sync data using self-managed connectors, you need to deploy the {enterprise-search-ref}/build-connector.html[Elastic connector service] on your own infrastructure. This service runs automatically on Elastic Cloud for native connectors.
 * The `connector_id` parameter should reference an existing connector.
 
 [[check-in-connector-api-path-params]]

docs/reference/connector/apis/check-in-connector-sync-job-api.asciidoc

@@ -15,7 +15,7 @@ Checks in a connector sync job (updates `last_seen` to the current time).
 [[check-in-connector-sync-job-api-prereqs]]
 ==== {api-prereq-title}
 
-* To sync data using connectors, it's essential to have the Elastic connectors service running.
+* To sync data using self-managed connectors, you need to deploy the {enterprise-search-ref}/build-connector.html[Elastic connector service] on your own infrastructure. This service runs automatically on Elastic Cloud for native connectors.
 * The `connector_sync_job_id` parameter should reference an existing connector sync job.
 
 [[check-in-connector-sync-job-api-desc]]

docs/reference/connector/apis/connector-apis.asciidoc

@@ -5,7 +5,7 @@ preview::[]
 
 The connector and sync jobs APIs provide a convenient way to create and manage Elastic {enterprise-search-ref}/connectors.html[connectors^] and sync jobs in an internal index.
 
-Connectors are third-party {es} integrations which can be deployed on {ecloud} or hosted on your own infrastructure:
+Connectors are {es} integrations that bring content from third-party data sources, which can be deployed on {ecloud} or hosted on your own infrastructure:
 
 * *Native connectors* are a managed service on {ecloud}
 * *Connector clients* are self-managed on your infrastructure

docs/reference/connector/apis/create-connector-api.asciidoc

@@ -7,7 +7,7 @@
 preview::[]
 
 Creates an Elastic connector.
-Connectors are third-party {es} integrations which can be deployed on {ecloud} or hosted on your own infrastructure:
+Connectors are {es} integrations that bring content from third-party data sources, which can be deployed on {ecloud} or hosted on your own infrastructure:
 
 * *Native connectors* are a managed service on {ecloud}
 * *Connector clients* are self-managed on your infrastructure
@@ -41,8 +41,8 @@ DELETE _connector/my-connector
 [[create-connector-api-prereqs]]
 ==== {api-prereq-title}
 
-* To sync data using connectors, it's essential to have the Elastic connectors service running.
-* The `service_type` parameter should reference an existing connector service type.
+* To sync data using self-managed connectors, you need to deploy the {enterprise-search-ref}/build-connector.html[Elastic connector service] on your own infrastructure. This service runs automatically on Elastic Cloud for native connectors.
+* The `service_type` parameter should reference a supported third-party service. See the available service types for {enterprise-search-ref}/native-connectors.html[native] and {enterprise-search-ref}/build-connector.html[self-managed] connectors. This can also reference the service type of your custom connector.
 
 
 [[create-connector-api-desc]]
@@ -65,10 +65,10 @@ Creates a connector document in the internal index and initializes its configura
 (Optional, string) The description of the connector.
 
 `index_name`::
-(Required, string) The target index for syncing data by the connector.
+(Optional, string) The target index to sync data. If the index doesn't exist, it will be created upon the first sync.
 
 `name`::
-(Optional, string) The name of the connector.
+(Optional, string) The name of the connector. Setting the connector name is recommended when managing connectors in {kib}.
 
 `is_native`::
 (Optional, boolean) Indicates if it's a native connector. Defaults to `false`.
@@ -77,7 +77,7 @@ Creates a connector document in the internal index and initializes its configura
 (Optional, string) Language analyzer for the data. Limited to supported languages.
 
 `service_type`::
-(Optional, string) Connector service type. Can reference Elastic-supported connector types or a custom connector type.
+(Optional, string) Connector service type. Can reference Elastic-supported third-party services or a custom connector type. See the available service types for {enterprise-search-ref}/native-connectors.html[native] and {enterprise-search-ref}/build-connector.html[self-managed] connectors.
 
 
 [role="child_attributes"]

docs/reference/connector/apis/create-connector-sync-job-api.asciidoc

@@ -29,7 +29,7 @@ POST _connector/_sync_job
 [[create-connector-sync-job-api-prereqs]]
 ==== {api-prereq-title}
 
-* To sync data using connectors, it's essential to have the Elastic connectors service running.
+* To sync data using self-managed connectors, you need to deploy the {enterprise-search-ref}/build-connector.html[Elastic connector service] on your own infrastructure. This service runs automatically on Elastic Cloud for native connectors.
 * The `id` parameter should reference an existing connector.
 
 [[create-connector-sync-job-api-desc]]

docs/reference/connector/apis/delete-connector-api.asciidoc

@@ -6,9 +6,11 @@
 
 preview::[]
 
-Removes a connector and its associated data.
+Removes a connector and associated sync jobs.
 This is a destructive action that is not recoverable.
 
+Note: this action doesn't delete any API key, ingest pipeline or data index associated with the connector. These need to be removed manually.
+
 [[delete-connector-api-request]]
 ==== {api-request-title}
 
@@ -17,7 +19,7 @@ This is a destructive action that is not recoverable.
 [[delete-connector-api-prereq]]
 ==== {api-prereq-title}
 
-* To sync data using connectors, it's essential to have the Elastic connectors service running.
+* To sync data using self-managed connectors, you need to deploy the {enterprise-search-ref}/build-connector.html[Elastic connector service] on your own infrastructure. This service runs automatically on Elastic Cloud for native connectors.
 * The `connector_id` parameter should reference an existing connector.
 
 [[delete-connector-api-path-params]]

docs/reference/connector/apis/delete-connector-sync-job-api.asciidoc

@@ -17,7 +17,7 @@ This is a destructive action that is not recoverable.
 [[delete-connector-sync-job-api-prereq]]
 ==== {api-prereq-title}
 
-* To sync data using connectors, it's essential to have the Elastic connectors service running.
+* To sync data using self-managed connectors, you need to deploy the {enterprise-search-ref}/build-connector.html[Elastic connector service] on your own infrastructure. This service runs automatically on Elastic Cloud for native connectors.
 
 [[delete-connector-sync-job-api-path-params]]
 ==== {api-path-parms-title}

docs/reference/connector/apis/get-connector-api.asciidoc

@@ -16,7 +16,7 @@ Retrieves the details about a connector.
 [[get-connector-api-prereq]]
 ==== {api-prereq-title}
 
-* To sync data using connectors, it's essential to have the Elastic connectors service running.
+* To sync data using self-managed connectors, you need to deploy the {enterprise-search-ref}/build-connector.html[Elastic connector service] on your own infrastructure. This service runs automatically on Elastic Cloud for native connectors.
 
 [[get-connector-api-path-params]]
 ==== {api-path-parms-title}

docs/reference/connector/apis/get-connector-sync-job-api.asciidoc

@@ -16,7 +16,7 @@ Retrieves the details about a connector sync job.
 [[get-connector-sync-job-api-prereq]]
 ==== {api-prereq-title}
 
-* To sync data using connectors, it's essential to have the Elastic connectors service running.
+* To sync data using self-managed connectors, you need to deploy the {enterprise-search-ref}/build-connector.html[Elastic connector service] on your own infrastructure. This service runs automatically on Elastic Cloud for native connectors.
 
 [[get-connector-sync-job-api-path-params]]
 ==== {api-path-parms-title}

docs/reference/connector/apis/list-connector-sync-jobs-api.asciidoc

@@ -18,7 +18,7 @@ Returns information about all stored connector sync jobs ordered by their creati
 [[list-connector-sync-jobs-api-prereq]]
 ==== {api-prereq-title}
 
-* To sync data using connectors, it's essential to have the Elastic connectors service running.
+* To sync data using self-managed connectors, you need to deploy the {enterprise-search-ref}/build-connector.html[Elastic connector service] on your own infrastructure. This service runs automatically on Elastic Cloud for native connectors.
 
 [[list-connector-sync-jobs-api-path-params]]
 ==== {api-path-parms-title}

docs/reference/connector/apis/list-connectors-api.asciidoc

@@ -7,7 +7,7 @@
 
 preview::[]
 
-Returns information about all stored connectors.
+Returns information about all created connectors.
 
 
 [[list-connector-api-request]]
@@ -18,19 +18,19 @@ Returns information about all stored connectors.
 [[list-connector-api-prereq]]
 ==== {api-prereq-title}
 
-* To sync data using connectors, it's essential to have the Elastic connectors service running.
+* To sync data using self-managed connectors, you need to deploy the {enterprise-search-ref}/build-connector.html[Elastic connector service] on your own infrastructure. This service runs automatically on Elastic Cloud for native connectors.
 
 [[list-connector-api-path-params]]
 ==== {api-path-parms-title}
 
 `size`::
-(Optional, integer) Maximum number of results to retrieve.
+(Optional, integer) Maximum number of results to retrieve. Defaults to `100`.
 
 `from`::
-(Optional, integer) The offset from the first result to fetch.
+(Optional, integer) The offset from the first result to fetch. Defaults to `0`.
 
 `index_name`::
-(Optional, string) A comma-separated list of data index names associated with connectors, used to filter search results.
+(Optional, string) A comma-separated list of index names associated with connectors, used to filter search results.
 
 `connector_name`::
 (Optional, string) A comma-separated list of connector names, used to filter search results.
@@ -81,27 +81,27 @@ The following example lists the first two connectors:
 
 [source,console]
 ----
-GET _connector/?from=0&size=2
+GET _connector?from=0&size=2
 ----
 
-An example to list connectors associated with `search-google-drive` data index name:
+An example to list a connector associated with the `search-google-drive` Elasticsearch index:
 
 [source,console]
 ----
-GET _connector/?index_name=search-google-drive
+GET _connector?index_name=search-google-drive
 ----
 
 
 An example to list all connectors with `sharepoint_online` service type:
 
 [source,console]
 ----
-GET _connector/?service_type=sharepoint_online
+GET _connector?service_type=sharepoint_online
 ----
 
 An example to list all connectors with `sharepoint_online` or `google_drive` service type:
 
 [source,console]
 ----
-GET _connector/?service_type=sharepoint_online,google_drive
+GET _connector?service_type=sharepoint_online,google_drive
 ----

docs/reference/connector/apis/set-connector-sync-job-error-api.asciidoc

@@ -15,7 +15,7 @@ Sets a connector sync job error.
 [[set-connector-sync-job-error-api-prereqs]]
 ==== {api-prereq-title}
 
-* To sync data using connectors, it's essential to have the Elastic connectors service running.
+* To sync data using self-managed connectors, you need to deploy the {enterprise-search-ref}/build-connector.html[Elastic connector service] on your own infrastructure. This service runs automatically on Elastic Cloud for native connectors.
 * The `connector_sync_job_id` parameter should reference an existing connector sync job.
 
 [[set-connector-sync-job-error-api-desc]]

docs/reference/connector/apis/set-connector-sync-job-stats-api.asciidoc

@@ -15,7 +15,7 @@ Sets connector sync job stats.
 [[set-connector-sync-job-stats-api-prereqs]]
 ==== {api-prereq-title}
 
-* To sync data using connectors, it's essential to have the Elastic connectors service running.
+* To sync data using self-managed connectors, you need to deploy the {enterprise-search-ref}/build-connector.html[Elastic connector service] on your own infrastructure. This service runs automatically on Elastic Cloud for native connectors.
 * The `connector_sync_job_id` parameter should reference an existing connector sync job.
 
 [[set-connector-sync-job-stats-api-desc]]

docs/reference/connector/apis/update-connector-api-key-id-api.asciidoc

@@ -23,7 +23,7 @@ See the documentation for {enterprise-search-ref}/native-connectors.html#native-
 [[update-connector-api-key-id-api-prereq]]
 ==== {api-prereq-title}
 
-* To sync data using connectors, it's essential to have the Elastic connectors service running.
+* To sync data using self-managed connectors, you need to deploy the {enterprise-search-ref}/build-connector.html[Elastic connector service] on your own infrastructure. This service runs automatically on Elastic Cloud for native connectors.
 * The `connector_id` parameter should reference an existing connector.
 * The `api_key_id` parameter should reference an existing API key.
 * The `api_key_secret_id` parameter should reference an existing Connector Secret containing an encoded API key value.

docs/reference/connector/apis/update-connector-configuration-api.asciidoc

@@ -17,7 +17,7 @@ Updates a connector's `configuration`, allowing for complete schema modification
 [[update-connector-configuration-api-prereq]]
 ==== {api-prereq-title}
 
-* To sync data using connectors, it's essential to have the Elastic connectors service running.
+* To sync data using self-managed connectors, you need to deploy the {enterprise-search-ref}/build-connector.html[Elastic connector service] on your own infrastructure. This service runs automatically on Elastic Cloud for native connectors.
 * The `connector_id` parameter should reference an existing connector.
 * The configuration fields definition must be compatible with the specific connector type being used.
 

docs/reference/connector/apis/update-connector-error-api.asciidoc

@@ -16,7 +16,7 @@ Updates the `error` field of a connector.
 [[update-connector-error-api-prereq]]
 ==== {api-prereq-title}
 
-* To sync data using connectors, it's essential to have the Elastic connectors service running.
+* To sync data using self-managed connectors, you need to deploy the {enterprise-search-ref}/build-connector.html[Elastic connector service] on your own infrastructure. This service runs automatically on Elastic Cloud for native connectors.
 * The `connector_id` parameter should reference an existing connector.
 
 [[update-connector-error-api-path-params]]

docs/reference/connector/apis/update-connector-filtering-api.asciidoc

@@ -17,7 +17,7 @@ Updates the `filtering` configuration of a connector. Learn more about filtering
 [[update-connector-filtering-api-prereq]]
 ==== {api-prereq-title}
 
-* To sync data using connectors, it's essential to have the Elastic connectors service running.
+* To sync data using self-managed connectors, you need to deploy the {enterprise-search-ref}/build-connector.html[Elastic connector service] on your own infrastructure. This service runs automatically on Elastic Cloud for native connectors.
 * The `connector_id` parameter should reference an existing connector.
 
 [[update-connector-filtering-api-path-params]]