From 2475eb0a56319eb29771b1aafa8862790052e18e Mon Sep 17 00:00:00 2001 From: jillguyonnet Date: Thu, 27 Mar 2025 12:37:12 +0100 Subject: [PATCH 1/5] [Fleet] Add documentation for remote ES output integrations syncing --- .../fleet/remote-elasticsearch-output.md | 43 ++++++++++++++++--- 1 file changed, 37 insertions(+), 6 deletions(-) diff --git a/reference/fleet/remote-elasticsearch-output.md b/reference/fleet/remote-elasticsearch-output.md index 2e7bbf10f4..5f60e18550 100644 --- a/reference/fleet/remote-elasticsearch-output.md +++ b/reference/fleet/remote-elasticsearch-output.md @@ -18,6 +18,7 @@ A bug has been found that causes {{elastic-defend}} response actions to stop wor Using a remote {{es}} output with a target cluster that has [traffic filters](/deploy-manage/security/traffic-filtering.md) enabled is not currently supported. :::: +## Configuration To configure a remote {{es}} cluster for your {{agent}} data: @@ -42,12 +43,13 @@ To configure a remote {{es}} cluster for your {{agent}} data: To prevent unauthorized access the {{es}} Service Token is stored as a secret value. While secret storage is recommended, you can choose to override this setting and store the password as plain text in the agent policy definition. Secret storage requires {{fleet-server}} version 8.12 or higher. This setting can also be stored as a secret value or as plain text for preconfigured outputs. See [Preconfiguration settings](kibana://reference/configuration-reference/fleet-settings.md#_preconfiguration_settings_for_advanced_use_cases) in the {{kib}} Guide to learn more. :::: -6. Choose whether or not the remote output should be the default for agent integrations or for agent monitoring data. When set, {{agent}}s use this output to send data if no other output is set in the [agent policy](/reference/fleet/agent-policy.md). -7. Select which [performance tuning settings](/reference/fleet/es-output-settings.md#es-output-settings-performance-tuning-settings) you’d prefer in order to optimize {{agent}} for throughput, scale, or latency, or leave the default `balanced` setting. -8. Add any [advanced YAML configuration settings](/reference/fleet/es-output-settings.md#es-output-settings-yaml-config) that you’d like for the output. -9. Click **Save and apply settings**. +6. Choose whether integrations should automatically be synchronized on the remote {{es}} cluster. Refer to [Automatic integrations synchronization](#automatic-integrations-synchronization) below to configure this feature. +7. Choose whether or not the remote output should be the default for agent integrations or for agent monitoring data. When set, {{agent}}s use this output to send data if no other output is set in the [agent policy](/reference/ingestion-tools/fleet/agent-policy.md). +8. Select which [performance tuning settings](/reference/ingestion-tools/fleet/es-output-settings.md#es-output-settings-performance-tuning-settings) you’d prefer in order to optimize {{agent}} for throughput, scale, or latency, or leave the default `balanced` setting. +9. Add any [advanced YAML configuration settings](/reference/ingestion-tools/fleet/es-output-settings.md#es-output-settings-yaml-config) that you’d like for the output. +10. Click **Save and apply settings**. -After the output is created, you can update an {{agent}} policy to use the new remote {{es}} cluster: +After the output is created, you can update an {{agent}} policy to use the new output and send data to the remote {{es}} cluster: 1. In {{fleet}}, open the **Agent policies** tab. 2. Click the agent policy to edit it, then click **Settings**. @@ -57,4 +59,33 @@ After the output is created, you can update an {{agent}} policy to use the new r The remote {{es}} cluster is now configured. -As a final step before using the remote {{es}} output, you need to make sure that for any integrations that have been [added to your {{agent}} policy](/reference/fleet/add-integration-to-policy.md), the integration assets have been installed on the remote {{es}} cluster. Refer to [Install and uninstall {{agent}} integration assets](/reference/fleet/install-uninstall-integration-assets.md) for the steps. +If you have chosen not to automatically synchronize integrations, you need to make sure that for any integrations that have been [added to your {{agent}} policy](/reference/ingestion-tools/fleet/add-integration-to-policy.md), the integration assets have been installed on the remote {{es}} cluster. Refer to [Install and uninstall {{agent}} integration assets](/reference/ingestion-tools/fleet/install-uninstall-integration-assets.md) for the steps. + +## Automatic integrations synchronization + +Beginning in version 9.1.0, you can choose to keep integrations synced between your main {{es}} cluster and remote {{es}} clusters. + +### Requirements + +This feature requires setting up [{{ccr}}](/deploy-manage/tools/cross-cluster-replication.md), which is available to Platinum and Enterprise [subscriptions](https://www.elastic.co/subscriptions). Remote clusters must be running the same version of Elasticsearch as the main cluster or a newer version that is compatible with {{ccr}}. + +Remote clusters require access to [{{package-registry}}](/reference/ingestion-tools/fleet/index.md#package-registry-intro) to install integrations. + +### Configuration + +1. Configure {{ccr}} on the remote cluster. + + 1. In the remote cluster, open the {{kib}} menu and go to **Stack Management → Remote Clusters**. + 2. Refer to [Remote clusters](https://www.elastic.co/guide/en/elasticsearch/reference/current/remote-clusters.html) to add your main cluster (where the remote {{es}} output is configured) as a remote cluster. + 3. Go to **Stack Management > Cross-Cluster Replication**. + 4. Create a follower index named `fleet-synced-integrations-ccr-` that replicates the `fleet-synced-integrations` leader index on the main cluster. + +2. In the main cluster, in the **Remote Kibana URL** field, add the Kibana URL of the remote cluster. + +3. Create an API key to access Kibana on the remote cluster. + + 1. Below the **Remote Kibana API Key** field, copy the API request. + 2. In the remote cluster, open the {{kib}} menu and go to **Management → Dev Tools**. + 3. Run the API request. + 4. Copy the encoded value of the generated API key. + 5. Back in the main cluster, paste the value you copied into the output **Remote Kibana API Key** field. From e56e4a3e5e51d0a333439b3e3aafed62ca58895e Mon Sep 17 00:00:00 2001 From: jillguyonnet Date: Thu, 27 Mar 2025 15:28:07 +0100 Subject: [PATCH 2/5] Feedback --- reference/fleet/remote-elasticsearch-output.md | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/reference/fleet/remote-elasticsearch-output.md b/reference/fleet/remote-elasticsearch-output.md index 5f60e18550..8e19851c9d 100644 --- a/reference/fleet/remote-elasticsearch-output.md +++ b/reference/fleet/remote-elasticsearch-output.md @@ -34,7 +34,7 @@ To configure a remote {{es}} cluster for your {{agent}} data: 5. Create a service token to access the remote cluster. 1. Below the **Service Token** field, copy the API request. - 2. In the remote cluster, open the {{kib}} menu and go to **Management → Dev Tools**. + 2. In the remote cluster, open the {{kib}} menu and go to **Management > Dev Tools**. 3. Run the API request. 4. Copy the value for the generated token. 5. Back in your main cluster, paste the value you copied into the output **Service Token** field. @@ -44,7 +44,7 @@ To configure a remote {{es}} cluster for your {{agent}} data: :::: 6. Choose whether integrations should automatically be synchronized on the remote {{es}} cluster. Refer to [Automatic integrations synchronization](#automatic-integrations-synchronization) below to configure this feature. -7. Choose whether or not the remote output should be the default for agent integrations or for agent monitoring data. When set, {{agent}}s use this output to send data if no other output is set in the [agent policy](/reference/ingestion-tools/fleet/agent-policy.md). +7. Choose whether or not the remote output should be the default for agent integrations or for agent monitoring data. When set, {{agents}} use this output to send data if no other output is set in the [agent policy](/reference/ingestion-tools/fleet/agent-policy.md). 8. Select which [performance tuning settings](/reference/ingestion-tools/fleet/es-output-settings.md#es-output-settings-performance-tuning-settings) you’d prefer in order to optimize {{agent}} for throughput, scale, or latency, or leave the default `balanced` setting. 9. Add any [advanced YAML configuration settings](/reference/ingestion-tools/fleet/es-output-settings.md#es-output-settings-yaml-config) that you’d like for the output. 10. Click **Save and apply settings**. @@ -63,19 +63,23 @@ If you have chosen not to automatically synchronize integrations, you need to ma ## Automatic integrations synchronization -Beginning in version 9.1.0, you can choose to keep integrations synced between your main {{es}} cluster and remote {{es}} clusters. +```{applies_to} +stack: ga 9.1 +``` + +When enabled, this feature keeps integrations synced between your main {{es}} cluster and remote {{es}} clusters. ### Requirements -This feature requires setting up [{{ccr}}](/deploy-manage/tools/cross-cluster-replication.md), which is available to Platinum and Enterprise [subscriptions](https://www.elastic.co/subscriptions). Remote clusters must be running the same version of Elasticsearch as the main cluster or a newer version that is compatible with {{ccr}}. +This feature requires setting up [{{ccr}}](/deploy-manage/tools/cross-cluster-replication.md), which is available to Platinum and Enterprise [subscriptions](https://www.elastic.co/subscriptions). Remote clusters must be running the same version of {{es}} as the main cluster or a newer version that is compatible with {{ccr}}. -Remote clusters require access to [{{package-registry}}](/reference/ingestion-tools/fleet/index.md#package-registry-intro) to install integrations. +Remote clusters require access to the [{{package-registry}}](/reference/ingestion-tools/fleet/index.md#package-registry-intro) to install integrations. ### Configuration 1. Configure {{ccr}} on the remote cluster. - 1. In the remote cluster, open the {{kib}} menu and go to **Stack Management → Remote Clusters**. + 1. In the remote cluster, open the {{kib}} menu and go to **Stack Management > Remote Clusters**. 2. Refer to [Remote clusters](https://www.elastic.co/guide/en/elasticsearch/reference/current/remote-clusters.html) to add your main cluster (where the remote {{es}} output is configured) as a remote cluster. 3. Go to **Stack Management > Cross-Cluster Replication**. 4. Create a follower index named `fleet-synced-integrations-ccr-` that replicates the `fleet-synced-integrations` leader index on the main cluster. @@ -85,7 +89,7 @@ Remote clusters require access to [{{package-registry}}](/reference/ingestion-to 3. Create an API key to access Kibana on the remote cluster. 1. Below the **Remote Kibana API Key** field, copy the API request. - 2. In the remote cluster, open the {{kib}} menu and go to **Management → Dev Tools**. + 2. In the remote cluster, open the {{kib}} menu and go to **Management > Dev Tools**. 3. Run the API request. 4. Copy the encoded value of the generated API key. 5. Back in the main cluster, paste the value you copied into the output **Remote Kibana API Key** field. From 47929c4d6e61cb611d3dbac76fab16fcd6ffd488 Mon Sep 17 00:00:00 2001 From: jillguyonnet Date: Thu, 27 Mar 2025 15:43:43 +0100 Subject: [PATCH 3/5] Fix reference links --- reference/fleet/remote-elasticsearch-output.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/reference/fleet/remote-elasticsearch-output.md b/reference/fleet/remote-elasticsearch-output.md index 8e19851c9d..d42a626602 100644 --- a/reference/fleet/remote-elasticsearch-output.md +++ b/reference/fleet/remote-elasticsearch-output.md @@ -44,9 +44,9 @@ To configure a remote {{es}} cluster for your {{agent}} data: :::: 6. Choose whether integrations should automatically be synchronized on the remote {{es}} cluster. Refer to [Automatic integrations synchronization](#automatic-integrations-synchronization) below to configure this feature. -7. Choose whether or not the remote output should be the default for agent integrations or for agent monitoring data. When set, {{agents}} use this output to send data if no other output is set in the [agent policy](/reference/ingestion-tools/fleet/agent-policy.md). -8. Select which [performance tuning settings](/reference/ingestion-tools/fleet/es-output-settings.md#es-output-settings-performance-tuning-settings) you’d prefer in order to optimize {{agent}} for throughput, scale, or latency, or leave the default `balanced` setting. -9. Add any [advanced YAML configuration settings](/reference/ingestion-tools/fleet/es-output-settings.md#es-output-settings-yaml-config) that you’d like for the output. +7. Choose whether or not the remote output should be the default for agent integrations or for agent monitoring data. When set, {{agents}} use this output to send data if no other output is set in the [agent policy](/reference/fleet/agent-policy.md). +8. Select which [performance tuning settings](/reference/fleet/es-output-settings.md#es-output-settings-performance-tuning-settings) you’d prefer in order to optimize {{agent}} for throughput, scale, or latency, or leave the default `balanced` setting. +9. Add any [advanced YAML configuration settings](/reference/fleet/es-output-settings.md#es-output-settings-yaml-config) that you’d like for the output. 10. Click **Save and apply settings**. After the output is created, you can update an {{agent}} policy to use the new output and send data to the remote {{es}} cluster: @@ -59,7 +59,7 @@ After the output is created, you can update an {{agent}} policy to use the new o The remote {{es}} cluster is now configured. -If you have chosen not to automatically synchronize integrations, you need to make sure that for any integrations that have been [added to your {{agent}} policy](/reference/ingestion-tools/fleet/add-integration-to-policy.md), the integration assets have been installed on the remote {{es}} cluster. Refer to [Install and uninstall {{agent}} integration assets](/reference/ingestion-tools/fleet/install-uninstall-integration-assets.md) for the steps. +If you have chosen not to automatically synchronize integrations, you need to make sure that for any integrations that have been [added to your {{agent}} policy](/reference/fleet/add-integration-to-policy.md), the integration assets have been installed on the remote {{es}} cluster. Refer to [Install and uninstall {{agent}} integration assets](/reference/fleet/install-uninstall-integration-assets.md) for the steps. ## Automatic integrations synchronization @@ -73,7 +73,7 @@ When enabled, this feature keeps integrations synced between your main {{es}} cl This feature requires setting up [{{ccr}}](/deploy-manage/tools/cross-cluster-replication.md), which is available to Platinum and Enterprise [subscriptions](https://www.elastic.co/subscriptions). Remote clusters must be running the same version of {{es}} as the main cluster or a newer version that is compatible with {{ccr}}. -Remote clusters require access to the [{{package-registry}}](/reference/ingestion-tools/fleet/index.md#package-registry-intro) to install integrations. +Remote clusters require access to the [{{package-registry}}](/reference/fleet/index.md#package-registry-intro) to install integrations. ### Configuration From b2a4a2d7d073a8011dbcdc83ab8e262f0e4455d1 Mon Sep 17 00:00:00 2001 From: jillguyonnet Date: Thu, 27 Mar 2025 15:53:16 +0100 Subject: [PATCH 4/5] Remove remote ES output min version --- reference/fleet/remote-elasticsearch-output.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/fleet/remote-elasticsearch-output.md b/reference/fleet/remote-elasticsearch-output.md index d42a626602..3fffcf43a0 100644 --- a/reference/fleet/remote-elasticsearch-output.md +++ b/reference/fleet/remote-elasticsearch-output.md @@ -5,7 +5,7 @@ mapped_pages: # Remote Elasticsearch output [remote-elasticsearch-output] -Beginning in version 8.12.0, you can send {{agent}} data to a remote {{es}} cluster. This is especially useful for data that you want to keep separate and independent from the deployment where you use {{fleet}} to manage the agents. +Remote {{es}} outputs allow you to send {{agent}} data to a remote {{es}} cluster. This is especially useful for data that you want to keep separate and independent from the deployment where you use {{fleet}} to manage the agents. A remote {{es}} cluster supports the same [output settings](/reference/fleet/es-output-settings.md) as your main {{es}} cluster. From 1ff83ec3e2e09f5b544b80e15f10e0d39690a611 Mon Sep 17 00:00:00 2001 From: jillguyonnet Date: Mon, 31 Mar 2025 14:59:58 +0200 Subject: [PATCH 5/5] Feedback --- reference/fleet/remote-elasticsearch-output.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/reference/fleet/remote-elasticsearch-output.md b/reference/fleet/remote-elasticsearch-output.md index 3fffcf43a0..cc22d9955b 100644 --- a/reference/fleet/remote-elasticsearch-output.md +++ b/reference/fleet/remote-elasticsearch-output.md @@ -83,6 +83,7 @@ Remote clusters require access to the [{{package-registry}}](/reference/fleet/in 2. Refer to [Remote clusters](https://www.elastic.co/guide/en/elasticsearch/reference/current/remote-clusters.html) to add your main cluster (where the remote {{es}} output is configured) as a remote cluster. 3. Go to **Stack Management > Cross-Cluster Replication**. 4. Create a follower index named `fleet-synced-integrations-ccr-` that replicates the `fleet-synced-integrations` leader index on the main cluster. + 5. Resume replication once the follower index is created. 2. In the main cluster, in the **Remote Kibana URL** field, add the Kibana URL of the remote cluster. @@ -92,4 +93,4 @@ Remote clusters require access to the [{{package-registry}}](/reference/fleet/in 2. In the remote cluster, open the {{kib}} menu and go to **Management > Dev Tools**. 3. Run the API request. 4. Copy the encoded value of the generated API key. - 5. Back in the main cluster, paste the value you copied into the output **Remote Kibana API Key** field. + 5. Back in the main cluster, paste the value you copied into the **Remote Kibana API Key** field.