diff --git a/deploy-manage/cloud-organization/billing/cloud-hosted-deployment-billing-dimensions.md b/deploy-manage/cloud-organization/billing/cloud-hosted-deployment-billing-dimensions.md index 4e426bafb1..bb13d910b7 100644 --- a/deploy-manage/cloud-organization/billing/cloud-hosted-deployment-billing-dimensions.md +++ b/deploy-manage/cloud-organization/billing/cloud-hosted-deployment-billing-dimensions.md @@ -60,7 +60,7 @@ Data transfer out of deployments and between nodes of the cluster is hard to con The largest contributor to inter-node data transfer is usually shard movement between nodes in a cluster. The only way to prevent shard movement is by having a single node in a single availability zone. This solution is only possible for clusters up to 64GB RAM and is not recommended as it creates a risk of data loss. [Oversharding](/deploy-manage/production-guidance/optimize-performance/size-shards.md) can cause excessive shard movement. Avoiding oversharding can also help control costs and improve performance. Note that creating snapshots generates inter-node data transfer. The *storage* cost of snapshots is detailed later in this document. -The exact root cause of unusual data transfer is not always something we can identify as it can have many causes, some of which are out of our control and not associated with Cloud configuration changes. It may help to [enable monitoring](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) and examine index and shard activity on your cluster. +The exact root cause of unusual data transfer is not always something we can identify as it can have many causes, some of which are out of our control and not associated with Cloud configuration changes. It may help to [enable monitoring](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md) and examine index and shard activity on your cluster. ## Storage [storage] diff --git a/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-create-templates.md b/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-create-templates.md index c838a8d1e0..3f3946b355 100644 --- a/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-create-templates.md +++ b/deploy-manage/deploy/cloud-enterprise/ece-configuring-ece-create-templates.md @@ -76,7 +76,7 @@ Before you start creating your own deployment templates, you should have: [tagge 9. On this page you can [configure index management](ece-configure-templates-index-management.md) by assigning attributes to each of the data nodes in the deployment template. In Kibana, you can configure an index lifecycle management (ILM) policy, based on the node attributes, to control how data moves across the nodes in your deployment. 10. Select **Stack features**. 11. You can select a [snapshot repository](../../tools/snapshot-and-restore/cloud-enterprise.md) to be used by default for deployment backups. -12. You can choose to [enable logging and monitoring](../../monitor/stack-monitoring/ece-stack-monitoring.md) by default, so that deployment logs and metrics are send to a dedicated monitoring deployment, and so that additional log types, retention options, and Kibana visualizations are available on all deployments created using this template. +12. You can choose to [enable logging and monitoring](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md) by default, so that deployment logs and metrics are send to a dedicated monitoring deployment, and so that additional log types, retention options, and Kibana visualizations are available on all deployments created using this template. 13. Select **Extensions**. 14. Select any Elasticsearch extensions that you would like to be available automatically to all deployments created using the template. 15. Select **Save and create template**. diff --git a/deploy-manage/deploy/cloud-enterprise/system-deployments-configuration.md b/deploy-manage/deploy/cloud-enterprise/system-deployments-configuration.md index f0a89b3206..8f89099625 100644 --- a/deploy-manage/deploy/cloud-enterprise/system-deployments-configuration.md +++ b/deploy-manage/deploy/cloud-enterprise/system-deployments-configuration.md @@ -69,7 +69,7 @@ In the case of the `admin-console-elasticsearch` and `security` system deploymen The `logging-and-metrics` cluster is different since, as an ECE admin, you likely want to provide users with access to the cluster in order to troubleshoot issues without your assistance, for example. In order to manage access to that cluster, you can configure roles that will provide access to the relevant indices, map those to users, and manage access to Kibana by leveraging the Elastic security integration with your authentication provider, such as LDAP, SAML, or AD. To configure one of those security realms, check [LDAP](../../users-roles/cluster-or-deployment-auth/ldap.md), [Active Directory](../../users-roles/cluster-or-deployment-auth/active-directory.md) or [SAML](../../users-roles/cluster-or-deployment-auth/saml.md). ::::{note} -The `logging-and-metrics` cluster is only intended for troubleshooting ECE deployment issues. If your use case involves modifying or normalizing logs from {{es}} or {{kib}}, use a separate [dedicated monitoring deployment](../../monitor/stack-monitoring/ece-stack-monitoring.md) instead. +The `logging-and-metrics` cluster is only intended for troubleshooting ECE deployment issues. If your use case involves modifying or normalizing logs from {{es}} or {{kib}}, use a separate [dedicated monitoring deployment](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md) instead. :::: You can’t use ECE’s single sign-on (SSO) to access system deployments. diff --git a/deploy-manage/deploy/cloud-enterprise/working-with-deployments.md b/deploy-manage/deploy/cloud-enterprise/working-with-deployments.md index 5c40541300..cb3bec22f3 100644 --- a/deploy-manage/deploy/cloud-enterprise/working-with-deployments.md +++ b/deploy-manage/deploy/cloud-enterprise/working-with-deployments.md @@ -44,7 +44,7 @@ From the deployment main page, you can quickly access the following configuratio * Select **{{es}} > snapshots** to associate a [snapshots repository](../../tools/snapshot-and-restore/cloud-enterprise.md#ece-manage-repositories-clusters) with the deployment. -* Select **Monitoring > Logs and metrics** to set up [Stack monitoring](../../monitor/stack-monitoring/ece-stack-monitoring.md) for your deployment, forwarding its logs and metrics to a dedicated monitoring deployment. +* Select **Monitoring > Logs and metrics** to set up [Stack monitoring](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md) for your deployment, forwarding its logs and metrics to a dedicated monitoring deployment. ::::{note} In addition to the monitoring of clusters that is described here, don’t forget that {{ece}} also provides [monitoring information for your entire installation](../../../deploy-manage/monitor/orchestrators/ece-platform-monitoring.md). diff --git a/deploy-manage/deploy/elastic-cloud/azure-native-isv-service.md b/deploy-manage/deploy/elastic-cloud/azure-native-isv-service.md index beeef156dd..60b38581f6 100644 --- a/deploy-manage/deploy/elastic-cloud/azure-native-isv-service.md +++ b/deploy-manage/deploy/elastic-cloud/azure-native-isv-service.md @@ -326,7 +326,7 @@ $$$azure-integration-modify-deployment$$$How can I modify my {{ecloud}} deployme * [Update {{stack}} user settings](edit-stack-settings.md) in the component YML files. * [Add or remove custom plugins](add-plugins-extensions.md). * [Configure IP filtering](../../security/traffic-filtering.md). - * [Monitor your {{ecloud}} deployment](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) to ensure it remains healthy. + * [Monitor your {{ecloud}} deployment](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md) to ensure it remains healthy. * Add or remove API keys to use the [REST API](cloud://reference/cloud-hosted/ec-api-restful.md). * [And more](cloud-hosted.md) diff --git a/deploy-manage/deploy/elastic-cloud/create-an-elastic-cloud-hosted-deployment.md b/deploy-manage/deploy/elastic-cloud/create-an-elastic-cloud-hosted-deployment.md index 55b81a0540..b64a23782c 100644 --- a/deploy-manage/deploy/elastic-cloud/create-an-elastic-cloud-hosted-deployment.md +++ b/deploy-manage/deploy/elastic-cloud/create-an-elastic-cloud-hosted-deployment.md @@ -71,5 +71,5 @@ To make sure you’re all set for production, consider the following actions: * [Add extensions and plugins](/deploy-manage/deploy/elastic-cloud/add-plugins-extensions.md) to use Elastic supported extensions or add your own custom dictionaries and scripts. * [Edit settings and defaults](/deploy-manage/deploy/elastic-cloud/edit-stack-settings.md) to fine tune the performance of specific features. * [Manage your deployment](/deploy-manage/deploy/elastic-cloud/manage-deployments.md) as a whole to restart, upgrade, stop routing, or delete. -* [Set up monitoring](/deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) to learn how to configure your deployments for observability, which includes metric and log collection, troubleshooting views, and cluster alerts to automate performance monitoring. +* [Set up monitoring](/deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md) to learn how to configure your deployments for observability, which includes metric and log collection, troubleshooting views, and cluster alerts to automate performance monitoring. diff --git a/deploy-manage/deploy/elastic-cloud/ec-customize-deployment-components.md b/deploy-manage/deploy/elastic-cloud/ec-customize-deployment-components.md index 568143e4f3..049adc7ae2 100644 --- a/deploy-manage/deploy/elastic-cloud/ec-customize-deployment-components.md +++ b/deploy-manage/deploy/elastic-cloud/ec-customize-deployment-components.md @@ -19,7 +19,7 @@ Autoscaling reduces some of the manual effort required to manage a deployment by ## {{es}} [ec-cluster-size] -Depending upon how much data you have and what queries you plan to run, you need to select a cluster size that fits your needs. There is no silver bullet for deciding how much memory you need other than simply testing it. The [cluster performance metrics](../../monitor/stack-monitoring.md) in the [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body) can tell you if your cluster is sized appropriately. You can also [enable deployment monitoring](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) for more detailed performance metrics. Fortunately, you can change the amount of memory allocated to the cluster later without any downtime for HA deployments. +Depending upon how much data you have and what queries you plan to run, you need to select a cluster size that fits your needs. There is no silver bullet for deciding how much memory you need other than simply testing it. The [cluster performance metrics](../../monitor/stack-monitoring.md) in the [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body) can tell you if your cluster is sized appropriately. You can also [enable deployment monitoring](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md) for more detailed performance metrics. Fortunately, you can change the amount of memory allocated to the cluster later without any downtime for HA deployments. To change a cluster’s topology, from deployment management, select **Edit deployment** from the **Actions** dropdown. Next, select a storage and RAM setting from the **Size per zone** drop-down list, and save your changes. When downsizing the cluster, make sure to have enough resources to handle the current load, otherwise your cluster will be under stress. diff --git a/deploy-manage/monitor/monitoring-data/ec-vcpu-boost-instance.md b/deploy-manage/deploy/elastic-cloud/ec-vcpu-boost-instance.md similarity index 95% rename from deploy-manage/monitor/monitoring-data/ec-vcpu-boost-instance.md rename to deploy-manage/deploy/elastic-cloud/ec-vcpu-boost-instance.md index f580133440..2ee9ecc31b 100644 --- a/deploy-manage/monitor/monitoring-data/ec-vcpu-boost-instance.md +++ b/deploy-manage/deploy/elastic-cloud/ec-vcpu-boost-instance.md @@ -42,12 +42,12 @@ For more information, check [{{ech}} default provider instance configurations](c You can check the **Monitoring > Performance > CPU Credits** section of the [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body), and find the related metrics: -:::{image} ../../../images/cloud-metrics-credits.png +:::{image} /images/cloud-metrics-credits.png :alt: CPU usage versus CPU credits over time ::: ## What to do if my vCPU credits get depleted constantly? [ec_what_to_do_if_my_vcpu_credits_get_depleted_constantly] -If you need your cluster to be able to sustain a certain level of performance, you cannot rely on CPU boosting to handle the workload except temporarily. To ensure that performance can be sustained, consider increasing the size of your cluster. Read [this page](../../../troubleshoot/monitoring/performance.md) for more guidance. +If you need your cluster to be able to sustain a certain level of performance, you cannot rely on CPU boosting to handle the workload except temporarily. To ensure that performance can be sustained, consider increasing the size of your cluster. Refer to [](/troubleshoot/monitoring/performance.md) for more guidance. diff --git a/deploy-manage/deploy/elastic-cloud/ech-restrictions.md b/deploy-manage/deploy/elastic-cloud/ech-restrictions.md index 66d09f2e5d..f9e47b4422 100644 --- a/deploy-manage/deploy/elastic-cloud/ech-restrictions.md +++ b/deploy-manage/deploy/elastic-cloud/ech-restrictions.md @@ -19,7 +19,7 @@ When using Elasticsearch Add-On for Heroku, there are some limitations you shoul * [Regions and Availability Zones](#ech-regions-and-availability-zone) * [Known problems](#ech-known-problems) -For limitations related to logging and monitoring, check the [Restrictions and limitations](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) section of the logging and monitoring page. +For limitations related to logging and monitoring, check the [Restrictions and limitations](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md) section of the logging and monitoring page. Occasionally, we also publish information about [Known problems](#ech-known-problems) with our Elasticsearch Add-On for Heroku or the Elastic Stack. @@ -53,7 +53,7 @@ Generally, if a feature is shown as available in the [Elasticsearch Add-On for H * Elasticsearch plugins, are not enabled by default for security purposes. Please reach out to support if you would like to enable Elasticsearch plugins support on your account. * Some Elasticsearch plugins do not apply to Elasticsearch Add-On for Heroku. For example, you won’t ever need to change discovery, as Elasticsearch Add-On for Heroku handles how nodes discover one another. * In Elasticsearch 5.0 and later, site plugins are no longer supported. This change does not affect the site plugins Elasticsearch Add-On for Heroku might provide out of the box, such as Kopf or Head, since these site plugins are serviced by our proxies and not Elasticsearch itself. -* In Elasticsearch 5.0 and later, site plugins such as Kopf and Paramedic are no longer provided. We recommend that you use our [cluster performance metrics](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md), [X-Pack monitoring features](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) and Kibana’s (6.3+) [Index Management UI](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-mgmt.html) if you want more detailed information or perform index management actions. +* In Elasticsearch 5.0 and later, site plugins such as Kopf and Paramedic are no longer provided. We recommend that you use our [cluster performance metrics](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md), [X-Pack monitoring features](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md) and Kibana’s (6.3+) [Index Management UI](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-mgmt.html) if you want more detailed information or perform index management actions. ## Private Link and SSO to Kibana URLs [ech-restrictions-traffic-filters-kibana-sso] diff --git a/deploy-manage/deploy/elastic-cloud/manage-deployments.md b/deploy-manage/deploy/elastic-cloud/manage-deployments.md index 7eea735082..8418804900 100644 --- a/deploy-manage/deploy/elastic-cloud/manage-deployments.md +++ b/deploy-manage/deploy/elastic-cloud/manage-deployments.md @@ -17,7 +17,7 @@ mapped_pages: * Ensure the health of your deployment over time - * [Keep track of your deployment's activity](keep-track-of-deployment-activity.md) or [Enable logging and monitoring](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) of the deployment performance. + * [Keep track of your deployment's activity](keep-track-of-deployment-activity.md) or [Enable logging and monitoring](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md) of the deployment performance. * Perform maintenance operations to ensure the health of your deployment, such as [restarting your deployment](../../maintenance/start-stop-services/restart-cloud-hosted-deployment.md) or [stopping routing](../../maintenance/ece/start-stop-routing-requests.md). * Manage the lifecycle of your deployment: diff --git a/deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md b/deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md index 3a98858196..61bb7c45b5 100644 --- a/deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md +++ b/deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md @@ -26,7 +26,7 @@ When using {{ecloud}}, there are some limitations you should be aware of: * [Regions and Availability Zones](#ec-regions-and-availability-zone) % * [Known problems](#ec-known-problems) -For limitations related to logging and monitoring, check the [Restrictions and limitations](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ec-restrictions-monitoring) section of the logging and monitoring page. +For limitations related to logging and monitoring, check the [Restrictions and limitations](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md#restrictions-monitoring) section of the logging and monitoring page. % Occasionally, we also publish information about [Known problems](#ec-known-problems) with our {{ecloud}} or the Elastic Stack. diff --git a/deploy-manage/monitor.md b/deploy-manage/monitor.md index 2c426b91bb..4b82269337 100644 --- a/deploy-manage/monitor.md +++ b/deploy-manage/monitor.md @@ -1,33 +1,91 @@ --- mapped_urls: - https://www.elastic.co/guide/en/elasticsearch/reference/current/monitor-elasticsearch-cluster.html - - https://www.elastic.co/guide/en/elasticsearch/reference/current/secure-monitoring.html + - https://www.elastic.co/guide/en/cloud/current/ec-monitoring.html applies_to: deployment: ess: all ece: all eck: all self: all - serverless: all --- # Monitoring -% What needs to be done: Write from scratch +Keeping on top of the health of your cluster or deployment, as well as your orchestrator, is an important part of maintenance. It also helps you to identify and troubleshoot issues. When you move to [production](/deploy-manage/production-guidance.md), detecting and resolving issues when they arise is a key component of keeping your deployment highly available. -% GitHub issue: https://github.com/elastic/docs-projects/issues/350 +Depending on your deployment type, you can use a variety of solutions for monitoring your Elastic components. -% Scope notes: one link left for redirection purposes. we have to write an introduction to this big section about Monitoring and Logging. Explain what is monitoring about (metrics and logs) and how orchestrators can help by automatically setting up the beats or monitoring agents. +## Monitoring your cluster or deployment -% Use migrated content from existing pages that map to this page: +Depending on your deployment type and context, you have several options for monitoring your cluster or deployment. -% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/monitor-elasticsearch-cluster.md -% Notes: Existing articles about monitoring: Elasticsearch, Cloud, Cloud-enterprise, Cloud on Kubernetes, Kibana books Might need a new landing page -% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/secure-monitoring.md +### AutoOps (recommended) -⚠️ **This page is a work in progress.** ⚠️ +```{applies_to} +deployment: + ech: +``` -The documentation team is working to combine content pulled from the following pages: +:::{include} /deploy-manage/monitor/_snippets/autoops.md +::: + +### Stack monitoring + +```{applies_to} +deployment: + ech: + ece: + eck: + self: +``` + +:::{include} /deploy-manage/monitor/_snippets/stack-monitoring-def.md +::: + +In {{ece}} and {{ech}}, Elastic manages the installation and configuration of the monitoring agent for you, simplifying the stack monitoring setup process. + +:::{include} /deploy-manage/monitor/_snippets/stack-monitoring-prod.md +::: + +### Cluster health and performance metrics + +```{applies_to} +deployment: + ece: + ech: +``` + +{{ece}} and {{ech}} provide out of the box tools for monitoring the health of your deployment and resolving health issues when they arise: + +* [Cluster health information](/deploy-manage/monitor/cloud-health-perf.md#ec-es-cluster-health), including [health warnings](/deploy-manage/monitor/cloud-health-perf.md#ec-es-health-warnings) +* A [JVM memory pressure indicator](/deploy-manage/monitor/ec-memory-pressure.md) + +{{ech}} only: +* [Cluster performance information](/deploy-manage/monitor/access-performance-metrics-on-elastic-cloud.md) +* [Preconfigured logs and metrics](/deploy-manage/monitor/cloud-health-perf.md#ec-es-health-preconfigured) + +{{ece}} only: +* [Platform monitoring](/deploy-manage/monitor/orchestrators/ece-platform-monitoring.md), including logs, metrics, and proxy logs + +:::{tip} +Out of the box logs and metrics tools, including ECH preconfigured logs and metrics and ECE platform monitoring logs and metrics, are useful for providing information in a non-production environment. In a production environment, it’s important set up either AutoOps or stack monitoring to retain the logs and metrics that can be used to troubleshoot any health issues in your deployments. In the event of that you need to [contact our support team](/troubleshoot/index.md#contact-us), they can use the retained data to help diagnose any problems that you may encounter. +::: + +To learn more about the health and performance tools in {{ecloud}}, refer to [](/deploy-manage/monitor/cloud-health-perf.md). + +## Monitoring your orchestrator +```{applies_to} +deployment: + ece: + eck: +``` + +TODO + +## Logging + +TODO + +% * [*Elasticsearch application logging*](../../../deploy-manage/monitor/logging-configuration/update-elasticsearch-logging-levels.md) -* [/raw-migrated-files/elasticsearch/elasticsearch-reference/monitor-elasticsearch-cluster.md](/raw-migrated-files/elasticsearch/elasticsearch-reference/monitor-elasticsearch-cluster.md) -* [/raw-migrated-files/elasticsearch/elasticsearch-reference/secure-monitoring.md](/raw-migrated-files/elasticsearch/elasticsearch-reference/secure-monitoring.md) \ No newline at end of file diff --git a/deploy-manage/monitor/_snippets/autoops.md b/deploy-manage/monitor/_snippets/autoops.md new file mode 100644 index 0000000000..1b0ae39b27 --- /dev/null +++ b/deploy-manage/monitor/_snippets/autoops.md @@ -0,0 +1 @@ +If you’re using {{ech}}, then you can use AutoOps to monitor your cluster. AutoOps significantly simplifies cluster management with performance recommendations, resource utilization visibility, real-time issue detection and resolution paths. AutoOps is [rolling out](/deploy-manage/monitor/autoops/ec-autoops-regions.md) in phases across {{ech}} regions and cloud service providers. It will be automatically activated for your deployment, with no installation required. For more information, refer to [Monitor with AutoOps](/deploy-manage/monitor/autoops.md). \ No newline at end of file diff --git a/deploy-manage/monitor/_snippets/stack-monitoring-def.md b/deploy-manage/monitor/_snippets/stack-monitoring-def.md new file mode 100644 index 0000000000..13b54e3318 --- /dev/null +++ b/deploy-manage/monitor/_snippets/stack-monitoring-def.md @@ -0,0 +1,3 @@ +Stack monitoring allows you to collect logs and metrics from various Elastic products, including {{es}} nodes, {{ls}} nodes, {{kib}} instances, APM Server, and Beats in your cluster. You can also collect logs. + +All of the monitoring metrics are stored in {{es}}, which enables you to easily visualize the data in {{kib}}. \ No newline at end of file diff --git a/deploy-manage/monitor/_snippets/stack-monitoring-prod.md b/deploy-manage/monitor/_snippets/stack-monitoring-prod.md new file mode 100644 index 0000000000..d03be20016 --- /dev/null +++ b/deploy-manage/monitor/_snippets/stack-monitoring-prod.md @@ -0,0 +1,3 @@ +::::{tip} +By default, the monitoring metrics are stored in local indices. In production, we strongly recommend using a [separate monitoring cluster](/deploy-manage/monitor/stack-monitoring.md#production-architecture). Using a separate monitoring cluster prevents production cluster outages from impacting your ability to access your monitoring data. It also prevents monitoring activities from impacting the performance of your production cluster. For the same reason, we also recommend using a separate {{kib}} instance for viewing the monitoring data. +:::: diff --git a/deploy-manage/monitor/monitoring-data/ec-saas-metrics-accessing.md b/deploy-manage/monitor/access-performance-metrics-on-elastic-cloud.md similarity index 52% rename from deploy-manage/monitor/monitoring-data/ec-saas-metrics-accessing.md rename to deploy-manage/monitor/access-performance-metrics-on-elastic-cloud.md index 373f45c536..2534980936 100644 --- a/deploy-manage/monitor/monitoring-data/ec-saas-metrics-accessing.md +++ b/deploy-manage/monitor/access-performance-metrics-on-elastic-cloud.md @@ -1,5 +1,5 @@ --- -mapped_pages: +mapped_urls: - https://www.elastic.co/guide/en/cloud/current/ec-saas-metrics-accessing.html - https://www.elastic.co/guide/en/cloud-heroku/current/ech-saas-metrics-accessing.html applies_to: @@ -7,11 +7,15 @@ applies_to: ess: all --- -# Access performance metrics [ec-saas-metrics-accessing] +# Performance metrics on {{ecloud}} [ec-saas-metrics-accessing] -Cluster performance metrics are available directly in the [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body). The graphs on this page include a subset of {{ech}}-specific performance metrics. +For {{ech}} deployments, cluster performance metrics are available directly in the [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body). The graphs on this page include a subset of {{ech}}-specific performance metrics. -For advanced views or production monitoring, [enable logging and monitoring](../stack-monitoring/elastic-cloud-stack-monitoring.md). The monitoring application provides more advanced views for Elasticsearch and JVM metrics, and includes a configurable retention period. +For advanced views or production monitoring, [enable logging and monitoring](/deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md). The monitoring application provides more advanced views for {{es}} and JVM metrics, and includes a configurable retention period. + +:::{tip} +For {{ece}} deployments, you can use [platform monitoring](/deploy-manage/monitor/orchestrators/ece-platform-monitoring.md) to access preconfigured performance metrics. +::: To access cluster performance metrics: @@ -27,30 +31,28 @@ The following metrics are available: ### CPU usage [ec_cpu_usage] -:::{image} ../../../images/cloud-metrics-cpu-usage.png +:::{image} /images/cloud-metrics-cpu-usage.png :alt: Graph showing CPU usage ::: -Shows the maximum usage of the CPU resources assigned to your Elasticsearch cluster, as a percentage. CPU resources are relative to the size of your cluster, so that a cluster with 32GB of RAM gets assigned twice as many CPU resources as a cluster with 16GB of RAM. All clusters are guaranteed their share of CPU resources, as {{ech}} infrastructure does not overcommit any resources. CPU credits permit boosting the performance of smaller clusters temporarily, so that CPU usage can exceed 100%. +Shows the maximum usage of the CPU resources assigned to your {{es}} cluster, as a percentage. CPU resources are relative to the size of your cluster, so that a cluster with 32GB of RAM gets assigned twice as many CPU resources as a cluster with 16GB of RAM. All clusters are guaranteed their share of CPU resources, as {{ech}} infrastructure does not overcommit any resources. CPU credits permit boosting the performance of smaller clusters temporarily, so that CPU usage can exceed 100%. ::::{tip} -This chart reports the maximum CPU values over the sampling period. [Logs and Metrics](../stack-monitoring/elastic-cloud-stack-monitoring.md) ingested into [Stack Monitoring](visualizing-monitoring-data.md)'s "CPU Usage" instead reflects the average CPU over the sampling period. Therefore, you should not expect the two graphs to look exactly the same. When investigating [CPU-related performance issues](../../../troubleshoot/monitoring/performance.md), you should default to [Stack Monitoring](visualizing-monitoring-data.md). +This chart reports the maximum CPU values over the sampling period. [Logs and metrics](/deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md) ingested into [Stack Monitoring](/deploy-manage/monitor/monitoring-data/visualizing-monitoring-data.md)'s "CPU Usage" instead reflects the average CPU over the sampling period. Therefore, you should not expect the two graphs to look exactly the same. When investigating [CPU-related performance issues](/troubleshoot/monitoring/performance.md), you should default to [Stack Monitoring](/deploy-manage/monitor/monitoring-data/visualizing-monitoring-data.md). :::: - - ### CPU credits [ec_cpu_credits] -:::{image} ../../../images/cloud-metrics-cpu-credits.png +:::{image} /images/cloud-metrics-cpu-credits.png :alt: Graph showing available CPU credits ::: -Shows your remaining CPU credits, measured in seconds of CPU time. CPU credits enable the boosting of CPU resources assigned to your cluster to improve performance temporarily when it is needed most. For more details check [How to use vCPU to boost your instance](ec-vcpu-boost-instance.md). +Shows your remaining CPU credits, measured in seconds of CPU time. CPU credits enable the boosting of CPU resources assigned to your cluster to improve performance temporarily when it is needed most. For more details check [How to use vCPU to boost your instance](/deploy-manage/deploy/elastic-cloud/ec-vcpu-boost-instance.md). ### Number of requests [ec_number_of_requests] -:::{image} ../../../images/cloud-metrics-number-of-requests.png +:::{image} /images/cloud-metrics-number-of-requests.png :alt: Graph showing the number of requests ::: @@ -59,34 +61,34 @@ Shows the number of requests that your cluster receives per second, separated in ### Search response times [ec_search_response_times] -:::{image} ../../../images/cloud-metrics-search-response-times.png +:::{image} /images/cloud-metrics-search-response-times.png :alt: Graph showing search response times ::: -Indicates the amount of time that it takes for your Elasticsearch cluster to complete a search query, in milliseconds. Response times won’t tell you about the cause of a performance issue, but they are often a first indicator that something is amiss with the performance of your Elasticsearch cluster. +Indicates the amount of time that it takes for your {{es}} cluster to complete a search query, in milliseconds. Response times won’t tell you about the cause of a performance issue, but they are often a first indicator that something is amiss with the performance of your {{es}} cluster. ### Index response times [ec_index_response_times] -:::{image} ../../../images/cloud-metrics-index-response-times.png +:::{image} /images/cloud-metrics-index-response-times.png :alt: Graph showing index response times ::: -Indicates the amount of time that it takes for your Elasticsearch cluster to complete an indexing operation, in milliseconds. Response times won’t tell you about the cause of a performance issue, but they are often a first indicator that something is amiss with the performance of your Elasticsearch cluster. +Indicates the amount of time that it takes for your {{es}} cluster to complete an indexing operation, in milliseconds. Response times won’t tell you about the cause of a performance issue, but they are often a first indicator that something is amiss with the performance of your {{es}} cluster. ### Memory pressure per node [ec_memory_pressure_per_node] -:::{image} ../../../images/cloud-metrics-memory-pressure-per-node.png +:::{image} /images/cloud-metrics-memory-pressure-per-node.png :alt: Graph showing memory pressure per node ::: -Indicates the total memory used by the JVM heap over time. We’ve configured {{es}}'s garbage collector to keep memory usage below 75% for heaps of 8GB or larger. For heaps smaller than 8GB, the threshold is 85%. If memory pressure consistently remains above this threshold, you might need to resize your cluster or reduce memory consumption. Check [how high memory pressure can cause performance issues](../../../troubleshoot/monitoring/high-memory-pressure.md). +Indicates the total memory used by the JVM heap over time. We’ve configured {{es}}'s garbage collector to keep memory usage below 75% for heaps of 8GB or larger. For heaps smaller than 8GB, the threshold is 85%. If memory pressure consistently remains above this threshold, you might need to resize your cluster or reduce memory consumption. Check [how high memory pressure can cause performance issues](/troubleshoot/monitoring/high-memory-pressure.md). ### GC overhead per node [ec_gc_overhead_per_node] -:::{image} ../../../images/cloud-metrics-gc-overhead-per-node.png +:::{image} /images/cloud-metrics-gc-overhead-per-node.png :alt: Graph showing the garbage collection overhead per node ::: @@ -97,12 +99,12 @@ Indicates the overhead involved in JVM garbage collection to reclaim memory. Performance correlates directly with resources assigned to your cluster, and many of these metrics will show some sort of correlation with each other when you are trying to determine the cause of a performance issue. Take a look at some of the scenarios included in this section to learn how you can determine the cause of performance issues. -It is not uncommon for performance issues on {{ech}} to be caused by an undersized cluster that cannot cope with the workload it is being asked to handle. If your cluster performance metrics often shows high CPU usage or excessive memory pressure, consider increasing the size of your cluster soon to improve performance. This is especially true for clusters that regularly reach 100% of CPU usage or that suffer out-of-memory failures; it is better to resize your cluster early when it is not yet maxed out than to have to resize a cluster that is already overwhelmed. [Changing the configuration of your cluster](../../deploy/elastic-cloud/configure.md) may add some overhead if data needs to be migrated to the new nodes, which can increase the load on a cluster further and delay configuration changes. +It is not uncommon for performance issues on {{ech}} to be caused by an undersized cluster that cannot cope with the workload it is being asked to handle. If your cluster performance metrics often shows high CPU usage or excessive memory pressure, consider increasing the size of your cluster soon to improve performance. This is especially true for clusters that regularly reach 100% of CPU usage or that suffer out-of-memory failures; it is better to resize your cluster early when it is not yet maxed out than to have to resize a cluster that is already overwhelmed. [Changing the configuration of your cluster](/deploy-manage/deploy/elastic-cloud/configure.md) may add some overhead if data needs to be migrated to the new nodes, which can increase the load on a cluster further and delay configuration changes. -To help diagnose high CPU usage you can also use the Elasticsearch [nodes hot threads API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-nodes-hot-threads), which identifies the threads on each node that have the highest CPU usage or that have been executing for a longer than normal period of time. +To help diagnose high CPU usage you can also use the {{es}} [nodes hot threads API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-nodes-hot-threads), which identifies the threads on each node that have the highest CPU usage or that have been executing for a longer than normal period of time. ::::{tip} -Got an overwhelmed cluster that needs to be upsized? [Try enabling maintenance mode first](../../maintenance/ece/start-stop-routing-requests.md). It will likely help with configuration changes. +Got an overwhelmed cluster that needs to be upsized? [Try enabling maintenance mode first](/deploy-manage/maintenance/ece/start-stop-routing-requests.md). It will likely help with configuration changes. :::: @@ -110,27 +112,17 @@ Work with the metrics shown in **Cluster Performance Metrics** section to help y * Hover on any part of a graph to get additional information. For example, hovering on a section of a graph that shows response times reveals the percentile that responses fall into at that point in time: - :::{image} ../../../images/cloud-metrics-hover.png + :::{image} /images/cloud-metrics-hover.png :alt: Hover over the metric graph ::: * Zoom in on a graph by drawing a rectangle to select a specific time window. As you zoom in one metric, other performance metrics change to show data for the same time window. - :::{image} ../../../images/cloud-metrics-zoom.png + :::{image} /images/cloud-metrics-zoom.png :alt: Zoom the metric graph ::: -* Pan around with ![Pan in a metric graph](../../../images/cloud-metrics-pan.png "") to make sure that you can get the right parts of a metric graph as you zoom in. -* Reset the metric graph axes with ![Reset the metric graph](../../../images/cloud-metrics-reset.png ""), which returns the graphs to their original scale. - -Cluster performance metrics are shown per node and are color-coded to indicate which running Elasticsearch instance they belong to. - - -## Cluster restarts after out-of-memory failures [ec_cluster_restarts_after_out_of_memory_failures] - -For clusters that suffer out-of-memory failures, it can be difficult to determine whether the clusters are in a completely healthy state afterwards. For this reason, {{ech}} automatically reboots clusters that suffer out-of-memory failures. - -You will receive an email notification to let you know that a restart occurred. For repeated alerts, the emails are aggregated so that you do not receive an excessive number of notifications. Either [resizing your cluster to reduce memory pressure](../../deploy/elastic-cloud/ec-customize-deployment-components.md#ec-cluster-size) or reducing the workload that a cluster is being asked to handle can help avoid these cluster restarts. - - +* Pan around with ![Pan in a metric graph](/images/cloud-metrics-pan.png "") to make sure that you can get the right parts of a metric graph as you zoom in. +* Reset the metric graph axes with ![Reset the metric graph](/images/cloud-metrics-reset.png ""), which returns the graphs to their original scale. +Cluster performance metrics are shown per node and are color-coded to indicate which running {{es}} instance they belong to. \ No newline at end of file diff --git a/deploy-manage/monitor/autoops.md b/deploy-manage/monitor/autoops.md index 6d72f6b102..378078c6e9 100644 --- a/deploy-manage/monitor/autoops.md +++ b/deploy-manage/monitor/autoops.md @@ -10,6 +10,8 @@ applies_to: AutoOps diagnoses issues in Elasticsearch by analyzing hundreds of metrics, providing root-cause analysis and accurate resolution paths. With AutoOps, customers can prevent and resolve issues, cut down administration time, and optimize resource utilization. +AutoOps is currently only available for [{{ech}} deployments](/deploy-manage/deploy/elastic-cloud/cloud-hosted.md). + :::{image} ../../images/cloud-autoops-overview-page.png :alt: The Overview page ::: @@ -41,23 +43,22 @@ AutoOps diagnoses issues in Elasticsearch by analyzing hundreds of metrics, prov ## AutoOps retention period [ec_autoops_retention_period] -AutoOps currently has a four-day retention period for all Cloud Hosted customers. +AutoOps currently has a four-day retention period for all {{ech}} customers. ## AutoOps scope [ec_autoops_scope] -AutoOps currently monitors only {{es}}, not the entire {{stack}}. Any deployment information pertains solely to {{es}}. AutoOps supports {{es}} version according to the [supported Elastic Stack versions](https://www.elastic.co/support/eol). There are plans to expand AutoOps monitoring to the entire stack. - - - - - - - - - - +AutoOps currently monitors only {{es}}, not the entire {{stack}}. Any deployment information pertains solely to {{es}}. AutoOps supports {{es}} versions according to the [supported {{es}} versions](https://www.elastic.co/support/eol). There are plans to expand AutoOps monitoring to the entire stack. +## Section overview +In this section, you'll find the following information: +* How to [open AutoOps](/deploy-manage/monitor/autoops/ec-autoops-how-to-access.md) for your deployment. +* The contents of [AutoOps events](/deploy-manage/monitor/autoops/ec-autoops-events.md). +* The [views](/deploy-manage/monitor/autoops/views.md) AutoOps offers to gain insight into facets of your deployment. +* [Notification settings](/deploy-manage/monitor/autoops/ec-autoops-notifications-settings.md) that allow you to specify when and how to be notified. +* [Event settings](/deploy-manage/monitor/autoops/ec-autoops-event-settings.md) that allow you to fine-tune when events are triggered, and a method to [dismiss](/deploy-manage/monitor/autoops/ec-autoops-dismiss-event.md) certain categories of events. +* The [regions](/deploy-manage/monitor/autoops/ec-autoops-regions.md) where AutoOps is available. +* Additional [frequently asked questions](/deploy-manage/monitor/autoops/ec-autoops-faq.md). diff --git a/deploy-manage/monitor/autoops/ec-autoops-deployment-view.md b/deploy-manage/monitor/autoops/ec-autoops-deployment-view.md index 29717bf64f..c5480d5ade 100644 --- a/deploy-manage/monitor/autoops/ec-autoops-deployment-view.md +++ b/deploy-manage/monitor/autoops/ec-autoops-deployment-view.md @@ -10,7 +10,7 @@ applies_to: The **Deployment** view is the event control panel that allows you to see which issues are affecting the {{es}} cluster and get a list of action items to address them. -:::{image} ../../../images/cloud-autoops-deployment-view.png +:::{image} /images/cloud-autoops-deployment-view.png :alt: The Deployment view ::: @@ -35,7 +35,7 @@ The **Events History** panel lists events that happened at some point and that h The **Deployment Info** panel provides a quick overview of the {{es}} cluster resources in the selected deployment, such as {{es}} version, cluster status (indicated by the colors green, yellow, or red) at the top right, number of nodes distributed by role, and resources metrics. -:::{image} ../../../images/cloud-autoops-deployment-info.png +:::{image} /images/cloud-autoops-deployment-info.png :alt: The AutoOps Deployment Info ::: @@ -44,7 +44,7 @@ The **Deployment Info** panel provides a quick overview of the {{es}} cluster re The **Performance** panel shows key performance metrics, aggregated at both the cluster level and the selected tier levels: -:::{image} ../../../images/cloud-autoops-deployment-performance.png +:::{image} /images/cloud-autoops-deployment-performance.png :alt: The AutoOps Deployment Performance ::: diff --git a/deploy-manage/monitor/autoops/ec-autoops-dismiss-event.md b/deploy-manage/monitor/autoops/ec-autoops-dismiss-event.md index 6f279aaee5..5aabfebabd 100644 --- a/deploy-manage/monitor/autoops/ec-autoops-dismiss-event.md +++ b/deploy-manage/monitor/autoops/ec-autoops-dismiss-event.md @@ -10,7 +10,7 @@ applies_to: Some events that are triggered may not require your attention immediately, or at all. Users with the appropriate permissions can choose to dismiss an event, preventing AutoOps from opening it. This action can be reversed using the Dismiss events report. -:::{image} ../../../images/cloud-autoops-dismiss-events.png +:::{image} /images/cloud-autoops-dismiss-events.png :alt: Dismiss events ::: diff --git a/deploy-manage/monitor/autoops/ec-autoops-event-settings.md b/deploy-manage/monitor/autoops/ec-autoops-event-settings.md index 5ebe7d8e98..f3955a2496 100644 --- a/deploy-manage/monitor/autoops/ec-autoops-event-settings.md +++ b/deploy-manage/monitor/autoops/ec-autoops-event-settings.md @@ -6,12 +6,12 @@ applies_to: ess: all --- -# Events Settings [ec-autoops-event-settings] +# Event Settings [ec-autoops-event-settings] AutoOps events are triggered when specific conditions are met and are closed when those conditions are no longer satisfied. An event can be triggered by multiple conditions, and each event comes with a default setting that can be adjusted differently for each connected deployment. ::::{note} -Only a user with Cloud Organization Owner role can set up notifications. +Only **Organization owners** can set up notifications. :::: @@ -23,7 +23,7 @@ The event settings include: * Index patterns to exclude - AutoOps will exclude system indices to prevent unnecessary events from opening. You can add or remove indices from the list. * Data roles tier to exclude from indications - Add threshold based on the type of data tier. -:::{image} ../../../images/cloud-autoops-event-settings.png +:::{image} /images/cloud-autoops-event-settings.png :alt: Event settings ::: @@ -34,7 +34,7 @@ The **Event Settings** report provides a list of all the events for which the se From the **Event Settings** report, you can click **Add** to add new settings, or select the edit icon to modify the existing settings. -:::{image} ../../../images/cloud-autoops-events-settings-report.png +:::{image} /images/cloud-autoops-events-settings-report.png :alt: Event settings report ::: diff --git a/deploy-manage/monitor/autoops/ec-autoops-events.md b/deploy-manage/monitor/autoops/ec-autoops-events.md index 498a7fb23d..bc1fdf0bd1 100644 --- a/deploy-manage/monitor/autoops/ec-autoops-events.md +++ b/deploy-manage/monitor/autoops/ec-autoops-events.md @@ -8,45 +8,22 @@ applies_to: # AutoOps events [ec-autoops-events] -An AutoOps event provides a detailed analysis of a specific issue, including why it was triggered and the steps needed to resolve it. The following sections provide you with comprehensive insights and context around issues, the reasons why the event was created, as well as the affected nodes and indices with high indexing activity. +An AutoOps event provides a detailed analysis of a specific issue, including why it was triggered and the steps needed to resolve it. -:::{image} ../../../images/cloud-autoops-events.png +:::{image} /images/cloud-autoops-events.png :alt: AutoOps events ::: +The following sections provide you with comprehensive insights and context around issues, the reasons why the event was created, as well as the affected nodes and indices with high indexing activity. -## What was detected [ec-autoops-what-was-detected] - -This section describes the reasons for which the event was created, as well as links to drill down into the issue. - - -## Recommendations [ec-autoops-recommendations] - -AutoOps provides a set of recommendations. The sequence of their appearance indicates the suggested order of steps to address the issue. - - -## Event duration [ec-autoops-event-duration] - -The time the event was detected (opened at) and the time AutoOps identified that the issue no longer exists (closed at). The closing of an event does not necessarily indicate that the customer resolved the issue, but rather that AutoOps no longer detects it. - - -## Background and impact [ec-autoops-background-impact] - -Provides background and context as to why an event is important, and the impact it can have on performance and stability. - - -## Event timeline chart [ec-autoops-event-timeline] - -This chart visually represents metrics related to an issue. It appears only for events with dynamic metrics. For example, load issues will have this section, while settings-related issues will not. The event timeline chart displays just the last 15 minutes. - - -## Event severity [ec-autoops-event-severity] - -Events are categorized into three levels of severity - high, medium, and low - based on their potential impact on cluster performance and stability: - -* **High**: Events can immediately cause significant usability, performance and stability problems. -* **Medium**: Events may lead to severe problems if not addressed. -* **Low**: Events have minimal/not urgent impact. +| Section | Description | +| --- | --- | +| What was detected | This section describes the reasons for which the event was created, as well as links to drill down into the issue. | +| Recommendations | AutoOps provides a set of recommendations. The sequence of their appearance indicates the suggested order of steps to address the issue. | +| Event duration | The time the event was detected (opened at) and the time AutoOps identified that the issue no longer exists (closed at). The closing of an event does not necessarily indicate that the customer resolved the issue, but rather that AutoOps no longer detects it. | +| Background and impact | Provides background and context as to why an event is important, and the impact it can have on performance and stability. | +| Event timeline chart | This chart visually represents metrics related to an issue. It appears only for events with dynamic metrics. For example, load issues will have this section, while settings-related issues will not. The event timeline chart displays just the last 15 minutes. | +| Event severity | Events are categorized into three levels of severity - high, medium, and low - based on their potential impact on cluster performance and stability:

- **High**: Events can immediately cause significant usability, performance and stability problems.
- **Medium**: Events may lead to severe problems if not addressed.
- **Low**: Events have minimal/not urgent impact. | ## Event settings [ec-autoops-event-customize] diff --git a/deploy-manage/monitor/autoops/ec-autoops-how-to-access.md b/deploy-manage/monitor/autoops/ec-autoops-how-to-access.md index e3f44f066d..043f9c2119 100644 --- a/deploy-manage/monitor/autoops/ec-autoops-how-to-access.md +++ b/deploy-manage/monitor/autoops/ec-autoops-how-to-access.md @@ -9,7 +9,7 @@ applies_to: # How to access AutoOps [ec-autoops-how-to-access] ::::{note} -AutoOps supports {{es}} version according to the [supported Elastic Stack versions](https://www.elastic.co/support/eol). +AutoOps supports {{es}} versions according to the [supported Elastic Stack versions](https://www.elastic.co/support/eol). :::: @@ -20,6 +20,6 @@ To access AutoOps from your Elastic Cloud console, follow these steps: 3. Click **Manage** on the right side of the selected deployment. 4. On the deployment details page, click **Open AutoOps**. -:::{image} ../../../images/cloud-autoops-how-to-access.png +:::{image} /images/cloud-autoops-how-to-access.png :alt: How to access AutoOps ::: diff --git a/deploy-manage/monitor/autoops/ec-autoops-index-view.md b/deploy-manage/monitor/autoops/ec-autoops-index-view.md index 2a93858b1c..3d5b1b28a2 100644 --- a/deploy-manage/monitor/autoops/ec-autoops-index-view.md +++ b/deploy-manage/monitor/autoops/ec-autoops-index-view.md @@ -10,7 +10,7 @@ applies_to: The **Indices** view provides detailed statistics for each {{es}} index in your deployment. -:::{image} ../../../images/cloud-autoops-index-view.png +:::{image} /images/cloud-autoops-index-view.png :alt: The Indices view ::: diff --git a/deploy-manage/monitor/autoops/ec-autoops-nodes-view.md b/deploy-manage/monitor/autoops/ec-autoops-nodes-view.md index bce59205a1..171199adbe 100644 --- a/deploy-manage/monitor/autoops/ec-autoops-nodes-view.md +++ b/deploy-manage/monitor/autoops/ec-autoops-nodes-view.md @@ -10,7 +10,7 @@ applies_to: The **Nodes** view provides a thorough overview on the essential metrics for all monitored deployment nodes. You can delve into specific nodes to observe metrics over extended periods. This includes data on the indexing rate and latency, search rate and latency, as well as details concerning thread pools, data, circuit breakers, network, disk, and additional elements. -:::{image} ../../../images/cloud-autoops-node-view.png +:::{image} /images/cloud-autoops-node-view.png :alt: The Node view ::: diff --git a/deploy-manage/monitor/autoops/ec-autoops-notifications-settings.md b/deploy-manage/monitor/autoops/ec-autoops-notifications-settings.md index 0878fe464c..6f80cb46cb 100644 --- a/deploy-manage/monitor/autoops/ec-autoops-notifications-settings.md +++ b/deploy-manage/monitor/autoops/ec-autoops-notifications-settings.md @@ -6,7 +6,7 @@ applies_to: ess: all --- -# Notifications settings [ec-autoops-notifications-settings] +# Notifications Settings [ec-autoops-notifications-settings] AutoOps can notify you of new events opened or closed through various methods and operation management tools. With a customizable mechanism, you can specify which events you want to be notified about, how you wish to receive these notifications, and their frequency. @@ -60,18 +60,22 @@ The following connectors are available with AutoOps: * [Microsoft Teams Configuration](#ec-autoops-ms-configuration) * [Webhook](#ec-autoops-webhook) -### Email [email] +:::{dropdown} Email +$$$email$$$ To set up notifications via email, follow these steps: 1. Add a new **Email** connector. 2. Add a list of emails. You can add up to 40 emails for a single email connector, and opt in to get alerts also when events close. -4. To receive notifications, scroll down the **Notification** page and click **Add**. -5. Fill in the filter details. -6. Select the events that you want to send to this connector. +3. To receive notifications, scroll down the **Notification** page and click **Add**. +4. Fill in the filter details. +5. Select the events that you want to send to this connector. +::: -### PagerDuty [ec-autoops-pagerduty] +:::{dropdown} PagerDuty + +$$$ec-autoops-pagerduty$$$ The PagerDuty integration consists of the following parts: @@ -86,9 +90,12 @@ The PagerDuty integration consists of the following parts: 2. To receive Slack notifications, add a notification filter. Scroll down the Notification page and click **Add**. 3. Fill in the filter details. 4. Select the events that should be sent to this output. +::: -### Slack [ec-autoops-slack] +:::{dropdown} Slack + +$$$ec-autoops-slack$$$ To set up a webhook to send AutoOps notifications to a Slack channel, go through the following steps. @@ -103,8 +110,12 @@ To set up a webhook to send AutoOps notifications to a Slack channel, go through 9. Copy the webhook URL to set up the webhook notification connector in AutoOps. 10. Add the webhook URL when creating the connector. +::: + -### VictorOps [ec-autoops-victorops] +:::{dropdown} VictorOps + +$$$ec-autoops-victorops$$$ The VictorOps integration consists of the following parts: @@ -119,9 +130,12 @@ The VictorOps integration consists of the following parts: 2. To receive Slack notifications, add a notification filter. Scroll down the Notification page and click Add. 3. Fill in the filter details. 4. Select the events that should be sent to this output. +::: + +:::{dropdown} Opsgenie -### Opsgenie [ec-autoops-opsgenie] +$$$ec-autoops-opsgenie$$$ The Opsgenie integration consists of the following parts: @@ -141,9 +155,11 @@ The Opsgenie integration consists of the following parts: 4. To receive notifications on Opsgenie, you need to add a notification filter. Scroll down the **Notification** page and click **Add**. 5. Fill in the filter details. 6. Select events that should be sent to this output. +::: +:::{dropdown} Microsoft Teams -### Microsoft Teams Configuration [ec-autoops-ms-configuration] +$$$ec-autoops-ms-configuration$$$ To create an incoming webhook on your Microsoft Teams, follow [these instructions](https://docs.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook). @@ -155,39 +171,42 @@ Save the URL displayed during the creation of the incoming webhook, as you will 2. To receive notifications into Microsoft Teams, you need to add a notification filter. Scroll down the Notification page and click Add. 3. Fill in the filter details. 4. Select events that should be sent to this output. +::: + +::::{dropdown} Webhook -### Webhook [ec-autoops-webhook] +$$$ec-autoops-webhook$$$ A webhook enables an application to provide other applications with real-time information. A webhook is a user-defined HTTP callback (HTTP POST), which is triggered by specific events. **How to add a webhook notification** -1. Go to **Settings** → **Notifications*** → ***Connector settings** and click **Add**. +1. Go to **Settings** > **Notifications** > **Connector settings** and click **Add**. 2. Select Webhook from the drop-dowon list and enter the following details: - * Name: It must be a unique name for this webhook. - * URL: This is the endpoint to which HTTP POST requests will be sent when events occur. - * Method: POST - * Header: Content-Type, application/Json + * **Name**: It must be a unique name for this webhook. + * **URL**: This is the endpoint to which HTTP POST requests will be sent when events occur. + * **Method**: POST + * **Header**: Content-Type, application/Json 3. Review and update the message as it appears in the body section. AutoOps provides a set of optional fields to use in the message. Read your application documentation for the expected message schema. - * RESOURCE_ID – Customer Deployment ID - * RESOURCE_NAME – Customer Deployment name - * TITLE – The title of the event. - * DESCRIPTION – The description of the issue that was found. - * SEVERITY – One of the 3 severity levels (High, Medium and Low). - * STATUS – Indicate if the event is currently open or close. - * MESSAGE – The background and impact of the issue - * START_TIME – The time the event was open. - * END_TIME – The time the event was closed. - * ENDPOINT_TYPE – The type of the endpoint (Slack, PagerDuty, Webhook, Opsgenie, VictorOps and MS Teams). - * AFFECTED_NODES – List of node names. - * AFFECTED_INDICES – List of indices names. - * EVENT_LINK – Direct link to the event in AutoOps. - -4. Click Validate to check your settings and click **Save**. + * `RESOURCE_ID`: Customer Deployment ID + * `RESOURCE_NAME`: Customer Deployment name + * `TITLE`: The title of the event. + * `DESCRIPTION`: The description of the issue that was found. + * `SEVERITY`: One of the 3 severity levels (High, Medium and Low). + * `STATUS`: Indicate if the event is currently open or close. + * `MESSAGE`: The background and impact of the issue + * `START_TIME`: The time the event was open. + * `END_TIME`: The time the event was closed. + * `ENDPOINT_TYPE`: The type of the endpoint (Slack, PagerDuty, Webhook, Opsgenie, VictorOps and MS Teams). + * `AFFECTED_NODES`: List of node names. + * `AFFECTED_INDICES`: List of indices names. + * `EVENT_LINK`: Direct link to the event in AutoOps. + +4. Click **Validate** to check your settings, and then click **Save**. 5. Optionally, you can test the webhook integration by using the [webhook.site](https://webhook.site/#!/view/fe9d630e-2f01-44b7-9e41-ef9520fbe9a7). ::::{note} @@ -198,7 +217,7 @@ When the connector settings have been completed, continue to set up the notifica From the **Notifications** report, you can check all the notifications sent. The report lists all the events that were set up in the notification filters and provide their status. -:::{image} ../../../images/cloud-autoops-notifications-report.png +:::{image} /images/cloud-autoops-notifications-report.png :alt: The Notifications report ::: @@ -213,6 +232,6 @@ The notification can have one of the following statuses: The notification status appears also in the event details window. -:::{image} ../../../images/cloud-autoops-notification-status.png +:::{image} /images/cloud-autoops-notification-status.png :alt: Notification status ::: diff --git a/deploy-manage/monitor/autoops/ec-autoops-overview-view.md b/deploy-manage/monitor/autoops/ec-autoops-overview-view.md index 15c3719b0c..df0f555908 100644 --- a/deploy-manage/monitor/autoops/ec-autoops-overview-view.md +++ b/deploy-manage/monitor/autoops/ec-autoops-overview-view.md @@ -10,7 +10,7 @@ applies_to: The **Overview** page displays the current status of customer deployments in Elastic Cloud Hosted that are linked to the same Elastic organization. -:::{image} ../../../images/cloud-autoops-overview-page.png +:::{image} /images/cloud-autoops-overview-page.png :alt: The Overview page ::: diff --git a/deploy-manage/monitor/autoops/ec-autoops-shards-view.md b/deploy-manage/monitor/autoops/ec-autoops-shards-view.md index d492807a80..b94f38df77 100644 --- a/deploy-manage/monitor/autoops/ec-autoops-shards-view.md +++ b/deploy-manage/monitor/autoops/ec-autoops-shards-view.md @@ -10,7 +10,7 @@ applies_to: The **Shards** view allows you to manage and monitor shards allocated to each node, ensuring optimal performance and reliability of your search and indexing operations. -:::{image} ../../../images/cloud-autoops-shard-view.png +:::{image} /images/cloud-autoops-shard-view.png :alt: The Shards view ::: diff --git a/deploy-manage/monitor/autoops/ec-autoops-template-optimizer.md b/deploy-manage/monitor/autoops/ec-autoops-template-optimizer.md index 6810c97926..79de068ce4 100644 --- a/deploy-manage/monitor/autoops/ec-autoops-template-optimizer.md +++ b/deploy-manage/monitor/autoops/ec-autoops-template-optimizer.md @@ -10,7 +10,7 @@ applies_to: The **Template Optimizer View** offers detailed recommendations for optimizing templates. -:::{image} ../../../images/cloud-autoops-template-optimizer.png +:::{image} /images/cloud-autoops-template-optimizer.png :alt: The Template Optimizer view ::: diff --git a/deploy-manage/monitor/autoops/views.md b/deploy-manage/monitor/autoops/views.md new file mode 100644 index 0000000000..5de5be0656 --- /dev/null +++ b/deploy-manage/monitor/autoops/views.md @@ -0,0 +1,17 @@ +--- +applies_to: + deployment: + ess: all +navigation_title: Views +--- + +# AutoOps views + +AutoOps offers the following views to gain further insight into difference facets of your deployment: + +* [](/deploy-manage/monitor/autoops/ec-autoops-overview-view.md) +* [](/deploy-manage/monitor/autoops/ec-autoops-deployment-view.md) +* [](/deploy-manage/monitor/autoops/ec-autoops-nodes-view.md) +* [](/deploy-manage/monitor/autoops/ec-autoops-index-view.md) +* [](/deploy-manage/monitor/autoops/ec-autoops-shards-view.md) +* [](/deploy-manage/monitor/autoops/ec-autoops-template-optimizer.md) \ No newline at end of file diff --git a/deploy-manage/monitor/cloud-health-perf.md b/deploy-manage/monitor/cloud-health-perf.md new file mode 100644 index 0000000000..8e78b1bbe2 --- /dev/null +++ b/deploy-manage/monitor/cloud-health-perf.md @@ -0,0 +1,153 @@ +--- +mapped_urls: + - https://www.elastic.co/guide/en/elasticsearch/reference/current/monitor-elasticsearch-cluster.html +navigation_title: "Cloud deployment health" +applies_to: + deployment: + ess: all + ece: all +--- + +# Cloud deployment health and performance metrics + +{{ece}} and {{ech}} provide out of the box tools for monitoring the health of your deployment and resolving health issues when they arise. + +These features augment [AutoOps](/deploy-manage/monitor/autoops.md) and [stack monitoring](/deploy-manage/monitor/stack-monitoring.md) features. These more complete monitoring features should be used if you plan to use your cluster in production. + +## Cluster health [ec-es-cluster-health] + +Health is reported based on the following areas: Shard availability, master node health, Snapshot Lifecycle Management (SLM), Index Lifecycle Management (ILM), and repository integrity. + +The deployment **Health** page provides detailed information on health issues, impacted areas, and troubleshooting support. + +To view the health for a deployment: + +1. Log in to the [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body) or [Cloud UI](/deploy-manage/deploy/cloud-enterprise/log-into-cloud-ui.md). +2. On the **Deployments** page, select your deployment. +3. In your deployment menu, select **Health**. + +The **Health** page provides the following information: + +* Health issues for {{kib}}, Enterprise Search, APM, and plan changes are reported in the health banner. +* Health issues for {{es}} clusters are broken down into a table with more details on Severity, Description and Affected capabilities. + +:::{image} /images/cloud-es-health-page.png +:alt: {{es}} Health page +::: + +* **Severity**: A critical issue impacts operations such as search and ingest and should be addressed as soon as possible. Warnings don’t impact the cluster immediately but might lead to more critical issues over time such as a corrupted repository might lead to no backups being available in the future. | +* **Description**: For most issues, you can click the description to get more details page on the specific issue and on how to fix it. +* **Affected capabilities**: Each of these areas might impact search, ingest, backups, or deployment management capabilities. + +You can also search and filter the table based on affected resources, such as indices, repositories, nodes, or SLM policies. Individual issues can be further expanded to get more details and guided troubleshooting. + +:::{image} /images/cloud-es-health-page-table.png +:alt: {{es}} Health page with details and troubleshooting +::: + +For each issue you can either use a troubleshooting link or get a suggestion to contact support, in case you need help. The [troubleshooting documentation](/troubleshoot/elasticsearch/elasticsearch.md) for {{es}} provides more details on specific errors. + +### Health warnings [ec-es-health-warnings] + +In the normal course of using your {{ech}} and {{ece}} deployments, health warnings and errors might appear from time to time. Following are the most common scenarios and methods to resolve them. + +| Category | Resolution | +| --- | --- | +| Health warning messages | Health warning messages will sometimes appear on the main page for one of your deployments, as well as on the **Logs and metrics** page.

For {{ech}} deployments, these messages might reflect routine maintenance activity occurring on {{ecloud}}.| +| Configuration change failures | In more serious cases, your deployment may be unable to restart. The failure can be due to a variety of causes, the most frequent of these being invalid secure settings, expired plugins or bundles, or out of memory errors. The problem is often not detected until an attempt is made to restart the deployment following a routine configuration change, such as a deployment resizing. | +| Out of memory errors | Out of memory errors (OOMs) may occur during your deployment’s normal operations, and these can have a very negative impact on performance. Common causes of memory shortages are oversharding, data retention oversights, and the overall request volume.

On your deployment page, you can check the [JVM memory pressure indicator](/deploy-manage/monitor/ec-memory-pressure.md) to get the current memory usage of each node of your deployment. You can also review the [common causes of high JVM memory usage](/deploy-manage/monitor/ec-memory-pressure.md#ec-memory-pressure-causes) to help diagnose the source of unexpectedly high memory pressure levels. To learn more, refer to [](/troubleshoot/monitoring/high-memory-pressure.md). | + +#### Cluster restarts after out-of-memory failures [ec_cluster_restarts_after_out_of_memory_failures] +```{applies_to} +deployment: + ess: +``` + +For clusters that suffer out-of-memory failures, it can be difficult to determine whether the clusters are in a completely healthy state afterwards. For this reason, {{ech}} automatically reboots clusters that suffer out-of-memory failures. + +You will receive an email notification to let you know that a restart occurred. For repeated alerts, the emails are aggregated so that you do not receive an excessive number of notifications. Either [resizing your cluster to reduce memory pressure](/deploy-manage/deploy/elastic-cloud/ec-customize-deployment-components.md#ec-cluster-size) or reducing the workload that a cluster is being asked to handle can help avoid these cluster restarts. + +## Cluster performance [ec-es-cluster-performance] + +```{applies_to} +deployment: + ess: +``` + +{{ess}} deployments offer an additional **Performance** page to get further information about your cluster performance. + +If you observe issues on search and ingest operations in terms of increased latency or throughput for queries, these might not be directly reported on the **Health** page, unless they are related to shard health or master node availability. + +The **Performance** page and the out-of-the-box logs allow you to monitor your cluster performance, but for production applications we strongly recommend setting up a dedicated monitoring cluster. Refer to [Understanding deployment health](#ec-health-best-practices), for more guidelines on how to monitor you cluster performance. + +[Learn more about the performance page and the metrics it captures](/deploy-manage/monitor/access-performance-metrics-on-elastic-cloud.md). + +:::{tip} +For {{ece}} deployments, you can use [platform monitoring](/deploy-manage/monitor/orchestrators/ece-platform-monitoring.md) to access preconfigured performance metrics. +::: + +## JVM memory pressure indicator + +{{ech}} and {{ece}} also provide a [JVM memory pressure indicator](/deploy-manage/monitor/ec-memory-pressure.md) for each node in your cluster in your deployment overview. This indicator can help you to determine when you need to upgrade to a larger cluster. + +## Preconfigured logs and metrics [ec-es-health-preconfigured] + +Both {{ech}} and {{ece}} offer out-of-the-box logs and metrics that you can access to get insight into your deployment's performance. + +:::{admonition} Monitoring in production environments +In a production environment, it’s important set up dedicated health monitoring using [stack monitoring](/deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md). Stack monitoring allows you to retain the logs and metrics that can be used to troubleshoot any health issues in your deployments. In the event of that you need to [contact our support team](/troubleshoot/index.md#contact-us), they can use the retained data to help diagnose any problems that you may encounter. + +You have the option of sending logs and metrics to a separate, specialized monitoring deployment, which ensures that they’re available in the event of a deployment outage. The monitoring deployment also gives you access to {{kib}} stack monitoring features, through which you can view health and performance data for all of your deployment resources. +::: + +### In {{ech}} + +In a non-production {{ech}} environment, you may choose to rely on the logs and metrics that are available for your deployment by default. The deployment **Logs and metrics** page displays any current deployment health warnings, and from there you can also view standard log files from the last 24 hours. + +The logs capture any activity related to your deployments, their component resources, snapshotting behavior, and more. You can use the search bar to filter the logs by, for example, a specific instance (`instance-0000000005`), a configuration file (`roles.yml`), an operation type (`snapshot`, `autoscaling`), or a component (`kibana`, `ent-search`). + +### In {{ece}} + +In {{ece}}, you can use [platform monitoring](/deploy-manage/monitor/orchestrators/ece-platform-monitoring.md) to view deployment logs and metrics, as well as proxy logs. From your deployment page, select either the **Elasticsearch**, **Kibana**, and **Integrations Server** components, and use the external links to access each service's logs and metrics. + +## vCPU credit monitoring + +```{applies_to} +deployment: + ess: +``` + +Elastic Cloud allows smaller instance sizes to get temporarily boosted vCPU when under heavy load. [vCPU boosting](/deploy-manage/deploy/elastic-cloud/ec-vcpu-boost-instance.md) is governed by vCPU credits that instances can earn over time when vCPU usage is less than the assigned amount. + +You can check the **Monitoring > Performance > CPU Credits** section of the [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body), and find the related metrics: + +:::{image} /images/cloud-metrics-credits.png +:alt: CPU usage versus CPU credits over time +::: + +If you need your cluster to be able to sustain a certain level of performance, you can't rely on CPU boosting to handle the workload except temporarily. To ensure that performance can be sustained, consider increasing the size of your cluster. Refer to [](/troubleshoot/monitoring/performance.md) for more guidance. + +## Understanding deployment health [ec-health-best-practices] + +We’ve compiled some guidelines to help you ensure the health of your deployments over time. These can help you to better understand the available performance metrics, and to make decisions involving performance and high availability. + +[Why is my node(s) unavailable?](/troubleshoot/monitoring/unavailable-nodes.md) +: Learn about common symptoms and possible actions that you can take to resolve issues when one or more nodes become unhealthy or unavailable. + +[Why are my shards unavailable?](/troubleshoot/monitoring/unavailable-shards.md) +: Provide instructions on how to troubleshoot issues related to unassigned shards. + +[Why is performance degrading over time?](/troubleshoot/monitoring/performance.md) +: Address performance degradation on a smaller size Elasticsearch cluster. + +[Is my cluster really highly available?](/troubleshoot/monitoring/high-availability.md) +: High availability involves more than setting multiple availability zones (although that’s really important!). Learn how to assess performance and workloads to determine if your deployment has adequate resources to mitigate a potential node failure. + +[How does high memory pressure affect performance?](/troubleshoot/monitoring/high-memory-pressure.md) +: Learn about typical memory usage patterns, how to assess when the deployment memory usage levels are problematic, how this impacts performance, and how to resolve memory-related issues. + +[Why are my cluster response times suddenly so much worse?](/troubleshoot/monitoring/cluster-response-time.md) +: Learn about the common causes of increased query response times and decreased performance in your deployment. + +[Why did my node move to a different host?](/troubleshoot/monitoring/node-moves-outages.md) +: Learn about why we may, from time to time, relocate your {{ech}} deployments across hosts. diff --git a/deploy-manage/monitor/monitoring-data/ec-memory-pressure.md b/deploy-manage/monitor/ec-memory-pressure.md similarity index 83% rename from deploy-manage/monitor/monitoring-data/ec-memory-pressure.md rename to deploy-manage/monitor/ec-memory-pressure.md index 5cd900b4f3..1bfd8d9ed4 100644 --- a/deploy-manage/monitor/monitoring-data/ec-memory-pressure.md +++ b/deploy-manage/monitor/ec-memory-pressure.md @@ -10,11 +10,11 @@ applies_to: # JVM memory pressure indicator [ec-memory-pressure] -In addition to the more detailed [cluster performance metrics](../stack-monitoring.md), the [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body) also includes a JVM memory pressure indicator for each node in your cluster. This indicator can help you to determine when you need to upgrade to a larger cluster. +In addition to the more detailed [cluster performance metrics](/deploy-manage/monitor/stack-monitoring.md), {{ech}} and {{ece}} also include a JVM memory pressure indicator for each node in your cluster in your deployment overview. This indicator can help you to determine when you need to upgrade to a larger cluster. -The percentage number used in the JVM memory pressure indicator is actually the fill rate of the old generation pool. For a detailed explanation of why this metric is used, check [Understanding Memory Pressure](https://www.elastic.co/blog/found-understanding-memory-pressure-indicator/). +The percentage number used in the JVM memory pressure indicator is actually the fill rate of the old generation pool. For a detailed explanation of why this metric is used, refer to [Understanding memory pressure](https://www.elastic.co/blog/found-understanding-memory-pressure-indicator/). -:::{image} ../../../images/cloud-memory-pressure-indicator.png +:::{image} /images/cloud-memory-pressure-indicator.png :alt: Memory pressure indicator ::: @@ -25,7 +25,6 @@ When the JVM memory pressure reaches 75%, the indicator turns red. At this level When the JVM memory pressure indicator rises above 95%, {{es}}'s [real memory circuit breaker](elasticsearch://reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#parent-circuit-breaker) triggers to prevent your instance from running out of memory. This situation can reduce the stability of your cluster and the integrity of your data. Unless you expect the load to drop soon, we recommend that you resize to a larger cluster before you reach this level of memory pressure. Even if you’re planning to optimize your memory usage, it is best to resize the cluster first. Resizing the cluster to increase capacity can give you more time to apply other changes, and also provides the cluster with more resource for when those changes are applied. - ## Common causes of high JVM memory usage [ec-memory-pressure-causes] The two most common reasons for a high JVM memory pressure reading are: diff --git a/deploy-manage/monitor/logging-configuration.md b/deploy-manage/monitor/logging-configuration.md index 9002f7e1b8..19ab67334b 100644 --- a/deploy-manage/monitor/logging-configuration.md +++ b/deploy-manage/monitor/logging-configuration.md @@ -12,4 +12,43 @@ applies_to: % GitHub issue: https://github.com/elastic/docs-projects/issues/350 -⚠️ **This page is a work in progress.** ⚠️ \ No newline at end of file +⚠️ **This page is a work in progress.** ⚠️ + + +## Logging features [ECE/ECH] [extra-logging-features] + +When shipping logs to a monitoring deployment there are more logging features available to you. These features include: + + +### For {{es}} [extra-logging-features-elasticsearch] + +* [Audit logging](/deploy-manage/security/logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment +* [Slow query and index logging](elasticsearch://reference/elasticsearch/index-settings/slow-log.md) - helps find and debug slow queries and indexing +* Verbose logging - helps debug stack issues by increasing component logs + +After you’ve enabled log delivery on your deployment, you can [add the Elasticsearch user settings](/deploy-manage/deploy/cloud-enterprise/edit-stack-settings.md) to enable these features. + + +### For {{kib}} [extra-logging-features-kibana] + +* [Audit logging](/deploy-manage/security/logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment + +After you’ve enabled log delivery on your deployment, you can [add the {{kib}} user settings](/deploy-manage/deploy/cloud-enterprise/edit-stack-settings.md) to enable this feature. + + +### Other components [extra-logging-features-enterprise-search] + +Enabling log collection also supports collecting and indexing the following types of logs from other components in your deployments: + +**APM** + +* `apm*.log*` + +**Fleet and Elastic Agent** + +* `fleet-server-json.log-*` +* `elastic-agent-json.log-*` + +The `*` indicates that we also index the archived files of each type of log. + +Check the respective product documentation for more information about the logging capabilities of each product. \ No newline at end of file diff --git a/deploy-manage/monitor/logging-configuration/update-elasticsearch-logging-levels.md b/deploy-manage/monitor/logging-configuration/update-elasticsearch-logging-levels.md index 63bd147452..92b3909817 100644 --- a/deploy-manage/monitor/logging-configuration/update-elasticsearch-logging-levels.md +++ b/deploy-manage/monitor/logging-configuration/update-elasticsearch-logging-levels.md @@ -19,22 +19,12 @@ You can use {{es}}'s application logs to monitor your cluster and diagnose issue On [Docker](../../deploy/self-managed/install-elasticsearch-with-docker.md), log messages go to the console and are handled by the configured Docker logging driver. To access logs, run `docker logs`. :::::: -::::::{tab-item} Debian (APT) -For [Debian installations](../../deploy/self-managed/install-elasticsearch-with-debian-package.md), {{es}} writes logs to `/var/log/elasticsearch`. +::::::{tab-item} Debian (APT) and RPM +For [Debian](../../deploy/self-managed/install-elasticsearch-with-debian-package.md) and [RPM](../../deploy/self-managed/install-elasticsearch-with-rpm.md) installations, {{es}} writes logs to `/var/log/elasticsearch`. :::::: -::::::{tab-item} RPM -For [RPM installations](../../deploy/self-managed/install-elasticsearch-with-rpm.md), {{es}} writes logs to `/var/log/elasticsearch`. -:::::: - -::::::{tab-item} macOS -For [macOS `.tar.gz`](../../deploy/self-managed/install-elasticsearch-from-archive-on-linux-macos.md) installations, {{es}} writes logs to `$ES_HOME/logs`. - -Files in `$ES_HOME` risk deletion during an upgrade. In production, we strongly recommend you set `path.logs` to a location outside of `$ES_HOME`. See [Path settings](../../deploy/self-managed/important-settings-configuration.md#path-settings). -:::::: - -::::::{tab-item} Linux -For [Linux `.tar.gz`](../../deploy/self-managed/install-elasticsearch-from-archive-on-linux-macos.md) installations, {{es}} writes logs to `$ES_HOME/logs`. +::::::{tab-item} macOS and Linux +For [macOS and Linux `.tar.gz`](../../deploy/self-managed/install-elasticsearch-from-archive-on-linux-macos.md) installations, {{es}} writes logs to `$ES_HOME/logs`. Files in `$ES_HOME` risk deletion during an upgrade. In production, we strongly recommend you set `path.logs` to a location outside of `$ES_HOME`. See [Path settings](../../deploy/self-managed/important-settings-configuration.md#path-settings). :::::: diff --git a/deploy-manage/monitor/monitoring-data.md b/deploy-manage/monitor/monitoring-data.md deleted file mode 100644 index 36b0da280b..0000000000 --- a/deploy-manage/monitor/monitoring-data.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -applies_to: - deployment: - ess: all - ece: all - eck: all - self: all ---- -# Managing monitoring data - -% Probably ALL THIS NEEDS TO BE UNDER STACK MONITORING - -% What needs to be done: Write from scratch - -% GitHub issue: https://github.com/elastic/docs-projects/issues/350 - -% Scope notes: we can review the name of this section... - - -⚠️ **This page is a work in progress.** ⚠️ \ No newline at end of file diff --git a/deploy-manage/monitor/monitoring-data/access-performance-metrics-on-elastic-cloud.md b/deploy-manage/monitor/monitoring-data/access-performance-metrics-on-elastic-cloud.md deleted file mode 100644 index d01d3ce25a..0000000000 --- a/deploy-manage/monitor/monitoring-data/access-performance-metrics-on-elastic-cloud.md +++ /dev/null @@ -1,26 +0,0 @@ ---- -mapped_urls: - - https://www.elastic.co/guide/en/cloud/current/ec-saas-metrics-accessing.html - - https://www.elastic.co/guide/en/cloud-heroku/current/ech-saas-metrics-accessing.html -applies_to: - deployment: - ess: all ---- - -# Access performance metrics on Elastic Cloud - -% What needs to be done: Lift-and-shift - -% Scope notes: only for elastic cloud SaaS and Heroku. We can get rid of the Heroku doc - -% Use migrated content from existing pages that map to this page: - -% - [ ] ./raw-migrated-files/cloud/cloud/ec-saas-metrics-accessing.md -% - [ ] ./raw-migrated-files/cloud/cloud-heroku/ech-saas-metrics-accessing.md - -⚠️ **This page is a work in progress.** ⚠️ - -The documentation team is working to combine content pulled from the following pages: - -* [/raw-migrated-files/cloud/cloud/ec-saas-metrics-accessing.md](/raw-migrated-files/cloud/cloud/ec-saas-metrics-accessing.md) -* [/raw-migrated-files/cloud/cloud-heroku/ech-saas-metrics-accessing.md](/raw-migrated-files/cloud/cloud-heroku/ech-saas-metrics-accessing.md) \ No newline at end of file diff --git a/deploy-manage/monitor/monitoring-data/beats-page.md b/deploy-manage/monitor/monitoring-data/beats-page.md index 4ef25cc27b..74919921b4 100644 --- a/deploy-manage/monitor/monitoring-data/beats-page.md +++ b/deploy-manage/monitor/monitoring-data/beats-page.md @@ -1,5 +1,4 @@ --- -navigation_title: "Beats Metrics" mapped_pages: - https://www.elastic.co/guide/en/kibana/current/beats-page.html applies_to: @@ -12,12 +11,12 @@ applies_to: -# Beats Metrics [beats-page] +# Beats metrics [beats-page] If you are monitoring Beats, the **Stack Monitoring** page in {{kib}} contains a panel for Beats in the cluster overview. -:::{image} ../../../images/kibana-monitoring-beats.png +:::{image} /images/kibana-monitoring-beats.png :alt: Monitoring Beats :screenshot: ::: @@ -26,7 +25,7 @@ To view an overview of the Beats data in the cluster, click **Overview**. The ov To view a listing of the individual Beat instances in the cluster, click **Beats**. The table listing shows each Beat instance that reports data to the monitoring cluster. All columns are sortable. Clicking a Beat name takes you to the detail page. For example: -:::{image} ../../../images/kibana-monitoring-beats-detail.png +:::{image} /images/kibana-monitoring-beats-detail.png :alt: Monitoring details for Filebeat :screenshot: ::: diff --git a/deploy-manage/monitor/monitoring-data/config-monitoring-data-streams-elastic-agent.md b/deploy-manage/monitor/monitoring-data/config-monitoring-data-streams-elastic-agent.md index 07d6813db7..60748dcd65 100644 --- a/deploy-manage/monitor/monitoring-data/config-monitoring-data-streams-elastic-agent.md +++ b/deploy-manage/monitor/monitoring-data/config-monitoring-data-streams-elastic-agent.md @@ -3,17 +3,33 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/config-monitoring-data-streams-elastic-agent.html applies_to: deployment: - ess: all - ece: all - eck: all self: all --- # Configuring data streams created by Elastic Agent [config-monitoring-data-streams-elastic-agent] -When [monitoring using {{agent}}](../stack-monitoring/collecting-monitoring-data-with-elastic-agent.md), data is stored in a set of data streams named `metrics-{{product}}.stack_monitoring.{{dataset}}-{namespace}`. For example: `metrics-elasticsearch.stack_monitoring.shard-default`. +When [monitoring using {{agent}}](../stack-monitoring/collecting-monitoring-data-with-elastic-agent.md), data is stored in a set of data streams with the following pattern: -The settings and mappings for these data streams are determined by an index template named `metrics-{{product}}.stack_monitoring.{{dataset}}`. For example: `metrics-elasticsearch.stack_monitoring.shard`. +``` +metrics-{{product}}.stack_monitoring.{{dataset}}-{namespace} +``` + +For example: + +``` +metrics-elasticsearch.stack_monitoring.shard-default +``` + +The settings and mappings for these data streams are determined by an index template with the following pattern: + +``` +metrics-{{product}}.stack_monitoring.{{dataset}} +``` +For example: + +``` +metrics-elasticsearch.stack_monitoring.shard +``` To change the settings of each data stream, edit the `metrics-{{product}}.stack_monitoring.{{dataset}}@custom` component template that already exists. You can do this in {{kib}}: @@ -28,4 +44,3 @@ You can also use the {{es}} API: * Store the updated component template using the [update component template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template). After changing the component template, the updated settings are only applied to the data stream’s new backing indices. [Roll over the data stream](../../../manage-data/data-store/data-streams/use-data-stream.md#manually-roll-over-a-data-stream) to immediately apply the updated settings to the data stream’s write index. - diff --git a/deploy-manage/monitor/monitoring-data/config-monitoring-data-streams-metricbeat-8.md b/deploy-manage/monitor/monitoring-data/config-monitoring-data-streams-metricbeat-8.md index 84672b5660..2cbe1dcc09 100644 --- a/deploy-manage/monitor/monitoring-data/config-monitoring-data-streams-metricbeat-8.md +++ b/deploy-manage/monitor/monitoring-data/config-monitoring-data-streams-metricbeat-8.md @@ -11,15 +11,35 @@ applies_to: # Configuring data streams created by Metricbeat 8 [config-monitoring-data-streams-metricbeat-8] -When [monitoring using {{metricbeat}} 8](../stack-monitoring/collecting-monitoring-data-with-metricbeat.md), data is stored in a set of data streams called `.monitoring-{{product}}-8-mb`. For example: `.monitoring-es-8-mb`. +When monitoring using {{metricbeat}}, data is stored in a set of data streams with the following pattern: -The settings and mappings for these data streams are determined by an index template named `.monitoring-{{product}}-mb`. For example: `.monitoring-es-mb`. You can alter the settings of each data stream by cloning this index template and editing it. +``` +.monitoring-{{product}}-8-mb +``` +For example: + +``` +.monitoring-es-8-mb +``` + +The settings and mappings for these data streams are determined by an index template with the following pattern: + +``` +.monitoring-{{product}}-mb +``` + +For example: + +``` +.monitoring-es-mb +``` + +You can alter the settings of each data stream by cloning this index template and editing it. ::::{warning} You need to repeat this procedure when upgrading the {{stack}} to get the latest updates to the default monitoring index templates. :::: - You can clone index templates in {{kib}}: * Navigate to **Stack Management** > **Index Management** > **Index Templates**. @@ -41,6 +61,4 @@ You can also use the {{es}} API: {{metricbeat}} 8 uses [composable templates](../../../manage-data/data-store/templates.md), rather than legacy templates. :::: - -After changing the index template, the updated settings are only applied to the data stream’s new backing indices. [Roll over the data stream](../../../manage-data/data-store/data-streams/use-data-stream.md#manually-roll-over-a-data-stream) to immediately apply the updated settings to the data stream’s write index. - +After changing the index template, the updated settings are only applied to the data stream’s new backing indices. [Roll over the data stream](../../../manage-data/data-store/data-streams/use-data-stream.md#manually-roll-over-a-data-stream) to immediately apply the updated settings to the data stream’s write index. \ No newline at end of file diff --git a/deploy-manage/monitor/monitoring-data/config-monitoring-indices-metricbeat-7-internal-collection.md b/deploy-manage/monitor/monitoring-data/config-monitoring-indices-metricbeat-7-internal-collection.md index 1aa88232f2..ea443fff9f 100644 --- a/deploy-manage/monitor/monitoring-data/config-monitoring-indices-metricbeat-7-internal-collection.md +++ b/deploy-manage/monitor/monitoring-data/config-monitoring-indices-metricbeat-7-internal-collection.md @@ -3,9 +3,6 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/config-monitoring-indices-metricbeat-7-internal-collection.html applies_to: deployment: - ess: all - ece: all - eck: all self: all --- diff --git a/deploy-manage/monitor/monitoring-data/configure-stack-monitoring-alerts.md b/deploy-manage/monitor/monitoring-data/configure-stack-monitoring-alerts.md index 992eb61f1e..141a461d83 100644 --- a/deploy-manage/monitor/monitoring-data/configure-stack-monitoring-alerts.md +++ b/deploy-manage/monitor/monitoring-data/configure-stack-monitoring-alerts.md @@ -1,45 +1,147 @@ --- mapped_pages: - https://www.elastic.co/guide/en/cloud/current/ec-cluster-health-notifications.html + - https://www.elastic.co/guide/en/kibana/current/kibana-alerts.html applies_to: deployment: ess: all + ece: all + eck: all + self: all --- -% NEEDS MERGING WITH kibana-alerts.md -% this one is written for Elastic Cloud but needs to be generic, except if it's really about Elastic cloud. +# Stack monitoring alerts [kibana-alerts] -# Configure Stack monitoring alerts [ec-cluster-health-notifications] +The {{stack}} {{monitor-features}} provide [Alerting rules](../../../explore-analyze/alerts-cases/alerts.md) out-of-the box to notify you of potential issues in the {{stack}}. These rules are preconfigured based on the best practices recommended by Elastic. However, you can tailor them to meet your specific needs. -You can configure Stack monitoring alerts to be sent to you by email when health related events occur in your deployments. To set up email notifications: +:::{image} /images/kibana-monitoring-kibana-alerting-notification.png +:alt: {{kib}} alerting notifications in {{stack-monitor-app}} +:screenshot: +::: -1. [Enable logging and monitoring](../stack-monitoring/elastic-cloud-stack-monitoring.md) on deployments for which you want to receive notifications. You need to enable only metrics data being shipped for the notifications to work. -2. In Kibana, configure the email connector to [send email from Elastic Cloud](kibana://reference/connectors-kibana/email-action-type.md#elasticcloud). If you want to use the preconfigured `Elastic-Cloud-SMTP` connector in Elastic Cloud, then you can skip this step. -3. From the Kibana main menu, go to **Stack Monitoring**. On this page you can find a summary of monitoring metrics for your deployment as well as any alerts. -4. Select **Enter setup mode**. -5. On any card showing available alerts, select the **alerts** indicator. Use the menu to select the type of alert for which you’d like to be notified. There are many alert types, including: +::::{note} +The default {{watcher}} based "cluster alerts" for {{stack-monitor-app}} have been recreated as rules in {{kib}} {{alert-features}}. For this reason, the existing {{watcher}} email action `monitoring.cluster_alerts.email_notifications.email_address` no longer works. The default action for all {{stack-monitor-app}} rules is to write to {{kib}} logs and display a notification in the UI. +:::: - * CPU usage threshold - * Disk usage threshold - * JVM memory threshold - * Missing monitoring data - * Thread pool rejections (search/write) - * CCR read exceptions - * Large shard size - * Cluster alerting, including: +## Create default rules [_create_default_rules] - * Elasticsearch cluster health status - * Elasticsearch, Kibana, or Logstash version mismatches - * Elasticsearch nodes changed +When you open **{{stack-monitor-app}}** for the first time, you will be asked to allow {{kib}} to create the default set of rules. They are initially configured to detect and notify on various conditions across your monitored clusters. You can view notifications for **Cluster health**, **Resource utilization**, and **Errors and exceptions** for {{es}} in real time. - All of these alerts are described in detail in [Kibana alerts](kibana-alerts.md). Check the documentation matching your Kibana version to find which alerts are available in that release. +If you denied creation of the default rules initially, or to recreate any deleted rules, then you can trigger {{kib}} to create the rules by going to **Alerts and rules** > **Create default rules**. +To receive external notifications for these alerts, you need to [configure a connector](/deploy-manage/manage-connectors.md) and modify the relevant rule to use the connector. If you're using {{ech}}, then you can use the default `Elastic-Cloud-SMTP` email connector or configure your own. - ::::{note} - For the `Elasticsearch nodes changed` alert, if you have only one master node in your cluster, during the master node vacate no notification will be sent. Kibana needs to communicate with the master node in order to send a notification. One way to avoid this is by shipping your deployment metrics to a dedicated monitoring cluster, which you can configure when you enable logging and monitoring in Step 2. +::::{note} +Some action types are subscription features, while others are free. For a comparison of the Elastic subscription levels, see the alerting section of the [Subscriptions page](https://www.elastic.co/subscriptions). +:::: - :::: +## Modify rules -6. In the **Edit rule** pane, set how often to check for the condition and how often to send notifications. -7. In the **Actions** section, select **Email** as a connector type and select the email connector that you configured in Step 3 (or the preconfigured `Elastic-Cloud-SMTP` connector). -8. Configure the email contents and select **Save**. Make sure that the email address you specify is the one that you allowed in Step 1. +To review and modify existing **{{stack-monitor-app}}** rules, click **Enter setup mode** on the **Cluster overview** page. Cards with alerts configured are annotated with an indicator. + +:::{tip} +Alternatively, to manage all rules, including create and delete functionality go to **{{stack-manage-app}} > {{rules-ui}}**. +::: + +1. On any card showing available alerts, select the **alerts** indicator. Use the menu to select the type of alert for which you’d like to be notified. +2. In the **Edit rule** pane, set how often to check for the condition and how often to send notifications. +3. In the **Actions** section, select the connector that you'd like to use for notifications. +4. Configure the connector message contents and select **Save**. + +## Default rules + +The following rules are [preconfigured](#_create_default_rules) for stack monitoring. + +:::{dropdown} CPU usage threshold + +$$$kibana-alerts-cpu-threshold$$$ + +This rule checks for {{es}} nodes that run a consistently high CPU load. + +By default, the condition is set at 85% or more averaged over the last 5 minutes. The default rule checks on a schedule time of 1 minute with a re-notify interval of 1 day. +::: + +:::{dropdown} Disk usage threshold + +$$$kibana-alerts-disk-usage-threshold$$$ + +This rule checks for {{es}} nodes that are nearly at disk capacity. + +By default, the condition is set at 80% or more averaged over the last 5 minutes. The default rule checks on a schedule time of 1 minute with a re-notify interval of 1 day. +::: + +:::{dropdown} JVM memory threshold + +$$$kibana-alerts-jvm-memory-threshold$$$ + +This rule checks for {{es}} nodes that use a high amount of JVM memory. + +By default, the condition is set at 85% or more averaged over the last 5 minutes. The default rule checks on a schedule time of 1 minute with a re-notify interval of 1 day. +::: + +:::{dropdown} Missing monitoring data + +$$$kibana-alerts-missing-monitoring-data$$$ + +This rule checks for {{es}} nodes that stop sending monitoring data. + +By default, the condition is set to missing for 15 minutes looking back 1 day. The default rule checks on a schedule time of 1 minute with a re-notify interval of 6 hours. +::: + +:::{dropdown} Thread pool rejections (search/write) + +$$$kibana-alerts-thread-pool-rejections$$$ + +This rule checks for {{es}} nodes that experience thread pool rejections. + +By default, the condition is set at 300 or more over the last 5 minutes. The default rule checks on a schedule time of 1 minute with a re-notify interval of 1 day. Thresholds can be set independently for `search` and `write` type rejections. +::: + +:::{dropdown} CCR read exceptions + +$$$kibana-alerts-ccr-read-exceptions$$$ + +This rule checks for read exceptions on any of the replicated {{es}} clusters. + +The condition is met if 1 or more read exceptions are detected in the last hour. The default rule checks on a schedule time of 1 minute with a re-notify interval of 6 hours. +::: + +:::{dropdown} Large shard size + +$$$kibana-alerts-large-shard-size$$$ + +This rule checks for a large average shard size (across associated primaries) on any of the specified data views in an {{es}} cluster. + +The condition is met if an index’s average shard size is 55gb or higher in the last 5 minutes. The default rule matches the pattern of `-.*` by running checks on a schedule time of 1 minute with a re-notify interval of 12 hours. +::: + +::::{dropdown} Cluster alerting + +$$$kibana-alerts-cluster-alerts$$$ + +These rules check the current status of your {{stack}}. You can drill down into the metrics to view more information about your cluster and specific nodes, instances, and indices. + +An action is triggered if any of the following conditions are met within the last minute: + +* {{es}} cluster health status is yellow (missing at least one replica) or red (missing at least one primary). +* {{es}} version mismatch. You have {{es}} nodes with different versions in the same cluster. +* {{kib}} version mismatch. You have {{kib}} instances with different versions running against the same {{es}} cluster. +* Logstash version mismatch. You have Logstash nodes with different versions reporting stats to the same monitoring cluster. +* {{es}} nodes changed. You have {{es}} nodes that were recently added or removed. +* {{es}} license expiration. The cluster’s license is about to expire. + + If you do not preserve the data directory when upgrading a {{kib}} or Logstash node, the instance is assigned a new persistent UUID and shows up as a new instance. + +* Subscription license expiration. When the expiration date approaches, you will get notifications with a severity level relative to how soon the expiration date is: + + * 60 days: Informational alert + * 30 days: Low-level alert + * 15 days: Medium-level alert + * 7 days: Severe-level alert + + The 60-day and 30-day thresholds are skipped for Trial licenses, which are only valid for 30 days. + +:::{note} +For the `Elasticsearch nodes changed` alert, if you have only one master node in your cluster, during the master node vacate no notification will be sent. Kibana needs to communicate with the master node in order to send a notification. One way to avoid this is by shipping your deployment metrics to a dedicated monitoring cluster. +::: +:::: \ No newline at end of file diff --git a/deploy-manage/monitor/monitoring-data/configuring-data-streamsindices-for-monitoring.md b/deploy-manage/monitor/monitoring-data/configuring-data-streamsindices-for-monitoring.md index a1aa885e41..98a6561621 100644 --- a/deploy-manage/monitor/monitoring-data/configuring-data-streamsindices-for-monitoring.md +++ b/deploy-manage/monitor/monitoring-data/configuring-data-streamsindices-for-monitoring.md @@ -9,19 +9,14 @@ applies_to: self: all --- -# Configuring data streams/indices for monitoring [config-monitoring-indices] +# Configuring monitoring data streams and indices [config-monitoring-indices] Monitoring data is stored in data streams or indices in {{es}}. The default data stream or index settings may not work for your situation. For example, you might want to change index lifecycle management (ILM) settings, add custom mappings, or change the number of shards and replicas. The steps to change these settings depend on the monitoring method: * [Configuring data streams created by {{agent}}](config-monitoring-data-streams-elastic-agent.md) -* [Configuring data streams created by {{metricbeat}} 8](config-monitoring-data-streams-metricbeat-8.md) (the default for version 8 {{ech}} deployments on {{ecloud}}) +* [Configuring data streams created by {{metricbeat}} 8](config-monitoring-data-streams-metricbeat-8.md) (the default for ECH, ECE, and ECK deployments) * [Configuring indices created by {{metricbeat}} 7 or internal collection](config-monitoring-indices-metricbeat-7-internal-collection.md) ::::{important} Changing mappings or settings can cause your monitoring dashboards to stop working correctly. :::: - - - - - diff --git a/deploy-manage/monitor/monitoring-data/elasticsearch-metrics.md b/deploy-manage/monitor/monitoring-data/elasticsearch-metrics.md index 023aa29636..9838fdfe72 100644 --- a/deploy-manage/monitor/monitoring-data/elasticsearch-metrics.md +++ b/deploy-manage/monitor/monitoring-data/elasticsearch-metrics.md @@ -1,5 +1,4 @@ --- -navigation_title: "{{es}} Metrics" mapped_pages: - https://www.elastic.co/guide/en/kibana/current/elasticsearch-metrics.html applies_to: @@ -12,12 +11,12 @@ applies_to: -# Elasticsearch Metrics [elasticsearch-metrics] +# Elasticsearch metrics [elasticsearch-metrics] You can drill down into the status of your {{es}} cluster in {{kib}} by clicking the [Overview](#cluster-overview-page), [Nodes](#nodes-page), [Indices](#indices-overview-page) and [Logs](#logs-monitor-page) links on the **Stack Monitoring** page. -:::{image} ../../../images/kibana-monitoring-elasticsearch.png +:::{image} /images/kibana-monitoring-elasticsearch.png :alt: Monitoring clusters :screenshot: ::: @@ -36,7 +35,7 @@ Conditions that require your attention are listed at the top of the Clusters pag The panel at the top shows the current cluster statistics, the charts show the search and indexing performance over time, and the table at the bottom shows information about any shards that are being recovered. If you use {{filebeat}} to collect log data from this cluster, you can also see its recent logs. -:::{image} ../../../images/kibana-monitoring-overview.png +:::{image} /images/kibana-monitoring-overview.png :alt: Elasticsearch Cluster Overview :screenshot: ::: diff --git a/deploy-manage/monitor/monitoring-data/kibana-alerts.md b/deploy-manage/monitor/monitoring-data/kibana-alerts.md deleted file mode 100644 index 5245ade6fa..0000000000 --- a/deploy-manage/monitor/monitoring-data/kibana-alerts.md +++ /dev/null @@ -1,105 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/kibana/current/kibana-alerts.html -applies_to: - deployment: - ess: all - ece: all - eck: all - self: all ---- - -% NEEDS TO BE MERGED WITH configure-stack-monitoring-alerts.md - -# Kibana alerts [kibana-alerts] - -The {{stack}} {{monitor-features}} provide [Alerting rules](../../../explore-analyze/alerts-cases/alerts.md) out-of-the box to notify you of potential issues in the {{stack}}. These rules are preconfigured based on the best practices recommended by Elastic. However, you can tailor them to meet your specific needs. - -:::{image} ../../../images/kibana-monitoring-kibana-alerting-notification.png -:alt: {{kib}} alerting notifications in {{stack-monitor-app}} -:screenshot: -::: - -When you open **{{stack-monitor-app}}** for the first time, you will be asked to acknowledge the creation of these default rules. They are initially configured to detect and notify on various conditions across your monitored clusters. You can view notifications for: **Cluster health**, **Resource utilization**, and **Errors and exceptions** for {{es}} in real time. - -::::{note} -The default {{watcher}} based "cluster alerts" for {{stack-monitor-app}} have been recreated as rules in {{kib}} {{alert-features}}. For this reason, the existing {{watcher}} email action `monitoring.cluster_alerts.email_notifications.email_address` no longer works. The default action for all {{stack-monitor-app}} rules is to write to {{kib}} logs and display a notification in the UI. -:::: - - -To review and modify existing **{{stack-monitor-app}}** rules, click **Enter setup mode** on the **Cluster overview** page. Alternatively, to manage all rules, including create and delete functionality go to **{{stack-manage-app}} > {{rules-ui}}**. - - -## CPU usage threshold [kibana-alerts-cpu-threshold] - -This rule checks for {{es}} nodes that run a consistently high CPU load. By default, the condition is set at 85% or more averaged over the last 5 minutes. The default rule checks on a schedule time of 1 minute with a re-notify interval of 1 day. - - -## Disk usage threshold [kibana-alerts-disk-usage-threshold] - -This rule checks for {{es}} nodes that are nearly at disk capacity. By default, the condition is set at 80% or more averaged over the last 5 minutes. The default rule checks on a schedule time of 1 minute with a re-notify interval of 1 day. - - -## JVM memory threshold [kibana-alerts-jvm-memory-threshold] - -This rule checks for {{es}} nodes that use a high amount of JVM memory. By default, the condition is set at 85% or more averaged over the last 5 minutes. The default rule checks on a schedule time of 1 minute with a re-notify interval of 1 day. - - -## Missing monitoring data [kibana-alerts-missing-monitoring-data] - -This rule checks for {{es}} nodes that stop sending monitoring data. By default, the condition is set to missing for 15 minutes looking back 1 day. The default rule checks on a schedule time of 1 minute with a re-notify interval of 6 hours. - - -## Thread pool rejections (search/write) [kibana-alerts-thread-pool-rejections] - -This rule checks for {{es}} nodes that experience thread pool rejections. By default, the condition is set at 300 or more over the last 5 minutes. The default rule checks on a schedule time of 1 minute with a re-notify interval of 1 day. Thresholds can be set independently for `search` and `write` type rejections. - - -## CCR read exceptions [kibana-alerts-ccr-read-exceptions] - -This rule checks for read exceptions on any of the replicated {{es}} clusters. The condition is met if 1 or more read exceptions are detected in the last hour. The default rule checks on a schedule time of 1 minute with a re-notify interval of 6 hours. - - -## Large shard size [kibana-alerts-large-shard-size] - -This rule checks for a large average shard size (across associated primaries) on any of the specified data views in an {{es}} cluster. The condition is met if an index’s average shard size is 55gb or higher in the last 5 minutes. The default rule matches the pattern of `-.*` by running checks on a schedule time of 1 minute with a re-notify interval of 12 hours. - - -## Cluster alerting [kibana-alerts-cluster-alerts] - -These rules check the current status of your {{stack}}. You can drill down into the metrics to view more information about your cluster and specific nodes, instances, and indices. - -An action is triggered if any of the following conditions are met within the last minute: - -* {{es}} cluster health status is yellow (missing at least one replica) or red (missing at least one primary). -* {{es}} version mismatch. You have {{es}} nodes with different versions in the same cluster. -* {{kib}} version mismatch. You have {{kib}} instances with different versions running against the same {{es}} cluster. -* Logstash version mismatch. You have Logstash nodes with different versions reporting stats to the same monitoring cluster. -* {{es}} nodes changed. You have {{es}} nodes that were recently added or removed. -* {{es}} license expiration. The cluster’s license is about to expire. - - If you do not preserve the data directory when upgrading a {{kib}} or Logstash node, the instance is assigned a new persistent UUID and shows up as a new instance. - -* Subscription license expiration. When the expiration date approaches, you will get notifications with a severity level relative to how soon the expiration date is: - - * 60 days: Informational alert - * 30 days: Low-level alert - * 15 days: Medium-level alert - * 7 days: Severe-level alert - - The 60-day and 30-day thresholds are skipped for Trial licenses, which are only valid for 30 days. - - - -## Alerts and rules [_alerts_and_rules] - - -### Create default rules [_create_default_rules] - -This option can be used to create default rules in this Kibana space. This is useful for scenarios when you didn’t choose to create these default rules initially or anytime later if the rules were accidentally deleted. - -::::{note} -Some action types are subscription features, while others are free. For a comparison of the Elastic subscription levels, see the alerting section of the [Subscriptions page](https://www.elastic.co/subscriptions). -:::: - - diff --git a/deploy-manage/monitor/monitoring-data/kibana-page.md b/deploy-manage/monitor/monitoring-data/kibana-page.md index 6d4b194239..d44d8103ec 100644 --- a/deploy-manage/monitor/monitoring-data/kibana-page.md +++ b/deploy-manage/monitor/monitoring-data/kibana-page.md @@ -1,5 +1,4 @@ --- -navigation_title: "{{kib}} Metrics" mapped_pages: - https://www.elastic.co/guide/en/kibana/current/kibana-page.html applies_to: @@ -12,12 +11,12 @@ applies_to: -# Kibana Metrics [kibana-page] +# Kibana metrics [kibana-page] To view the key metrics that indicate the overall health of {{kib}} itself, click **Overview** in the {{kib}} section of the **Stack Monitoring** page. -:::{image} ../../../images/kibana-monitoring-kibana-overview.png +:::{image} /images/kibana-monitoring-kibana-overview.png :alt: Kibana Overview :screenshot: ::: diff --git a/deploy-manage/monitor/monitoring-data/logstash-page.md b/deploy-manage/monitor/monitoring-data/logstash-page.md index 3b8e220465..ed43c4096f 100644 --- a/deploy-manage/monitor/monitoring-data/logstash-page.md +++ b/deploy-manage/monitor/monitoring-data/logstash-page.md @@ -1,5 +1,4 @@ --- -navigation_title: "Logstash Metrics" mapped_pages: - https://www.elastic.co/guide/en/kibana/current/logstash-page.html applies_to: @@ -12,12 +11,12 @@ applies_to: -# Logstash Metrics [logstash-page] +# Logstash metrics [logstash-page] If you are monitoring Logstash nodes, click **Overview** in the Logstash section of the **Stack Monitoring** page in {{kib}}. You can view the overall health of the Logstash nodes. -:::{image} ../../../images/kibana-monitoring-logstash-overview.png +:::{image} /images/kibana-monitoring-logstash-overview.png :alt: Logstash Overview :screenshot: ::: diff --git a/deploy-manage/monitor/monitoring-data/visualizing-monitoring-data.md b/deploy-manage/monitor/monitoring-data/visualizing-monitoring-data.md index d9f3489019..9e31924758 100644 --- a/deploy-manage/monitor/monitoring-data/visualizing-monitoring-data.md +++ b/deploy-manage/monitor/monitoring-data/visualizing-monitoring-data.md @@ -11,14 +11,21 @@ applies_to: # Visualizing monitoring data [xpack-monitoring] -The {{kib}} {{monitor-features}} serve two separate purposes: +From within the **Stack Monitoring** section of {{kib}}, you can view health and performance data for {{es}}, {{ls}}, {{kib}}, and Beats in real time, as well as analyze past performance. -1. To visualize monitoring data from across the {{stack}}. You can view health and performance data for {{es}}, {{ls}}, APM, and Beats in real time, as well as analyze past performance. -2. To monitor {{kib}} itself and route that data to the monitoring cluster. +Before you can see this data inside of {{kib}}, you need to [configure stack monitoring](/deploy-manage/monitor/stack-monitoring.md) and, for {{eck}} and self-managed deployments, [configure {{kib}} to retrieve and display monitoring data](/deploy-manage/monitor/stack-monitoring/kibana-monitoring-data.md). -If you enable monitoring across the {{stack}}, each monitored component is considered unique based on its persistent UUID, which is written to the [`path.data`](kibana://reference/configuration-reference/general-settings.md#path-data) directory when the node or instance starts. +Review the following topics to learn more about the metrics and visualizations available in the **Stack Monitoring** area: + +* [](/deploy-manage/monitor/monitoring-data/elasticsearch-metrics.md) +* [](/deploy-manage/monitor/monitoring-data/kibana-page.md) +* [](/deploy-manage/monitor/monitoring-data/beats-page.md) +* [](/deploy-manage/monitor/monitoring-data/logstash-page.md) + +From the **Stack Monitoring** section, you can also configure [Kibana alerts](/deploy-manage/monitor/monitoring-data/configure-stack-monitoring-alerts.md) for stack monitoring metrics. -For more information, see [Configure monitoring](../stack-monitoring/kibana-monitoring-self-managed.md) and [Monitor a cluster](../../monitor.md). +If you're having trouble accessing your monitoring data within the **Stack Monitoring** section, then refer to [](/deploy-manage/monitor/monitoring-data/monitor-troubleshooting.md). + +If you enable monitoring across the {{stack}}, each monitored component is considered unique based on its persistent UUID, which is written to the [`path.data`](kibana://reference/configuration-reference/general-settings.md#path-data) directory when the node or instance starts. -Want to monitor your fleet of {{agent}}s, too? Use {{fleet}} instead of the Stack Monitoring UI. To learn more, refer to [Monitor {{agent}}s](/reference/ingestion-tools/fleet/monitor-elastic-agent.md). diff --git a/deploy-manage/monitor/orchestrators.md b/deploy-manage/monitor/orchestrators.md index 72fa3074b8..d5fe00adbe 100644 --- a/deploy-manage/monitor/orchestrators.md +++ b/deploy-manage/monitor/orchestrators.md @@ -5,7 +5,7 @@ applies_to: eck: all --- -# Monitoring Orchestrators +# Monitoring orchestrators % What needs to be done: Write from scratch diff --git a/deploy-manage/monitor/orchestrators/ece-platform-monitoring.md b/deploy-manage/monitor/orchestrators/ece-platform-monitoring.md index 7fa3662dcd..4a8fc79a00 100644 --- a/deploy-manage/monitor/orchestrators/ece-platform-monitoring.md +++ b/deploy-manage/monitor/orchestrators/ece-platform-monitoring.md @@ -13,7 +13,7 @@ Elastic Cloud Enterprise by default collects monitoring data for your installati * Logs for all core services that are a part of Elastic Cloud Enterprise and monitoring metrics for core Elastic Cloud Enterprise services, system processes on the host, and any third-party software * Logs and monitoring metrics for Elasticsearch clusters and for Kibana instances -These monitoring indices are collected in addition to the [monitoring you might have enabled for specific clusters](../stack-monitoring/ece-stack-monitoring.md), which also provides monitoring metrics that you can access in Kibana (note that the `logging-and-metrics` deployment is used for monitoring data from system deployments only; for non-system deployments, monitoring data must be sent to a deployment other than `logging-and-metrics`). +These monitoring indices are collected in addition to the [monitoring you might have enabled for specific clusters](/deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md), which also provides monitoring metrics that you can access in Kibana (note that the `logging-and-metrics` deployment is used for monitoring data from system deployments only; for non-system deployments, monitoring data must be sent to a deployment other than `logging-and-metrics`). In this section: diff --git a/deploy-manage/monitor/stack-monitoring.md b/deploy-manage/monitor/stack-monitoring.md index 9e02fc3158..fd9a9ae69f 100644 --- a/deploy-manage/monitor/stack-monitoring.md +++ b/deploy-manage/monitor/stack-monitoring.md @@ -2,7 +2,6 @@ mapped_urls: - https://www.elastic.co/guide/en/elasticsearch/reference/current/monitoring-overview.html - https://www.elastic.co/guide/en/elasticsearch/reference/current/how-monitoring-works.html - - https://www.elastic.co/guide/en/cloud/current/ec-monitoring.html applies_to: deployment: ess: all @@ -11,36 +10,92 @@ applies_to: self: all --- -# Stack Monitoring +# Stack monitoring -% What needs to be done: Refine +:::{include} _snippets/stack-monitoring-def.md +::: -% GitHub issue: https://github.com/elastic/docs-projects/issues/350 +::::{admonition} Better monitoring with AutoOps +:::{include} _snippets/autoops.md +::: +:::: -% Scope notes: Merge both docs into 1 as an overview. Ensure its valid for all deployment types. Consider bringing part of the info from the elastic cloud monitoring introduction. +## How it works -% Use migrated content from existing pages that map to this page: +Each monitored {{stack}} component is considered unique in the cluster based on its persistent UUID, which is written to the [`path.data`](/deploy-manage/deploy/self-managed/important-settings-configuration.md#path-settings) directory when the node or instance starts. -% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/monitoring-overview.md -% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/how-monitoring-works.md -% - [ ] ./raw-migrated-files/cloud/cloud/ec-monitoring.md +Monitoring documents are just ordinary JSON documents built by monitoring each {{stack}} component at a specified collection interval. If you want to alter how these documents are structured or stored, refer to [Configuring data streams/indices for monitoring](/deploy-manage/monitor/monitoring-data/configuring-data-streamsindices-for-monitoring.md). -% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc): +You can use {{agent}} or {{metricbeat}} to collect monitoring data and to ship it directly to the monitoring cluster. -$$$ec-es-cluster-health$$$ +In {{ech}}, {{ece}}, and {{eck}}, Elastic manages the installation and configuration of the monitoring agent for you. -$$$ec-es-cluster-performance$$$ +## Production architecture -$$$ec-es-health-dedicated$$$ +You can collect and ship data directly to your monitoring cluster rather than routing it through your production cluster. -$$$ec-es-health-preconfigured$$$ +The following diagram illustrates a typical monitoring architecture with separate production and monitoring clusters. This example shows {{metricbeat}}, but you can use {{agent}} instead. -$$$ec-es-health-warnings$$$ +:::{image} /images/elasticsearch-reference-architecture.png +:alt: A typical monitoring environment +::: -$$$ec-health-best-practices$$$ +If you have the appropriate license, you can route data from multiple production clusters to a single monitoring cluster. [Learn about the differences between various subscription levels](https://www.elastic.co/subscriptions). -**This page is a work in progress.** The documentation team is working to combine content pulled from the following pages: +::::{important} +In general, the monitoring cluster and the clusters being monitored should be running the same version of the stack. A monitoring cluster cannot monitor production clusters running newer versions of the stack. If necessary, the monitoring cluster can monitor production clusters running the latest release of the previous major version. +:::: -* [/raw-migrated-files/elasticsearch/elasticsearch-reference/monitoring-overview.md](/raw-migrated-files/elasticsearch/elasticsearch-reference/monitoring-overview.md) -* [/raw-migrated-files/elasticsearch/elasticsearch-reference/how-monitoring-works.md](/raw-migrated-files/elasticsearch/elasticsearch-reference/how-monitoring-works.md) -* [/raw-migrated-files/cloud/cloud/ec-monitoring.md](/raw-migrated-files/cloud/cloud/ec-monitoring.md) \ No newline at end of file +## Configure and use stack monitoring + +Refer to the following topics to learn how to configure stack monitoring: + +### Configure for ECH, ECE, and ECK deployments + +* [](/deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md) +* [](/deploy-manage/monitor/stack-monitoring/eck-stack-monitoring.md) + + +### Configure for self-managed deployments + +* **{{es}}**: + * [](/deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-elastic-agent.md) (recommended): Uses a single agent to gather logs and metrics. Can be managed from a central location in {{fleet}}. + * [](/deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-metricbeat.md): Uses a lightweight {{beats}} shipper to gather metrics. May be preferred if you have an existing investment in {{beats}} or are not yet ready to use {{agent}}. + * [](/deploy-manage/monitor/stack-monitoring/collecting-log-data-with-filebeat.md): Uses a lightweight {{beats}} shipper to gather logs. + * [](/deploy-manage/monitor/stack-monitoring/es-legacy-collection-methods.md): Uses internal exporters to gather metrics. Not recommended. If you have previously configured legacy collection methods, you should migrate to using {{agent}} or {{metricbeat}}. +* **{{kib}}**: + * [](/deploy-manage/monitor/stack-monitoring/kibana-monitoring-elastic-agent.md) (recommended): Uses a single agent to gather logs and metrics. Can be managed from a central location in {{fleet}}. + * [](/deploy-manage/monitor/stack-monitoring/kibana-monitoring-metricbeat.md): Uses a lightweight {{beats}} shipper to gather metrics. May be preferred if you have an existing investment in {{beats}} or are not yet ready to use {{agent}}. + * [](/deploy-manage/monitor/stack-monitoring/kibana-monitoring-legacy.md): Uses internal exporters to gather metrics. Not recommended. If you have previously configured legacy collection methods, you should migrate to using {{agent}} or {{metricbeat}}. + + +### Monitor other {{stack}} components + +:::{tip} +Most of these methods require that you configure monitoring of {{es}} before monitoring other components. +::: + +* **Logstash**: + * [Monitoring {{ls}} with {{agent}}](logstash://reference/monitoring-logstash-with-elastic-agent.md) (recommended): Uses a single agent to gather logs and metrics. Can be managed from a central location in {{fleet}}. + * [Monitoring {{ls}} with legacy collection methods](logstash://reference/monitoring-logstash-legacy.md): Use {{metricbeat}} or legacy methods to collect monitoring data from {{ls}}. +* **{{beats}}**: + + * [Auditbeat](beats://reference/auditbeat/monitoring.md) + * [Filebeat](beats://reference/filebeat/monitoring.md) + * [Heartbeat](beats://reference/heartbeat/monitoring.md) + * [Metricbeat](beats://reference/metricbeat/monitoring.md) + * [Packetbeat](beats://reference/packetbeat/monitoring.md) + * [Winlogbeat](beats://reference/winlogbeat/monitoring.md) + +* [**APM Server**](/solutions/observability/apps/monitor-apm-server.md) + +* **{{agent}}s**: + * [{{fleet}}-managed {{agent}}s](/reference/ingestion-tools/fleet/monitor-elastic-agent.md) + * [Standalone {{agent}}s](/reference/ingestion-tools/fleet/elastic-agent-monitoring-configuration.md) + +### Access, view, and use monitoring data + +* [](/deploy-manage/monitor/stack-monitoring/kibana-monitoring-data.md): After you collect monitoring data for one or more products in the {{stack}}, configure {{kib}} to retrieve that information and display it in on the **Stack Monitoring** page. +* [](/deploy-manage/monitor/monitoring-data/visualizing-monitoring-data.md): View health and performance data for {{stack}} components in real time, as well as analyze past performance. +* [](/deploy-manage/monitor/monitoring-data/configure-stack-monitoring-alerts.md): Configure alerts that trigger based on stack monitoring metrics. +* [](/deploy-manage/monitor/monitoring-data/configuring-data-streamsindices-for-monitoring.md): Adjust the data streams and indices used by stack monitoring. diff --git a/deploy-manage/monitor/stack-monitoring/_snippets/cloud-monitoring-access.md b/deploy-manage/monitor/stack-monitoring/_snippets/cloud-monitoring-access.md new file mode 100644 index 0000000000..10499a2e7b --- /dev/null +++ b/deploy-manage/monitor/stack-monitoring/_snippets/cloud-monitoring-access.md @@ -0,0 +1,9 @@ +1. Log into [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body) (ECH) or [the Cloud UI](/deploy-manage/deploy/cloud-enterprise/log-into-cloud-ui.md) (ECE). +2. On the **Deployments** page, select your deployment. + + Narrow the list by name, ID, or choose from several other filters. To further define the list, use a combination of filters. + +3. From your deployment menu, go to the **Logs and Metrics** page. +4. Select the corresponding **View** button to check the logs or metrics data. + +Alternatively, you can access logs and metrics directly on the {{kib}} **Logs** and **Stack Monitoring** pages in the target monitoring deployment. \ No newline at end of file diff --git a/deploy-manage/monitor/stack-monitoring/_snippets/legacy-warning.md b/deploy-manage/monitor/stack-monitoring/_snippets/legacy-warning.md new file mode 100644 index 0000000000..c7e26505b7 --- /dev/null +++ b/deploy-manage/monitor/stack-monitoring/_snippets/legacy-warning.md @@ -0,0 +1,5 @@ +:::{important} +{{agent}} and {{metricbeat}} are the recommended methods for collecting and shipping monitoring data to a monitoring cluster. + +If you have previously configured legacy collection methods, you should migrate to using [{{agent}}](/deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-elastic-agent.md) or [{{metricbeat}}](/deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-metricbeat.md) collection. Do not use legacy collection alongside other collection methods. +::: \ No newline at end of file diff --git a/deploy-manage/monitor/stack-monitoring/collecting-log-data-with-filebeat.md b/deploy-manage/monitor/stack-monitoring/collecting-log-data-with-filebeat.md index 543d5b429f..143c0e6f95 100644 --- a/deploy-manage/monitor/stack-monitoring/collecting-log-data-with-filebeat.md +++ b/deploy-manage/monitor/stack-monitoring/collecting-log-data-with-filebeat.md @@ -22,7 +22,9 @@ If you’re using {{agent}}, do not deploy {{filebeat}} for log collection. Inst 1. Verify that {{es}} is running and that the monitoring cluster is ready to receive data from {{filebeat}}. ::::{tip} - In production environments, we strongly recommend using a separate cluster (referred to as the *monitoring cluster*) to store the data. Using a separate monitoring cluster prevents production cluster outages from impacting your ability to access your monitoring data. It also prevents monitoring activities from impacting the performance of your production cluster. See [*Monitoring in a production environment*](elasticsearch-monitoring-self-managed.md). + In production environments, we strongly recommend using a separate cluster (referred to as the *monitoring cluster*) to store the data. Using a separate monitoring cluster prevents production cluster outages from impacting your ability to access your monitoring data. It also prevents monitoring activities from impacting the performance of your production cluster. + + For more information, refer to [](/deploy-manage/monitor/stack-monitoring/es-self-monitoring-prod.md). :::: 2. Identify which logs you want to monitor. diff --git a/deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-elastic-agent.md b/deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-elastic-agent.md index f3f7568f05..d672494ab6 100644 --- a/deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-elastic-agent.md +++ b/deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-elastic-agent.md @@ -11,13 +11,11 @@ applies_to: # Collecting monitoring data with Elastic Agent [configuring-elastic-agent] - -In 8.5 and later, you can use {{agent}} to collect data about {{es}} and ship it to the monitoring cluster, rather than [using {{metricbeat}}](collecting-monitoring-data-with-metricbeat.md) or routing it through exporters as described in [Legacy collection methods](es-legacy-collection-methods.md). - +You can use {{agent}} to collect data about {{es}} and ship it to the monitoring cluster. ## Prerequisites [_prerequisites_11] -* (Optional) Create a monitoring cluster as described in [*Monitoring in a production environment*](elasticsearch-monitoring-self-managed.md). +* (Optional) Create a monitoring cluster as described in [](elasticsearch-monitoring-self-managed.md). * Create a user on the production cluster that has the `remote_monitoring_collector` [built-in role](../../users-roles/cluster-or-deployment-auth/built-in-roles.md). @@ -48,7 +46,7 @@ To collect {{es}} monitoring data, add an {{es}} integration to an {{agent}} and 7. Click **Save and continue**. This step takes a minute or two to complete. When it’s done, you’ll have an agent policy that contains an integration for collecting monitoring data from {{es}}. 8. If an {{agent}} is already assigned to the policy and deployed to the host where {{es}} is running, you’re done. Otherwise, you need to deploy an {{agent}}. To deploy an {{agent}}: - 1. Go to **{{fleet}} → Agents**, then click **Add agent**. + 1. Go to **{{fleet}} > Agents**, then click **Add agent**. 2. Follow the steps in the **Add agent** flyout to download, install, and enroll the {{agent}}. Make sure you choose the agent policy you created earlier. 9. Wait a minute or two until incoming data is confirmed. diff --git a/deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-metricbeat.md b/deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-metricbeat.md index 5d09fd70f4..a02f7b5293 100644 --- a/deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-metricbeat.md +++ b/deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-metricbeat.md @@ -12,15 +12,24 @@ applies_to: # Collecting monitoring data with Metricbeat [configuring-metricbeat] -In 6.5 and later, you can use {{metricbeat}} to collect data about {{es}} and ship it to the monitoring cluster, rather than routing it through exporters as described in [Legacy collection methods](es-legacy-collection-methods.md). +You can use {{metricbeat}} to collect data about {{es}} and ship it to the monitoring cluster. +:::{tip} Want to use {{agent}} instead? Refer to [Collecting monitoring data with {{agent}}](collecting-monitoring-data-with-elastic-agent.md). +::: -:::{image} ../../../images/elasticsearch-reference-metricbeat.png +:::{image} /images/elasticsearch-reference-metricbeat.png :alt: Example monitoring architecture +:width: 550px ::: -1. [Install {{metricbeat}}](beats://reference/metricbeat/metricbeat-installation-configuration.md). Ideally install a single {{metricbeat}} instance configured with `scope: cluster` and configure `hosts` to point to an endpoint (e.g. a load-balancing proxy) which directs requests to the master-ineligible nodes in the cluster. If this is not possible then install one {{metricbeat}} instance for each {{es}} node in the production cluster and use the default `scope: node`. When {{metricbeat}} is monitoring {{es}} with `scope: node` then you must install a {{metricbeat}} instance for each {{es}} node. If you don’t, some metrics will not be collected. {{metricbeat}} with `scope: node` collects most of the metrics from the elected master of the cluster, so you must scale up all your master-eligible nodes to account for this extra load and you should not use this mode if you have dedicated master nodes. +1. [Install {{metricbeat}}](beats://reference/metricbeat/metricbeat-installation-configuration.md). + + Ideally, install a single {{metricbeat}} instance configured with `scope: cluster` and configure `hosts` to point to an endpoint, such as a load-balancing proxy, which directs requests to the master-ineligible nodes in the cluster. + + If this is not possible, then install one {{metricbeat}} instance for each {{es}} node in the production cluster and use the default `scope: node`. When {{metricbeat}} is monitoring {{es}} with `scope: node` then you must install a {{metricbeat}} instance for each {{es}} node. If you don’t, some metrics will not be collected. + + {{metricbeat}} with `scope: node` collects most of the metrics from the elected master of the cluster, so you must scale up all your master-eligible nodes to account for this extra load. You should not use this mode if you have dedicated master nodes. 2. Enable the {{es}} module in {{metricbeat}} on each {{es}} node. For example, to enable the default configuration for the {{stack-monitor-features}} in the `modules.d` directory, run the following command: @@ -71,7 +80,9 @@ Want to use {{agent}} instead? Refer to [Collecting monitoring data with {{agent 5. Identify where to send the monitoring data. ::::{tip} - In production environments, we strongly recommend using a separate cluster (referred to as the *monitoring cluster*) to store the data. Using a separate monitoring cluster prevents production cluster outages from impacting your ability to access your monitoring data. It also prevents monitoring activities from impacting the performance of your production cluster. + In production environments, we strongly recommend using a separate cluster (referred to as the *monitoring cluster*) to store the data. Using a separate monitoring cluster prevents production cluster outages from impacting your ability to access your monitoring data. It also prevents monitoring activities from impacting the performance of your production cluster. + + For more information, refer to [](/deploy-manage/monitor/stack-monitoring/es-self-monitoring-prod.md). :::: diff --git a/deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md b/deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md new file mode 100644 index 0000000000..7ea03caf5f --- /dev/null +++ b/deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md @@ -0,0 +1,152 @@ +--- +navigation_title: "Enable on ECH and ECE" +mapped_pages: + - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-enable-logging-and-monitoring.html + - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-restrictions-monitoring.html + - https://www.elastic.co/guide/en/cloud-heroku/current/ech-monitoring.html + - https://www.elastic.co/guide/en/cloud/current/ec-monitoring-setup.html + - https://www.elastic.co/guide/en/cloud/current/ec-enable-logging-and-monitoring.html + - https://www.elastic.co/guide/en/cloud-heroku/current/ech-enable-logging-and-monitoring.html + - https://www.elastic.co/guide/en/cloud-heroku/current/ech-monitoring-setup.html + - https://www.elastic.co/guide/en/cloud-heroku/current/ech-restrictions-monitoring.html +applies_to: + deployment: + ece: all + ess: all +--- + +# Enable stack monitoring on ECH and ECE deployments + +The deployment logging and monitoring feature lets you monitor your deployment in {{kib}} by shipping logs and metrics to a monitoring deployment. You can: + +* View your deployment’s health and performance in real time and analyze past cluster, index, and node metrics. +* View your deployment’s logs to debug issues, discover slow queries, surface deprecations, and analyze access to your deployment. + +Monitoring consists of two components: + +* A monitoring and logging agent that is installed on each node in your deployment. The agents collect and index metrics to {{es}}, either on the same deployment or by sending logs and metrics to an external monitoring deployment. Elastic manages the installation and configuration of the monitoring agent for you, and you should not modify any of the settings. +* The stack monitoring application in {{kib}} that visualizes the monitoring metrics through a dashboard, and the logs application that allows you to search and analyze deployment logs. + +With logging and monitoring enabled for a deployment, metrics are collected for {{es}}, {{kib}}, and APM with Fleet Server. + +:::{admonition} Simplify monitoring with AutoOps +If you’re using Elastic Cloud Hosted, then you can use AutoOps to monitor your cluster. AutoOps significantly simplifies cluster management with performance recommendations, resource utilization visibility, real-time issue detection and resolution paths. + +For more information, refer to [Monitor with AutoOps](/deploy-manage/monitor/autoops.md). +::: + +## Before you begin [logging-and-monitoring-limitations] + +* Some limitations apply when you use monitoring on ECH or ECE. To learn more, check the monitoring [restrictions and limitations](#restrictions-monitoring). +* Enabling logs and monitoring consumes extra resources on a deployment. For production systems, we recommend sizing deployments with logs and monitoring enabled to at least 4 GB of RAM. + +## Monitoring for production use [logging-and-monitoring-production] + +For production use, you should send your deployment logs and metrics to a dedicated monitoring deployment. Monitoring indexes logs and metrics into {{es}} and these indexes consume storage, memory, and CPU cycles like any other index. By using a separate monitoring deployment, you avoid affecting your other production deployments and can view the logs and metrics even when a production deployment is unavailable. + +How many monitoring deployments you use depends on your requirements: + +* You can ship logs and metrics for many deployments to a single monitoring deployment, if your business requirements permit it. +* Although monitoring will work with a deployment running a single node, you need a minimum of three monitoring nodes to make monitoring highly available. +* You might need to create dedicated monitoring deployments for isolation purposes in some cases. For example: + + * If you have many deployments and some of them are much larger than others, creating separate monitoring deployments prevents a large deployment from potentially affecting monitoring performance for smaller deployments. + * If you need to silo {{es}} data for different business departments. Deployments that have been configured to ship logs and metrics to a target monitoring deployment have access to indexing data and can manage monitoring index templates, which is addressed by creating separate monitoring deployments. + +Logs and metrics that get sent to a dedicated monitoring {{es}} deployment [may not be cleaned up automatically](#logging-and-monitoring-retention) and might require some additional steps to remove excess data periodically. + +## Retention of logging and monitoring indices [logging-and-monitoring-retention] + +When sending monitoring and logging data to a deployment, an ILM policy is pre-configured to control data retention. To view or edit the policies, open {{kib}} **Stack management > Data > Index Lifecycle Policies**. + +For monitoring indices, the retention period is configured in the `.monitoring-8-ilm-policy` index lifecycle policy. + +## Enable logging and monitoring [enable-logging-and-monitoring-steps] + +Elastic manages the installation and configuration of the monitoring agent for you. When you enable monitoring on a deployment, you are configuring where the monitoring agent for your current deployment should send its logs and metrics. + +To enable monitoring on your deployment: + +1. Log into [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body) (ECH) or [the Cloud UI](/deploy-manage/deploy/cloud-enterprise/log-into-cloud-ui.md) (ECE). +2. On the **Deployments** page, select your deployment. + + Narrow the list by name, ID, or choose from several other filters. To further define the list, use a combination of filters. + +3. From your deployment menu, go to the **Logs and metrics** page. +4. Under **Ship to a deployment**, select **Enable**. +5. Choose where to send your logs and metrics. Select **Save**. + + If a deployment is not listed, make sure that it is running a compatible version. The monitoring deployment and production deployment must be on the same major version, cloud provider, and region. + + ::::{tip} + Remember to send logs and metrics for production deployments to a dedicated monitoring deployment, so that your production deployments are not impacted by the overhead of indexing and storing monitoring data. A dedicated monitoring deployment also gives you more control over the retention period for monitoring data. + :::: + +::::{note} +Enabling logs and monitoring may trigger a plan change on your deployment. You can monitor the plan change progress from the deployment’s **Activity** page. +:::: + +## Access the monitoring application in {{kib}} [access-kibana-monitoring] + +With monitoring enabled for your deployment, you can access the [logs](//solutions/observability.md) and [stack monitoring](../monitoring-data/visualizing-monitoring-data.md) through {{kib}}. + +:::{include} /deploy-manage/monitor/stack-monitoring/_snippets/cloud-monitoring-access.md +::: + +You can also create an `elastic-cloud-logs-*` data view (formerly *index pattern*) to view your deployment’s logs in the {{kib}} **Discover** tab. + +Several fields are available for you to view logs based on key details, such as the source deployment: + +| Field | Description | Example value | +| --- | --- | --- | +| `service.id` | The ID of the deployment that generated the log | `6ff525333d2844539663f3b1da6c04b6` | +| `service.name` | The name of the deployment that generated the log | `My Production Deployment` | +| `cloud.availability_zone` | The availability zone in which the instance that generated the log is deployed | `ap-northeast-1d` | +| `service.node.name` | The ID of the instance that generated the log | `instance-0000000008` | +| `service.type` | The type of instance that generated the log | `elasticsearch` | +| `service.version` | The version of the stack resource that generated the log | `9.0.0` | + +## Logging features [extra-logging-features] + +When shipping logs to a monitoring deployment there are more logging features available to you. These features include: + + +### For {{es}} [extra-logging-features-elasticsearch] + +* [Audit logging](/deploy-manage/security/logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment +* [Slow query and index logging](elasticsearch://reference/elasticsearch/index-settings/slow-log.md) - helps find and debug slow queries and indexing +* Verbose logging - helps debug stack issues by increasing component logs + +After you’ve enabled log delivery on your deployment, you can [add the Elasticsearch user settings](/deploy-manage/deploy/cloud-enterprise/edit-stack-settings.md) to enable these features. + + +### For {{kib}} [extra-logging-features-kibana] + +* [Audit logging](/deploy-manage/security/logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment + +After you’ve enabled log delivery on your deployment, you can [add the {{kib}} user settings](/deploy-manage/deploy/cloud-enterprise/edit-stack-settings.md) to enable this feature. + + +### Other components [extra-logging-features-enterprise-search] + +Enabling log collection also supports collecting and indexing the following types of logs from other components in your deployments: + +**APM** + +* `apm*.log*` + +**Fleet and Elastic Agent** + +* `fleet-server-json.log-*` +* `elastic-agent-json.log-*` + +The `*` indicates that we also index the archived files of each type of log. + +Check the respective product documentation for more information about the logging capabilities of each product. + +## Restrictions and limitations [restrictions-monitoring] + +* To avoid compatibility issues, ensure your monitoring cluster and production cluster run on the same {{stack}} version. Monitoring clusters that use 9.x do work with production clusters that use the latest release of 8.x, but this setup should only occur when upgrading clusters to the same version. +* $$$cross-region-monitor$$$ Monitoring across regions is not supported. If you need to move your existing monitoring to the same region, you can do a reindex or create a new deployment and select the snapshot from the old deployment. +* The logs shipped to a monitoring cluster use an ILM managed data stream (`elastic-cloud-logs-`). If you need to delete indices due to space, do not delete the current `is_write_enabled: true` index. +* When sending metrics to a dedicated monitoring deployment, the graph for IO Operations Rate(/s) is blank. This is due to the fact that this graph actually contains metrics from of all of the virtualized resources from the provider. \ No newline at end of file diff --git a/deploy-manage/monitor/stack-monitoring/ece-restrictions-monitoring.md b/deploy-manage/monitor/stack-monitoring/ece-restrictions-monitoring.md deleted file mode 100644 index 30a1025dff..0000000000 --- a/deploy-manage/monitor/stack-monitoring/ece-restrictions-monitoring.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-restrictions-monitoring.html -applies_to: - deployment: - ece: all ---- - -# Restrictions and limitations [ece-restrictions-monitoring] - -* To avoid compatibility issues, ensure your monitoring cluster and production cluster run on the same {{stack}} version. Monitoring clusters that use 8.x do work with production clusters that use the latest release of 7.x, but this setup should only occur when upgrading clusters to the same version. -* $$$cross-region-monitor$$$ Monitoring across regions is not supported. If you need to move your existing monitoring to the same region, you can do a reindex or create a new deployment and select the snapshot from the old deployment. -* The logs shipped to a monitoring cluster use an ILM managed data stream (elastic-cloud-logs-). If you need to delete indices due to space, do not delete the current is_write_enabled: true index. -* When sending metrics to a dedicated monitoring deployment, the graph for IO Operations Rate(/s) is blank. This is due to the fact that this graph actually contains metrics from of all of the virtualized resources from the provider. - diff --git a/deploy-manage/monitor/stack-monitoring/ece-stack-monitoring.md b/deploy-manage/monitor/stack-monitoring/ece-stack-monitoring.md deleted file mode 100644 index af3d682d76..0000000000 --- a/deploy-manage/monitor/stack-monitoring/ece-stack-monitoring.md +++ /dev/null @@ -1,238 +0,0 @@ ---- -navigation_title: "Elastic Cloud Enterprise (ECE)" -mapped_pages: - - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-enable-logging-and-monitoring.html -applies_to: - deployment: - ece: all ---- - -# Enable stack monitoring on ECE deployments [ece-enable-logging-and-monitoring] - -The deployment logging and monitoring feature lets you monitor your deployment in [Kibana](/get-started/the-stack.md) by shipping logs and metrics to a monitoring deployment. You can: - -* View your deployment’s health and performance in real time and analyze past cluster, index, and node metrics. -* View your deployment’s logs to debug issues, discover slow queries, surface deprecations, and analyze access to your deployment. - -Monitoring consists of two components: - -* A monitoring and logging agent that is installed on each node in your deployment. The agents collect and index metrics to {{es}}, either on the same deployment or by sending logs and metrics to an external monitoring deployment. Elastic Cloud Enterprise manages the installation and configuration of the monitoring agent for you, and you should not modify any of the settings. -* The stack monitoring application in Kibana that visualizes the monitoring metrics through a dashboard and the logs application that allows you to search and analyze deployment logs. - -The steps in this section cover only the enablement of the monitoring and logging features in Elastic Cloud Enterprise. For more information on how to use the monitoring features, refer to [Monitor a cluster](../../monitor.md). - - -### Before you begin [ece-logging-and-monitoring-limitations] - -Some limitations apply when you use monitoring on Elastic Cloud Enterprise. To learn more, check the monitoring [restrictions and limitations](ece-restrictions-monitoring.md). - - -### Monitoring for production use [ece-logging-and-monitoring-production] - -For production use, you should send your deployment logs and metrics to a dedicated monitoring deployment. Monitoring indexes logs and metrics into {{es}} and these indexes consume storage, memory, and CPU cycles like any other index. By using a separate monitoring deployment, you avoid affecting your other production deployments and can view the logs and metrics even when a production deployment is unavailable. - -How many monitoring deployments you use depends on your requirements: - -* You can ship logs and metrics for many deployments to a single monitoring deployment, if your business requirements permit it. -* Although monitoring will work with a deployment running a single node, you need a minimum of three monitoring nodes to make monitoring highly available. -* You might need to create dedicated monitoring deployments for isolation purposes in some cases. For example: - - * If you have many deployments and some of them are much larger than others, creating separate monitoring deployments prevents a large deployment from potentially affecting monitoring performance for smaller deployments. - * If you need to silo {{es}} data for different business departments. Deployments that have been configured to ship logs and metrics to a target monitoring deployment have access to indexing data and can manage monitoring index templates, which is addressed by creating separate monitoring deployments. - - -Logs and metrics that get sent to a dedicated monitoring {{es}} deployment [may not be cleaned up automatically](#ece-logging-and-monitoring-retention) and might require some additional steps to remove excess data periodically. - - -### Retention of monitoring daily indices [ece-logging-and-monitoring-retention] - - -#### Stack versions 8.0 and above [ece-logging-and-monitoring-retention-8] - -When you enable monitoring in Elastic Cloud Enterprise, your monitoring indices are retained for a certain period by default. After the retention period has passed, the monitoring indices are deleted automatically. The retention period is configured in the `.monitoring-8-ilm-policy` index lifecycle policy. To view or edit the policy open {{kib}} **Stack management > Data > Index Lifecycle Policies**. - - -### Sending monitoring data to itself (self monitoring) [ece-logging-and-monitoring-retention-self-monitoring] - -$$$ece-logging-and-monitoring-retention-7$$$ -When you enable self-monitoring in Elastic Cloud Enterprise, your monitoring indices are retained for a certain period by default. After the retention period has passed, the monitoring indices are deleted automatically. Monitoring data is retained for three days by default or as specified by the [`xpack.monitoring.history.duration` user setting](https://www.elastic.co/guide/en/cloud-enterprise/current/ece-change-user-settings-examples.html#xpack-monitoring-history-duration). - -To retain monitoring indices as is without deleting them automatically, you must disable the [cleaner service](es-local-exporter.md#local-exporter-cleaner) by adding a disabled local exporter in your cluster settings. - -For example - -```sh -PUT /_cluster/settings -{ - "persistent": { - "xpack.monitoring.exporters.__no-default-local__": { - "type": "local", - "enabled": false - } - } -} -``` - - -### Sending monitoring data to a dedicated monitoring deployment [ece-logging-and-monitoring-retention-dedicated-monitoring] - -When [monitoring for production use](#ece-logging-and-monitoring-production), where you configure your deployments **to send monitoring data to a dedicated monitoring deployment** for indexing, this retention period does not apply. Monitoring indices on a dedicated monitoring deployment are retained until you remove them. There are three options open to you: - -* To enable the automatic deletion of monitoring indices from dedicated monitoring deployments, [enable monitoring](#ece-enable-logging-and-monitoring-steps) on your dedicated monitoring deployment in Elastic Cloud Enterprise to send monitoring data to itself. When an {{es}} deployment sends monitoring data to itself, all monitoring indices are deleted automatically after the retention period, regardless of the origin of the monitoring data. -* Alternatively, you can enable the cleaner service on the monitoring deployment by creating a local exporter. You can define the retention period at the same time. - - For example - - ```sh - PUT _cluster/settings - { - "persistent": { - "xpack" : { - "monitoring" : { - "exporters" : { - "found-user-defined" : { - "type" : "local", - "enabled" : "true" - } - }, - "history" : { - "duration" : "24h" - } - } - } - } - } - ``` - -* To retain monitoring indices on a dedicated monitoring deployment as is without deleting them automatically, no additional steps are required other than making sure that you do not enable the monitoring deployment to send monitoring data to itself. You should also monitor the deployment for disk space usage and upgrade your deployment periodically, if necessary. - - -### Retention of logging indices [ece-logging-and-monitoring-log-retention] - -An ILM policy is pre-configured to manage log retention. The policy can be adjusted according to your requirements. - - -### Index management [ece-logging-and-monitoring-index-management-ilm] - -When sending monitoring data to a deployment, you can configure [Index Lifecycle Management (ILM)](/manage-data/lifecycle/index-lifecycle-management.md) to manage retention of your monitoring and logging indices. When sending logs to a deployment, an ILM policy is pre-configured to manage log retention and the policy can be customized to your needs. - - -### Enable logging and monitoring [ece-enable-logging-and-monitoring-steps] - -Elastic Cloud Enterprise manages the installation and configuration of the monitoring agent for you. When you enable monitoring on a deployment, you are configuring where the monitoring agent for your current deployment should send its logs and metrics. - -To enable monitoring on your deployment: - -1. [Log into the Cloud UI](../../deploy/cloud-enterprise/log-into-cloud-ui.md). -2. On the **Deployments** page, select your deployment. - - Narrow the list by name, ID, or choose from several other filters. To further define the list, use a combination of filters. - -3. From your deployment menu, go to the **Logs and metrics** page. -4. Select **Enable**. -5. Choose where to send your logs and metrics. Select **Save**. - - If a deployment is not listed, make sure that it is running a compatible version. The monitoring deployment and production deployment must be on the same major version, cloud provider, and region. - - ::::{tip} - Remember to send logs and metrics for production deployments to a dedicated monitoring deployment, so that your production deployments are not impacted by the overhead of indexing and storing monitoring data. A dedicated monitoring deployment also gives you more control over the retention period for monitoring data. - :::: - - -::::{note} -Enabling logs and monitoring may trigger a plan change on your deployment. You can monitor the plan change progress from the deployment’s **Activity** page. -:::: - - -::::{note} -Enabling logs and monitoring requires some extra resource on a deployment. For production systems, we recommend sizing deployments with logs and monitoring enabled to at least 4 GB of RAM. -:::: - - - -### Access the monitoring application in Kibana [ece-access-kibana-monitoring] - -With monitoring enabled for your deployment, you can access the [logs](https://www.elastic.co/guide/en/kibana/current/observability.html) and [stack monitoring](../monitoring-data/visualizing-monitoring-data.md) through Kibana. - -1. [Log into the Cloud UI](../../deploy/cloud-enterprise/log-into-cloud-ui.md). -2. On the **Deployments** page, select your deployment. - - Narrow the list by name, ID, or choose from several other filters. To further define the list, use a combination of filters. - -3. From your deployment menu, go to the **Logs and Metrics** page. -4. Select the corresponding **View** button to check the logs or metrics data. - -Alternatively, you can access logs and metrics directly on the Kibana **Logs** and **Stack Monitoring** pages in the target monitoring deployment. You can also create an `elastic-cloud-logs-*` data view (formerly *index pattern*) to view your deployment’s logs in the Kibana **Discover** tab. Several fields are available for you to view logs based on key details, such as the source deployment: - -| Field | Description | Example value | -| --- | --- | --- | -| `service.id` | The ID of the deployment that generated the log | `6ff525333d2844539663f3b1da6c04b6` | -| `service.name` | The name of the deployment that generated the log | `My Production Deployment` | -| `cloud.availability_zone` | The availability zone in which the instance that generated the log is deployed | `ap-northeast-1d` | -| `service.node.name` | The ID of the instance that generated the log | `instance-0000000008` | -| `service.type` | The type of instance that generated the log | `elasticsearch` | -| `service.version` | The version of the stack resource that generated the log | `8.13.1` | - - -### Logging features [ece-extra-logging-features] - -When shipping logs to a monitoring deployment there are more logging features available to you. These features include: - - -#### For {{es}}: [ece-extra-logging-features-elasticsearch] - -* [Audit logging](../../security/logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment -* [Slow query and index logging](elasticsearch://reference/elasticsearch/index-settings/slow-log.md) - helps find and debug slow queries and indexing -* Verbose logging - helps debug stack issues by increasing component logs - -After you’ve enabled log delivery on your deployment, you can [add the Elasticsearch user settings](../../deploy/cloud-enterprise/edit-stack-settings.md) to enable these features. - - -#### For Kibana: [ece-extra-logging-features-kibana] - -* [Audit logging](../../security/logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment - -After you’ve enabled log delivery on your deployment, you can [add the Kibana user settings](../../deploy/cloud-enterprise/edit-stack-settings.md) to enable this feature. - - -### Other components [ece-extra-logging-features-enterprise-search] - -Enabling log collection also supports collecting and indexing the following types of logs from other components in your deployments: - -**APM** - -* apm*.log* - -**Fleet and Elastic Agent** - -* fleet-server-json.log-* -* elastic-agent-json.log-* - -The ˆ*ˆ indicates that we also index the archived files of each type of log. - -Check the respective product documentation for more information about the logging capabilities of each product. - - -## Metrics features [ece-extra-metrics-features] - -With logging and monitoring enabled for a deployment, metrics are collected for Elasticsearch, Kibana, and APM with Fleet Server. - - -#### Enabling Elasticsearch/Kibana audit logs on your deployment [ece-enable-audit-logs] - -Audit logs are useful for tracking security events on your {{es}} and/or {{kib}} clusters. To enable {{es}} audit logs on your deployment: - -1. [Log into the Cloud UI](../../deploy/cloud-enterprise/log-into-cloud-ui.md). -2. On the **Deployments** page, select your deployment. - - Narrow the list by name, ID, or choose from several other filters. To further define the list, use a combination of filters. - -3. From your deployment menu, go to the **Edit** page. -4. To enable audit logs in {{es}}, in the **Elasticsearch** section select **Manage user settings and extensions**. For deployments with existing user settings, you may have to expand the **Edit elasticsearch.yml** caret for each node instead. -5. To enable audit logs in {{kib}}, in the **Kibana** section select **Edit user settings**. For deployments with existing user settings, you may have to expand the **Edit kibana.yml** caret instead. -6. Add `xpack.security.audit.enabled: true` to the user settings. -7. Select **Save changes**. - -A plan change will run on your deployment. When it finishes, audit logs will be delivered to your monitoring deployment. - - diff --git a/deploy-manage/monitor/stack-monitoring/eck-stack-monitoring.md b/deploy-manage/monitor/stack-monitoring/eck-stack-monitoring.md index 169846eca5..a5732a7830 100644 --- a/deploy-manage/monitor/stack-monitoring/eck-stack-monitoring.md +++ b/deploy-manage/monitor/stack-monitoring/eck-stack-monitoring.md @@ -1,7 +1,12 @@ --- -navigation_title: "Elastic Cloud on Kubernetes (ECK)" +navigation_title: "Enable on ECK" mapped_pages: - https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-stack-monitoring.html + - https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s_how_it_works.html + - https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s_when_to_use_it.html + - https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s_override_the_beats_pod_template.html + - https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s_connect_to_an_external_monitoring_elasticsearch_cluster.html + applies_to: deployment: eck: all @@ -11,9 +16,35 @@ applies_to: You can enable [Stack Monitoring](/deploy-manage/monitor.md) on Elasticsearch, Kibana, Beats and Logstash to collect and ship their metrics and logs to a monitoring cluster. Although self-monitoring is possible, it is advised to use a [separate monitoring cluster](/deploy-manage/monitor/stack-monitoring.md). -To enable Stack Monitoring, simply reference the monitoring Elasticsearch cluster in the `spec.monitoring` section of their specification. +## How stack monitoring works in ECK + +In the background, Metricbeat and Filebeat are deployed as sidecar containers in the same Pod as {{es}} and {{kib}}. + +Metricbeat is used to collect monitoring metrics, and Filebeat to monitor the {{es}} log files and collect log events. + +The two Beats are configured to ship data directly to the monitoring cluster(s) using HTTPS and dedicated Elastic users managed by ECK. + +## When to use it + +This feature is a good solution if you need to monitor your Elastic applications in restricted Kubernetes environments where you cannot grant the following advanced permissions: + +* To Metricbeat, to allow querying the k8s API +* To Filebeat, to deploy a privileged DaemonSet -The following example shows how Elastic Stack components can be configured to send their monitoring data to a separate Elasticsearch cluster in the same Kubernetes cluster. +However, for maximum efficiency and minimizing resource consumption, or advanced use cases that require specific Beats configurations, you can deploy a standalone Metricbeat deployment and a Filebeat DaemonSet. Refer to [Beats configuration examples](/deploy-manage/deploy/cloud-on-k8s/configuration-examples-beats.md) for more information. + +## Enable stack monitoring + +To enable stack monitoring, reference the monitoring {{es}} cluster in the `spec.monitoring` section of their specification. + +The monitoring cluster must be managed by ECK in the same Kubernetes cluster as the monitored one. To learn how to connect an external monitoring cluster, refer to [Connect ot an external monitoring {{es}} cluster](#k8s_connect_to_an_external_monitoring_elasticsearch_cluster). + +The following example shows how {{stack}} components can be configured to send their monitoring data to a separate {{es}} cluster in the same Kubernetes cluster. + +You can also do the following: +* Configure an {{es}} cluster to monitor itself. This is not recommended for production deployments. +* Send metrics and logs to two different {{es}} monitoring clusters. +* Enable stack monitoring on a single {{stack}} component only. If {{es}} is not monitored, other {{stack}} components will not be available on the [Stack Monitoring](/deploy-manage/monitor/stack-monitoring/kibana-monitoring-data.md) {{kib}} page. ```yaml apiVersion: elasticsearch.k8s.elastic.co/v1 @@ -73,18 +104,93 @@ spec: ``` 1. The same monitoring cluster is used for metrics and logs, but separate clusters could be used. -2. The use of `namespace` is optional if the monitoring Elasticsearch cluster and the monitored Elastic Stack resource are running in the same namespace. +2. The use of `namespace` is optional if the monitoring {{es}} cluster and the monitored {{stack}} resource are running in the same namespace. +::::{note} +If stack monitoring is configured for a Beat, but the corresponding {{es}} cluster is not monitored, the Kibana stack monitoring page will not show the Beats data. +:::: ::::{note} -If Logs Stack Monitoring is configured for a Beat, and custom container arguments (`podTemplate.spec.containers[].args`) include `-e`, which enables logging to stderr and disables log file output, this argument will be removed from the Pod to allow the Filebeat sidecar to consume the Beat’s log files. +If Logs stack monitoring is configured for a Beat, and custom container arguments (`podTemplate.spec.containers[].args`) include `-e`, which enables logging to stderr and disables log file output, this argument will be removed from the Pod to allow the Filebeat sidecar to consume the Beat’s log files. :::: +## Connect to an external monitoring {{es}} cluster [k8s_connect_to_an_external_monitoring_elasticsearch_cluster] -You can also enable Stack Monitoring on a single Stack component only. In case Elasticsearch is not monitored, other Stack components will not be available on the Stack Monitoring Kibana page (check [View monitoring data in Kibana](/deploy-manage/monitor/stack-monitoring/kibana-monitoring-data.md)). +If you want to connect to a monitoring {{es}} cluster not managed by ECK, you can reference a Secret instead of an {{es}} cluster in the `monitoring` section through the `secretName` attribute: +```yaml +apiVersion: elasticsearch.k8s.elastic.co/v1 +kind: Elasticsearch +metadata: + name: monitored-sample + namespace: production +spec: + version: 8.16.1 + monitoring: + metrics: + elasticsearchRefs: + - secretName: monitoring-metrics-es-ref <1> + logs: + elasticsearchRefs: + - name: monitoring-logs + namespace: observability <2> + serviceName: monitoring-logs-es-coordinating-nodes <2> + nodeSets: + - name: default + count: 1 + config: + node.store.allow_mmap: false +``` + +1. The `secretName` and `name` attributes are mutually exclusive, you have to choose one or the other. +2. The `namespace` and `serviceName` attributes can only be used in conjunction with `name`, not with `secretName`. + +The referenced Secret must contain the following connection information: + +* `url`: The URL to reach the {{es}} cluster +* `username`: The username of the user to be authenticated to the {{es}} cluster +* `password`: The password of the user to be authenticated to the {{es}} cluster +* `ca.crt`: The contents of the CA certificate in PEM format to secure communication to the {{es}} cluster (optional) +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: monitoring-metrics-es-ref +stringData: + url: https://mon1.es.abcd-42.xyz.elastic-cloud.com:9243 + username: monitoring-user + password: +``` +The user referenced in the Secret must have been created beforehand. +## Override the Beats Pod template +You can customize the Filebeat and Metricbeat containers through the Pod template. Your configuration is merged with the values of the default Pod template that ECK uses. +```yaml +apiVersion: elasticsearch.k8s.elastic.co/v1 +kind: Elasticsearch +spec: + monitoring: + metrics: + elasticsearchRef: + name: monitoring + namespace: observability + logs: + elasticsearchRef: + name: monitoring + namespace: observability + nodeSets: + - name: default + podTemplate: + spec: + containers: + - name: metricbeat + env: + - foo: bar + - name: filebeat + env: + - foo: bar +``` \ No newline at end of file diff --git a/deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md b/deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md deleted file mode 100644 index d0d011d708..0000000000 --- a/deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md +++ /dev/null @@ -1,77 +0,0 @@ ---- -navigation_title: "Elastic Cloud Hosted (ECH)" -mapped_urls: - - https://www.elastic.co/guide/en/cloud-heroku/current/ech-monitoring.html - - https://www.elastic.co/guide/en/cloud/current/ec-monitoring-setup.html - - https://www.elastic.co/guide/en/cloud/current/ec-enable-logging-and-monitoring.html - - https://www.elastic.co/guide/en/cloud-heroku/current/ech-enable-logging-and-monitoring.html - - https://www.elastic.co/guide/en/cloud-heroku/current/ech-monitoring-setup.html - - https://www.elastic.co/guide/en/cloud-heroku/current/ech-restrictions-monitoring.html -applies_to: - deployment: - ess: all ---- - -# Stack Monitoring on Elastic Cloud deployments - -% What needs to be done: Refine - -% GitHub issue: https://github.com/elastic/docs-projects/issues/350 - -% Scope notes: deal with ech redirect. Refine all these docs, part of the info should go to the stack monitoring intro doc. - -% Use migrated content from existing pages that map to this page: - -% - [ ] ./raw-migrated-files/cloud/cloud-heroku/ech-monitoring.md -% - [ ] ./raw-migrated-files/cloud/cloud/ec-monitoring-setup.md -% Notes: redirect only -% - [ ] ./raw-migrated-files/cloud/cloud/ec-enable-logging-and-monitoring.md -% - [ ] ./raw-migrated-files/cloud/cloud-heroku/ech-enable-logging-and-monitoring.md -% - [ ] ./raw-migrated-files/cloud/cloud-heroku/ech-monitoring-setup.md -% - [ ] ./raw-migrated-files/cloud/cloud-heroku/ech-restrictions-monitoring.md - -% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc): - -$$$ec-check-logs$$$ - -$$$ec-restrictions-monitoring$$$ - -$$$ec-enable-logging-and-monitoring-steps$$$ - -$$$ec-logging-and-monitoring-production$$$ - -$$$ec-logging-and-monitoring-retention$$$ - -$$$ech-enable-logging-and-monitoring-steps$$$ - -$$$ech-es-cluster-health$$$ - -$$$ech-es-cluster-performance$$$ - -$$$ech-es-health-dedicated$$$ - -$$$ech-es-health-preconfigured$$$ - -$$$ech-es-health-warnings$$$ - -$$$ech-health-best-practices$$$ - -$$$ech-logging-and-monitoring-production$$$ - -$$$ech-logging-and-monitoring-retention$$$ - -% Please leave the AutoOps banner in the final content of this page - -:::{important} - If you’re using Elastic Cloud Hosted, then you can use AutoOps to monitor your cluster. AutoOps significantly simplifies cluster management with performance recommendations, resource utilization visibility, real-time issue detection and resolution paths. For more information, refer to [Monitor with AutoOps](/deploy-manage/monitor/autoops.md). -::: - - -**This page is a work in progress.** The documentation team is working to combine content pulled from the following pages: - -* [/raw-migrated-files/cloud/cloud-heroku/ech-monitoring.md](/raw-migrated-files/cloud/cloud-heroku/ech-monitoring.md) -* [/raw-migrated-files/cloud/cloud/ec-monitoring-setup.md](/raw-migrated-files/cloud/cloud/ec-monitoring-setup.md) -* [/raw-migrated-files/cloud/cloud/ec-enable-logging-and-monitoring.md](/raw-migrated-files/cloud/cloud/ec-enable-logging-and-monitoring.md) -* [/raw-migrated-files/cloud/cloud-heroku/ech-enable-logging-and-monitoring.md](/raw-migrated-files/cloud/cloud-heroku/ech-enable-logging-and-monitoring.md) -* [/raw-migrated-files/cloud/cloud-heroku/ech-monitoring-setup.md](/raw-migrated-files/cloud/cloud-heroku/ech-monitoring-setup.md) -* [/raw-migrated-files/cloud/cloud-heroku/ech-restrictions-monitoring.md](/raw-migrated-files/cloud/cloud-heroku/ech-restrictions-monitoring.md) \ No newline at end of file diff --git a/deploy-manage/monitor/stack-monitoring/elasticsearch-monitoring-self-managed.md b/deploy-manage/monitor/stack-monitoring/elasticsearch-monitoring-self-managed.md index 04639b89e8..04755bb2a9 100644 --- a/deploy-manage/monitor/stack-monitoring/elasticsearch-monitoring-self-managed.md +++ b/deploy-manage/monitor/stack-monitoring/elasticsearch-monitoring-self-managed.md @@ -1,5 +1,5 @@ --- -navigation_title: "Elasticsearch self-managed" +navigation_title: "Self-managed: {{es}}" mapped_urls: - https://www.elastic.co/guide/en/elasticsearch/reference/current/monitoring-production.html - https://www.elastic.co/guide/en/elasticsearch/reference/current/secure-monitoring.html @@ -8,20 +8,27 @@ applies_to: self: all --- -# Elasticsearch monitoring self-managed +# Enable stack monitoring for {{es}} on a self-managed cluster -% What needs to be done: Refine +The {{stack}} {{monitor-features}} consist of two components: an agent that you install on each {{es}} and Logstash node, and a Monitoring UI in {{kib}}. The monitoring agent collects and indexes metrics from the nodes and you visualize the data through the Monitoring dashboards in {{kib}}. The agent can index data on the same {{es}} cluster, or send it to an external monitoring cluster. -% Scope notes: Consider refine vs lift-and-shift. The title needs to be updated at least. +## Requirements -% Use migrated content from existing pages that map to this page: +To use the {{monitor-features}} with the {{security-features}} enabled, you need to [set up {{kib}} to work with the {{security-features}}](/deploy-manage/security.md) and create at least one user for the Monitoring UI. If you are using an external monitoring cluster, you also need to configure a user for the monitoring agent and configure the agent to use the appropriate credentials when communicating with the monitoring cluster. -% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/monitoring-production.md -% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/secure-monitoring.md +## Collection methods -⚠️ **This page is a work in progress.** ⚠️ +You can collect monitoring and logging data in the following ways: -The documentation team is working to combine content pulled from the following pages: +* [Collect monitoring data with Elastic Agent](/deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-elastic-agent.md) +* [Collect monitoring data with Metricbeat](/deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-metricbeat.md) +* [Collect log data with Filebeat](/deploy-manage/monitor/stack-monitoring/collecting-log-data-with-filebeat.md) -* [/raw-migrated-files/elasticsearch/elasticsearch-reference/monitoring-production.md](/raw-migrated-files/elasticsearch/elasticsearch-reference/monitoring-production.md) -* [/raw-migrated-files/elasticsearch/elasticsearch-reference/secure-monitoring.md](/raw-migrated-files/elasticsearch/elasticsearch-reference/secure-monitoring.md) \ No newline at end of file +If you're building a production environment, then you should send monitoring data to a separate monitoring cluster so that historical data is available even when the nodes you are monitoring are not. To learn how to store monitoring data in a separate cluster, refer to [](/deploy-manage/monitor/stack-monitoring/es-self-monitoring-prod.md). + +### Legacy collection methods + +You can also monitor your stack using legacy {{es}} monitoring features. This method is deprecated and should not be used. To learn more, refer to [](/deploy-manage/monitor/stack-monitoring/es-legacy-collection-methods.md). + +::::{include} _snippets/legacy-warning.md +:::: \ No newline at end of file diff --git a/deploy-manage/monitor/stack-monitoring/es-legacy-collection-methods.md b/deploy-manage/monitor/stack-monitoring/es-legacy-collection-methods.md index 546caa3e66..4ac84607d5 100644 --- a/deploy-manage/monitor/stack-monitoring/es-legacy-collection-methods.md +++ b/deploy-manage/monitor/stack-monitoring/es-legacy-collection-methods.md @@ -13,9 +13,7 @@ applies_to: # Legacy collection methods [collecting-monitoring-data] -::::{admonition} Deprecated in 7.16. -:class: warning - +::::{admonition} Deprecated in 7.16 Using the {{es}} Monitoring plugin to collect and ship monitoring data is deprecated. {{agent}} and {{metricbeat}} are the recommended methods for collecting and shipping monitoring data to a monitoring cluster. If you previously configured legacy collection methods, you should migrate to using [{{agent}}](collecting-monitoring-data-with-elastic-agent.md) or [{{metricbeat}}](collecting-monitoring-data-with-metricbeat.md) collection methods. :::: diff --git a/deploy-manage/monitor/stack-monitoring/es-local-exporter.md b/deploy-manage/monitor/stack-monitoring/es-local-exporter.md index a392e3fe33..744193d63a 100644 --- a/deploy-manage/monitor/stack-monitoring/es-local-exporter.md +++ b/deploy-manage/monitor/stack-monitoring/es-local-exporter.md @@ -8,13 +8,8 @@ applies_to: # Local exporters [local-exporter] -::::{important} -{{agent}} and {{metricbeat}} are the recommended methods for collecting and shipping monitoring data to a monitoring cluster. - -If you have previously configured legacy collection methods, you should migrate to using [{{agent}}](collecting-monitoring-data-with-elastic-agent.md) or [{{metricbeat}}](collecting-monitoring-data-with-metricbeat.md) collection. Do not use legacy collection alongside other collection methods. - -:::: - +:::{include} _snippets/legacy-warning.md +::: The `local` exporter is the default exporter in {{monitoring}}. It routes data back into the same (local) cluster. In other words, it uses the production cluster as the monitoring cluster. For example: diff --git a/deploy-manage/monitor/stack-monitoring/es-monitoring-collectors.md b/deploy-manage/monitor/stack-monitoring/es-monitoring-collectors.md index 9bfe497909..ff816bedaa 100644 --- a/deploy-manage/monitor/stack-monitoring/es-monitoring-collectors.md +++ b/deploy-manage/monitor/stack-monitoring/es-monitoring-collectors.md @@ -9,13 +9,8 @@ applies_to: # Collectors [es-monitoring-collectors] -::::{important} -{{agent}} and {{metricbeat}} are the recommended methods for collecting and shipping monitoring data to a monitoring cluster. - -If you have previously configured legacy collection methods, you should migrate to using [{{agent}}](collecting-monitoring-data-with-elastic-agent.md) or [{{metricbeat}}](collecting-monitoring-data-with-metricbeat.md) collection. Do not use legacy collection alongside other collection methods. - -:::: - +:::{include} _snippets/legacy-warning.md +::: Collectors, as their name implies, collect things. Each collector runs once for each collection interval to obtain data from the public APIs in {{es}} and {{xpack}} that it chooses to monitor. When the data collection is finished, the data is handed in bulk to the [exporters](es-monitoring-exporters.md) to be sent to the monitoring clusters. Regardless of the number of exporters, each collector only runs once per collection interval. diff --git a/deploy-manage/monitor/stack-monitoring/es-monitoring-exporters.md b/deploy-manage/monitor/stack-monitoring/es-monitoring-exporters.md index d15f010a97..cba50e865d 100644 --- a/deploy-manage/monitor/stack-monitoring/es-monitoring-exporters.md +++ b/deploy-manage/monitor/stack-monitoring/es-monitoring-exporters.md @@ -9,13 +9,8 @@ applies_to: # Exporters [es-monitoring-exporters] -::::{important} -{{agent}} and {{metricbeat}} are the recommended methods for collecting and shipping monitoring data to a monitoring cluster. - -If you have previously configured legacy collection methods, you should migrate to using [{{agent}}](collecting-monitoring-data-with-elastic-agent.md) or [{{metricbeat}}](collecting-monitoring-data-with-metricbeat.md) collection. Do not use legacy collection alongside other collection methods. - -:::: - +:::{include} _snippets/legacy-warning.md +::: The purpose of exporters is to take data collected from any Elastic Stack source and route it to the monitoring cluster. It is possible to configure more than one exporter, but the general and default setup is to use a single exporter. diff --git a/deploy-manage/monitor/stack-monitoring/es-pause-export.md b/deploy-manage/monitor/stack-monitoring/es-pause-export.md index a9f66531a2..03937ef6ec 100644 --- a/deploy-manage/monitor/stack-monitoring/es-pause-export.md +++ b/deploy-manage/monitor/stack-monitoring/es-pause-export.md @@ -9,6 +9,9 @@ applies_to: # Pausing data collection [pause-export] +:::{include} _snippets/legacy-warning.md +::: + To stop generating {{monitoring}} data in {{es}}, disable data collection: ```yaml diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/monitoring-production.md b/deploy-manage/monitor/stack-monitoring/es-self-monitoring-prod.md similarity index 63% rename from raw-migrated-files/elasticsearch/elasticsearch-reference/monitoring-production.md rename to deploy-manage/monitor/stack-monitoring/es-self-monitoring-prod.md index bfb85a994d..657de84557 100644 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/monitoring-production.md +++ b/deploy-manage/monitor/stack-monitoring/es-self-monitoring-prod.md @@ -1,16 +1,18 @@ +--- +mapped_urls: + - https://www.elastic.co/guide/en/elasticsearch/reference/current/monitoring-production.html +applies_to: + deployment: + self: all +--- + # Monitoring in a production environment [monitoring-production] In production, you should send monitoring data to a separate *monitoring cluster* so that historical data is available even when the nodes you are monitoring are not. -::::{important} -{{agent}} and {{metricbeat}} are the recommended methods for collecting and shipping monitoring data to a monitoring cluster. - -If you have previously configured legacy collection methods, you should migrate to using [{{agent}}](../../../deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-elastic-agent.md) or [{{metricbeat}}](../../../deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-metricbeat.md) collection. Do not use legacy collection alongside other collection methods. - -:::: +If you have at least a Gold subscription, using a dedicated monitoring cluster also enables you to monitor multiple clusters from a central location. - -If you have at least a Gold Subscription, using a dedicated monitoring cluster also enables you to monitor multiple clusters from a central location. +## Set up your monitoring cluster and {{kib}} instance To store monitoring data in a separate cluster: @@ -62,30 +64,7 @@ To store monitoring data in a separate cluster: Alternatively, use the `remote_monitoring_user` [built-in user](../../../deploy-manage/users-roles/cluster-or-deployment-auth/built-in-users.md). -2. Configure your production cluster to collect data and send it to the monitoring cluster: - - * [{{agent}} collection methods](../../../deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-elastic-agent.md) - * [{{metricbeat}} collection methods](../../../deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-metricbeat.md) - * [Legacy collection methods](../../../deploy-manage/monitor/stack-monitoring/es-legacy-collection-methods.md) - -3. (Optional) [Configure {{ls}} to collect data and send it to the monitoring cluster](logstash://reference/monitoring-logstash-legacy.md). -4. (Optional) Configure the {{beats}} to collect data and send it to the monitoring cluster. Skip this step for {{beats}} that are managed by {{agent}}. - - * [Auditbeat](beats://reference/auditbeat/monitoring.md) - * [Filebeat](beats://reference/filebeat/monitoring.md) - * [Heartbeat](beats://reference/heartbeat/monitoring.md) - * [Metricbeat](beats://reference/metricbeat/monitoring.md) - * [Packetbeat](beats://reference/packetbeat/monitoring.md) - * [Winlogbeat](beats://reference/winlogbeat/monitoring.md) - -5. (Optional) [Configure APM Server monitoring](/solutions/observability/apps/monitor-apm-server.md) -6. (Optional) Configure {{kib}} to collect data and send it to the monitoring cluster: - - * [{{agent}} collection methods](../../../deploy-manage/monitor/stack-monitoring/kibana-monitoring-elastic-agent.md) - * [{{metricbeat}} collection methods](../../../deploy-manage/monitor/stack-monitoring/kibana-monitoring-metricbeat.md) - * [Legacy collection methods](../../../deploy-manage/monitor/stack-monitoring/kibana-monitoring-legacy.md) - -7. (Optional) Create a dedicated {{kib}} instance for monitoring, rather than using a single {{kib}} instance to access both your production cluster and monitoring cluster. +2. (Optional) Create a dedicated {{kib}} instance for monitoring, rather than using a single {{kib}} instance to access both your production cluster and monitoring cluster. ::::{note} If you log in to {{kib}} using SAML, Kerberos, PKI, OpenID Connect, or token authentication providers, a dedicated {{kib}} instance is **required**. The security tokens that are used in these contexts are cluster-specific; therefore you cannot use a single {{kib}} instance to connect to both production and monitoring clusters. @@ -94,5 +73,10 @@ To store monitoring data in a separate cluster: 1. (Optional) Disable the collection of monitoring data in this {{kib}} instance. Set the `xpack.monitoring.kibana.collection.enabled` setting to `false` in the `kibana.yml` file. For more information about this setting, see [Monitoring settings in {{kib}}](kibana://reference/configuration-reference/monitoring-settings.md). -8. [Configure {{kib}} to retrieve and display the monitoring data](../../../deploy-manage/monitor/stack-monitoring/kibana-monitoring-data.md). +## Send data to your cluster + +After your monitoring cluster is set up, configure your {{stack}} components to send data to the cluster. Refer to [](/deploy-manage/monitor/stack-monitoring.md) to learn about the available methods for each component. + +## Configure {{kib}} to retrieve and display the monitoring data +After data is being shipped to your monitoring cluster, you can [configure {{kib}} to retrieve and display the monitoring data](../../../deploy-manage/monitor/stack-monitoring/kibana-monitoring-data.md). \ No newline at end of file diff --git a/deploy-manage/monitor/stack-monitoring/k8s_audit_logging.md b/deploy-manage/monitor/stack-monitoring/k8s_audit_logging.md deleted file mode 100644 index 638cc8517f..0000000000 --- a/deploy-manage/monitor/stack-monitoring/k8s_audit_logging.md +++ /dev/null @@ -1,48 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s_audit_logging.html -applies_to: - deployment: - eck: all ---- - -# Audit logging [k8s_audit_logging] - -Audit logs are collected and shipped to the monitoring cluster referenced in the `monitoring.logs` section when audit logging is enabled (it is disabled by default). - -```yaml -apiVersion: elasticsearch.k8s.elastic.co/v1 -kind: Elasticsearch -spec: - monitoring: - metrics: - elasticsearchRefs: - - name: monitoring - namespace: observability - logs: - elasticsearchRefs: - - name: monitoring - namespace: observability - nodeSets: - - name: default - config: - # https://www.elastic.co/guide/en/elasticsearch/reference/current/enable-audit-logging.html - xpack.security.audit.enabled: true ---- -apiVersion: kibana.k8s.elastic.co/v1 -kind: Kibana -spec: - monitoring: - metrics: - elasticsearchRefs: - - name: monitoring - namespace: observability - logs: - elasticsearchRefs: - - name: monitoring - namespace: observability - config: - # https://www.elastic.co/guide/en/kibana/current/xpack-security-audit-logging.html - xpack.security.audit.enabled: true -``` - diff --git a/deploy-manage/monitor/stack-monitoring/k8s_connect_to_an_external_monitoring_elasticsearch_cluster.md b/deploy-manage/monitor/stack-monitoring/k8s_connect_to_an_external_monitoring_elasticsearch_cluster.md deleted file mode 100644 index 71497acb7f..0000000000 --- a/deploy-manage/monitor/stack-monitoring/k8s_connect_to_an_external_monitoring_elasticsearch_cluster.md +++ /dev/null @@ -1,61 +0,0 @@ ---- -navigation_title: "Connect to an external cluster" -mapped_pages: - - https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s_connect_to_an_external_monitoring_elasticsearch_cluster.html -applies_to: - deployment: - eck: all ---- - -# Connect to an external monitoring Elasticsearch cluster [k8s_connect_to_an_external_monitoring_elasticsearch_cluster] - -If you want to connect to a monitoring Elasticsearch cluster not managed by ECK, you can reference a Secret instead of an Elastisearch cluster in the `monitoring` section through the `secretName` attribute: - -```yaml -apiVersion: elasticsearch.k8s.elastic.co/v1 -kind: Elasticsearch -metadata: - name: monitored-sample - namespace: production -spec: - version: 8.16.1 - monitoring: - metrics: - elasticsearchRefs: - - secretName: monitoring-metrics-es-ref <1> - logs: - elasticsearchRefs: - - name: monitoring-logs - namespace: observability <2> - serviceName: monitoring-logs-es-coordinating-nodes <2> - nodeSets: - - name: default - count: 1 - config: - node.store.allow_mmap: false -``` - -1. The `secretName` and `name` attributes are mutually exclusive, you have to choose one or the other. -2. The `namespace` and `serviceName` attributes can only be used in conjunction with `name`, not with `secretName`. - - -The referenced Secret must contain the following connection information: - -* `url`: the URL to reach the Elasticsearch cluster -* `username`: the username of the user to be authenticated to the Elasticsearch cluster -* `password`: the password of the user to be authenticated to the Elasticsearch cluster -* `ca.crt`: the contents of the CA certificate in PEM format to secure communication to the Elasticsearch cluster (optional) - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: monitoring-metrics-es-ref -stringData: - url: https://mon1.es.abcd-42.xyz.elastic-cloud.com:9243 - username: monitoring-user - password: REDACTED -``` - -The user referenced in the Secret must have been created beforehand. - diff --git a/deploy-manage/monitor/stack-monitoring/k8s_how_it_works.md b/deploy-manage/monitor/stack-monitoring/k8s_how_it_works.md deleted file mode 100644 index e96b713807..0000000000 --- a/deploy-manage/monitor/stack-monitoring/k8s_how_it_works.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s_how_it_works.html -applies_to: - deployment: - eck: all ---- - -# How it works [k8s_how_it_works] - -In the background, Metricbeat and Filebeat are deployed as sidecar containers in the same Pod as Elasticsearch and Kibana. - -Metricbeat is used to collect monitoring metrics and Filebeat to monitor the Elasticsearch log files and collect log events. - -The two Beats are configured to ship data directly to the monitoring cluster(s) using HTTPS and dedicated Elastic users managed by ECK. - diff --git a/deploy-manage/monitor/stack-monitoring/k8s_override_the_beats_pod_template.md b/deploy-manage/monitor/stack-monitoring/k8s_override_the_beats_pod_template.md deleted file mode 100644 index 118d4b2dee..0000000000 --- a/deploy-manage/monitor/stack-monitoring/k8s_override_the_beats_pod_template.md +++ /dev/null @@ -1,38 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s_override_the_beats_pod_template.html -applies_to: - deployment: - eck: all ---- - -# Override the Beats Pod Template [k8s_override_the_beats_pod_template] - -You can customize the Filebeat and Metricbeat containers through the Pod template. Your configuration is merged with the values of the default Pod template that ECK uses. - -```yaml -apiVersion: elasticsearch.k8s.elastic.co/v1 -kind: Elasticsearch -spec: - monitoring: - metrics: - elasticsearchRef: - name: monitoring - namespace: observability - logs: - elasticsearchRef: - name: monitoring - namespace: observability - nodeSets: - - name: default - podTemplate: - spec: - containers: - - name: metricbeat - env: - - foo: bar - - name: filebeat - env: - - foo: bar -``` - diff --git a/deploy-manage/monitor/stack-monitoring/k8s_when_to_use_it.md b/deploy-manage/monitor/stack-monitoring/k8s_when_to_use_it.md deleted file mode 100644 index b774e0dd2c..0000000000 --- a/deploy-manage/monitor/stack-monitoring/k8s_when_to_use_it.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s_when_to_use_it.html -applies_to: - deployment: - eck: all ---- - -# When to use it [k8s_when_to_use_it] - -This feature is a good solution if you need to monitor your Elastic applications in restricted Kubernetes environments where you cannot grant advanced permissions: - -* to Metricbeat to allow queriying the k8s API -* to Filebeat to deploy a privileged DaemonSet - -However, for maximum efficiency and minimising resource consumption, or advanced use cases that require specific Beats configurations, you can deploy a standalone Metricbeat Deployment and a Filebeat Daemonset. Check the [Beats configuration Examples](/deploy-manage/deploy/cloud-on-k8s/configuration-examples-beats.md) for more information. - diff --git a/deploy-manage/monitor/stack-monitoring/kibana-monitoring-data.md b/deploy-manage/monitor/stack-monitoring/kibana-monitoring-data.md index a07b9f5709..0b2959484d 100644 --- a/deploy-manage/monitor/stack-monitoring/kibana-monitoring-data.md +++ b/deploy-manage/monitor/stack-monitoring/kibana-monitoring-data.md @@ -1,5 +1,4 @@ --- -navigation_title: "View monitoring data" mapped_pages: - https://www.elastic.co/guide/en/kibana/current/monitoring-data.html applies_to: @@ -10,20 +9,26 @@ applies_to: self: all --- - -% hola -# View monitoring data [monitoring-data] +# Access monitoring data in {{kib}} [monitoring-data] After you collect monitoring data for one or more products in the {{stack}}, you can configure {{kib}} to retrieve that information and display it in on the **Stack Monitoring** page. At a minimum, you must have monitoring data for the {{es}} production cluster. Once that data exists, {{kib}} can display monitoring data for other products in the cluster. +In {{ece}} and {{ech}}, this configuration is performed automatically. Skip to [View monitoring data in {{kib}}](#view-monitoring-data-in-kibana). + ::::{tip} If you use a separate monitoring cluster to store the monitoring data, it is strongly recommended that you use a separate {{kib}} instance to view it. If you log in to {{kib}} using SAML, Kerberos, PKI, OpenID Connect, or token authentication providers, a dedicated {{kib}} instance is **required**. The security tokens that are used in these contexts are cluster-specific, therefore you cannot use a single {{kib}} instance to connect to both production and monitoring clusters. For more information about the recommended configuration, see [Monitoring overview](../stack-monitoring.md). :::: +## Configure {{kib}} to consume monitoring data +```{applies_to} +deployment: + eck: + self: +``` 1. Identify where to retrieve monitoring data from. @@ -44,37 +49,50 @@ If you use a separate monitoring cluster to store the monitoring data, it is str 2. Add the `monitoring.ui.elasticsearch.username` and `monitoring.ui.elasticsearch.password` settings in the `kibana.yml` file. If these settings are omitted, {{kib}} uses the `elasticsearch.username` and `elasticsearch.password` setting values. For more information, see [Configuring security in {{kib}}](../../security.md). -4. (Optional) Configure {{kib}} to encrypt communications between the {{kib}} server and the monitoring cluster. See [*Encrypt TLS communications in {{kib}}*](/deploy-manage/security/set-up-basic-security-plus-https.md#encrypt-kibana-http). +4. (Optional) If you're using a self-managed cluster, then optionally configure {{kib}} to encrypt communications between the {{kib}} server and the monitoring cluster. See [Encrypt TLS communications in {{kib}}](/deploy-manage/security/set-up-basic-security-plus-https.md#encrypt-kibana-http). 5. If the Elastic {{security-features}} are enabled on the {{kib}} server, only users that have the authority to access {{kib}} indices and to read the monitoring indices can use the monitoring dashboards. + Create users that have the `monitoring_user` and `kibana_admin` [built-in roles](../../users-roles/cluster-or-deployment-auth/built-in-roles.md). If you created a new role with read privileges on `metrics-*` indices, also assign that role to the users. + ::::{note} These users must exist on the monitoring cluster. If you are accessing a remote monitoring cluster, you must use credentials that are valid on both the {{kib}} server and the monitoring cluster. :::: +## View monitoring data in {{kib}} [view-monitoring-data-in-kibana] - 1. Create users that have the `monitoring_user` and `kibana_admin` [built-in roles](../../users-roles/cluster-or-deployment-auth/built-in-roles.md). If you created a new role with read privileges on `metrics-*` indices, also assign that role to the users. +:::::{tab-set} +::::{tab-item} In ECK and self-managed -6. Open {{kib}} in your web browser. +1. Open the {{kib}} monitoring instance in your web browser. By default, if you are running {{kib}} locally, go to `http://localhost:5601/`. If the Elastic {{security-features}} are enabled, log in. -7. Go to the **Stack Monitoring** page using the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). +2. Go to the **Stack Monitoring** page using the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). If data collection is disabled, you are prompted to turn on data collection. If {{es}} {{security-features}} are enabled, you must have `manage` cluster privileges to turn on data collection. - ::::{note} - If you are using a separate monitoring cluster, you do not need to turn on data collection. The dashboards appear when there is data in the monitoring cluster. - :::: +:::{note} +If you are using a separate monitoring cluster, you do not need to turn on data collection. The dashboards appear when there is data in the monitoring cluster. +::: +:::: +::::{tab-item} In ECH and ECE +:::{include} /deploy-manage/monitor/stack-monitoring/_snippets/cloud-monitoring-access.md +::: +:::: +::::: -You’ll see cluster alerts that require your attention and a summary of the available monitoring metrics for {{es}}, Logstash, {{kib}}, and Beats. To view additional information, click the Overview, Nodes, Indices, or Instances links. See [Stack Monitoring](../monitoring-data/visualizing-monitoring-data.md). +On the **Stack Monitoring** page, you’ll see cluster alerts that require your attention and a summary of the available monitoring metrics for {{es}}, Logstash, {{kib}}, and Beats. To view additional information, click the **Overview**, **Nodes**, **Indices**, or **Instances** links. For more information about these metrics, refer to [](../monitoring-data/visualizing-monitoring-data.md). For information about configuring alerts for these metrics, refer to [](/deploy-manage/monitor/monitoring-data/configure-stack-monitoring-alerts.md). -:::{image} ../../../images/kibana-monitoring-dashboard.png +:::{image} /images/kibana-monitoring-dashboard.png :alt: Monitoring dashboard :screenshot: ::: If you encounter problems, see [Troubleshooting monitoring](../monitoring-data/monitor-troubleshooting.md). +:::{tip} +If you're using {{ech}} or {{ece}}, then you can also get a direct link to the relevant **Stack Monitoring** page from the **Deployments** > **Logs and metrics** page. [Learn more](/deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md#access-kibana-monitoring). +:::: \ No newline at end of file diff --git a/deploy-manage/monitor/stack-monitoring/kibana-monitoring-elastic-agent.md b/deploy-manage/monitor/stack-monitoring/kibana-monitoring-elastic-agent.md index 842c3fd9c8..9d00dfdbe4 100644 --- a/deploy-manage/monitor/stack-monitoring/kibana-monitoring-elastic-agent.md +++ b/deploy-manage/monitor/stack-monitoring/kibana-monitoring-elastic-agent.md @@ -11,15 +11,13 @@ applies_to: # Collect monitoring data with Elastic Agent [monitoring-elastic-agent] +You can use {{agent}} to collect data about {{kib}} and ship it to the monitoring cluster. -In 8.5 and later, you can use {{agent}} to collect data about {{kib}} and ship it to the monitoring cluster, rather than [using {{metricbeat}}](/deploy-manage/monitor/stack-monitoring/kibana-monitoring-metricbeat.md) or routing data through the production cluster as described in [Legacy collection methods](/deploy-manage/monitor/stack-monitoring/kibana-monitoring-legacy.md). - -To learn about monitoring in general, see [Monitor a cluster](../../monitor.md). - +To learn about monitoring in general, refer to [](/deploy-manage/monitor/stack-monitoring.md). ## Prerequisites [_prerequisites] -* Set up {{es}} monitoring and optionally create a monitoring cluster as described in the [{{es}} monitoring documentation](elasticsearch-monitoring-self-managed.md). +* [Set up {{es}} monitoring](/deploy-manage/monitor/stack-monitoring/elasticsearch-monitoring-self-managed.md) and optionally [create a monitoring cluster](/deploy-manage/monitor/stack-monitoring/es-self-monitoring-prod.md). * Create a user on the production cluster that has the `remote_monitoring_collector` [built-in role](../../users-roles/cluster-or-deployment-auth/built-in-roles.md). @@ -47,12 +45,12 @@ To collect {{kib}} monitoring data, add a {{kib}} integration to an {{agent}} an * Under **Collect Kibana metrics**, make sure the hosts setting points to your Kibana host URLs. By default, the integration collects {{kib}} monitoring metrics from `localhost:5601`. If that host and port number are not correct, update the `hosts` setting. If you configured {{kib}} to use encrypted communications, you must access it via HTTPS. For example, use a `hosts` setting like `https://localhost:5601`. * If the Elastic {{security-features}} are enabled, expand **Advanced options** under the Hosts setting and enter the username and password of a user that has the `remote_monitoring_collector` role. -6. Choose where to add the integration policy. Click **New hosts*** to add it to new agent policy or ***Existing hosts** to add it to an existing agent policy. +6. Choose where to add the integration policy. Click **New hosts** to add it to new agent policy or **Existing hosts** to add it to an existing agent policy. 7. Click **Save and continue**. This step takes a minute or two to complete. When it’s done, you’ll have an agent policy that contains an integration for collecting monitoring data from {{kib}}. 8. If an {{agent}} is already assigned to the policy and deployed to the host where {{kib}} is running, you’re done. Otherwise, you need to deploy an {{agent}}. To deploy an {{agent}}: - 1. Go to **{{fleet}} → Agents***, then click ***Add agent**. + 1. Go to **{{fleet}} > Agents**, then click **Add agent**. 2. Follow the steps in the **Add agent** flyout to download, install, and enroll the {{agent}}. Make sure you choose the agent policy you created earlier. 9. Wait a minute or two until incoming data is confirmed. -10. [View the monitoring data in {{kib}}](/deploy-manage/monitor/monitoring-data.md). +10. [View the monitoring data in {{kib}}](/deploy-manage/monitor/stack-monitoring/kibana-monitoring-data.md). diff --git a/deploy-manage/monitor/stack-monitoring/kibana-monitoring-legacy.md b/deploy-manage/monitor/stack-monitoring/kibana-monitoring-legacy.md index 99881adc9f..6a70604b73 100644 --- a/deploy-manage/monitor/stack-monitoring/kibana-monitoring-legacy.md +++ b/deploy-manage/monitor/stack-monitoring/kibana-monitoring-legacy.md @@ -20,14 +20,14 @@ If you enable the Elastic {{monitor-features}} in your cluster, you can optional If you have previously configured legacy collection methods, you should migrate to using {{agent}} or {{metricbeat}} collection. Do not use legacy collection alongside other collection methods. -For more information, refer to [Collect monitoring data with {{agent}}](kibana-monitoring-elastic-agent.md) and [Collect monitoring data with {{metricbeat}}](kibana-monitoring-metricbeat.md). +For more information, refer to [](kibana-monitoring-elastic-agent.md) and [](kibana-monitoring-metricbeat.md). :::: The following method involves sending the metrics to the production cluster, which ultimately routes them to the monitoring cluster. -To learn about monitoring in general, see [Monitor a cluster](../../monitor.md). +To learn about monitoring in general, refer to [](/deploy-manage/monitor/stack-monitoring.md). 1. Set the `xpack.monitoring.collection.enabled` setting to `true` on each node in the production cluster. By default, it is is disabled (`false`). diff --git a/deploy-manage/monitor/stack-monitoring/kibana-monitoring-metricbeat.md b/deploy-manage/monitor/stack-monitoring/kibana-monitoring-metricbeat.md index 3009d7b636..7a23905e34 100644 --- a/deploy-manage/monitor/stack-monitoring/kibana-monitoring-metricbeat.md +++ b/deploy-manage/monitor/stack-monitoring/kibana-monitoring-metricbeat.md @@ -12,15 +12,17 @@ applies_to: # Collect monitoring data with Metricbeat [monitoring-metricbeat] -In 6.4 and later, you can use {{metricbeat}} to collect data about {{kib}} and ship it to the monitoring cluster, rather than routing it through the production cluster as described in [Legacy collection methods](/deploy-manage/monitor/stack-monitoring/kibana-monitoring-legacy.md). +Yu can use {{metricbeat}} to collect data about {{kib}} and ship it to the monitoring cluster. -:::{image} ../../../images/kibana-metricbeat.png +To learn about monitoring in general, refer to [](/deploy-manage/monitor/stack-monitoring.md). + +:::{image} /images/kibana-metricbeat.png :alt: Example monitoring architecture +:width: 450px ::: -To learn about monitoring in general, see [Monitor a cluster](../../monitor.md). -1. Disable the default collection of {{kib}} monitoring metrics.
+1. Disable the default collection of {{kib}} monitoring metrics. Add the following setting in the {{kib}} configuration file (`kibana.yml`): @@ -52,9 +54,11 @@ To learn about monitoring in general, see [Monitor a cluster](../../monitor.md). For example, you can use the following APIs to review and change this setting: - ```js + ```console GET _cluster/settings + ``` + ```console PUT _cluster/settings { "persistent": { @@ -63,7 +67,7 @@ To learn about monitoring in general, see [Monitor a cluster](../../monitor.md). } ``` - For more information, see [Monitoring settings in {{es}}](elasticsearch://reference/elasticsearch/configuration-reference/monitoring-settings.md) and [Cluster update settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). + For more information, see [Monitoring settings in {{es}}](elasticsearch://reference/elasticsearch/configuration-reference/monitoring-settings.md) and [the Cluster update settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). 4. [Install {{metricbeat}}](beats://reference/metricbeat/metricbeat-installation-configuration.md) on the same server as {{kib}}. 5. Enable the {{kib}} {{xpack}} module in {{metricbeat}}.
@@ -111,6 +115,8 @@ To learn about monitoring in general, see [Monitor a cluster](../../monitor.md). ::::{tip} In production environments, we strongly recommend using a separate cluster (referred to as the *monitoring cluster*) to store the data. Using a separate monitoring cluster prevents production cluster outages from impacting your ability to access your monitoring data. It also prevents monitoring activities from impacting the performance of your production cluster. + + For more information, refer to [](/deploy-manage/monitor/stack-monitoring/es-self-monitoring-prod.md). :::: @@ -145,5 +151,5 @@ To learn about monitoring in general, see [Monitor a cluster](../../monitor.md). For more information about these configuration options, see [Configure the {{es}} output](beats://reference/metricbeat/elasticsearch-output.md). 9. [Start {{metricbeat}}](beats://reference/metricbeat/metricbeat-starting.md). -10. [View the monitoring data in {{kib}}](/deploy-manage/monitor/monitoring-data.md). +10. [View the monitoring data in {{kib}}](/deploy-manage/monitor/stack-monitoring/kibana-monitoring-data.md). diff --git a/deploy-manage/monitor/stack-monitoring/kibana-monitoring-self-managed.md b/deploy-manage/monitor/stack-monitoring/kibana-monitoring-self-managed.md index 4f975d6a58..50778f39c5 100644 --- a/deploy-manage/monitor/stack-monitoring/kibana-monitoring-self-managed.md +++ b/deploy-manage/monitor/stack-monitoring/kibana-monitoring-self-managed.md @@ -1,5 +1,5 @@ --- -navigation_title: "Kibana self-managed" +navigation_title: "Self-managed: {{kib}}" mapped_pages: - https://www.elastic.co/guide/en/kibana/current/configuring-monitoring.html applies_to: @@ -19,7 +19,7 @@ If you enable the {{monitor-features}} in your cluster, there are a few methods You can also use {{kib}} to [visualize monitoring data from across the {{stack}}](kibana-monitoring-data.md). -To learn about monitoring in general, see [Monitor a cluster](../../monitor.md). +To learn about monitoring in general, refer to [](/deploy-manage/monitor/stack-monitoring.md). diff --git a/deploy-manage/security/logging-configuration/enabling-audit-logs.md b/deploy-manage/security/logging-configuration/enabling-audit-logs.md index 5c2a5d55a0..f6a73999e4 100644 --- a/deploy-manage/security/logging-configuration/enabling-audit-logs.md +++ b/deploy-manage/security/logging-configuration/enabling-audit-logs.md @@ -30,6 +30,11 @@ In orchestrated deployments, audit logs must be shipped to a monitoring deployme When audit logging is enabled, security events are persisted to a dedicated `_audit.json` file on the host’s file system, on every cluster node. For more information, refer to [{{es}} logfile audit output](./logfile-audit-output.md). +## Before you begin + +* If you're using ECE or ECH, then you need to [enable monitoring and logging](/deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md) on your deployment. As part of enabling monitoring and logging, you choose a location where logs will be delivered. +* In production environments, consider creating a separate monitoring cluster that can consume the logs. + ## Enable audit logging [enable-audit-logging-procedure] To enable {{es}} or {{kib}} audit logs, configure `xpack.security.audit.enabled` to `true` in **all {{es}} or {{kib}} nodes**, then restart the nodes to apply the changes. For detailed instructions, select your deployment type: @@ -53,11 +58,11 @@ Audit logs are disabled by default and must be explicitly enabled. 1. Set `xpack.security.audit.enabled` to `true` in `kibana.yml`. 2. Restart {{kib}}. +To learn how to consume these logs in an {{es}} cluster, refer to [](/deploy-manage/monitor/stack-monitoring/collecting-log-data-with-filebeat.md). ::::: :::::{tab-item} Elastic Cloud Hosted - To enable audit logging in an {{ech}} deployment: 1. Log in to the [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body). diff --git a/deploy-manage/toc.yml b/deploy-manage/toc.yml index 4582f3d70a..065de2aed6 100644 --- a/deploy-manage/toc.yml +++ b/deploy-manage/toc.yml @@ -80,6 +80,7 @@ toc: children: - file: deploy/elastic-cloud/switch-from-apm-to-integrations-server-payload.md - file: deploy/elastic-cloud/find-cloud-id.md + - file: deploy/elastic-cloud/ec-vcpu-boost-instance.md - file: deploy/elastic-cloud/change-hardware.md - file: deploy/elastic-cloud/manage-deployments-using-elastic-cloud-api.md - file: deploy/elastic-cloud/keep-track-of-deployment-activity.md @@ -657,12 +658,14 @@ toc: children: - file: monitor/autoops/ec-autoops-how-to-access.md - file: monitor/autoops/ec-autoops-events.md - - file: monitor/autoops/ec-autoops-overview-view.md - - file: monitor/autoops/ec-autoops-deployment-view.md - - file: monitor/autoops/ec-autoops-nodes-view.md - - file: monitor/autoops/ec-autoops-index-view.md - - file: monitor/autoops/ec-autoops-shards-view.md - - file: monitor/autoops/ec-autoops-template-optimizer.md + - file: monitor/autoops/views.md + children: + - file: monitor/autoops/ec-autoops-overview-view.md + - file: monitor/autoops/ec-autoops-deployment-view.md + - file: monitor/autoops/ec-autoops-nodes-view.md + - file: monitor/autoops/ec-autoops-index-view.md + - file: monitor/autoops/ec-autoops-shards-view.md + - file: monitor/autoops/ec-autoops-template-optimizer.md - file: monitor/autoops/ec-autoops-notifications-settings.md - file: monitor/autoops/ec-autoops-event-settings.md - file: monitor/autoops/ec-autoops-dismiss-event.md @@ -670,22 +673,14 @@ toc: - file: monitor/autoops/ec-autoops-faq.md - file: monitor/stack-monitoring.md children: - - file: monitor/stack-monitoring/elastic-cloud-stack-monitoring.md + - file: monitor/stack-monitoring/ece-ech-stack-monitoring.md - file: monitor/stack-monitoring/eck-stack-monitoring.md - children: - - file: monitor/stack-monitoring/k8s_connect_to_an_external_monitoring_elasticsearch_cluster.md - - file: monitor/stack-monitoring/k8s_when_to_use_it.md - - file: monitor/stack-monitoring/k8s_how_it_works.md - - file: monitor/stack-monitoring/k8s_audit_logging.md - - file: monitor/stack-monitoring/k8s_override_the_beats_pod_template.md - - file: monitor/stack-monitoring/ece-stack-monitoring.md - children: - - file: monitor/stack-monitoring/ece-restrictions-monitoring.md - file: monitor/stack-monitoring/elasticsearch-monitoring-self-managed.md children: - file: monitor/stack-monitoring/collecting-monitoring-data-with-elastic-agent.md - file: monitor/stack-monitoring/collecting-monitoring-data-with-metricbeat.md - file: monitor/stack-monitoring/collecting-log-data-with-filebeat.md + - file: monitor/stack-monitoring/es-self-monitoring-prod.md - file: monitor/stack-monitoring/es-legacy-collection-methods.md children: - file: monitor/stack-monitoring/es-monitoring-collectors.md @@ -697,42 +692,37 @@ toc: children: - file: monitor/stack-monitoring/kibana-monitoring-elastic-agent.md - file: monitor/stack-monitoring/kibana-monitoring-metricbeat.md - - file: monitor/stack-monitoring/kibana-monitoring-data.md - - file: monitor/stack-monitoring/kibana-monitoring-legacy.md - - file: monitor/orchestrators.md - children: - - file: monitor/orchestrators/eck-metrics-configuration.md - children: - - file: monitor/orchestrators/k8s-enabling-metrics-endpoint.md - - file: monitor/orchestrators/k8s-securing-metrics-endpoint.md - - file: monitor/orchestrators/k8s-prometheus-requirements.md - - file: monitor/orchestrators/ece-platform-monitoring.md - children: - - file: monitor/orchestrators/ece-monitoring-ece-access.md - - file: monitor/orchestrators/ece-proxy-log-fields.md - - file: monitor/orchestrators/ece-monitoring-ece-set-retention.md - - file: monitor/monitoring-data.md - children: + - file: monitor/stack-monitoring/kibana-monitoring-legacy.md + - file: monitor/stack-monitoring/kibana-monitoring-data.md - file: monitor/monitoring-data/visualizing-monitoring-data.md children: - file: monitor/monitoring-data/beats-page.md - file: monitor/monitoring-data/elasticsearch-metrics.md - - file: monitor/monitoring-data/kibana-alerts.md - file: monitor/monitoring-data/kibana-page.md - file: monitor/monitoring-data/logstash-page.md - file: monitor/monitoring-data/monitor-troubleshooting.md - - file: monitor/monitoring-data/access-performance-metrics-on-elastic-cloud.md - children: - - file: monitor/monitoring-data/ec-saas-metrics-accessing.md - children: - - file: monitor/monitoring-data/ec-memory-pressure.md - - file: monitor/monitoring-data/ec-vcpu-boost-instance.md - file: monitor/monitoring-data/configure-stack-monitoring-alerts.md - file: monitor/monitoring-data/configuring-data-streamsindices-for-monitoring.md children: - file: monitor/monitoring-data/config-monitoring-data-streams-elastic-agent.md - file: monitor/monitoring-data/config-monitoring-data-streams-metricbeat-8.md - file: monitor/monitoring-data/config-monitoring-indices-metricbeat-7-internal-collection.md + - file: monitor/cloud-health-perf.md + children: + - file: monitor/access-performance-metrics-on-elastic-cloud.md + - file: monitor/ec-memory-pressure.md + - file: monitor/orchestrators.md + children: + - file: monitor/orchestrators/eck-metrics-configuration.md + children: + - file: monitor/orchestrators/k8s-enabling-metrics-endpoint.md + - file: monitor/orchestrators/k8s-securing-metrics-endpoint.md + - file: monitor/orchestrators/k8s-prometheus-requirements.md + - file: monitor/orchestrators/ece-platform-monitoring.md + children: + - file: monitor/orchestrators/ece-monitoring-ece-access.md + - file: monitor/orchestrators/ece-proxy-log-fields.md + - file: monitor/orchestrators/ece-monitoring-ece-set-retention.md - file: monitor/kibana-task-manager-health-monitoring.md - file: monitor/logging-configuration.md children: diff --git a/deploy-manage/users-roles/cloud-enterprise-orchestrator/manage-users-roles.md b/deploy-manage/users-roles/cloud-enterprise-orchestrator/manage-users-roles.md index 8d32f1784f..eb4f307c2b 100644 --- a/deploy-manage/users-roles/cloud-enterprise-orchestrator/manage-users-roles.md +++ b/deploy-manage/users-roles/cloud-enterprise-orchestrator/manage-users-roles.md @@ -64,7 +64,7 @@ We strongly recommend using three availability zones with at least 1 GB {{es}} n 1. [Log into the Cloud UI](/deploy-manage/deploy/cloud-enterprise/log-into-cloud-ui.md). 2. Go to **Deployments** a select the **security-cluster**. 3. Configure regular [snapshots](/deploy-manage/tools/snapshot-and-restore/create-snapshots.md) of the security deployment. This is critical if you plan to create any native users. -4. Optional: [Enable monitoring](/deploy-manage/monitor/stack-monitoring/ece-stack-monitoring.md) on the security deployment to a dedicated monitoring deployment. +4. Optional: [Enable monitoring](/deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md) on the security deployment to a dedicated monitoring deployment. If you have authentication issues, you check out the security deployment {{es}} [logs](/deploy-manage/monitor/logging-configuration.md). diff --git a/explore-analyze/alerts-cases/alerts/create-manage-rules.md b/explore-analyze/alerts-cases/alerts/create-manage-rules.md index 064741f556..a7504e957d 100644 --- a/explore-analyze/alerts-cases/alerts/create-manage-rules.md +++ b/explore-analyze/alerts-cases/alerts/create-manage-rules.md @@ -166,7 +166,7 @@ Some rule types cannot be exported through this interface: **Security rules** can be imported and exported using the [Security UI](../../../solutions/security/detect-and-alert/manage-detection-rules.md#import-export-rules-ui). -**Stack monitoring rules** are [automatically created](../../../deploy-manage/monitor/monitoring-data/kibana-alerts.md) for you and therefore cannot be managed in **Saved Objects**. +**Stack monitoring rules** are [automatically created](../../../deploy-manage/monitor/monitoring-data/configure-stack-monitoring-alerts.md) for you and therefore cannot be managed in **Saved Objects**. :::: diff --git a/explore-analyze/alerts-cases/alerts/maintenance-windows.md b/explore-analyze/alerts-cases/alerts/maintenance-windows.md index 0f1ffdfd8c..83ea7882df 100644 --- a/explore-analyze/alerts-cases/alerts/maintenance-windows.md +++ b/explore-analyze/alerts-cases/alerts/maintenance-windows.md @@ -57,7 +57,7 @@ If you turn on **Filter alerts**, you can use KQL to filter the alerts affected ::::{note} * You can select only a single category when you turn on filters. -* Some rules are not affected by maintenance window filters because their alerts do not contain requisite data. In particular, [{{stack-monitor-app}}](../../../deploy-manage/monitor/monitoring-data/kibana-alerts.md), [tracking containment](../../../explore-analyze/alerts-cases/alerts/geo-alerting.md), [{{anomaly-jobs}} health](../../../explore-analyze/machine-learning/anomaly-detection/ml-configuring-alerts.md), and [transform health](../../../explore-analyze/transforms/transform-alerts.md) rules are not affected by the filters. +* Some rules are not affected by maintenance window filters because their alerts do not contain requisite data. In particular, [{{stack-monitor-app}}](../../../deploy-manage/monitor/monitoring-data/configure-stack-monitoring-alerts.md), [tracking containment](../../../explore-analyze/alerts-cases/alerts/geo-alerting.md), [{{anomaly-jobs}} health](../../../explore-analyze/machine-learning/anomaly-detection/ml-configuring-alerts.md), and [transform health](../../../explore-analyze/transforms/transform-alerts.md) rules are not affected by the filters. :::: diff --git a/manage-data/lifecycle/data-tiers.md b/manage-data/lifecycle/data-tiers.md index 885cd48be3..8ba1965602 100644 --- a/manage-data/lifecycle/data-tiers.md +++ b/manage-data/lifecycle/data-tiers.md @@ -115,7 +115,7 @@ To add a data tier to an existing deployment: Disabling a data tier, attempting to scale nodes down in size, reducing availability zones, or reverting an [autoscaling](/deploy-manage/autoscaling.md) change can all result in cluster instability, cluster inaccessibility, and even data corruption or loss in extreme cases. To avoid this, especially for [production environments](/deploy-manage/production-guidance.md), and in addition to making configuration changes to your indices and ILM as described on this page: -* Review the disk size, CPU, JVM memory pressure, and other [performance metrics](/deploy-manage/monitor/monitoring-data/access-performance-metrics-on-elastic-cloud.md) of your deployment **before** attempting to perform the scaling down action. +* Review the disk size, CPU, JVM memory pressure, and other [performance metrics](/deploy-manage/monitor/access-performance-metrics-on-elastic-cloud.md) of your deployment **before** attempting to perform the scaling down action. * Make sure that you have enough resources and [availability zones](/deploy-manage/production-guidance/availability-and-resilience.md) to handle your workloads after scaling down. * Check that your [deployment hardware profile](/deploy-manage/deploy/elastic-cloud/ec-change-hardware-profile.md) (for {{ech}}) or [deployment template](/deploy-manage/deploy/cloud-enterprise/configure-deployment-templates.md) (for {{ece}}) is correct for your business use case. For example, if you need to scale due to CPU pressure increases and are using a *Storage Optimized* hardware profile, consider switching to a *CPU Optimized* configuration instead. diff --git a/raw-migrated-files/cloud/cloud-heroku/ech-add-user-settings.md b/raw-migrated-files/cloud/cloud-heroku/ech-add-user-settings.md index f9ed62a51c..861c66aa49 100644 --- a/raw-migrated-files/cloud/cloud-heroku/ech-add-user-settings.md +++ b/raw-migrated-files/cloud/cloud-heroku/ech-add-user-settings.md @@ -275,7 +275,7 @@ The following audit settings are supported: : A list of action names or wildcards. The specified policy will not print audit events for actions matching these values. ::::{note} -To enable auditing you must first [enable deployment logging](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md). +To enable auditing you must first [enable deployment logging](../../../deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md). :::: diff --git a/raw-migrated-files/cloud/cloud-heroku/ech-enable-logging-and-monitoring.md b/raw-migrated-files/cloud/cloud-heroku/ech-enable-logging-and-monitoring.md deleted file mode 100644 index d3c48bc78e..0000000000 --- a/raw-migrated-files/cloud/cloud-heroku/ech-enable-logging-and-monitoring.md +++ /dev/null @@ -1,229 +0,0 @@ -# Enable logging and monitoring [ech-enable-logging-and-monitoring] - -The deployment logging and monitoring feature lets you monitor your deployment in [Kibana](../../../get-started/the-stack.md) by shipping logs and metrics to a monitoring deployment. You can: - -* View your deployment’s health and performance in real time and analyze past cluster, index, and node metrics. -* View your deployment’s logs to debug issues, discover slow queries, surface deprecations, and analyze access to your deployment. - -Monitoring consists of two components: - -* A monitoring and logging agent that is installed on each node in your deployment. The agents collect and index metrics to {{es}}, either on the same deployment or by sending logs and metrics to an external monitoring deployment. Elasticsearch Add-On for Heroku manages the installation and configuration of the monitoring agent for you, and you should not modify any of the settings. -* The stack monitoring application in Kibana that visualizes the monitoring metrics through a dashboard and the logs application that allows you to search and analyze deployment logs. - -The steps in this section cover only the enablement of the monitoring and logging features in Elasticsearch Add-On for Heroku. For more information on how to use the monitoring features, refer to [Monitor a cluster](../../../deploy-manage/monitor.md). - - -### Before you begin [ech-logging-and-monitoring-limitations] - -Some limitations apply when you use monitoring on Elasticsearch Add-On for Heroku. To learn more, check the monitoring [restrictions and limitations](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md). - - -### Monitoring for production use [ech-logging-and-monitoring-production] - -For production use, you should send your deployment logs and metrics to a dedicated monitoring deployment. Monitoring indexes logs and metrics into {{es}} and these indexes consume storage, memory, and CPU cycles like any other index. By using a separate monitoring deployment, you avoid affecting your other production deployments and can view the logs and metrics even when a production deployment is unavailable. - -How many monitoring deployments you use depends on your requirements: - -* You can ship logs and metrics for many deployments to a single monitoring deployment, if your business requirements permit it. -* Although monitoring will work with a deployment running a single node, you need a minimum of three monitoring nodes to make monitoring highly available. -* You might need to create dedicated monitoring deployments for isolation purposes in some cases. For example: - - * If you have many deployments and some of them are much larger than others, creating separate monitoring deployments prevents a large deployment from potentially affecting monitoring performance for smaller deployments. - * If you need to silo {{es}} data for different business departments. Deployments that have been configured to ship logs and metrics to a target monitoring deployment have access to indexing data and can manage monitoring index templates, which is addressed by creating separate monitoring deployments. - - -Logs and metrics that get sent to a dedicated monitoring {{es}} deployment [may not be cleaned up automatically](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ech-logging-and-monitoring-retention) and might require some additional steps to remove excess data periodically. - - -### Retention of monitoring daily indices [ech-logging-and-monitoring-retention] - - -#### Stack versions 8.0 and above [ech-logging-and-monitoring-retention-8] - -When you enable monitoring in Elasticsearch Add-On for Heroku, your monitoring indices are retained for a certain period by default. After the retention period has passed, the monitoring indices are deleted automatically. The retention period is configured in the `.monitoring-8-ilm-policy` index lifecycle policy. To view or edit the policy open {{kib}} **Stack management > Data > Index Lifecycle Policies**. - - -### Sending monitoring data to itself (self monitoring) [ech-logging-and-monitoring-retention-self-monitoring] - -$$$ech-logging-and-monitoring-retention-7$$$ -When you enable self-monitoring in Elasticsearch Add-On for Heroku, your monitoring indices are retained for a certain period by default. After the retention period has passed, the monitoring indices are deleted automatically. Monitoring data is retained for three days by default or as specified by the [`xpack.monitoring.history.duration` user setting](../../../deploy-manage/deploy/elastic-cloud/edit-stack-settings.md#xpack-monitoring-history-duration). - -To retain monitoring indices as is without deleting them automatically, you must disable the [cleaner service](../../../deploy-manage/monitor/stack-monitoring/es-local-exporter.md#local-exporter-cleaner) by adding a disabled local exporter in your cluster settings. - -For example - -```sh -PUT /_cluster/settings -{ - "persistent": { - "xpack.monitoring.exporters.__no-default-local__": { - "type": "local", - "enabled": false - } - } -} -``` - - -### Sending monitoring data to a dedicated monitoring deployment [ech-logging-and-monitoring-retention-dedicated-monitoring] - -When [monitoring for production use](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ech-logging-and-monitoring-production), where you configure your deployments **to send monitoring data to a dedicated monitoring deployment** for indexing, this retention period does not apply. Monitoring indices on a dedicated monitoring deployment are retained until you remove them. There are three options open to you: - -* To enable the automatic deletion of monitoring indices from dedicated monitoring deployments, [enable monitoring](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ech-enable-logging-and-monitoring-steps) on your dedicated monitoring deployment in Elasticsearch Add-On for Heroku to send monitoring data to itself. When an {{es}} deployment sends monitoring data to itself, all monitoring indices are deleted automatically after the retention period, regardless of the origin of the monitoring data. -* Alternatively, you can enable the cleaner service on the monitoring deployment by creating a local exporter. You can define the retention period at the same time. - - For example - - ```sh - PUT _cluster/settings - { - "persistent": { - "xpack" : { - "monitoring" : { - "exporters" : { - "found-user-defined" : { - "type" : "local", - "enabled" : "true" - } - }, - "history" : { - "duration" : "24h" - } - } - } - } - } - ``` - -* To retain monitoring indices on a dedicated monitoring deployment as is without deleting them automatically, no additional steps are required other than making sure that you do not enable the monitoring deployment to send monitoring data to itself. You should also monitor the deployment for disk space usage and upgrade your deployment periodically, if necessary. - - -### Retention of logging indices [ech-logging-and-monitoring-log-retention] - -An ILM policy is pre-configured to manage log retention. The policy can be adjusted according to your requirements. - - -### Index management [ech-logging-and-monitoring-index-management-ilm] - -When sending monitoring data to a deployment, you can configure [Index Lifecycle Management (ILM)](../../../manage-data/lifecycle/index-lifecycle-management.md) to manage retention of your monitoring and logging indices. When sending logs to a deployment, an ILM policy is pre-configured to manage log retention and the policy can be customized to your needs. - - -### Enable logging and monitoring [ech-enable-logging-and-monitoring-steps] - -Elasticsearch Add-On for Heroku manages the installation and configuration of the monitoring agent for you. When you enable monitoring on a deployment, you are configuring where the monitoring agent for your current deployment should send its logs and metrics. - -To enable monitoring on your deployment: - -1. Log in to the [Elasticsearch Add-On for Heroku console](https://cloud.elastic.co?page=docs&placement=docs-body). -2. On the **Deployments** page, select your deployment. - - Narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list. - -3. From your deployment menu, go to the **Logs and metrics** page. -4. Select **Enable**. -5. Choose where to send your logs and metrics. Select **Save**. - - If a deployment is not listed, make sure that it is running a compatible version. The monitoring deployment and production deployment must be on the same major version, cloud provider, and region. - - ::::{tip} - Remember to send logs and metrics for production deployments to a dedicated monitoring deployment, so that your production deployments are not impacted by the overhead of indexing and storing monitoring data. A dedicated monitoring deployment also gives you more control over the retention period for monitoring data. - :::: - - -::::{note} -Enabling logs and monitoring may trigger a plan change on your deployment. You can monitor the plan change progress from the deployment’s **Activity** page. -:::: - - -::::{note} -Enabling logs and monitoring requires some extra resource on a deployment. For production systems, we recommend sizing deployments with logs and monitoring enabled to at least 4 GB of RAM. -:::: - - - -### Access the monitoring application in Kibana [ech-access-kibana-monitoring] - -With monitoring enabled for your deployment, you can access the [logs](https://www.elastic.co/guide/en/kibana/current/observability.html) and [stack monitoring](../../../deploy-manage/monitor/monitoring-data/visualizing-monitoring-data.md) through Kibana. - -1. Log in to the [Elasticsearch Add-On for Heroku console](https://cloud.elastic.co?page=docs&placement=docs-body). -2. On the **Deployments** page, select your deployment. - - Narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list. - -3. From your deployment menu, go to the **Logs and Metrics** page. -4. Select the corresponding **View** button to check the logs or metrics data. - -Alternatively, you can access logs and metrics directly on the Kibana **Logs** and **Stack Monitoring** pages in the target monitoring deployment. You can also create an `elastic-cloud-logs-*` data view (formerly *index pattern*) to view your deployment’s logs in the Kibana **Discover** tab. Several fields are available for you to view logs based on key details, such as the source deployment: - -| Field | Description | Example value | -| --- | --- | --- | -| `service.id` | The ID of the deployment that generated the log | `6ff525333d2844539663f3b1da6c04b6` | -| `service.name` | The name of the deployment that generated the log | `My Production Deployment` | -| `cloud.availability_zone` | The availability zone in which the instance that generated the log is deployed | `ap-northeast-1d` | -| `service.node.name` | The ID of the instance that generated the log | `instance-0000000008` | -| `service.type` | The type of instance that generated the log | `elasticsearch` | -| `service.version` | The version of the stack resource that generated the log | `8.13.1` | - - -### Logging features [ech-extra-logging-features] - -When shipping logs to a monitoring deployment there are more logging features available to you. These features include: - - -#### For {{es}}: [ech-extra-logging-features-elasticsearch] - -* [Audit logging](../../../deploy-manage/security/logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment -* [Slow query and index logging](elasticsearch://reference/elasticsearch/index-settings/slow-log.md) - helps find and debug slow queries and indexing -* Verbose logging - helps debug stack issues by increasing component logs - -After you’ve enabled log delivery on your deployment, you can [add the Elasticsearch user settings](../../../deploy-manage/deploy/elastic-cloud/edit-stack-settings.md) to enable these features. - - -#### For Kibana: [ech-extra-logging-features-kibana] - -* [Audit logging](../../../deploy-manage/security/logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment - -After you’ve enabled log delivery on your deployment, you can [add the Kibana user settings](../../../deploy-manage/deploy/elastic-cloud/edit-stack-settings.md) to enable this feature. - - -### Other components [ech-extra-logging-features-enterprise-search] - -Enabling log collection also supports collecting and indexing the following types of logs from other components in your deployments: - -**APM** - -* apm*.log* - -**Fleet and Elastic Agent** - -* fleet-server-json.log-* -* elastic-agent-json.log-* - -The ˆ*ˆ indicates that we also index the archived files of each type of log. - -Check the respective product documentation for more information about the logging capabilities of each product. - - -## Metrics features [ech-extra-metrics-features] - -With logging and monitoring enabled for a deployment, metrics are collected for Elasticsearch, Kibana, and APM with Fleet Server. - - -#### Enabling Elasticsearch/Kibana audit logs on your deployment [ech-enable-audit-logs] - -Audit logs are useful for tracking security events on your {{es}} and/or {{kib}} clusters. To enable {{es}} audit logs on your deployment: - -1. Log in to the [Elasticsearch Add-On for Heroku console](https://cloud.elastic.co?page=docs&placement=docs-body). -2. On the **Deployments** page, select your deployment. - - Narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list. - -3. From your deployment menu, go to the **Edit** page. -4. To enable audit logs in {{es}}, in the **Elasticsearch** section select **Manage user settings and extensions**. For deployments with existing user settings, you may have to expand the **Edit elasticsearch.yml** caret for each node instead. -5. To enable audit logs in {{kib}}, in the **Kibana** section select **Edit user settings**. For deployments with existing user settings, you may have to expand the **Edit kibana.yml** caret instead. -6. Add `xpack.security.audit.enabled: true` to the user settings. -7. Select **Save changes**. - -A plan change will run on your deployment. When it finishes, audit logs will be delivered to your monitoring deployment. - - diff --git a/raw-migrated-files/cloud/cloud-heroku/ech-manage-apm-settings.md b/raw-migrated-files/cloud/cloud-heroku/ech-manage-apm-settings.md index e9ba2386ec..0273036530 100644 --- a/raw-migrated-files/cloud/cloud-heroku/ech-manage-apm-settings.md +++ b/raw-migrated-files/cloud/cloud-heroku/ech-manage-apm-settings.md @@ -361,7 +361,7 @@ Allow anonymous access only for specified agents and/or services. This is primar : The period after which to log the internal metrics. Defaults to *30s*. ::::{note} -To change logging settings you must first [enable deployment logging](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md). +To change logging settings you must first [enable deployment logging](../../../deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md). :::: diff --git a/raw-migrated-files/cloud/cloud-heroku/ech-manage-kibana-settings.md b/raw-migrated-files/cloud/cloud-heroku/ech-manage-kibana-settings.md index 059c643555..45212b9174 100644 --- a/raw-migrated-files/cloud/cloud-heroku/ech-manage-kibana-settings.md +++ b/raw-migrated-files/cloud/cloud-heroku/ech-manage-kibana-settings.md @@ -784,7 +784,7 @@ If search latency in {{es}} is sufficiently high, such as if you are using cross ## Logging and audit settings [echlogging_and_audit_settings] ::::{note} -To change logging settings or to enable auditing you must first [enable deployment logging](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md). +To change logging settings or to enable auditing you must first [enable deployment logging](../../../deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md). :::: diff --git a/raw-migrated-files/cloud/cloud-heroku/ech-monitoring-setup.md b/raw-migrated-files/cloud/cloud-heroku/ech-monitoring-setup.md deleted file mode 100644 index e41fd678f2..0000000000 --- a/raw-migrated-files/cloud/cloud-heroku/ech-monitoring-setup.md +++ /dev/null @@ -1,152 +0,0 @@ -# How to set up monitoring [ech-monitoring-setup] - -Learn how to configure your deployments for observability, which includes metric and log collection, troubleshooting views, and cluster alerts to automate performance monitoring. - -These steps are helpful to set yourself up for success by making monitoring readily available and to automate alerts for the future. - - -## Before you begin [echbefore_you_begin_12] - -As you manage, monitor, and troubleshoot your deployment, make sure you have an understanding of the [shared responsibilities](https://www.elastic.co/cloud/shared-responsibility) between Elastic and yourself, so you know what you need to do to keep your deployments running smoothly. - -You may also consider subscribing to incident notices reported on the Elasticsearch Add-On for Heroku [status page](https://status.elastic.co). - - -## Enable logs and metrics [echenable_logs_and_metrics] - -After you have created a new deployment, you should enable shipping logs and metrics to a monitoring deployment: - -1. Go to the [Deployments](https://cloud.elastic.co/deployments) page in {{ecloud}}. -2. Find your deployment and go to the **Logs and Metrics** page. -3. Select **Enable**. -4. Choose where to send your logs and metrics. - - ::::{important} - Anything used for production should go to a separate deployment you create only for monitoring. For development or testing, you can send monitoring data to the same deployment. Check [Enable logging and monitoring](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ech-logging-and-monitoring-production). - :::: - -5. Select **Save**. - -Optionally, turn on [audit logging](elasticsearch://reference/elasticsearch/configuration-reference/auding-settings.md) to capture security-related events, such as authentication failures, refused connections, and data-access events through the proxy. To turn on audit logging, [edit your deployment’s elasticsearch.yml file](../../../deploy-manage/deploy/elastic-cloud/edit-stack-settings.md) to add these lines: - -```sh -xpack.security.audit.enabled: true -# xpack.security.audit.logfile.events.include: _all -# xpack.security.audit.logfile.events.emit_request_body: true -``` - -The last two lines are commented out for now but left there as placeholders to easily turn on in the future. These two settings generate large logs, but can be helpful to turn on temporarily when troubleshooting traffic request bodies. - - -## View your deployment health [echview_your_deployment_health] - -From the monitoring deployment, you can now view your deployment’s health in Kibana using [Stack Monitoring](/deploy-manage/monitor/monitoring-data/visualizing-monitoring-data.md): - -1. Select the **Kibana** link for your monitoring deployment. -2. From the app menu or the search bar, open **Stack Monitoring**. - - ::::{tip} - Stack monitoring comes with many out-of-the-box rules, but you need to enable them when prompted. - :::: - - -To learn more about what [Elasticsearch monitoring metrics](/deploy-manage/monitor/monitoring-data/elasticsearch-metrics.md) are available, take a look at the different tabs. For example: - -* The **Overview** tab includes information about the search and indexing performance of Elasticsearch and also provides log entries. -* The **Nodes** tab can help you monitor cluster CPU performance, JVM strain, and free disk space. - -:::{image} ../../../images/cloud-heroku-ec-ce-monitoring-nodes.png -:alt: Node tab in Kibana under Stack Monitoring -::: - -Some [performance metrics](/deploy-manage/monitor/monitoring-data/access-performance-metrics-on-elastic-cloud.md) are also available directly in the [Elasticsearch Add-On for Heroku console](https://cloud.elastic.co?page=docs&placement=docs-body) and don’t require looking at your monitoring deployment. If you’re ever in a rush to determine if there is a performance problem, you can get a quick overview by going to the **Performance** page from your deployment menu: - -:::{image} ../../../images/cloud-heroku-ec-ce-monitoring-performance.png -:alt: Performance page of the Elastic Cloud console -::: - - -## Check the logs [ech-check-logs] - -If you suspect a performance issue, you can use your monitoring deployment to investigate what is going in Kibana: - -* Through **Observability** > **Logs** > **Stream**: This page shows errors in real-time and is part of the same logs Elastic Support reviews when a deployment experiences issues. Check [Tail log files](/solutions/observability/logs/logs-stream.md). -* Through **Discover**: This page is a good option for investigating widespread historical patterns. Check [Discover](/explore-analyze/discover.md). - - Discover requires a quick setup in Kibana: - - 1. Go to **Stack Management** > **Data Views** (formerly *Index Patterns*). - 2. Create a [data view](/explore-analyze/find-and-organize/data-views.md) for `elastic-cloud-logs*` and set **Timestamp field** to `@timestamp`: - - :::{image} ../../../images/cloud-heroku-ec-ce-monitoring-logs.png - :alt: Create data view example in Kibana - ::: - - -Navigate to the **Discover** or **Stream** pages to check if you’ve misconfigured your SAML authentication setup by filtering for `WARN` and `ERROR` level logs and viewing the specific `message` fields, for example: - -:::{image} ../../../images/cloud-heroku-ec-ce-monitoring-saml.png -:alt: Log error in Stream page showing failed SAML authentication -::: - -You can also use this page to test how problematic proxy traffic requests show up in audit logs. To illustrate, create a spurious test request from the [Elasticsearch API console](../../../deploy-manage/deploy/elastic-cloud/ech-api-console.md): - -:::{image} ../../../images/cloud-heroku-ec-ce-monitoring-api-console.png -:alt: Elasticsearch API console showing a spurious request that fails -::: - -You will get this request reported as a new log. Audit logs do not currently report the HTTP response status code, but they do report a correlating `event.action` column: - -:::{image} ../../../images/cloud-heroku-ec-ce-monitoring-new-log.png -:alt: New log entry that shows failed spurious request issued from the API console -::: - - -## Get notified [echget_notified] - -You should take advantage of the default [Elastic Stack monitoring alerts](/deploy-manage/monitor/monitoring-data/kibana-alerts.md) that are available out-of-the-box. You don’t have to do anything other than enable shipping logs and metrics to have them made available to you (which you did earlier on). - -On top of these default alerts that write to indices you can investigate, you might want to add some custom actions, such as a [connector](kibana://reference/connectors-kibana.md) for Slack notifications. To set up these notifications, you first configure a Slack connector and then append it to the default alerts and actions. From Kibana: - -1. Go to **Stack Management** > **Rules and Connectors** > **Connectors** and create your Slack connector: - - 1. Select **Slack**. - 2. [Create a Slack Webhook URL](kibana://reference/connectors-kibana/slack-action-type.md#configuring-slack) and paste it into the **Webhook URL** field. - 3. Select **Save**. - -2. Go to **Stack Monitoring** and select **Enter setup mode**. -3. Edit an alert rule, such as [CPU usage](/deploy-manage/monitor/monitoring-data/kibana-alerts.md#kibana-alerts-cpu-threshold): - - 1. Select one of the alert rule fields and select **CPU Usage**. - 2. Choose **Edit rule** and scroll down to the bottom of the screen to select **Slack**. - 3. Optional: Set up a customized message that helps you identify what the message is for. - 4. Select **Save**. - - :::{image} ../../../images/cloud-heroku-ec-ce-monitoring-connector-action.png - :alt: Alert rule example showing settings to send a Slack notification based on CPU Usage - ::: - - -Now, when your CPU usage alert goes off, you will also get a Slack notification to investigate if your cluster is experiencing a traffic blip or if you need to scale out. (You can automate the latter with [deployment autoscaling](../../../deploy-manage/autoscaling.md)). - - -## Keep monitoring [echkeep_monitoring] - -As a managed service, {{ecloud}} is here to help you manage the maintenance and upkeep. As part of your responsibilities, you should monitor deployment health on an ongoing basis. There are two main activities to perform: - -* Review the deployment logs -* Act on automated alerts - -When issues come up that you need to troubleshoot, you’ll frequently start with the same queries to determine which rabbit hole to investigate further, such as [`_cluster/health`](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-health) to determine overall deployment health. - -:::{image} ../../../images/cloud-heroku-ec-ce-monitoring-ongoing.png -:alt: Elasticsearch API console showing queries useful for monitoring -::: - -You can run this query and many others from the API consoles available via: - -* **Kibana** > **Dev Tools**. Check [Run Elasticsearch API requests](/explore-analyze/query-filter/tools/console.md). -* **Elastic Cloud** > **Deployment** > **Elasticsearch** > **API Console**. Check [Access the Elasticsearch API console](../../../deploy-manage/deploy/elastic-cloud/ech-api-console.md). - -You can also learn more about the queries you should run for your deployment by reading our blog [Managing and Troubleshooting Elasticsearch Memory](https://www.elastic.co/blog/managing-and-troubleshooting-elasticsearch-memory). - diff --git a/raw-migrated-files/cloud/cloud-heroku/ech-monitoring.md b/raw-migrated-files/cloud/cloud-heroku/ech-monitoring.md deleted file mode 100644 index c50c30e02f..0000000000 --- a/raw-migrated-files/cloud/cloud-heroku/ech-monitoring.md +++ /dev/null @@ -1,121 +0,0 @@ -# Monitoring your deployment [ech-monitoring] - -Keeping on top of the health of your deployments is a key part of the [shared responsibilities](https://www.elastic.co/cloud/shared-responsibility) between Elastic and yourself. Elastic Cloud provides out of the box tools for monitoring the health of your deployment and resolving health issues when they arise. If you are ready to set up a deployment for production use cases, make sure you check the recommendations and best practices for [production readiness](../../../deploy-manage/production-guidance/plan-for-production-elastic-cloud.md). - -A deployment on Elastic Cloud is a combination of an Elasticsearch cluster, a Kibana instance and potentially an APM server instance, and an Integration Server instance. The health of an Elastic Cloud deployment comprises the health of the various components that are part of the deployment. - -The most important of these is the {{es}} cluster, because it is the heart of the system for searching and indexing data. - -This section provides some best practices to help you monitor and understand the ongoing state of your deployments and their resources. - -* [{{es}} cluster health](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ech-es-cluster-health) -* [{{es}} cluster performance](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ech-es-cluster-performance) -* [Health warnings](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ech-es-health-warnings) -* [Preconfigured logs and metrics](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ech-es-health-preconfigured) -* [Dedicated logs and metrics](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ech-es-health-dedicated) -* [Understanding deployment health](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ech-health-best-practices) - - -## {{es}} cluster health [ech-es-cluster-health] - -Health is reported based on the following areas: Shard availability, master node health, Snapshot Lifecycle Management (SLM), Index Lifecycle Management (ILM), and repository integrity. - -For {{stack}} versions 8.3 and below, the deployment **Health** page is limited. Health issues are displayed in a banner with no details on impacts and troubleshooting steps. Follow [these steps](../../../deploy-manage/upgrade/deployment-or-cluster.md) if you want to perform a version upgrade. - -For {{stack}} versions 8.4 and later, the deployment **Health** page provides detailed information on health issues, impacted areas, and troubleshooting support. - -To view the health for a deployment: - -1. Log in to the [Elasticsearch Add-On for Heroku console](https://cloud.elastic.co?page=docs&placement=docs-body). -2. On the **Deployments** page, select your deployment. -3. In your deployment menu, select **Health**. - -The **Health** page provides the following information: - -* Health issues for Kibana, APM and plan changes are reported in the health banner. -* Health issues for {{es}} clusters are broken down into a table with more details on Severity, Description and Affected capabilities. - -:::{image} ../../../images/cloud-heroku-es-health-page.png -:alt: {{es}} Health page -::: - -**Severity**: A critical issue impacts operations such as search and ingest and should be addressed as soon as possible. Warnings don’t impact the cluster immediately but might lead to more critical issues over time such as a corrupted repository might lead to no backups being available in the future. - -**Description**: For most issues, you can click the description to get more details page on the specific issue and on how to fix it. - -**Affected capabilities**: Each of these areas might impact Search, Ingest, Backups or Deployment Management capabilities. - -You can also search and filter the table based on affected resources, such as indices, repositories, nodes, or SLM policies. Individual issues can be further expanded to get more details and guided troubleshooting. - -:::{image} ../../../images/cloud-heroku-es-health-page-table.png -:alt: {{es}} Health page with details and troubleshooting -::: - -For each issue you can either use a troubleshooting link or get a suggestion to contact support, in case you need help. The [troubleshooting documentation](../../../troubleshoot/elasticsearch/elasticsearch.md) for {{es}} provides more details on specific errors. - - -## {{es}} cluster performance [ech-es-cluster-performance] - -The deployment **Health** page does not include information on cluster performance. If you observe issues on search and ingest operations in terms of increased latency or throughput for queries, these might not be directly reported on the **Health** page, unless they are related to shard health or master node availability. The performance page and the out-of-the-box logs allow you to monitor your cluster performance, but for production applications we strongly recommend setting up a dedicated monitoring cluster. Check [Understanding deployment health](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ech-health-best-practices), for more guidelines on how to monitor you cluster performance. - - -## Health warnings [ech-es-health-warnings] - -In the normal course of using your Elasticsearch Add-On for Heroku deployments, health warnings and errors might appear from time to time. Following are the most common scenarios and methods to resolve them. - -Health warning messages -: Health warning messages will sometimes appear on the main page for one of your deployments, as well as on the **Logs and metrics** page. - - A single warning is rarely cause for concern, as often it just reflects ongoing, routine maintenance activity occurring on the Elasticsearch Add-On for Heroku platform. - - -Configuration change failures -: In more serious cases, your deployment may be unable to restart. The failure can be due to a variety of causes, the most frequent of these being invalid secure settings, expired plugins or bundles, or out of memory errors. The problem is often not detected until an attempt is made to restart the deployment following a routine configuration change, such as a deployment resizing. - -Out of memory errors -: Out of memory errors (OOMs) may occur during your deployment’s normal operations, and these can have a very negative impact on performance. Common causes of memory shortages are oversharding, data retention oversights, and the overall request volume. - - On your deployment page, you can check the [JVM memory pressure indicator]/deploy-manage/monitor/monitoring-data/ec-memory-pressure.md) to get the current memory usage of each node of your deployment. You can also review the [common causes of high JVM memory usage](/deploy-manage/monitor/monitoring-data/ec-memory-pressure.md#ec-memory-pressure-causes) to help diagnose the source of unexpectedly high memory pressure levels. To learn more, check [How does high memory pressure affect performance?](../../../troubleshoot/monitoring/high-memory-pressure.md). - - - -## Preconfigured logs and metrics [ech-es-health-preconfigured] - -In a non-production environment, you may choose to rely on the logs and metrics that are available for your deployment by default. The deployment **Logs and metrics** page displays any current deployment health warnings, and from there you can also view standard log files from the last 24 hours. - -The logs capture any activity related to your deployments, their component resources, snapshotting behavior, and more. You can use the search bar to filter the logs by, for example, a specific instance (`instance-0000000005`), a configuration file (`roles.yml`), an operation type (`snapshot`, `autoscaling`), or a component (`kibana`, `ent-search`). - -In a production environment, we highly recommend storing your logs and metrics in another cluster. This gives you the ability to retain your logs and metrics over longer periods of time and setting custom alerts and watches. - - -## Dedicated logs and metrics [ech-es-health-dedicated] - -In a production environment, it’s important set up dedicated health monitoring in order to retain the logs and metrics that can be used to troubleshoot any health issues in your deployments. In the event of that you need to [contact our support team](../../../deploy-manage/deploy/elastic-cloud/ech-get-help.md), they can use the retained data to help diagnose any problems that you may encounter. - -You have the option of sending logs and metrics to a separate, specialized monitoring deployment, which ensures that they’re available in the event of a deployment outage. The monitoring deployment also gives you access to Kibana’s stack monitoring features, through which you can view health and performance data for all of your deployment resources. - -Check the guide on [how to set up monitoring](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) to learn more. - - -## Understanding deployment health [ech-health-best-practices] - -We’ve compiled some guidelines to help you ensure the health of your deployments over time. These can help you to better understand the available performance metrics, and to make decisions involving performance and high availability. - -[Why is my node(s) unavailable?](../../../troubleshoot/monitoring/unavailable-nodes.md) -: Learn about common symptoms and possible actions that you can take to resolve issues when one or more nodes become unhealthy or unavailable. - -[Why are my shards unavailable?](../../../troubleshoot/monitoring/unavailable-shards.md) -: Provide instructions on how to troubleshoot issues related to unassigned shards. - -[Why is performance degrading over time?](../../../troubleshoot/monitoring/performance.md) -: Address performance degradation on a smaller size Elasticsearch cluster. - -[Is my cluster really highly available?](../../../troubleshoot/monitoring/high-availability.md) -: High availability involves more than setting multiple availability zones (although that’s really important!). Learn how to assess performance and workloads to determine if your deployment has adequate resources to mitigate a potential node failure. - -[How does high memory pressure affect performance?](../../../troubleshoot/monitoring/high-memory-pressure.md) -: Learn about typical memory usage patterns, how to assess when the deployment memory usage levels are problematic, how this impacts performance, and how to resolve memory-related issues. - -[Why are my cluster response times suddenly so much worse?](../../../troubleshoot/monitoring/cluster-response-time.md) -: Learn about the common causes of increased query response times and decreased performance in your deployment. - diff --git a/raw-migrated-files/cloud/cloud-heroku/ech-planning.md b/raw-migrated-files/cloud/cloud-heroku/ech-planning.md index c55030dec0..a4629cf49c 100644 --- a/raw-migrated-files/cloud/cloud-heroku/ech-planning.md +++ b/raw-migrated-files/cloud/cloud-heroku/ech-planning.md @@ -44,7 +44,7 @@ Clusters that only have one master node are not highly available and are at risk ## Do you know when to scale? [ech-workloads] -Knowing how to scale your deployment is critical, especially when unexpected workloads hits. Don’t forget to [check your performance metrics](/deploy-manage/monitor/monitoring-data/access-performance-metrics-on-elastic-cloud.md) to make sure your deployments are healthy and can cope with your workloads. +Knowing how to scale your deployment is critical, especially when unexpected workloads hits. Don’t forget to [check your performance metrics](/deploy-manage/monitor/access-performance-metrics-on-elastic-cloud.md) to make sure your deployments are healthy and can cope with your workloads. Scaling with Elasticsearch Add-On for Heroku is easy: diff --git a/raw-migrated-files/cloud/cloud-heroku/ech-restrictions-monitoring.md b/raw-migrated-files/cloud/cloud-heroku/ech-restrictions-monitoring.md deleted file mode 100644 index 450c36ee77..0000000000 --- a/raw-migrated-files/cloud/cloud-heroku/ech-restrictions-monitoring.md +++ /dev/null @@ -1,7 +0,0 @@ -# Restrictions and limitations [ech-restrictions-monitoring] - -* To avoid compatibility issues, ensure your monitoring cluster and production cluster run on the same {{stack}} version. Monitoring clusters that use 8.x do work with production clusters that use the latest release of 7.x, but this setup should only occur when upgrading clusters to the same version. -* $$$cross-region-monitor$$$ Monitoring across regions is not supported. If you need to move your existing monitoring to the same region, you can do a reindex or create a new deployment and select the snapshot from the old deployment. -* The logs shipped to a monitoring cluster use an ILM managed data stream (elastic-cloud-logs-). If you need to delete indices due to space, do not delete the current is_write_enabled: true index. -* When sending metrics to a dedicated monitoring deployment, the graph for IO Operations Rate(/s) is blank. This is due to the fact that this graph actually contains metrics from of all of the virtualized resources from the provider. - diff --git a/raw-migrated-files/cloud/cloud-heroku/ech-saas-metrics-accessing.md b/raw-migrated-files/cloud/cloud-heroku/ech-saas-metrics-accessing.md deleted file mode 100644 index 73a5e27d76..0000000000 --- a/raw-migrated-files/cloud/cloud-heroku/ech-saas-metrics-accessing.md +++ /dev/null @@ -1,122 +0,0 @@ -# Access performance metrics [ech-saas-metrics-accessing] - -Cluster performance metrics are available directly in the [Elasticsearch Add-On for Heroku console](https://cloud.elastic.co?page=docs&placement=docs-body). The graphs on this page include a subset of Elasticsearch Add-On for Heroku-specific performance metrics. - -For advanced views or production monitoring, [enable logging and monitoring](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md). The monitoring application provides more advanced views for Elasticsearch and JVM metrics, and includes a configurable retention period. - -To access cluster performance metrics: - -1. Log in to the [Elasticsearch Add-On for Heroku console](https://cloud.elastic.co?page=docs&placement=docs-body). -2. On the **Deployments** page, select your deployment. - - Narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list. For example, you might want to select **Is unhealthy** and **Has master problems** to get a short list of deployments that need attention. - -3. From your deployment menu, go to the **Performance** page. - -The following metrics are available: - - -### CPU usage [echcpu_usage] - -:::{image} ../../../images/cloud-heroku-metrics-cpu-usage.png -:alt: Graph showing CPU usage -::: - -Shows the maximum usage of the CPU resources assigned to your Elasticsearch cluster, as a percentage. CPU resources are relative to the size of your cluster, so that a cluster with 32GB of RAM gets assigned twice as many CPU resources as a cluster with 16GB of RAM. All clusters are guaranteed their share of CPU resources, as Elasticsearch Add-On for Heroku infrastructure does not overcommit any resources. CPU credits permit boosting the performance of smaller clusters temporarily, so that CPU usage can exceed 100%. - - -### CPU credits [echcpu_credits] - -:::{image} ../../../images/cloud-heroku-metrics-cpu-credits.png -:alt: Graph showing available CPU credits -::: - -Shows your remaining CPU credits, measured in seconds of CPU time. CPU credits enable the boosting of CPU resources assigned to your cluster to improve performance temporarily when it is needed most. For more details check [How to use vCPU to boost your instance](/deploy-manage/monitor/monitoring-data/ec-vcpu-boost-instance.md). - - -### Number of requests [echnumber_of_requests] - -:::{image} ../../../images/cloud-heroku-metrics-number-of-requests.png -:alt: Graph showing the number of requests -::: - -Shows the number of requests that your cluster receives per second, separated into search requests and requests to index documents. This metric provides a good indicator of the volume of work that your cluster typically handles over time which, together with other performance metrics, helps you determine if your cluster is sized correctly. Also lets you check if there is a sudden increase in the volume of user requests that might explain an increase in response times. - - -### Search response times [echsearch_response_times] - -:::{image} ../../../images/cloud-heroku-metrics-search-response-times.png -:alt: Graph showing search response times -::: - -Indicates the amount of time that it takes for your Elasticsearch cluster to complete a search query, in milliseconds. Response times won’t tell you about the cause of a performance issue, but they are often a first indicator that something is amiss with the performance of your Elasticsearch cluster. - - -### Index response times [echindex_response_times] - -:::{image} ../../../images/cloud-heroku-metrics-index-response-times.png -:alt: Graph showing index response times -::: - -Indicates the amount of time that it takes for your Elasticsearch cluster to complete an indexing operation, in milliseconds. Response times won’t tell you about the cause of a performance issue, but they are often a first indicator that something is amiss with the performance of your Elasticsearch cluster. - - -### Memory pressure per node [echmemory_pressure_per_node] - -:::{image} ../../../images/cloud-heroku-metrics-memory-pressure-per-node.png -:alt: Graph showing memory pressure per node -::: - -Indicates the total memory used by the JVM heap over time. We’ve configured {{es}}'s garbage collector to keep memory usage below 75% for heaps of 8GB or larger. For heaps smaller than 8GB, the threshold is 85%. If memory pressure consistently remains above this threshold, you might need to resize your cluster or reduce memory consumption. Check [how high memory pressure can cause performance issues](../../../troubleshoot/monitoring/high-memory-pressure.md). - - -### GC overhead per node [echgc_overhead_per_node] - -:::{image} ../../../images/cloud-heroku-metrics-gc-overhead-per-node.png -:alt: Graph showing the garbage collection overhead per node -::: - -Indicates the overhead involved in JVM garbage collection to reclaim memory. - - -## Tips for working with performance metrics [echtips_for_working_with_performance_metrics] - -Performance correlates directly with resources assigned to your cluster, and many of these metrics will show some sort of correlation with each other when you are trying to determine the cause of a performance issue. Take a look at some of the scenarios included in this section to learn how you can determine the cause of performance issues. - -It is not uncommon for performance issues on Elasticsearch Add-On for Heroku to be caused by an undersized cluster that cannot cope with the workload it is being asked to handle. If your cluster performance metrics often shows high CPU usage or excessive memory pressure, consider increasing the size of your cluster soon to improve performance. This is especially true for clusters that regularly reach 100% of CPU usage or that suffer out-of-memory failures; it is better to resize your cluster early when it is not yet maxed out than to have to resize a cluster that is already overwhelmed. [Changing the configuration of your cluster](../../../deploy-manage/deploy/elastic-cloud/cloud-hosted.md) may add some overhead if data needs to be migrated to the new nodes, which can increase the load on a cluster further and delay configuration changes. - -To help diagnose high CPU usage you can also use the Elasticsearch [nodes hot threads API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-nodes-hot-threads), which identifies the threads on each node that have the highest CPU usage or that have been executing for a longer than normal period of time. - -::::{tip} -Got an overwhelmed cluster that needs to be upsized? [Try enabling maintenance mode first](https://www.elastic.co/guide/en/cloud-heroku/current/ech-upgrading-v5.html#ech-maintenance-mode-routing). It will likely help with configuration changes. -:::: - - -Work with the metrics shown in **Cluster Performance Metrics** section to help you find the information you need: - -* Hover on any part of a graph to get additional information. For example, hovering on a section of a graph that shows response times reveals the percentile that responses fall into at that point in time: - - :::{image} ../../../images/cloud-heroku-metrics-hover.png - :alt: Hover over the metric graph - ::: - -* Zoom in on a graph by drawing a rectangle to select a specific time window. As you zoom in one metric, other performance metrics change to show data for the same time window. - - :::{image} ../../../images/cloud-heroku-metrics-zoom.png - :alt: Zoom the metric graph - ::: - -* Pan around with ![Pan in a metric graph](../../../images/cloud-heroku-metrics-pan.png "") to make sure that you can get the right parts of a metric graph as you zoom in. -* Reset the metric graph axes with ![Reset the metric graph](../../../images/cloud-heroku-metrics-reset.png ""), which returns the graphs to their original scale. - -Cluster performance metrics are shown per node and are color-coded to indicate which running Elasticsearch instance they belong to. - - -## Cluster restarts after out-of-memory failures [echcluster_restarts_after_out_of_memory_failures] - -For clusters that suffer out-of-memory failures, it can be difficult to determine whether the clusters are in a completely healthy state afterwards. For this reason, Elasticsearch Add-On for Heroku automatically reboots clusters that suffer out-of-memory failures. - -You will receive an email notification to let you know that a restart occurred. For repeated alerts, the emails are aggregated so that you do not receive an excessive number of notifications. Either [resizing your cluster to reduce memory pressure](../../../deploy-manage/deploy/elastic-cloud/configure.md) or reducing the workload that a cluster is being asked to handle can help avoid these cluster restarts. - - - diff --git a/raw-migrated-files/cloud/cloud/ec-add-user-settings.md b/raw-migrated-files/cloud/cloud/ec-add-user-settings.md index 9828a4f416..c6725cee47 100644 --- a/raw-migrated-files/cloud/cloud/ec-add-user-settings.md +++ b/raw-migrated-files/cloud/cloud/ec-add-user-settings.md @@ -275,7 +275,7 @@ The following audit settings are supported: : A list of action names or wildcards. The specified policy will not print audit events for actions matching these values. ::::{note} -To enable auditing you must first [enable deployment logging](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md). +To enable auditing you must first [enable deployment logging](../../../deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md). :::: diff --git a/raw-migrated-files/cloud/cloud/ec-enable-logging-and-monitoring.md b/raw-migrated-files/cloud/cloud/ec-enable-logging-and-monitoring.md deleted file mode 100644 index 1ec417f0d2..0000000000 --- a/raw-migrated-files/cloud/cloud/ec-enable-logging-and-monitoring.md +++ /dev/null @@ -1,237 +0,0 @@ -# Enable logging and monitoring [ec-enable-logging-and-monitoring] - -The deployment logging and monitoring feature lets you monitor your deployment in [Kibana](../../../get-started/the-stack.md) by shipping logs and metrics to a monitoring deployment. You can: - -* View your deployment’s health and performance in real time and analyze past cluster, index, and node metrics. -* View your deployment’s logs to debug issues, discover slow queries, surface deprecations, and analyze access to your deployment. - -Monitoring consists of two components: - -* A monitoring and logging agent that is installed on each node in your deployment. The agents collect and index metrics to {{es}}, either on the same deployment or by sending logs and metrics to an external monitoring deployment. {{ech}} manages the installation and configuration of the monitoring agent for you, and you should not modify any of the settings. -* The stack monitoring application in Kibana that visualizes the monitoring metrics through a dashboard and the logs application that allows you to search and analyze deployment logs. - -The steps in this section cover only the enablement of the monitoring and logging features in {{ech}}. For more information on how to use the monitoring features, refer to [Monitor a cluster](../../../deploy-manage/monitor.md). - - -### Before you begin [ec-logging-and-monitoring-limitations] - -Some limitations apply when you use monitoring on {{ech}}. To learn more, check the monitoring [restrictions and limitations](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ec-restrictions-monitoring). - - -### Monitoring for production use [ec-logging-and-monitoring-production] - -For production use, you should send your deployment logs and metrics to a dedicated monitoring deployment. Monitoring indexes logs and metrics into {{es}} and these indexes consume storage, memory, and CPU cycles like any other index. By using a separate monitoring deployment, you avoid affecting your other production deployments and can view the logs and metrics even when a production deployment is unavailable. - -How many monitoring deployments you use depends on your requirements: - -* You can ship logs and metrics for many deployments to a single monitoring deployment, if your business requirements permit it. -* Although monitoring will work with a deployment running a single node, you need a minimum of three monitoring nodes to make monitoring highly available. -* You might need to create dedicated monitoring deployments for isolation purposes in some cases. For example: - - * If you have many deployments and some of them are much larger than others, creating separate monitoring deployments prevents a large deployment from potentially affecting monitoring performance for smaller deployments. - * If you need to silo {{es}} data for different business departments. Deployments that have been configured to ship logs and metrics to a target monitoring deployment have access to indexing data and can manage monitoring index templates, which is addressed by creating separate monitoring deployments. - - -Logs and metrics that get sent to a dedicated monitoring {{es}} deployment [may not be cleaned up automatically](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ec-logging-and-monitoring-retention) and might require some additional steps to remove excess data periodically. - - -### Retention of monitoring daily indices [ec-logging-and-monitoring-retention] - - -#### Stack versions 8.0 and above [ec-logging-and-monitoring-retention-8] - -When you enable monitoring in {{ech}}, your monitoring indices are retained for a certain period by default. After the retention period has passed, the monitoring indices are deleted automatically. The retention period is configured in the `.monitoring-8-ilm-policy` index lifecycle policy. To view or edit the policy open {{kib}} **Stack management > Data > Index Lifecycle Policies**. - - -### Sending monitoring data to itself (self monitoring) [ec-logging-and-monitoring-retention-self-monitoring] - -$$$ec-logging-and-monitoring-retention-7$$$ -When you enable self-monitoring in {{ech}}, your monitoring indices are retained for a certain period by default. After the retention period has passed, the monitoring indices are deleted automatically. Monitoring data is retained for three days by default or as specified by the [`xpack.monitoring.history.duration` user setting](../../../deploy-manage/deploy/elastic-cloud/edit-stack-settings.md#xpack-monitoring-history-duration). - -To retain monitoring indices as is without deleting them automatically, you must disable the [cleaner service](../../../deploy-manage/monitor/stack-monitoring/es-local-exporter.md#local-exporter-cleaner) by adding a disabled local exporter in your cluster settings. - -For example - -```sh -PUT /_cluster/settings -{ - "persistent": { - "xpack.monitoring.exporters.__no-default-local__": { - "type": "local", - "enabled": false - } - } -} -``` - - -### Sending monitoring data to a dedicated monitoring deployment [ec-logging-and-monitoring-retention-dedicated-monitoring] - -When [monitoring for production use](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ec-logging-and-monitoring-production), where you configure your deployments **to send monitoring data to a dedicated monitoring deployment** for indexing, this retention period does not apply. Monitoring indices on a dedicated monitoring deployment are retained until you remove them. There are three options open to you: - -* To enable the automatic deletion of monitoring indices from dedicated monitoring deployments, [enable monitoring](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ec-enable-logging-and-monitoring-steps) on your dedicated monitoring deployment in {{ech}} to send monitoring data to itself. When an {{es}} deployment sends monitoring data to itself, all monitoring indices are deleted automatically after the retention period, regardless of the origin of the monitoring data. -* Alternatively, you can enable the cleaner service on the monitoring deployment by creating a local exporter. You can define the retention period at the same time. - - For example - - ```sh - PUT _cluster/settings - { - "persistent": { - "xpack" : { - "monitoring" : { - "exporters" : { - "found-user-defined" : { - "type" : "local", - "enabled" : "true" - } - }, - "history" : { - "duration" : "24h" - } - } - } - } - } - ``` - -* To retain monitoring indices on a dedicated monitoring deployment as is without deleting them automatically, no additional steps are required other than making sure that you do not enable the monitoring deployment to send monitoring data to itself. You should also monitor the deployment for disk space usage and upgrade your deployment periodically, if necessary. - - -### Retention of logging indices [ec-logging-and-monitoring-log-retention] - -An ILM policy is pre-configured to manage log retention. The policy can be adjusted according to your requirements. - - -### Index management [ec-logging-and-monitoring-index-management-ilm] - -When sending monitoring data to a deployment, you can configure [Index Lifecycle Management (ILM)](../../../manage-data/lifecycle/index-lifecycle-management.md) to manage retention of your monitoring and logging indices. When sending logs to a deployment, an ILM policy is pre-configured to manage log retention and the policy can be customized to your needs. - - -### Enable logging and monitoring [ec-enable-logging-and-monitoring-steps] - -{{ech}} manages the installation and configuration of the monitoring agent for you. When you enable monitoring on a deployment, you are configuring where the monitoring agent for your current deployment should send its logs and metrics. - -To enable monitoring on your deployment: - -1. Log in to the [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body). -2. Find your deployment on the home page in the **Hosted deployments** card and select **Manage** to access it directly. Or, select **Hosted deployments** to go to the **Deployments** page to view all of your deployments. - - On the **Deployments** page you can narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list. - -3. From your deployment menu, go to the **Logs and metrics** page. -4. Select **Enable**. -5. Choose where to send your logs and metrics. Select **Save**. - - If a deployment is not listed, make sure that it is running a compatible version. The monitoring deployment and production deployment must be on the same major version, cloud provider, and region. - - ::::{tip} - Remember to send logs and metrics for production deployments to a dedicated monitoring deployment, so that your production deployments are not impacted by the overhead of indexing and storing monitoring data. A dedicated monitoring deployment also gives you more control over the retention period for monitoring data. - :::: - - -::::{note} -Enabling logs and monitoring may trigger a plan change on your deployment. You can monitor the plan change progress from the deployment’s **Activity** page. -:::: - - -::::{note} -Enabling logs and monitoring requires some extra resource on a deployment. For production systems, we recommend sizing deployments with logs and monitoring enabled to at least 4 GB of RAM. -:::: - - - -### Access the monitoring application in Kibana [ec-access-kibana-monitoring] - -With monitoring enabled for your deployment, you can access the [logs](https://www.elastic.co/guide/en/kibana/current/observability.html) and [stack monitoring](../../../deploy-manage/monitor/monitoring-data/visualizing-monitoring-data.md) through Kibana. - -1. Log in to the [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body). -2. Find your deployment on the home page in the **Hosted deployments** card and select **Manage** to access it directly. Or, select **Hosted deployments** to go to the **Deployments** page to view all of your deployments. - - On the **Deployments** page you can narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list. - -3. From your deployment menu, go to the **Logs and Metrics** page. -4. Select the corresponding **View** button to check the logs or metrics data. - -Alternatively, you can access logs and metrics directly on the Kibana **Logs** and **Stack Monitoring** pages in the target monitoring deployment. You can also create an `elastic-cloud-logs-*` data view (formerly *index pattern*) to view your deployment’s logs in the Kibana **Discover** tab. Several fields are available for you to view logs based on key details, such as the source deployment: - -| Field | Description | Example value | -| --- | --- | --- | -| `service.id` | The ID of the deployment that generated the log | `6ff525333d2844539663f3b1da6c04b6` | -| `service.name` | The name of the deployment that generated the log | `My Production Deployment` | -| `cloud.availability_zone` | The availability zone in which the instance that generated the log is deployed | `ap-northeast-1d` | -| `service.node.name` | The ID of the instance that generated the log | `instance-0000000008` | -| `service.type` | The type of instance that generated the log | `elasticsearch` | -| `service.version` | The version of the stack resource that generated the log | `8.13.1` | - - -### Logging features [ec-extra-logging-features] - -When shipping logs to a monitoring deployment there are more logging features available to you. These features include: - - -#### For {{es}}: [ec-extra-logging-features-elasticsearch] - -* [Audit logging](../../../deploy-manage/security/logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment -* [Slow query and index logging](elasticsearch://reference/elasticsearch/index-settings/slow-log.md) - helps find and debug slow queries and indexing -* Verbose logging - helps debug stack issues by increasing component logs - -After you’ve enabled log delivery on your deployment, you can [add the Elasticsearch user settings](../../../deploy-manage/deploy/elastic-cloud/edit-stack-settings.md) to enable these features. - - -#### For Kibana: [ec-extra-logging-features-kibana] - -* [Audit logging](../../../deploy-manage/security/logging-configuration/enabling-audit-logs.md) - logs security-related events on your deployment - -After you’ve enabled log delivery on your deployment, you can [add the Kibana user settings](../../../deploy-manage/deploy/elastic-cloud/edit-stack-settings.md) to enable this feature. - - -### Other components [ec-extra-logging-features-enterprise-search] - -Enabling log collection also supports collecting and indexing the following types of logs from other components in your deployments: - -**APM** - -* apm*.log* - -**Fleet and Elastic Agent** - -* fleet-server-json.log-* -* elastic-agent-json.log-* - -The ˆ*ˆ indicates that we also index the archived files of each type of log. - -Check the respective product documentation for more information about the logging capabilities of each product. - - -## Metrics features [ec-extra-metrics-features] - -With logging and monitoring enabled for a deployment, metrics are collected for Elasticsearch, Kibana, and APM with Fleet Server. - - -#### Enabling Elasticsearch/Kibana audit logs on your deployment [ec-enable-audit-logs] -% Added by eedugon to audit logging in deploy and manage -> monitoring -> logging section - -Audit logs are useful for tracking security events on your {{es}} and/or {{kib}} clusters. To enable {{es}} audit logs on your deployment: - -1. Log in to the [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body). -2. Find your deployment on the home page in the **Hosted deployments** card and select **Manage** to access it directly. Or, select **Hosted deployments** to go to the **Deployments** page to view all of your deployments. - - On the **Deployments** page you can narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list. - -3. From your deployment menu, go to the **Edit** page. -4. To enable audit logs in {{es}}, in the **Elasticsearch** section select **Manage user settings and extensions**. For deployments with existing user settings, you may have to expand the **Edit elasticsearch.yml** caret for each node instead. -5. To enable audit logs in {{kib}}, in the **Kibana** section select **Edit user settings**. For deployments with existing user settings, you may have to expand the **Edit kibana.yml** caret instead. -6. Add `xpack.security.audit.enabled: true` to the user settings. -7. Select **Save changes**. - -A plan change will run on your deployment. When it finishes, audit logs will be delivered to your monitoring deployment. - -## Restrictions and limitations [ec-restrictions-monitoring] - -* To avoid compatibility issues, ensure your monitoring cluster and production cluster run on the same {{stack}} version. Monitoring clusters that use 8.x do work with production clusters that use the latest release of 7.x, but this setup should only occur when upgrading clusters to the same version. -* $$$cross-region-monitor$$$ Monitoring across regions is not supported. If you need to move your existing monitoring to the same region, you can do a reindex or create a new deployment and select the snapshot from the old deployment. -* The logs shipped to a monitoring cluster use an ILM managed data stream (elastic-cloud-logs-). If you need to delete indices due to space, do not delete the current is_write_enabled: true index. -* When sending metrics to a dedicated monitoring deployment, the graph for IO Operations Rate(/s) is blank. This is due to the fact that this graph actually contains metrics from of all of the virtualized resources from the provider. - - diff --git a/raw-migrated-files/cloud/cloud/ec-manage-apm-settings.md b/raw-migrated-files/cloud/cloud/ec-manage-apm-settings.md index d9fa252999..e57b3ee671 100644 --- a/raw-migrated-files/cloud/cloud/ec-manage-apm-settings.md +++ b/raw-migrated-files/cloud/cloud/ec-manage-apm-settings.md @@ -361,7 +361,7 @@ Allow anonymous access only for specified agents and/or services. This is primar : The period after which to log the internal metrics. Defaults to *30s*. ::::{note} -To change logging settings you must first [enable deployment logging](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md). +To change logging settings you must first [enable deployment logging](../../../deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md). :::: diff --git a/raw-migrated-files/cloud/cloud/ec-manage-kibana-settings.md b/raw-migrated-files/cloud/cloud/ec-manage-kibana-settings.md index 88ff28115b..712c7e703e 100644 --- a/raw-migrated-files/cloud/cloud/ec-manage-kibana-settings.md +++ b/raw-migrated-files/cloud/cloud/ec-manage-kibana-settings.md @@ -784,7 +784,7 @@ If search latency in {{es}} is sufficiently high, such as if you are using cross ## Logging and audit settings [ec_logging_and_audit_settings] ::::{note} -To change logging settings or to enable auditing you must first [enable deployment logging](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md). +To change logging settings or to enable auditing you must first [enable deployment logging](../../../deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md). :::: diff --git a/raw-migrated-files/cloud/cloud/ec-monitoring-setup.md b/raw-migrated-files/cloud/cloud/ec-monitoring-setup.md index 96e4163493..976f5b4454 100644 --- a/raw-migrated-files/cloud/cloud/ec-monitoring-setup.md +++ b/raw-migrated-files/cloud/cloud/ec-monitoring-setup.md @@ -22,7 +22,7 @@ After you have created a new deployment, you should enable shipping logs and met 4. Choose where to send your logs and metrics. ::::{important} - Anything used for production should go to a separate deployment you create only for monitoring. For development or testing, you can send monitoring data to the same deployment. Check [Enable logging and monitoring](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ec-logging-and-monitoring-production). + Anything used for production should go to a separate deployment you create only for monitoring. For development or testing, you can send monitoring data to the same deployment. Check [Enable logging and monitoring](../../../deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md). :::: 5. Select **Save**. @@ -59,7 +59,7 @@ To learn more about what [Elasticsearch monitoring metrics](/deploy-manage/monit :alt: Node tab in Kibana under Stack Monitoring ::: -Some [performance metrics](../../../deploy-manage/monitor/monitoring-data/ec-saas-metrics-accessing.md) are also available directly in the [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body) and don’t require looking at your monitoring deployment. If you’re ever in a rush to determine if there is a performance problem, you can get a quick overview by going to the **Performance** page from your deployment menu: +Some [performance metrics](/deploy-manage/monitor/access-performance-metrics-on-elastic-cloud.md) are also available directly in the [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body) and don’t require looking at your monitoring deployment. If you’re ever in a rush to determine if there is a performance problem, you can get a quick overview by going to the **Performance** page from your deployment menu: :::{image} ../../../images/cloud-ec-ce-monitoring-performance.png :alt: Performance page of the Elastic Cloud console @@ -104,7 +104,7 @@ You will get this request reported as a new log. Audit logs do not currently rep ## Get notified [ec_get_notified] -You should take advantage of the default [Elastic Stack monitoring alerts](/deploy-manage/monitor/monitoring-data/kibana-alerts.md) that are available out-of-the-box. You don’t have to do anything other than enable shipping logs and metrics to have them made available to you (which you did earlier on). +You should take advantage of the default [Elastic Stack monitoring alerts](/deploy-manage/monitor/monitoring-data/configure-stack-monitoring-alerts.md) that are available out-of-the-box. You don’t have to do anything other than enable shipping logs and metrics to have them made available to you (which you did earlier on). On top of these default alerts that write to indices you can investigate, you might want to add some custom actions, such as a [connector](kibana://reference/connectors-kibana.md) for Slack notifications. To set up these notifications, you first configure a Slack connector and then append it to the default alerts and actions. From Kibana: @@ -115,7 +115,7 @@ On top of these default alerts that write to indices you can investigate, you mi 3. Select **Save**. 2. Go to **Stack Monitoring** and select **Enter setup mode**. -3. Edit an alert rule, such as [CPU usage](/deploy-manage/monitor/monitoring-data/kibana-alerts.md#kibana-alerts-cpu-threshold): +3. Edit an alert rule, such as [CPU usage](/deploy-manage/monitor/monitoring-data/configure-stack-monitoring-alerts.md#kibana-alerts-cpu-threshold): 1. Select one of the alert rule fields and select **CPU Usage**. 2. Choose **Edit rule** and scroll down to the bottom of the screen to select **Slack**. diff --git a/raw-migrated-files/cloud/cloud/ec-monitoring.md b/raw-migrated-files/cloud/cloud/ec-monitoring.md deleted file mode 100644 index 2b01e9b48f..0000000000 --- a/raw-migrated-files/cloud/cloud/ec-monitoring.md +++ /dev/null @@ -1,132 +0,0 @@ -# Monitoring your deployment [ec-monitoring] - -::::{admonition} -If you’re using Elastic Cloud Hosted, then you can use AutoOps to monitor your cluster. AutoOps significantly simplifies cluster management with performance recommendations, resource utilization visibility, real-time issue detection and resolution paths. AutoOps is [rolling out](../../../deploy-manage/monitor/autoops/ec-autoops-regions.md) in phases across Elastic Cloud Hosted regions and CSP. It will be automatically activated for your deployment, with no installation required. For more information, check [Monitor with AutoOps](../../../deploy-manage/monitor/autoops.md). - -:::: - - -Keeping on top of the health of your deployments is a key part of the [shared responsibilities](https://www.elastic.co/cloud/shared-responsibility) between Elastic and yourself. Elastic Cloud provides out of the box tools for monitoring the health of your deployment and resolving health issues when they arise. If you are ready to set up a deployment for production use cases, make sure you check the recommendations and best practices for [production readiness](../../../deploy-manage/production-guidance/plan-for-production-elastic-cloud.md). - -A deployment on Elastic Cloud is a combination of an Elasticsearch cluster, a Kibana instance and potentially an APM server instance, and an Integration Server instance. The health of an Elastic Cloud deployment comprises the health of the various components that are part of the deployment. - -The most important of these is the {{es}} cluster, because it is the heart of the system for searching and indexing data. - -This section provides some best practices to help you monitor and understand the ongoing state of your deployments and their resources. - -* [{{es}} cluster health](../../../deploy-manage/monitor/stack-monitoring.md#ec-es-cluster-health) -* [{{es}} cluster performance](../../../deploy-manage/monitor/stack-monitoring.md#ec-es-cluster-performance) -* [Health warnings](../../../deploy-manage/monitor/stack-monitoring.md#ec-es-health-warnings) -* [Preconfigured logs and metrics](../../../deploy-manage/monitor/stack-monitoring.md#ec-es-health-preconfigured) -* [Dedicated logs and metrics](../../../deploy-manage/monitor/stack-monitoring.md#ec-es-health-dedicated) -* [Understanding deployment health](../../../deploy-manage/monitor/stack-monitoring.md#ec-health-best-practices) - - -## {{es}} cluster health [ec-es-cluster-health] - -Health is reported based on the following areas: Shard availability, master node health, Snapshot Lifecycle Management (SLM), Index Lifecycle Management (ILM), and repository integrity. - -For {{stack}} versions 8.3 and below, the deployment **Health** page is limited. Health issues are displayed in a banner with no details on impacts and troubleshooting steps. Follow [these steps](../../../deploy-manage/upgrade/deployment-or-cluster.md) if you want to perform a version upgrade. - -For {{stack}} versions 8.4 and later, the deployment **Health** page provides detailed information on health issues, impacted areas, and troubleshooting support. - -To view the health for a deployment: - -1. Log in to the [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body). -2. On the **Deployments** page, select your deployment. -3. In your deployment menu, select **Health**. - -The **Health** page provides the following information: - -* Health issues for Kibana, Enterprise Search, APM and plan changes are reported in the health banner. -* Health issues for {{es}} clusters are broken down into a table with more details on Severity, Description and Affected capabilities. - -:::{image} ../../../images/cloud-es-health-page.png -:alt: {{es}} Health page -::: - -**Severity**: A critical issue impacts operations such as search and ingest and should be addressed as soon as possible. Warnings don’t impact the cluster immediately but might lead to more critical issues over time such as a corrupted repository might lead to no backups being available in the future. - -**Description**: For most issues, you can click the description to get more details page on the specific issue and on how to fix it. - -**Affected capabilities**: Each of these areas might impact Search, Ingest, Backups or Deployment Management capabilities. - -You can also search and filter the table based on affected resources, such as indices, repositories, nodes, or SLM policies. Individual issues can be further expanded to get more details and guided troubleshooting. - -:::{image} ../../../images/cloud-es-health-page-table.png -:alt: {{es}} Health page with details and troubleshooting -::: - -For each issue you can either use a troubleshooting link or get a suggestion to contact support, in case you need help. The [troubleshooting documentation](../../../troubleshoot/elasticsearch/elasticsearch.md) for {{es}} provides more details on specific errors. - - -## {{es}} cluster performance [ec-es-cluster-performance] - -The deployment **Health** page does not include information on cluster performance. If you observe issues on search and ingest operations in terms of increased latency or throughput for queries, these might not be directly reported on the **Health** page, unless they are related to shard health or master node availability. The performance page and the out-of-the-box logs allow you to monitor your cluster performance, but for production applications we strongly recommend setting up a dedicated monitoring cluster. Check [Understanding deployment health](../../../deploy-manage/monitor/stack-monitoring.md#ec-health-best-practices), for more guidelines on how to monitor you cluster performance. - - -## Health warnings [ec-es-health-warnings] - -In the normal course of using your {{ech}} deployments, health warnings and errors might appear from time to time. Following are the most common scenarios and methods to resolve them. - -Health warning messages -: Health warning messages will sometimes appear on the main page for one of your deployments, as well as on the **Logs and metrics** page. - - A single warning is rarely cause for concern, as often it just reflects ongoing, routine maintenance activity occurring on {{ecloud}}. - - -Configuration change failures -: In more serious cases, your deployment may be unable to restart. The failure can be due to a variety of causes, the most frequent of these being invalid secure settings, expired plugins or bundles, or out of memory errors. The problem is often not detected until an attempt is made to restart the deployment following a routine configuration change, such as a deployment resizing. - -Out of memory errors -: Out of memory errors (OOMs) may occur during your deployment’s normal operations, and these can have a very negative impact on performance. Common causes of memory shortages are oversharding, data retention oversights, and the overall request volume. - - On your deployment page, you can check the [JVM memory pressure indicator](../../../deploy-manage/monitor/monitoring-data/ec-memory-pressure.md) to get the current memory usage of each node of your deployment. You can also review the [common causes of high JVM memory usage](../../../deploy-manage/monitor/monitoring-data/ec-memory-pressure.md#ec-memory-pressure-causes) to help diagnose the source of unexpectedly high memory pressure levels. To learn more, check [How does high memory pressure affect performance?](../../../troubleshoot/monitoring/high-memory-pressure.md). - - - -## Preconfigured logs and metrics [ec-es-health-preconfigured] - -In a non-production environment, you may choose to rely on the logs and metrics that are available for your deployment by default. The deployment **Logs and metrics** page displays any current deployment health warnings, and from there you can also view standard log files from the last 24 hours. - -The logs capture any activity related to your deployments, their component resources, snapshotting behavior, and more. You can use the search bar to filter the logs by, for example, a specific instance (`instance-0000000005`), a configuration file (`roles.yml`), an operation type (`snapshot`, `autoscaling`), or a component (`kibana`, `ent-search`). - -In a production environment, we highly recommend storing your logs and metrics in another cluster. This gives you the ability to retain your logs and metrics over longer periods of time and setting custom alerts and watches. - - -## Dedicated logs and metrics [ec-es-health-dedicated] - -In a production environment, it’s important set up dedicated health monitoring in order to retain the logs and metrics that can be used to troubleshoot any health issues in your deployments. In the event of that you need to [contact our support team](../../../troubleshoot/index.md), they can use the retained data to help diagnose any problems that you may encounter. - -You have the option of sending logs and metrics to a separate, specialized monitoring deployment, which ensures that they’re available in the event of a deployment outage. The monitoring deployment also gives you access to Kibana’s stack monitoring features, through which you can view health and performance data for all of your deployment resources. - -As part of health monitoring, it’s also a best practice to [configure alerting](../../../deploy-manage/monitor/monitoring-data/configure-stack-monitoring-alerts.md), so that you can be notified right away about any deployment health issues. - -Check the guide on [how to set up monitoring](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) to learn more. - - -## Understanding deployment health [ec-health-best-practices] - -We’ve compiled some guidelines to help you ensure the health of your deployments over time. These can help you to better understand the available performance metrics, and to make decisions involving performance and high availability. - -[Why is my node(s) unavailable?](../../../troubleshoot/monitoring/unavailable-nodes.md) -: Learn about common symptoms and possible actions that you can take to resolve issues when one or more nodes become unhealthy or unavailable. - -[Why are my shards unavailable?](../../../troubleshoot/monitoring/unavailable-shards.md) -: Provide instructions on how to troubleshoot issues related to unassigned shards. - -[Why is performance degrading over time?](../../../troubleshoot/monitoring/performance.md) -: Address performance degradation on a smaller size Elasticsearch cluster. - -[Is my cluster really highly available?](../../../troubleshoot/monitoring/high-availability.md) -: High availability involves more than setting multiple availability zones (although that’s really important!). Learn how to assess performance and workloads to determine if your deployment has adequate resources to mitigate a potential node failure. - -[How does high memory pressure affect performance?](../../../troubleshoot/monitoring/high-memory-pressure.md) -: Learn about typical memory usage patterns, how to assess when the deployment memory usage levels are problematic, how this impacts performance, and how to resolve memory-related issues. - -[Why are my cluster response times suddenly so much worse?](../../../troubleshoot/monitoring/cluster-response-time.md) -: Learn about the common causes of increased query response times and decreased performance in your deployment. - -[Why did my node move to a different host?](../../../troubleshoot/monitoring/node-moves-outages.md) -: Learn about why we may, from time to time, relocate your {{ech}} deployments across hosts. - diff --git a/raw-migrated-files/cloud/cloud/ec-planning.md b/raw-migrated-files/cloud/cloud/ec-planning.md index 97eb60a553..753866730c 100644 --- a/raw-migrated-files/cloud/cloud/ec-planning.md +++ b/raw-migrated-files/cloud/cloud/ec-planning.md @@ -44,7 +44,7 @@ Clusters that only have one master node are not highly available and are at risk ## Do you know when to scale? [ec-workloads] -Knowing how to scale your deployment is critical, especially when unexpected workloads hits. Don’t forget to [check your performance metrics](../../../deploy-manage/monitor/monitoring-data/ec-saas-metrics-accessing.md) to make sure your deployments are healthy and can cope with your workloads. +Knowing how to scale your deployment is critical, especially when unexpected workloads hits. Don’t forget to [check your performance metrics](/deploy-manage/monitor/access-performance-metrics-on-elastic-cloud.md) to make sure your deployments are healthy and can cope with your workloads. Scaling with {{ech}} is easy: diff --git a/raw-migrated-files/cloud/cloud/ec-saas-metrics-accessing.md b/raw-migrated-files/cloud/cloud/ec-saas-metrics-accessing.md deleted file mode 100644 index b11d83c981..0000000000 --- a/raw-migrated-files/cloud/cloud/ec-saas-metrics-accessing.md +++ /dev/null @@ -1,131 +0,0 @@ -# Access performance metrics [ec-saas-metrics-accessing] - -Cluster performance metrics are available directly in the [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body). The graphs on this page include a subset of {{ech}}-specific performance metrics. - -For advanced views or production monitoring, [enable logging and monitoring](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md). The monitoring application provides more advanced views for Elasticsearch and JVM metrics, and includes a configurable retention period. - -:::{important} - If you’re using Elastic Cloud Hosted, then you can use AutoOps to monitor your cluster. AutoOps significantly simplifies cluster management with performance recommendations, resource utilization visibility, real-time issue detection and resolution paths. For more information, refer to [Monitor with AutoOps](/deploy-manage/monitor/autoops.md). -::: - -To access cluster performance metrics: - -1. Log in to the [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body). -2. Find your deployment on the home page in the **Hosted deployments** card and select **Manage** to access it directly. Or, select **Hosted deployments** to go to the **Deployments** page to view all of your deployments. - - On the **Deployments** page you can narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list. For example, you might want to select **Is unhealthy** and **Has master problems** to get a short list of deployments that need attention. - -3. From your deployment menu, go to the **Performance** page. - -The following metrics are available: - - -### CPU usage [ec_cpu_usage] - -:::{image} ../../../images/cloud-metrics-cpu-usage.png -:alt: Graph showing CPU usage -::: - -Shows the maximum usage of the CPU resources assigned to your Elasticsearch cluster, as a percentage. CPU resources are relative to the size of your cluster, so that a cluster with 32GB of RAM gets assigned twice as many CPU resources as a cluster with 16GB of RAM. All clusters are guaranteed their share of CPU resources, as {{ech}} infrastructure does not overcommit any resources. CPU credits permit boosting the performance of smaller clusters temporarily, so that CPU usage can exceed 100%. - -::::{tip} -This chart reports the maximum CPU values over the sampling period. [Logs and Metrics](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) ingested into [Stack Monitoring](../../../deploy-manage/monitor/monitoring-data/visualizing-monitoring-data.md)'s "CPU Usage" instead reflects the average CPU over the sampling period. Therefore, you should not expect the two graphs to look exactly the same. When investigating [CPU-related performance issues](../../../troubleshoot/monitoring/performance.md), you should default to [Stack Monitoring](../../../deploy-manage/monitor/monitoring-data/visualizing-monitoring-data.md). -:::: - - - -### CPU credits [ec_cpu_credits] - -:::{image} ../../../images/cloud-metrics-cpu-credits.png -:alt: Graph showing available CPU credits -::: - -Shows your remaining CPU credits, measured in seconds of CPU time. CPU credits enable the boosting of CPU resources assigned to your cluster to improve performance temporarily when it is needed most. For more details check [How to use vCPU to boost your instance](../../../deploy-manage/monitor/monitoring-data/ec-vcpu-boost-instance.md). - - -### Number of requests [ec_number_of_requests] - -:::{image} ../../../images/cloud-metrics-number-of-requests.png -:alt: Graph showing the number of requests -::: - -Shows the number of requests that your cluster receives per second, separated into search requests and requests to index documents. This metric provides a good indicator of the volume of work that your cluster typically handles over time which, together with other performance metrics, helps you determine if your cluster is sized correctly. Also lets you check if there is a sudden increase in the volume of user requests that might explain an increase in response times. - - -### Search response times [ec_search_response_times] - -:::{image} ../../../images/cloud-metrics-search-response-times.png -:alt: Graph showing search response times -::: - -Indicates the amount of time that it takes for your Elasticsearch cluster to complete a search query, in milliseconds. Response times won’t tell you about the cause of a performance issue, but they are often a first indicator that something is amiss with the performance of your Elasticsearch cluster. - - -### Index response times [ec_index_response_times] - -:::{image} ../../../images/cloud-metrics-index-response-times.png -:alt: Graph showing index response times -::: - -Indicates the amount of time that it takes for your Elasticsearch cluster to complete an indexing operation, in milliseconds. Response times won’t tell you about the cause of a performance issue, but they are often a first indicator that something is amiss with the performance of your Elasticsearch cluster. - - -### Memory pressure per node [ec_memory_pressure_per_node] - -:::{image} ../../../images/cloud-metrics-memory-pressure-per-node.png -:alt: Graph showing memory pressure per node -::: - -Indicates the total memory used by the JVM heap over time. We’ve configured {{es}}'s garbage collector to keep memory usage below 75% for heaps of 8GB or larger. For heaps smaller than 8GB, the threshold is 85%. If memory pressure consistently remains above this threshold, you might need to resize your cluster or reduce memory consumption. Check [how high memory pressure can cause performance issues](../../../troubleshoot/monitoring/high-memory-pressure.md). - - -### GC overhead per node [ec_gc_overhead_per_node] - -:::{image} ../../../images/cloud-metrics-gc-overhead-per-node.png -:alt: Graph showing the garbage collection overhead per node -::: - -Indicates the overhead involved in JVM garbage collection to reclaim memory. - - -## Tips for working with performance metrics [ec_tips_for_working_with_performance_metrics] - -Performance correlates directly with resources assigned to your cluster, and many of these metrics will show some sort of correlation with each other when you are trying to determine the cause of a performance issue. Take a look at some of the scenarios included in this section to learn how you can determine the cause of performance issues. - -It is not uncommon for performance issues on {{ech}} to be caused by an undersized cluster that cannot cope with the workload it is being asked to handle. If your cluster performance metrics often shows high CPU usage or excessive memory pressure, consider increasing the size of your cluster soon to improve performance. This is especially true for clusters that regularly reach 100% of CPU usage or that suffer out-of-memory failures; it is better to resize your cluster early when it is not yet maxed out than to have to resize a cluster that is already overwhelmed. [Changing the configuration of your cluster](../../../deploy-manage/deploy/elastic-cloud/configure.md) may add some overhead if data needs to be migrated to the new nodes, which can increase the load on a cluster further and delay configuration changes. - -To help diagnose high CPU usage you can also use the Elasticsearch [nodes hot threads API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-nodes-hot-threads), which identifies the threads on each node that have the highest CPU usage or that have been executing for a longer than normal period of time. - -::::{tip} -Got an overwhelmed cluster that needs to be upsized? [Try enabling maintenance mode first](../../../deploy-manage/maintenance/ece/start-stop-routing-requests.md). It will likely help with configuration changes. -:::: - - -Work with the metrics shown in **Cluster Performance Metrics** section to help you find the information you need: - -* Hover on any part of a graph to get additional information. For example, hovering on a section of a graph that shows response times reveals the percentile that responses fall into at that point in time: - - :::{image} ../../../images/cloud-metrics-hover.png - :alt: Hover over the metric graph - ::: - -* Zoom in on a graph by drawing a rectangle to select a specific time window. As you zoom in one metric, other performance metrics change to show data for the same time window. - - :::{image} ../../../images/cloud-metrics-zoom.png - :alt: Zoom the metric graph - ::: - -* Pan around with ![Pan in a metric graph](../../../images/cloud-metrics-pan.png "") to make sure that you can get the right parts of a metric graph as you zoom in. -* Reset the metric graph axes with ![Reset the metric graph](../../../images/cloud-metrics-reset.png ""), which returns the graphs to their original scale. - -Cluster performance metrics are shown per node and are color-coded to indicate which running Elasticsearch instance they belong to. - - -## Cluster restarts after out-of-memory failures [ec_cluster_restarts_after_out_of_memory_failures] - -For clusters that suffer out-of-memory failures, it can be difficult to determine whether the clusters are in a completely healthy state afterwards. For this reason, {{ech}} automatically reboots clusters that suffer out-of-memory failures. - -You will receive an email notification to let you know that a restart occurred. For repeated alerts, the emails are aggregated so that you do not receive an excessive number of notifications. Either [resizing your cluster to reduce memory pressure](../../../deploy-manage/deploy/elastic-cloud/ec-customize-deployment-components.md#ec-cluster-size) or reducing the workload that a cluster is being asked to handle can help avoid these cluster restarts. - - - diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/how-monitoring-works.md b/raw-migrated-files/elasticsearch/elasticsearch-reference/how-monitoring-works.md deleted file mode 100644 index a906700b93..0000000000 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/how-monitoring-works.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -navigation_title: "How it works" ---- - -# How monitoring works [how-monitoring-works] - - -Each monitored {{stack}} component is considered unique in the cluster based on its persistent UUID, which is written to the [`path.data`](../../../deploy-manage/deploy/self-managed/important-settings-configuration.md#path-settings) directory when the node or instance starts. - -Monitoring documents are just ordinary JSON documents built by monitoring each {{stack}} component at a specified collection interval. If you want to alter how these documents are structured or stored, refer to [*Configuring data streams/indices for monitoring*](../../../deploy-manage/monitor/monitoring-data/configuring-data-streamsindices-for-monitoring.md). - -You can use {{agent}} or {{metricbeat}} to collect monitoring data and to ship it directly to the monitoring cluster. - -To learn how to collect monitoring data, refer to: - -* One of the following topics depending on how you want to collect monitoring data from {{es}}: - - * [Collecting monitoring data with {{agent}}](../../../deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-elastic-agent.md): Uses a single agent to gather logs and metrics. Can be managed from a central location in {{fleet}}. - * [Collecting monitoring data with {{metricbeat}}](../../../deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-metricbeat.md): Uses a lightweight {{beats}} shipper to gather metrics. May be preferred if you have an existing investment in {{beats}} or are not yet ready to use {{agent}}. - * [Legacy collection methods](../../../deploy-manage/monitor/stack-monitoring/es-legacy-collection-methods.md): Uses internal exporters to gather metrics. Not recommended. If you have previously configured legacy collection methods, you should migrate to using {{agent}} or {{metricbeat}}. - -* [Monitoring {{kib}}](../../../deploy-manage/monitor/monitoring-data/visualizing-monitoring-data.md) -* [Monitoring {{ls}}](logstash://reference/monitoring-logstash-legacy.md) -* Monitoring {{beats}}: - - * [{{auditbeat}}](beats://reference/auditbeat/monitoring.md) - * [{{filebeat}}](beats://reference/filebeat/monitoring.md) - * [{{heartbeat}}](beats://reference/heartbeat/monitoring.md) - * [{{metricbeat}}](beats://reference/metricbeat/monitoring.md) - * [{{packetbeat}}](beats://reference/packetbeat/monitoring.md) - * [{{winlogbeat}}](beats://reference/winlogbeat/monitoring.md) - -* [Monitoring APM Server](/solutions/observability/apps/monitor-apm-server.md) -* [Monitoring {{agent}}s](/reference/ingestion-tools/fleet/monitor-elastic-agent.md) {{fleet}}-managed agents) or [Configure monitoring for standalone {{agent}}s](/reference/ingestion-tools/fleet/elastic-agent-monitoring-configuration.md) - diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/monitor-elasticsearch-cluster.md b/raw-migrated-files/elasticsearch/elasticsearch-reference/monitor-elasticsearch-cluster.md deleted file mode 100644 index 14f8da34f2..0000000000 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/monitor-elasticsearch-cluster.md +++ /dev/null @@ -1,15 +0,0 @@ -# Monitor a cluster [monitor-elasticsearch-cluster] - -The {{stack}} {{monitor-features}} provide a way to keep a pulse on the health and performance of your {{es}} cluster. - -* [Overview](../../../deploy-manage/monitor/stack-monitoring.md) -* [How it works](../../../deploy-manage/monitor/stack-monitoring.md) -* [*Elasticsearch application logging*](../../../deploy-manage/monitor/logging-configuration/update-elasticsearch-logging-levels.md) -* [*Monitoring in a production environment*](../../../deploy-manage/monitor/stack-monitoring/elasticsearch-monitoring-self-managed.md) -* [Collecting monitoring data with {{agent}}](../../../deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-elastic-agent.md) -* [Collecting monitoring data with {{metricbeat}}](../../../deploy-manage/monitor/stack-monitoring/collecting-monitoring-data-with-metricbeat.md) -* [Collecting log data with {{filebeat}}](../../../deploy-manage/monitor/stack-monitoring/collecting-log-data-with-filebeat.md) -* [*Configuring data streams/indices for monitoring*](../../../deploy-manage/monitor/monitoring-data/configuring-data-streamsindices-for-monitoring.md) -* [Legacy collection methods](../../../deploy-manage/monitor/stack-monitoring/es-legacy-collection-methods.md) -* [*Troubleshooting monitoring*](../../../troubleshoot/elasticsearch/monitoring-troubleshooting.md) - diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/monitoring-overview.md b/raw-migrated-files/elasticsearch/elasticsearch-reference/monitoring-overview.md deleted file mode 100644 index adf7be4c5c..0000000000 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/monitoring-overview.md +++ /dev/null @@ -1,37 +0,0 @@ ---- -navigation_title: "Overview" ---- - -# Monitoring overview [monitoring-overview] - - -When you monitor a cluster, you collect data from the {{es}} nodes, {{ls}} nodes, {{kib}} instances, APM Server, and Beats in your cluster. You can also collect logs. - -All of the monitoring metrics are stored in {{es}}, which enables you to easily visualize the data in {{kib}}. By default, the monitoring metrics are stored in local indices. - -::::{admonition} -If you’re using Elastic Cloud Hosted, then you can use AutoOps to monitor your cluster. AutoOps significantly simplifies cluster management with performance recommendations, resource utilization visibility, real-time issue detection and resolution paths. For more information, refer to [Monitor with AutoOps](/deploy-manage/monitor/autoops.md). - -:::: - - -::::{tip} -In production, we strongly recommend using a separate monitoring cluster. Using a separate monitoring cluster prevents production cluster outages from impacting your ability to access your monitoring data. It also prevents monitoring activities from impacting the performance of your production cluster. For the same reason, we also recommend using a separate {{kib}} instance for viewing the monitoring data. -:::: - - -You can use {{agent}} or {{metricbeat}} to collect and ship data directly to your monitoring cluster rather than routing it through your production cluster. - -The following diagram illustrates a typical monitoring architecture with separate production and monitoring clusters. This example shows {{metricbeat}}, but you can use {{agent}} instead. - -:::{image} ../../../images/elasticsearch-reference-architecture.png -:alt: A typical monitoring environment -::: - -If you have the appropriate license, you can route data from multiple production clusters to a single monitoring cluster. For more information about the differences between various subscription levels, see: [https://www.elastic.co/subscriptions](https://www.elastic.co/subscriptions) - -::::{important} -In general, the monitoring cluster and the clusters being monitored should be running the same version of the stack. A monitoring cluster cannot monitor production clusters running newer versions of the stack. If necessary, the monitoring cluster can monitor production clusters running the latest release of the previous major version. -:::: - - diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/secure-monitoring.md b/raw-migrated-files/elasticsearch/elasticsearch-reference/secure-monitoring.md deleted file mode 100644 index 5d8a71ebd8..0000000000 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/secure-monitoring.md +++ /dev/null @@ -1,12 +0,0 @@ -# Monitoring and security [secure-monitoring] - -The {{stack}} {{monitor-features}} consist of two components: an agent that you install on each {{es}} and Logstash node, and a Monitoring UI in {{kib}}. The monitoring agent collects and indexes metrics from the nodes and you visualize the data through the Monitoring dashboards in {{kib}}. The agent can index data on the same {{es}} cluster, or send it to an external monitoring cluster. - -To use the {{monitor-features}} with the {{security-features}} enabled, you need to [set up {{kib}} to work with the {{security-features}}](../../../deploy-manage/security.md) and create at least one user for the Monitoring UI. If you are using an external monitoring cluster, you also need to configure a user for the monitoring agent and configure the agent to use the appropriate credentials when communicating with the monitoring cluster. - -For more information, see: - -* [*Monitoring in a production environment*](../../../deploy-manage/monitor/stack-monitoring/elasticsearch-monitoring-self-managed.md) -* [Configuring monitoring in {{kib}}](/deploy-manage/monitor/stack-monitoring/kibana-monitoring-legacy.md) -* [Configuring monitoring for Logstash nodes](logstash://reference/monitoring-logstash-legacy.md) - diff --git a/raw-migrated-files/stack-docs/elastic-stack/upgrade-elastic-stack-for-elastic-cloud.md b/raw-migrated-files/stack-docs/elastic-stack/upgrade-elastic-stack-for-elastic-cloud.md index d69f17e7f0..76db772193 100644 --- a/raw-migrated-files/stack-docs/elastic-stack/upgrade-elastic-stack-for-elastic-cloud.md +++ b/raw-migrated-files/stack-docs/elastic-stack/upgrade-elastic-stack-for-elastic-cloud.md @@ -6,7 +6,7 @@ Minor version upgrades, upgrades from 8.17 to 9.0.0-beta1, and cluster configura {{ech}} and {{ece}} do not support the ability to upgrade to or from release candidate builds, such as 8.0.0-rc1. -If you use a separate [monitoring deployment](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md), you should upgrade the monitoring deployment before the production deployment. In general, the monitoring deployment and the deployments being monitored should be running the same version of the Elastic Stack. A monitoring deployment cannot monitor production deployments running newer versions of the stack. If necessary, the monitoring deployment can monitor production deployments running the latest release of the previous major version. +If you use a separate [monitoring deployment](../../../deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md), you should upgrade the monitoring deployment before the production deployment. In general, the monitoring deployment and the deployments being monitored should be running the same version of the Elastic Stack. A monitoring deployment cannot monitor production deployments running newer versions of the stack. If necessary, the monitoring deployment can monitor production deployments running the latest release of the previous major version. ::::{important} Although it’s simple to upgrade an Elastic Cloud deployment, the new version might include breaking changes that affect your application. Make sure you review the deprecation logs, make any necessary changes, and test against the new version before upgrading your production deployment. diff --git a/raw-migrated-files/toc.yml b/raw-migrated-files/toc.yml index 0c704b5981..9a7b1540e2 100644 --- a/raw-migrated-files/toc.yml +++ b/raw-migrated-files/toc.yml @@ -37,17 +37,12 @@ toc: - file: cloud/cloud-heroku/ech-delete-deployment.md - file: cloud/cloud-heroku/ech-editing-user-settings.md - file: cloud/cloud-heroku/ech-enable-kibana2.md - - file: cloud/cloud-heroku/ech-enable-logging-and-monitoring.md - file: cloud/cloud-heroku/ech-getting-started.md - file: cloud/cloud-heroku/ech-manage-apm-settings.md - file: cloud/cloud-heroku/ech-manage-kibana-settings.md - - file: cloud/cloud-heroku/ech-monitoring-setup.md - - file: cloud/cloud-heroku/ech-monitoring.md - file: cloud/cloud-heroku/ech-planning.md - file: cloud/cloud-heroku/ech-regional-deployment-aliases.md - file: cloud/cloud-heroku/ech-restoring-snapshots.md - - file: cloud/cloud-heroku/ech-restrictions-monitoring.md - - file: cloud/cloud-heroku/ech-saas-metrics-accessing.md - file: cloud/cloud-heroku/ech-security.md - file: cloud/cloud-heroku/ech-snapshot-restore.md - file: cloud/cloud-heroku/ech-upgrade-deployment.md @@ -64,7 +59,6 @@ toc: - file: cloud/cloud/ec-custom-repository.md - file: cloud/cloud/ec-delete-deployment.md - file: cloud/cloud/ec-editing-user-settings.md - - file: cloud/cloud/ec-enable-logging-and-monitoring.md - file: cloud/cloud/ec-faq-getting-started.md - file: cloud/cloud/ec-faq-technical.md - file: cloud/cloud/ec-getting-started-trial.md @@ -75,13 +69,11 @@ toc: - file: cloud/cloud/ec-manage-enterprise-search-settings.md - file: cloud/cloud/ec-manage-kibana-settings.md - file: cloud/cloud/ec-monitoring-setup.md - - file: cloud/cloud/ec-monitoring.md - file: cloud/cloud/ec-password-reset.md - file: cloud/cloud/ec-planning.md - file: cloud/cloud/ec-regional-deployment-aliases.md - file: cloud/cloud/ec-restore-across-clusters.md - file: cloud/cloud/ec-restoring-snapshots.md - - file: cloud/cloud/ec-saas-metrics-accessing.md - file: cloud/cloud/ec-security.md - file: cloud/cloud/ec-select-subscription-level.md - file: cloud/cloud/ec-service-status.md @@ -112,18 +104,13 @@ toc: children: - file: elasticsearch/elasticsearch-reference/documents-indices.md - file: elasticsearch/elasticsearch-reference/esql-using.md - - file: elasticsearch/elasticsearch-reference/how-monitoring-works.md - file: elasticsearch/elasticsearch-reference/index-modules-allocation.md - file: elasticsearch/elasticsearch-reference/index-modules-mapper.md - file: elasticsearch/elasticsearch-reference/ip-filtering.md - - file: elasticsearch/elasticsearch-reference/monitor-elasticsearch-cluster.md - - file: elasticsearch/elasticsearch-reference/monitoring-overview.md - - file: elasticsearch/elasticsearch-reference/monitoring-production.md - file: elasticsearch/elasticsearch-reference/recovery-prioritization.md - file: elasticsearch/elasticsearch-reference/scalability.md - file: elasticsearch/elasticsearch-reference/search-with-synonyms.md - file: elasticsearch/elasticsearch-reference/secure-cluster.md - - file: elasticsearch/elasticsearch-reference/secure-monitoring.md - file: elasticsearch/elasticsearch-reference/security-files.md - file: elasticsearch/elasticsearch-reference/security-limitations.md - file: elasticsearch/elasticsearch-reference/semantic-search-inference.md diff --git a/redirects.yml b/redirects.yml index 04643bcb5b..e15e6a0adc 100644 --- a/redirects.yml +++ b/redirects.yml @@ -24,6 +24,8 @@ redirects: 'deploy-manage/deploy/self-managed/deploy-cluster.md': '!deploy-manage/deploy/self-managed/installing-elasticsearch.md' 'deploy-manage/deploy/self-managed/configure.md': '!deploy-manage/deploy/self-managed/configure-kibana.md' 'deploy-manage/security/manage-traffic-filtering-through-api.md': '!deploy-manage/security/ec-traffic-filtering-through-the-api.md' + 'deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md': '!deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md' + 'deploy-manage/monitor/stack-monitoring/ece-stack-monitoring.md': '!deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md' 'deploy-manage/deploy/kibana-reporting-configuration.md': '!deploy-manage/kibana-reporting-configuration.md' ## audit logging movement to security section diff --git a/reference/ingestion-tools/cloud/apm-settings.md b/reference/ingestion-tools/cloud/apm-settings.md index b2c30ac0cc..c4ca425302 100644 --- a/reference/ingestion-tools/cloud/apm-settings.md +++ b/reference/ingestion-tools/cloud/apm-settings.md @@ -366,7 +366,7 @@ Allow anonymous access only for specified agents and/or services. This is primar : The period after which to log the internal metrics. Defaults to *30s*. ::::{note} -To change logging settings you must first [enable deployment logging](/deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md). +To change logging settings you must first [enable deployment logging](/deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md). :::: diff --git a/solutions/observability/apps/configure-apm-server.md b/solutions/observability/apps/configure-apm-server.md index cabbc58269..c8bdca17c2 100644 --- a/solutions/observability/apps/configure-apm-server.md +++ b/solutions/observability/apps/configure-apm-server.md @@ -393,5 +393,5 @@ Allow anonymous access only for specified agents and/or services. This is primar : The period after which to log the internal metrics. Defaults to *30s*. ::::{note} -To change logging settings you must first [enable deployment logging](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md). +To change logging settings you must first [enable deployment logging](../../../deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md). :::: \ No newline at end of file diff --git a/solutions/observability/apps/monitor-apm-server.md b/solutions/observability/apps/monitor-apm-server.md index 4fa00156ac..fc8f14fe29 100644 --- a/solutions/observability/apps/monitor-apm-server.md +++ b/solutions/observability/apps/monitor-apm-server.md @@ -20,5 +20,5 @@ Select your deployment method to get started: {{ecloud}} manages the installation and configuration of a monitoring agent for you — so all you have to do is flip a switch and watch the data pour in. -* **{{ech}}** user? See [Stack Monitoring on {{ecloud}} deployments](../../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md). -* **{{ece}}** user? See [Enable stack monitoring on ECE deployments](/deploy-manage/monitor/stack-monitoring/ece-stack-monitoring.md). +* **{{ech}}** user? See [Stack Monitoring on {{ecloud}} deployments](../../../deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md). +* **{{ece}}** user? See [Enable stack monitoring on ECE deployments](/deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md). diff --git a/troubleshoot/elasticsearch/high-cpu-usage.md b/troubleshoot/elasticsearch/high-cpu-usage.md index c04fa2c739..54f7f8c3e2 100644 --- a/troubleshoot/elasticsearch/high-cpu-usage.md +++ b/troubleshoot/elasticsearch/high-cpu-usage.md @@ -36,23 +36,23 @@ To track CPU usage over time, we recommend enabling monitoring: :::::::{tab-set} ::::::{tab-item} {{ech}} -* (Recommended) Enable [logs and metrics](../../deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md). When logs and metrics are enabled, monitoring information is visible on {{kib}}'s [Stack Monitoring](../../deploy-manage/monitor/monitoring-data/visualizing-monitoring-data.md) page. +* (Recommended) Enable [logs and metrics](../../deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md). When logs and metrics are enabled, monitoring information is visible on {{kib}}'s [Stack Monitoring](../../deploy-manage/monitor/monitoring-data/visualizing-monitoring-data.md) page. - You can also enable the [CPU usage threshold alert](../../deploy-manage/monitor/monitoring-data/kibana-alerts.md) to be notified about potential issues through email. + You can also enable the [CPU usage threshold alert](../../deploy-manage/monitor/monitoring-data/configure-stack-monitoring-alerts.md) to be notified about potential issues through email. -* From your deployment menu, view the [**Performance**](../../deploy-manage/monitor/monitoring-data/access-performance-metrics-on-elastic-cloud.md) page. On this page, you can view two key metrics: +* From your deployment menu, view the [**Performance**](../../deploy-manage/monitor/access-performance-metrics-on-elastic-cloud.md) page. On this page, you can view two key metrics: * **CPU usage**: Your deployment’s CPU usage, represented as a percentage. * **CPU credits**: Your remaining CPU credits, measured in seconds of CPU time. -{{ech}} grants [CPU credits](../../deploy-manage/monitor/monitoring-data/ec-vcpu-boost-instance.md) per deployment to provide smaller clusters with performance boosts when needed. High CPU usage can deplete these credits, which might lead to [performance degradation](../monitoring/performance.md) and [increased cluster response times](../monitoring/cluster-response-time.md). +{{ech}} grants [CPU credits](/deploy-manage/deploy/elastic-cloud/ec-vcpu-boost-instance.md) per deployment to provide smaller clusters with performance boosts when needed. High CPU usage can deplete these credits, which might lead to [performance degradation](../monitoring/performance.md) and [increased cluster response times](../monitoring/cluster-response-time.md). :::::: ::::::{tab-item} Self-managed * Enable [{{es}} monitoring](../../deploy-manage/monitor/stack-monitoring.md). When logs and metrics are enabled, monitoring information is visible on {{kib}}'s [Stack Monitoring](../../deploy-manage/monitor/monitoring-data/visualizing-monitoring-data.md) page. - You can also enable the [CPU usage threshold alert](../../deploy-manage/monitor/monitoring-data/kibana-alerts.md) to be notified about potential issues through email. + You can also enable the [CPU usage threshold alert](../../deploy-manage/monitor/monitoring-data/configure-stack-monitoring-alerts.md) to be notified about potential issues through email. :::::: ::::::: diff --git a/troubleshoot/monitoring/node-bootlooping.md b/troubleshoot/monitoring/node-bootlooping.md index 6cd072ff72..aa27cf87f7 100644 --- a/troubleshoot/monitoring/node-bootlooping.md +++ b/troubleshoot/monitoring/node-bootlooping.md @@ -14,7 +14,7 @@ When you attempt to apply a configuration change to a deployment, the attempt ma :alt: A screen capture of the deployment page showing an error: Latest change to {{es}} configuration failed. ::: -To help diagnose these and any other types of issues in your deployments, we recommend [setting up monitoring](/deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md). Then, you can easily view your deployment health and access log files to troubleshoot this configuration failure. +To help diagnose these and any other types of issues in your deployments, we recommend [setting up monitoring](/deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md). Then, you can easily view your deployment health and access log files to troubleshoot this configuration failure. To confirm if your Elasticsearch cluster is bootlooping, you can check the most recent plan under your [Deployment Activity page](/deploy-manage/deploy/elastic-cloud/keep-track-of-deployment-activity.md) for the error: @@ -125,7 +125,7 @@ To view any added plugins or bundles: Configuration change errors can occur when there is insufficient RAM configured for a data tier. In this case, the cluster typically also shows OOM (out of memory) errors. To resolve these, you need to increase the amount of heap memory, which is half of the amount of memory allocated to a cluster. You might also detect OOM in plan changes via their [related exit codes](/deploy-manage/maintenance/start-stop-services/start-stop-elasticsearch.md#fatal-errors) `127`, `137`, and `158`. -Check the [{{es}} cluster size](/deploy-manage/deploy/elastic-cloud/ec-customize-deployment-components.md#ec-cluster-size) and the [JVM memory pressure indicator](/deploy-manage/monitor/monitoring-data/ec-memory-pressure.md) documentation to learn more. +Check the [{{es}} cluster size](/deploy-manage/deploy/elastic-cloud/ec-customize-deployment-components.md#ec-cluster-size) and the [JVM memory pressure indicator](/deploy-manage/monitor/ec-memory-pressure.md) documentation to learn more. As well, you can read our detailed blog [Managing and troubleshooting {{es}} memory](https://www.elastic.co/blog/managing-and-troubleshooting-elasticsearch-memory). diff --git a/troubleshoot/monitoring/unavailable-nodes.md b/troubleshoot/monitoring/unavailable-nodes.md index e94b3c4161..89e15245da 100644 --- a/troubleshoot/monitoring/unavailable-nodes.md +++ b/troubleshoot/monitoring/unavailable-nodes.md @@ -16,7 +16,7 @@ mapped_urls: # Diagnose unavailable nodes [ec-scenario_why_is_my_node_unavailable] -This section provides a list of common symptoms and possible actions that you can take to resolve issues when one or more nodes become unhealthy or unavailable. This guide is particularly useful if you are not [shipping your logs and metrics](/deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) to a dedicated monitoring cluster. +This section provides a list of common symptoms and possible actions that you can take to resolve issues when one or more nodes become unhealthy or unavailable. This guide is particularly useful if you are not [shipping your logs and metrics](/deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md) to a dedicated monitoring cluster. **What are the symptoms?** diff --git a/troubleshoot/monitoring/unavailable-shards.md b/troubleshoot/monitoring/unavailable-shards.md index fd1d243010..ab3049bbc0 100644 --- a/troubleshoot/monitoring/unavailable-shards.md +++ b/troubleshoot/monitoring/unavailable-shards.md @@ -164,7 +164,7 @@ The response is as follows: #### Check {{es}} cluster logs [ec-check-es-cluster-logs] -To determine the allocation issue, you can [check the logs](/deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ec-check-logs). This is easier if you have set up a dedicated monitoring deployment. +To determine the allocation issue, you can [check the logs](/deploy-manage/monitor.md#logging). This is easier if you have set up a dedicated monitoring deployment. ## Analyze unassigned shards using the Kibana UI [ec-analyze_shards_with-kibana]