Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ Data transfer out of deployments and between nodes of the cluster is hard to con

The largest contributor to inter-node data transfer is usually shard movement between nodes in a cluster. The only way to prevent shard movement is by having a single node in a single availability zone. This solution is only possible for clusters up to 64GB RAM and is not recommended as it creates a risk of data loss. [Oversharding](/deploy-manage/production-guidance/optimize-performance/size-shards.md) can cause excessive shard movement. Avoiding oversharding can also help control costs and improve performance. Note that creating snapshots generates inter-node data transfer. The *storage* cost of snapshots is detailed later in this document.

The exact root cause of unusual data transfer is not always something we can identify as it can have many causes, some of which are out of our control and not associated with Cloud configuration changes. It may help to [enable monitoring](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) and examine index and shard activity on your cluster.
The exact root cause of unusual data transfer is not always something we can identify as it can have many causes, some of which are out of our control and not associated with Cloud configuration changes. It may help to [enable monitoring](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md) and examine index and shard activity on your cluster.


## Storage [storage]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ Before you start creating your own deployment templates, you should have: [tagge
9. On this page you can [configure index management](ece-configure-templates-index-management.md) by assigning attributes to each of the data nodes in the deployment template. In Kibana, you can configure an index lifecycle management (ILM) policy, based on the node attributes, to control how data moves across the nodes in your deployment.
10. Select **Stack features**.
11. You can select a [snapshot repository](../../tools/snapshot-and-restore/cloud-enterprise.md) to be used by default for deployment backups.
12. You can choose to [enable logging and monitoring](../../monitor/stack-monitoring/ece-stack-monitoring.md) by default, so that deployment logs and metrics are send to a dedicated monitoring deployment, and so that additional log types, retention options, and Kibana visualizations are available on all deployments created using this template.
12. You can choose to [enable logging and monitoring](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md) by default, so that deployment logs and metrics are send to a dedicated monitoring deployment, and so that additional log types, retention options, and Kibana visualizations are available on all deployments created using this template.
13. Select **Extensions**.
14. Select any Elasticsearch extensions that you would like to be available automatically to all deployments created using the template.
15. Select **Save and create template**.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ In the case of the `admin-console-elasticsearch` and `security` system deploymen
The `logging-and-metrics` cluster is different since, as an ECE admin, you likely want to provide users with access to the cluster in order to troubleshoot issues without your assistance, for example. In order to manage access to that cluster, you can configure roles that will provide access to the relevant indices, map those to users, and manage access to Kibana by leveraging the Elastic security integration with your authentication provider, such as LDAP, SAML, or AD. To configure one of those security realms, check [LDAP](../../users-roles/cluster-or-deployment-auth/ldap.md), [Active Directory](../../users-roles/cluster-or-deployment-auth/active-directory.md) or [SAML](../../users-roles/cluster-or-deployment-auth/saml.md).

::::{note}
The `logging-and-metrics` cluster is only intended for troubleshooting ECE deployment issues. If your use case involves modifying or normalizing logs from {{es}} or {{kib}}, use a separate [dedicated monitoring deployment](../../monitor/stack-monitoring/ece-stack-monitoring.md) instead.
The `logging-and-metrics` cluster is only intended for troubleshooting ECE deployment issues. If your use case involves modifying or normalizing logs from {{es}} or {{kib}}, use a separate [dedicated monitoring deployment](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md) instead.
::::

You can’t use ECE’s single sign-on (SSO) to access system deployments.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ From the deployment main page, you can quickly access the following configuratio

* Select **{{es}} > snapshots** to associate a [snapshots repository](../../tools/snapshot-and-restore/cloud-enterprise.md#ece-manage-repositories-clusters) with the deployment.

* Select **Monitoring > Logs and metrics** to set up [Stack monitoring](../../monitor/stack-monitoring/ece-stack-monitoring.md) for your deployment, forwarding its logs and metrics to a dedicated monitoring deployment.
* Select **Monitoring > Logs and metrics** to set up [Stack monitoring](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md) for your deployment, forwarding its logs and metrics to a dedicated monitoring deployment.

::::{note}
In addition to the monitoring of clusters that is described here, don’t forget that {{ece}} also provides [monitoring information for your entire installation](../../../deploy-manage/monitor/orchestrators/ece-platform-monitoring.md).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -326,7 +326,7 @@ $$$azure-integration-modify-deployment$$$How can I modify my {{ecloud}} deployme
* [Update {{stack}} user settings](edit-stack-settings.md) in the component YML files.
* [Add or remove custom plugins](add-plugins-extensions.md).
* [Configure IP filtering](../../security/traffic-filtering.md).
* [Monitor your {{ecloud}} deployment](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) to ensure it remains healthy.
* [Monitor your {{ecloud}} deployment](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md) to ensure it remains healthy.
* Add or remove API keys to use the [REST API](cloud://reference/cloud-hosted/ec-api-restful.md).
* [And more](cloud-hosted.md)

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,5 +71,5 @@ To make sure you’re all set for production, consider the following actions:
* [Add extensions and plugins](/deploy-manage/deploy/elastic-cloud/add-plugins-extensions.md) to use Elastic supported extensions or add your own custom dictionaries and scripts.
* [Edit settings and defaults](/deploy-manage/deploy/elastic-cloud/edit-stack-settings.md) to fine tune the performance of specific features.
* [Manage your deployment](/deploy-manage/deploy/elastic-cloud/manage-deployments.md) as a whole to restart, upgrade, stop routing, or delete.
* [Set up monitoring](/deploy-manage/monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) to learn how to configure your deployments for observability, which includes metric and log collection, troubleshooting views, and cluster alerts to automate performance monitoring.
* [Set up monitoring](/deploy-manage/monitor/stack-monitoring/ece-ech-stack-monitoring.md) to learn how to configure your deployments for observability, which includes metric and log collection, troubleshooting views, and cluster alerts to automate performance monitoring.

Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Autoscaling reduces some of the manual effort required to manage a deployment by

## {{es}} [ec-cluster-size]

Depending upon how much data you have and what queries you plan to run, you need to select a cluster size that fits your needs. There is no silver bullet for deciding how much memory you need other than simply testing it. The [cluster performance metrics](../../monitor/stack-monitoring.md) in the [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body) can tell you if your cluster is sized appropriately. You can also [enable deployment monitoring](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) for more detailed performance metrics. Fortunately, you can change the amount of memory allocated to the cluster later without any downtime for HA deployments.
Depending upon how much data you have and what queries you plan to run, you need to select a cluster size that fits your needs. There is no silver bullet for deciding how much memory you need other than simply testing it. The [cluster performance metrics](../../monitor/stack-monitoring.md) in the [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body) can tell you if your cluster is sized appropriately. You can also [enable deployment monitoring](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md) for more detailed performance metrics. Fortunately, you can change the amount of memory allocated to the cluster later without any downtime for HA deployments.

To change a cluster’s topology, from deployment management, select **Edit deployment** from the **Actions** dropdown. Next, select a storage and RAM setting from the **Size per zone** drop-down list, and save your changes. When downsizing the cluster, make sure to have enough resources to handle the current load, otherwise your cluster will be under stress.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,12 +42,12 @@ For more information, check [{{ech}} default provider instance configurations](c

You can check the **Monitoring > Performance > CPU Credits** section of the [{{ecloud}} Console](https://cloud.elastic.co?page=docs&placement=docs-body), and find the related metrics:

:::{image} ../../../images/cloud-metrics-credits.png
:::{image} /images/cloud-metrics-credits.png
:alt: CPU usage versus CPU credits over time
:::


## What to do if my vCPU credits get depleted constantly? [ec_what_to_do_if_my_vcpu_credits_get_depleted_constantly]

If you need your cluster to be able to sustain a certain level of performance, you cannot rely on CPU boosting to handle the workload except temporarily. To ensure that performance can be sustained, consider increasing the size of your cluster. Read [this page](../../../troubleshoot/monitoring/performance.md) for more guidance.
If you need your cluster to be able to sustain a certain level of performance, you cannot rely on CPU boosting to handle the workload except temporarily. To ensure that performance can be sustained, consider increasing the size of your cluster. Refer to [](/troubleshoot/monitoring/performance.md) for more guidance.

4 changes: 2 additions & 2 deletions deploy-manage/deploy/elastic-cloud/ech-restrictions.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ When using Elasticsearch Add-On for Heroku, there are some limitations you shoul
* [Regions and Availability Zones](#ech-regions-and-availability-zone)
* [Known problems](#ech-known-problems)

For limitations related to logging and monitoring, check the [Restrictions and limitations](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) section of the logging and monitoring page.
For limitations related to logging and monitoring, check the [Restrictions and limitations](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md) section of the logging and monitoring page.

Occasionally, we also publish information about [Known problems](#ech-known-problems) with our Elasticsearch Add-On for Heroku or the Elastic Stack.

Expand Down Expand Up @@ -53,7 +53,7 @@ Generally, if a feature is shown as available in the [Elasticsearch Add-On for H
* Elasticsearch plugins, are not enabled by default for security purposes. Please reach out to support if you would like to enable Elasticsearch plugins support on your account.
* Some Elasticsearch plugins do not apply to Elasticsearch Add-On for Heroku. For example, you won’t ever need to change discovery, as Elasticsearch Add-On for Heroku handles how nodes discover one another.
* In Elasticsearch 5.0 and later, site plugins are no longer supported. This change does not affect the site plugins Elasticsearch Add-On for Heroku might provide out of the box, such as Kopf or Head, since these site plugins are serviced by our proxies and not Elasticsearch itself.
* In Elasticsearch 5.0 and later, site plugins such as Kopf and Paramedic are no longer provided. We recommend that you use our [cluster performance metrics](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md), [X-Pack monitoring features](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) and Kibana’s (6.3+) [Index Management UI](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-mgmt.html) if you want more detailed information or perform index management actions.
* In Elasticsearch 5.0 and later, site plugins such as Kopf and Paramedic are no longer provided. We recommend that you use our [cluster performance metrics](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md), [X-Pack monitoring features](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md) and Kibana’s (6.3+) [Index Management UI](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-mgmt.html) if you want more detailed information or perform index management actions.


## Private Link and SSO to Kibana URLs [ech-restrictions-traffic-filters-kibana-sso]
Expand Down
2 changes: 1 addition & 1 deletion deploy-manage/deploy/elastic-cloud/manage-deployments.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ mapped_pages:

* Ensure the health of your deployment over time

* [Keep track of your deployment's activity](keep-track-of-deployment-activity.md) or [Enable logging and monitoring](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md) of the deployment performance.
* [Keep track of your deployment's activity](keep-track-of-deployment-activity.md) or [Enable logging and monitoring](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md) of the deployment performance.
* Perform maintenance operations to ensure the health of your deployment, such as [restarting your deployment](../../maintenance/start-stop-services/restart-cloud-hosted-deployment.md) or [stopping routing](../../maintenance/ece/start-stop-routing-requests.md).

* Manage the lifecycle of your deployment:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ When using {{ecloud}}, there are some limitations you should be aware of:
* [Regions and Availability Zones](#ec-regions-and-availability-zone)
% * [Known problems](#ec-known-problems)

For limitations related to logging and monitoring, check the [Restrictions and limitations](../../monitor/stack-monitoring/elastic-cloud-stack-monitoring.md#ec-restrictions-monitoring) section of the logging and monitoring page.
For limitations related to logging and monitoring, check the [Restrictions and limitations](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md#restrictions-monitoring) section of the logging and monitoring page.

% Occasionally, we also publish information about [Known problems](#ec-known-problems) with our {{ecloud}} or the Elastic Stack.

Expand Down
84 changes: 71 additions & 13 deletions deploy-manage/monitor.md
Original file line number Diff line number Diff line change
@@ -1,33 +1,91 @@
---
mapped_urls:
- https://www.elastic.co/guide/en/elasticsearch/reference/current/monitor-elasticsearch-cluster.html
- https://www.elastic.co/guide/en/elasticsearch/reference/current/secure-monitoring.html
- https://www.elastic.co/guide/en/cloud/current/ec-monitoring.html
applies_to:
deployment:
ess: all
ece: all
eck: all
self: all
serverless: all
---

# Monitoring

% What needs to be done: Write from scratch
Keeping on top of the health of your cluster or deployment, as well as your orchestrator, is an important part of maintenance. It also helps you to identify and troubleshoot issues. When you move to [production](/deploy-manage/production-guidance.md), detecting and resolving issues when they arise is a key component of keeping your deployment highly available.

% GitHub issue: https://github.com/elastic/docs-projects/issues/350
Depending on your deployment type, you can use a variety of solutions for monitoring your Elastic components.

% Scope notes: one link left for redirection purposes. we have to write an introduction to this big section about Monitoring and Logging. Explain what is monitoring about (metrics and logs) and how orchestrators can help by automatically setting up the beats or monitoring agents.
## Monitoring your cluster or deployment

% Use migrated content from existing pages that map to this page:
Depending on your deployment type and context, you have several options for monitoring your cluster or deployment.

% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/monitor-elasticsearch-cluster.md
% Notes: Existing articles about monitoring: Elasticsearch, Cloud, Cloud-enterprise, Cloud on Kubernetes, Kibana books Might need a new landing page
% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/secure-monitoring.md
### AutoOps (recommended)

⚠️ **This page is a work in progress.** ⚠️
```{applies_to}
deployment:
ech:
```

The documentation team is working to combine content pulled from the following pages:
:::{include} /deploy-manage/monitor/_snippets/autoops.md
:::

### Stack monitoring

```{applies_to}
deployment:
ech:
ece:
eck:
self:
```

:::{include} /deploy-manage/monitor/_snippets/stack-monitoring-def.md
:::

In {{ece}} and {{ech}}, Elastic manages the installation and configuration of the monitoring agent for you, simplifying the stack monitoring setup process.

:::{include} /deploy-manage/monitor/_snippets/stack-monitoring-prod.md
:::

### Cluster health and performance metrics

```{applies_to}
deployment:
ece:
ech:
```

{{ece}} and {{ech}} provide out of the box tools for monitoring the health of your deployment and resolving health issues when they arise:

* [Cluster health information](/deploy-manage/monitor/cloud-health-perf.md#ec-es-cluster-health), including [health warnings](/deploy-manage/monitor/cloud-health-perf.md#ec-es-health-warnings)
* A [JVM memory pressure indicator](/deploy-manage/monitor/ec-memory-pressure.md)

{{ech}} only:
* [Cluster performance information](/deploy-manage/monitor/access-performance-metrics-on-elastic-cloud.md)
* [Preconfigured logs and metrics](/deploy-manage/monitor/cloud-health-perf.md#ec-es-health-preconfigured)

{{ece}} only:
* [Platform monitoring](/deploy-manage/monitor/orchestrators/ece-platform-monitoring.md), including logs, metrics, and proxy logs

:::{tip}
Out of the box logs and metrics tools, including ECH preconfigured logs and metrics and ECE platform monitoring logs and metrics, are useful for providing information in a non-production environment. In a production environment, it’s important set up either AutoOps or stack monitoring to retain the logs and metrics that can be used to troubleshoot any health issues in your deployments. In the event of that you need to [contact our support team](/troubleshoot/index.md#contact-us), they can use the retained data to help diagnose any problems that you may encounter.
:::

To learn more about the health and performance tools in {{ecloud}}, refer to [](/deploy-manage/monitor/cloud-health-perf.md).

## Monitoring your orchestrator
```{applies_to}
deployment:
ece:
eck:
```

TODO

## Logging

TODO

% * [*Elasticsearch application logging*](../../../deploy-manage/monitor/logging-configuration/update-elasticsearch-logging-levels.md)

* [/raw-migrated-files/elasticsearch/elasticsearch-reference/monitor-elasticsearch-cluster.md](/raw-migrated-files/elasticsearch/elasticsearch-reference/monitor-elasticsearch-cluster.md)
* [/raw-migrated-files/elasticsearch/elasticsearch-reference/secure-monitoring.md](/raw-migrated-files/elasticsearch/elasticsearch-reference/secure-monitoring.md)
1 change: 1 addition & 0 deletions deploy-manage/monitor/_snippets/autoops.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
If you’re using {{ech}}, then you can use AutoOps to monitor your cluster. AutoOps significantly simplifies cluster management with performance recommendations, resource utilization visibility, real-time issue detection and resolution paths. AutoOps is [rolling out](/deploy-manage/monitor/autoops/ec-autoops-regions.md) in phases across {{ech}} regions and cloud service providers. It will be automatically activated for your deployment, with no installation required. For more information, refer to [Monitor with AutoOps](/deploy-manage/monitor/autoops.md).
Loading
Loading