Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] Updates Elasticsearch monitoring tasks #34339

Merged
merged 8 commits into from
Oct 25, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
195 changes: 195 additions & 0 deletions docs/reference/monitoring/configuring-metricbeat.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,195 @@
[role="xpack"]
[testenv="gold"]
[[configuring-metricbeat]]
=== Monitoring {es} with {metricbeat}

beta[] In 6.5 and later, you can use {metricbeat} to collect data about {es}
and ship it to the monitoring cluster, rather than routing it through exporters
as described in <<configuring-monitoring>>.

image::monitoring/images/metricbeat.png[Example monitoring architecture]

To learn about monitoring in general, see
{stack-ov}/xpack-monitoring.html[Monitoring the {stack}].

. Enable the collection of monitoring data. Set
`xpack.monitoring.collection.enabled` to `true` on the production cluster. +
+
--
For example, you can use the following APIs to review and change this setting:

[source,js]
----------------------------------
GET _cluster/settings

PUT _cluster/settings
{
"persistent": {
"xpack.monitoring.collection.enabled": true
}
}
----------------------------------
// CONSOLE

For more information, see <<monitoring-settings>> and <<cluster-update-settings>>.
--

. Disable the default collection of {es} monitoring metrics. Set
`xpack.monitoring.elasticsearch.collection.enabled` to `false` on the production
cluster. +
+
--
For example, you can use the following API to change this setting:

[source,js]
----------------------------------
PUT _cluster/settings
{
"persistent": {
"xpack.monitoring.elasticsearch.collection.enabled": false
}
}
----------------------------------
// CONSOLE

Leave `xpack.monitoring.enabled` set to its default value (`true`).
--

. On each {es} node in the production cluster:

.. {metricbeat-ref}/metricbeat-installation.html[Install {metricbeat}].

.. Enable the {es} module in {metricbeat}. +
+
--
For example, to enable the default configuration in the `modules.d` directory,
run the following command:

["source","sh",subs="attributes,callouts"]
----------------------------------------------------------------------
metricbeat modules enable elasticsearch
----------------------------------------------------------------------

For more information, see
{metricbeat-ref}/configuration-metricbeat.html[Specify which modules to run] and
{metricbeat-ref}/metricbeat-module-elasticsearch.html[{es} module].
--

.. Configure the {es} module in {metricbeat}. +
+
--
You must specify the following settings in the `modules.d/elasticsearch.yml` file:

[source,yaml]
----------------------------------
- module: elasticsearch
metricsets:
- ccr
- cluster_stats
- index
- index_recovery
- index_summary
- ml_job
- node_stats
- shard
period: 10s
hosts: ["http://localhost:9200"] <1>
xpack.enabled: true
----------------------------------
<1> This setting identifies the host and port number that are used to access {es}.
--

.. If {security} is enabled, you must also provide a user ID and password so that
{metricbeat} can collect metrics successfully.

... Create or identify a user that you want to use to collect the metrics.
+
--
TIP: There is a `remote_monitoring_user` built-in user that grants the privileges
necessary for {metricbeat} to monitor {stack} products. See
{stack-ov}/built-in-users.html[Built-in users].

Alternatively, you can choose a different user and give them the
`remote_monitoring_collector` {stack-ov}/built-in-roles.html[built-in role].
--

... Add the `username` and `password` settings to the {es} module configuration
file.
+
--
For example, add the following settings in the `modules.d/kibana.yml` file:

[source,yaml]
----------------------------------
- module: elasticsearch
...
username: remote_monitoring_user
password: YOUR_PASSWORD
----------------------------------
--

.. If you configured {es} to use <<configuring-tls,encrypted communications>>,
you must access it via HTTPS. For example, use a `hosts` setting like
`https://localhost:9200` in the `modules.d/elasticsearch.yml` file.

.. Identify where to send the monitoring data. +
+
--
TIP: In production environments, we strongly recommend using a separate cluster
(referred to as the _monitoring cluster_) to store the data. Using a separate
monitoring cluster prevents production cluster outages from impacting your
ability to access your monitoring data. It also prevents monitoring activities
from impacting the performance of your production cluster.

For example, specify the {es} output information in the {metricbeat}
configuration file (`metricbeat.yml`):

[source,yaml]
----------------------------------
output.elasticsearch:
hosts: ["http://es-mon-1:9200", "http://es-mon2:9200"] <1>
----------------------------------
<1> In this example, the data is stored on a monitoring cluster with nodes
`es-mon-1` and `es-mon-2`.

For more information about these configuration options, see
{metricbeat-ref}/elasticsearch-output.html[Configure the {es} output].
--

.. If {security} is enabled on the monitoring cluster, you must provide a valid
user ID and password so that {metricbeat} can send metrics successfully.

... Create or identify a user that you want to use to send the metrics.
+
--
TIP: There is a `remote_monitoring_user` built-in user that grants the privileges
necessary for {metricbeat} to monitor {stack} products. See
{stack-ov}/built-in-users.html[Built-in users].

Alternatively, you can choose a different user and give them the
`remote_monitoring_agent` {stack-ov}/built-in-roles.html[built-in role].
--

... Add the `username` and `password` settings to the {es} output information in
the {metricbeat} configuration file (`metricbeat.yml`):
+
--
[source,yaml]
----------------------------------
output.elasticsearch:
...
username: remote_monitoring_user
password: YOUR_PASSWORD
----------------------------------
--

.. If you configured the monitoring cluster to use
<<configuring-tls,encrypted communications>>, you must access it via
HTTPS. For example, use a `hosts` setting like `https://es-mon-1:9200` in the
`metricbeat.yml` file.

. <<starting-elasticsearch,Start {es}>>.

. {metricbeat-ref}/metricbeat-starting.html[Start {metricbeat}].

. {kibana-ref}/monitoring-data.html[View the monitoring data in {kib}].
40 changes: 29 additions & 11 deletions docs/reference/monitoring/configuring-monitoring.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,18 +6,27 @@
<titleabbrev>Configuring monitoring</titleabbrev>
++++

By default, {monitoring} is enabled but data collection is disabled. Advanced
monitoring settings enable you to control how frequently data is collected,
configure timeouts, and set the retention period for locally-stored monitoring
indices. You can also adjust how monitoring data is displayed.
If you enable the collection of monitoring data in your cluster, you can
optionally collect metrics about {es}. By default, {monitoring} is enabled but
data collection is disabled.

The following method involves sending the metrics to the monitoring cluster by
using exporters. For an alternative method, see <<configuring-metricbeat>>.

Advanced monitoring settings enable you to control how frequently data is
collected, configure timeouts, and set the retention period for locally-stored
monitoring indices. You can also adjust how monitoring data is displayed.

To learn about monitoring in general, see
{stack-ov}/xpack-monitoring.html[Monitoring the {stack}].

. To collect monitoring data about your {es} cluster:

.. Verify that the `xpack.monitoring.enabled`,
`xpack.monitoring.collection.enabled`, and
`xpack.monitoring.elasticsearch.collection.enabled` settings are `true` on each
node in the cluster. By default xpack.monitoring.collection.enabled is disabled
(`false`), and that overrides xpack.monitoring.elasticsearch.collection.enabled,
node in the cluster. By default `xpack.monitoring.collection.enabled` is disabled
(`false`), and that overrides `xpack.monitoring.elasticsearch.collection.enabled`,
which defaults to being enabled (`true`). Both settings can be set dynamically
at runtime. For more information, see <<monitoring-settings>>.

Expand Down Expand Up @@ -69,8 +78,9 @@ see {stack-ov}/how-monitoring-works.html[How Monitoring Works].
a dedicated monitoring cluster:

.. Create a user on the monitoring cluster that has the
{xpack-ref}/built-in-roles.html#built-in-roles-remote-monitoring-agent[`remote_monitoring_agent` built-in role]. For example, the following request
creates a `remote_monitor` user that has the `remote_monitoring_agent` role:
{stack-ov}/built-in-roles.html#built-in-roles-remote-monitoring-agent[`remote_monitoring_agent` built-in role].
For example, the following request creates a `remote_monitor` user that has the
`remote_monitoring_agent` role:
+
--
[source, sh]
Expand All @@ -87,12 +97,17 @@ POST /_xpack/security/user/remote_monitor
--

.. On each node in the cluster that is being monitored, configure the `http`
exporter to use the appropriate credentials when data is shipped to the monitoring cluster.
exporter to use the appropriate credentials when data is shipped to the
monitoring cluster.
+
--
If SSL/TLS is enabled on the monitoring cluster, you must use the HTTPS protocol in the `host` setting. You must also include the CA certificate in each node's trusted certificates in order to verify the identities of the nodes in the monitoring cluster.
If SSL/TLS is enabled on the monitoring cluster, you must use the HTTPS protocol
in the `host` setting. You must also include the CA certificate in each node's
trusted certificates in order to verify the identities of the nodes in the
monitoring cluster.

The following example specifies the location of the PEM encoded certificate with the `certificate_authorities` setting:
The following example specifies the location of the PEM encoded certificate with
the `certificate_authorities` setting:

[source,yaml]
--------------------------------------------------
Expand Down Expand Up @@ -144,5 +159,8 @@ stored, that is to say the monitoring cluster. To grant all of the necessary per
. Optional:
<<config-monitoring-indices,Configure the indices that store the monitoring data>>.

. {kibana-ref}/monitoring-data.html[View the monitoring data in {kib}].

include::configuring-metricbeat.asciidoc[]
include::indices.asciidoc[]
include::{es-repo-dir}/settings/monitoring-settings.asciidoc[]
Binary file added docs/reference/monitoring/images/metricbeat.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion docs/reference/monitoring/indices.asciidoc
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
[role="xpack"]
[testenv="basic"]
[[config-monitoring-indices]]
=== Configuring Indices for Monitoring
=== Configuring indices for monitoring

<<indices-templates,Index templates>> are used to configure the indices
that store the monitoring data collected from a cluster.
Expand Down
24 changes: 6 additions & 18 deletions docs/reference/settings/monitoring-settings.asciidoc
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
[role="xpack"]
[[monitoring-settings]]
=== Monitoring Settings in Elasticsearch
=== Monitoring settings in Elasticsearch
++++
<titleabbrev>Monitoring Settings</titleabbrev>
<titleabbrev>Monitoring settings</titleabbrev>
++++

By default, monitoring is enabled but data collection is disabled. To enable
Expand Down Expand Up @@ -43,17 +43,14 @@ to `true`. Its default value is `false`.
The `xpack.monitoring.collection` settings control how data is collected from
your Elasticsearch nodes.

`xpack.monitoring.collection.enabled`::
`xpack.monitoring.collection.enabled`:: (<<cluster-update-settings,Dynamic>>)

added[6.3.0] Set to `true` to enable the collection of monitoring data. When
this setting is `false` (default), {es} monitoring data is not collected and
all monitoring data from other sources such as {kib}, Beats, and Logstash is
ignored.
+
You can update this setting through the
<<cluster-update-settings,Cluster Update Settings API>>.

`xpack.monitoring.collection.interval`::
`xpack.monitoring.collection.interval`:: (<<cluster-update-settings,Dynamic>>)

Setting to `-1` to disable data collection is no longer supported beginning with
7.0.0. deprecated[6.3.0, Use `xpack.monitoring.collection.enabled` set to
Expand All @@ -62,35 +59,26 @@ Setting to `-1` to disable data collection is no longer supported beginning with
Controls how often data samples are collected. Defaults to `10s`. If you
modify the collection interval, set the `xpack.monitoring.min_interval_seconds`
option in `kibana.yml` to the same value.
+
You can update this setting through the
<<cluster-update-settings,Cluster Update Settings API>>.

`xpack.monitoring.elasticsearch.collection.enabled`::
`xpack.monitoring.elasticsearch.collection.enabled`:: (<<cluster-update-settings,Dynamic>>)

Controls whether statistics about your {es} cluster should be collected. Defaults to `true`.
This is different from xpack.monitoring.collection.enabled, which allows you to enable or disable
all monitoring collection. However, this setting simply disables the collection of Elasticsearch
data while still allowing other data (e.g., Kibana, Logstash, Beats, or APM Server monitoring data)
to pass through this cluster.
+
You can update this setting through the
<<cluster-update-settings,Cluster Update Settings API>>.

`xpack.monitoring.collection.cluster.stats.timeout`::

Sets the timeout for collecting the cluster statistics. Defaults to `10s`.

`xpack.monitoring.collection.indices`::
`xpack.monitoring.collection.indices`:: (<<cluster-update-settings,Dynamic>>)

Controls which indices Monitoring collects data from. Defaults to all indices. Specify the index names
as a comma-separated list, for example `test1,test2,test3`. Names can include wildcards, for
example `test*`. You can explicitly include or exclude indices by prepending
`+` to include the index, or `-` to exclude the index. For example, to include all indices that
start with `test` except `test3`, you could specify `+test*,-test3`.
+
You can update this setting through the
<<cluster-update-settings,Cluster Update Settings API>>.

`xpack.monitoring.collection.index.stats.timeout`::

Expand Down