diff --git a/explore-analyze/machine-learning/anomaly-detection/geographic-anomalies.md b/explore-analyze/machine-learning/anomaly-detection/geographic-anomalies.md index a6853ca8e4..5aa4856de2 100644 --- a/explore-analyze/machine-learning/anomaly-detection/geographic-anomalies.md +++ b/explore-analyze/machine-learning/anomaly-detection/geographic-anomalies.md @@ -41,7 +41,7 @@ There are a few limitations to consider before you create this type of job: 1. You cannot create forecasts for {{anomaly-jobs}} that contain geographic functions. 2. You cannot add [custom rules with conditions](/explore-analyze/machine-learning/anomaly-detection/ml-ad-run-jobs.md#ml-ad-rules) to detectors that use geographic functions. -If those limitations are acceptable, try creating an {{anomaly-job}} that uses the [`lat_long` function](/reference/data-analysis/machine-learning/ml-geo-functions.md#ml-lat-long) to analyze your own data or the sample data sets. +If those limitations are acceptable, try creating an {{anomaly-job}} that uses the [`lat_long` function](/reference/machine-learning/ml-geo-functions.md#ml-lat-long) to analyze your own data or the sample data sets. To create an {{anomaly-job}} that uses the `lat_long` function, navigate to the **Anomaly Detection Jobs** page in the main menu, or use the [global search field](../../find-and-organize/find-apps-and-objects.md). Then click **Create job** and select the appropriate job wizard. Alternatively, use the [create {{anomaly-jobs}} API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-put-job). diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-configuring-aggregation.md b/explore-analyze/machine-learning/anomaly-detection/ml-configuring-aggregation.md index efb24f0432..a66a5475c1 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-configuring-aggregation.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-configuring-aggregation.md @@ -40,8 +40,8 @@ There are a number of requirements for using aggregations in {{dfeeds}}. ## Recommendations [aggs-recommendations-dfeeds] -* When your detectors use [metric](/reference/data-analysis/machine-learning/ml-metric-functions.md) or [sum](/reference/data-analysis/machine-learning/ml-sum-functions.md) analytical functions, it’s recommended to set the `date_histogram` or `composite` aggregation interval to a tenth of the bucket span. This creates finer, more granular time buckets, which are ideal for this type of analysis. -* When your detectors use [count](/reference/data-analysis/machine-learning/ml-count-functions.md) or [rare](/reference/data-analysis/machine-learning/ml-rare-functions.md) functions, set the interval to the same value as the bucket span. +* When your detectors use [metric](/reference/machine-learning/ml-metric-functions.md) or [sum](/reference/machine-learning/ml-sum-functions.md) analytical functions, it’s recommended to set the `date_histogram` or `composite` aggregation interval to a tenth of the bucket span. This creates finer, more granular time buckets, which are ideal for this type of analysis. +* When your detectors use [count](/reference/machine-learning/ml-count-functions.md) or [rare](/reference/machine-learning/ml-rare-functions.md) functions, set the interval to the same value as the bucket span. * If you have multiple influencers or partition fields or if your field cardinality is more than 1000, use [composite aggregations](elasticsearch://reference/aggregations/search-aggregations-bucket-composite-aggregation.md). To determine the cardinality of your data, you can run searches such as: diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-configuring-categories.md b/explore-analyze/machine-learning/anomaly-detection/ml-configuring-categories.md index 9436279c79..9b8023f1e2 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-configuring-categories.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-configuring-categories.md @@ -10,7 +10,7 @@ products: # Detecting anomalous categories of data [ml-configuring-categories] -Categorization is a {{ml}} process that tokenizes a text field, clusters similar data together, and classifies it into categories. It works best on machine-written messages and application output that typically consist of repeated elements. [Categorization jobs](ml-anomaly-detection-job-types.md#categorization-jobs) enable you to find anomalous behavior in your categorized data. Categorization is not natural language processing (NLP). When you create a categorization {{anomaly-job}}, the {{ml}} model learns what volume and pattern is normal for each category over time. You can then detect anomalies and surface rare events or unusual types of messages by using [count](/reference/data-analysis/machine-learning/ml-count-functions.md) or [rare](/reference/data-analysis/machine-learning/ml-rare-functions.md) functions. Categorization works well on finite set of possible messages, for example: +Categorization is a {{ml}} process that tokenizes a text field, clusters similar data together, and classifies it into categories. It works best on machine-written messages and application output that typically consist of repeated elements. [Categorization jobs](ml-anomaly-detection-job-types.md#categorization-jobs) enable you to find anomalous behavior in your categorized data. Categorization is not natural language processing (NLP). When you create a categorization {{anomaly-job}}, the {{ml}} model learns what volume and pattern is normal for each category over time. You can then detect anomalies and surface rare events or unusual types of messages by using [count](/reference/machine-learning/ml-count-functions.md) or [rare](/reference/machine-learning/ml-rare-functions.md) functions. Categorization works well on finite set of possible messages, for example: ```js {"@timestamp":1549596476000, diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-configuring-transform.md b/explore-analyze/machine-learning/anomaly-detection/ml-configuring-transform.md index 5ef8dae9cc..9fce68cc47 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-configuring-transform.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-configuring-transform.md @@ -382,7 +382,7 @@ PUT _ml/anomaly_detectors/test3 GET _ml/datafeeds/datafeed-test3/_preview ``` -In {{es}}, location data can be stored in `geo_point` fields but this data type is not supported natively in {{ml}} analytics. This example of a runtime field transforms the data into an appropriate format. For more information, see [Geographic functions](/reference/data-analysis/machine-learning/ml-geo-functions.md). +In {{es}}, location data can be stored in `geo_point` fields but this data type is not supported natively in {{ml}} analytics. This example of a runtime field transforms the data into an appropriate format. For more information, see [Geographic functions](/reference/machine-learning/ml-geo-functions.md). The preview {{dfeed}} API returns the following results, which show that `41.44` and `90.5` have been combined into "41.44,90.5": diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-functions.md b/explore-analyze/machine-learning/anomaly-detection/ml-functions.md index b13fd45ca6..fef8cb4925 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-functions.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-functions.md @@ -21,10 +21,10 @@ You can specify a `summary_count_field_name` with any function except `metric`. If your data is sparse, there may be gaps in the data which means you might have empty buckets. You might want to treat these as anomalies or you might want these gaps to be ignored. Your decision depends on your use case and what is important to you. It also depends on which functions you use. The `sum` and `count` functions are strongly affected by empty buckets. For this reason, there are `non_null_sum` and `non_zero_count` functions, which are tolerant to sparse data. These functions effectively ignore empty buckets. -* [Count functions](/reference/data-analysis/machine-learning/ml-count-functions.md) -* [Geographic functions](/reference/data-analysis/machine-learning/ml-geo-functions.md) -* [Information content functions](/reference/data-analysis/machine-learning/ml-info-functions.md) -* [Metric functions](/reference/data-analysis/machine-learning/ml-metric-functions.md) -* [Rare functions](/reference/data-analysis/machine-learning/ml-rare-functions.md) -* [Sum functions](/reference/data-analysis/machine-learning/ml-sum-functions.md) -* [Time functions](/reference/data-analysis/machine-learning/ml-time-functions.md) +* [Count functions](/reference/machine-learning/ml-count-functions.md) +* [Geographic functions](/reference/machine-learning/ml-geo-functions.md) +* [Information content functions](/reference/machine-learning/ml-info-functions.md) +* [Metric functions](/reference/machine-learning/ml-metric-functions.md) +* [Rare functions](/reference/machine-learning/ml-rare-functions.md) +* [Sum functions](/reference/machine-learning/ml-sum-functions.md) +* [Time functions](/reference/machine-learning/ml-time-functions.md) diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-getting-started.md b/explore-analyze/machine-learning/anomaly-detection/ml-getting-started.md index becd20fd51..e3ca80bc40 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-getting-started.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-getting-started.md @@ -315,6 +315,6 @@ If you’re now thinking about where {{anomaly-detect}} can be most impactful fo In general, it is a good idea to start with single metric {{anomaly-jobs}} for your key performance indicators. After you examine these simple analysis results, you will have a better idea of what the influencers might be. You can create multi-metric jobs and split the data or create more complex analysis functions as necessary. For examples of more complicated configuration options, see [Examples](/explore-analyze/machine-learning/anomaly-detection/anomaly-how-tos.md). -If you want to find more sample jobs, see [Supplied configurations](ootb-ml-jobs.md). In particular, there are sample jobs for [Apache](/reference/data-analysis/machine-learning/ootb-ml-jobs-apache.md) and [Nginx](/reference/data-analysis/machine-learning/ootb-ml-jobs-nginx.md) that are quite similar to the examples in this tutorial. +If you want to find more sample jobs, see [Supplied configurations](ootb-ml-jobs.md). In particular, there are sample jobs for [Apache](/reference/machine-learning/ootb-ml-jobs-apache.md) and [Nginx](/reference/machine-learning/ootb-ml-jobs-nginx.md) that are quite similar to the examples in this tutorial. If you encounter problems, we’re here to help. If you are an existing Elastic customer with a support contract, create a ticket in the [Elastic Support portal](http://support.elastic.co). Or post in the [Elastic forum](https://discuss.elastic.co/). diff --git a/explore-analyze/machine-learning/anomaly-detection/ml-limitations.md b/explore-analyze/machine-learning/anomaly-detection/ml-limitations.md index 134450b68a..4c06d451e6 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ml-limitations.md +++ b/explore-analyze/machine-learning/anomaly-detection/ml-limitations.md @@ -140,22 +140,22 @@ The charts can also look odd in circumstances where there is very little data to | Detector functions | Function description | Supported | | --- | --- | --- | -| count, high_count, low_count, non_zero_count, low_non_zero_count | [Count functions](/reference/data-analysis/machine-learning/ml-count-functions.md) | yes | -| count, high_count, low_count, non_zero_count, low_non_zero_count with summary_count_field_name that is not doc_count (model plot not enabled) | [Count functions](/reference/data-analysis/machine-learning/ml-count-functions.md) | yes | -| non_zero_count with summary_count_field that is not doc_count using cardinality aggregation in datafeed config (model plot not enabled) | [Count functions](/reference/data-analysis/machine-learning/ml-count-functions.md) | yes | -| distinct_count, high_distinct_count, low_distinct_count | [Count functions](/reference/data-analysis/machine-learning/ml-count-functions.md) | yes | -| mean, high_mean, low_mean | [Mean, high_mean, low_mean](/reference/data-analysis/machine-learning/ml-metric-functions.md#ml-metric-mean) | yes | -| min | [Min](/reference/data-analysis/machine-learning/ml-metric-functions.md#ml-metric-min) | yes | -| max | [Max](/reference/data-analysis/machine-learning/ml-metric-functions.md#ml-metric-max) | yes | -| metric | [Metric](/reference/data-analysis/machine-learning/ml-metric-functions.md#ml-metric-metric) | yes | -| median, high_median, low_median | [Median, high_median, low_median](/reference/data-analysis/machine-learning/ml-metric-functions.md#ml-metric-median) | yes | -| sum, high_sum ,low_sum, non_null_sum, high_non_null_sum, low_non_null_sum | [Sum functions](/reference/data-analysis/machine-learning/ml-sum-functions.md) | yes | -| varp, high_varp, low_varp | [Varp, high_varp, low_varp](/reference/data-analysis/machine-learning/ml-metric-functions.md#ml-metric-varp) | yes (only if model plot is enabled) | -| lat_long | [Lat_long](/reference/data-analysis/machine-learning/ml-geo-functions.md#ml-lat-long) | no (but map is displayed in the Anomaly Explorer) | -| info_content, high_info_content, low_info_content | [Info_content, High_info_content, Low_info_content](/reference/data-analysis/machine-learning/ml-info-functions.md#ml-info-content) | yes (only if model plot is enabled) | -| rare | [Rare](/reference/data-analysis/machine-learning/ml-rare-functions.md#ml-rare) | yes | -| freq_rare | [Freq_rare](/reference/data-analysis/machine-learning/ml-rare-functions.md#ml-freq-rare) | no | -| time_of_day, time_of_week | [Time functions](/reference/data-analysis/machine-learning/ml-time-functions.md) | no | +| count, high_count, low_count, non_zero_count, low_non_zero_count | [Count functions](/reference/machine-learning/ml-count-functions.md) | yes | +| count, high_count, low_count, non_zero_count, low_non_zero_count with summary_count_field_name that is not doc_count (model plot not enabled) | [Count functions](/reference/machine-learning/ml-count-functions.md) | yes | +| non_zero_count with summary_count_field that is not doc_count using cardinality aggregation in datafeed config (model plot not enabled) | [Count functions](/reference/machine-learning/ml-count-functions.md) | yes | +| distinct_count, high_distinct_count, low_distinct_count | [Count functions](/reference/machine-learning/ml-count-functions.md) | yes | +| mean, high_mean, low_mean | [Mean, high_mean, low_mean](/reference/machine-learning/ml-metric-functions.md#ml-metric-mean) | yes | +| min | [Min](/reference/machine-learning/ml-metric-functions.md#ml-metric-min) | yes | +| max | [Max](/reference/machine-learning/ml-metric-functions.md#ml-metric-max) | yes | +| metric | [Metric](/reference/machine-learning/ml-metric-functions.md#ml-metric-metric) | yes | +| median, high_median, low_median | [Median, high_median, low_median](/reference/machine-learning/ml-metric-functions.md#ml-metric-median) | yes | +| sum, high_sum ,low_sum, non_null_sum, high_non_null_sum, low_non_null_sum | [Sum functions](/reference/machine-learning/ml-sum-functions.md) | yes | +| varp, high_varp, low_varp | [Varp, high_varp, low_varp](/reference/machine-learning/ml-metric-functions.md#ml-metric-varp) | yes (only if model plot is enabled) | +| lat_long | [Lat_long](/reference/machine-learning/ml-geo-functions.md#ml-lat-long) | no (but map is displayed in the Anomaly Explorer) | +| info_content, high_info_content, low_info_content | [Info_content, High_info_content, Low_info_content](/reference/machine-learning/ml-info-functions.md#ml-info-content) | yes (only if model plot is enabled) | +| rare | [Rare](/reference/machine-learning/ml-rare-functions.md#ml-rare) | yes | +| freq_rare | [Freq_rare](/reference/machine-learning/ml-rare-functions.md#ml-freq-rare) | no | +| time_of_day, time_of_week | [Time functions](/reference/machine-learning/ml-time-functions.md) | no | ### Jobs created in {{kib}} must use {{dfeeds}} [_jobs_created_in_kib_must_use_dfeeds] diff --git a/explore-analyze/machine-learning/anomaly-detection/ootb-ml-jobs.md b/explore-analyze/machine-learning/anomaly-detection/ootb-ml-jobs.md index 19d370a863..2f9308a15c 100644 --- a/explore-analyze/machine-learning/anomaly-detection/ootb-ml-jobs.md +++ b/explore-analyze/machine-learning/anomaly-detection/ootb-ml-jobs.md @@ -13,15 +13,15 @@ products: {{anomaly-jobs-cap}} contain the configuration information and metadata necessary to perform an analytics task. {{kib}} can recognize certain types of data and provide specialized wizards for that context. This page lists the categories of the {{anomaly-jobs}} that are ready to use via {{kib}} in **Machine learning**. Refer to [Create {{anomaly-jobs}}](/explore-analyze/machine-learning/anomaly-detection/ml-ad-run-jobs.md#ml-ad-create-job) to learn more about creating a job by using supplied configurations. Logs and Metrics supplied configurations are available and can be created via the related solution UI in {{kib}}. -* [Apache](/reference/data-analysis/machine-learning/ootb-ml-jobs-apache.md) -* [APM](/reference/data-analysis/machine-learning/ootb-ml-jobs-apm.md) -* [{{auditbeat}}](/reference/data-analysis/machine-learning/ootb-ml-jobs-auditbeat.md) -* [Logs](/reference/data-analysis/machine-learning/ootb-ml-jobs-logs-ui.md) -* [{{metricbeat}}](/reference/data-analysis/machine-learning/ootb-ml-jobs-metricbeat.md) -* [Metrics](/reference/data-analysis/machine-learning/ootb-ml-jobs-metrics-ui.md) -* [Nginx](/reference/data-analysis/machine-learning/ootb-ml-jobs-nginx.md) -* [Security](/reference/data-analysis/machine-learning/ootb-ml-jobs-siem.md) -* [Uptime](/reference/data-analysis/machine-learning/ootb-ml-jobs-uptime.md) +* [Apache](/reference/machine-learning/ootb-ml-jobs-apache.md) +* [APM](/reference/machine-learning/ootb-ml-jobs-apm.md) +* [{{auditbeat}}](/reference/machine-learning/ootb-ml-jobs-auditbeat.md) +* [Logs](/reference/machine-learning/ootb-ml-jobs-logs-ui.md) +* [{{metricbeat}}](/reference/machine-learning/ootb-ml-jobs-metricbeat.md) +* [Metrics](/reference/machine-learning/ootb-ml-jobs-metrics-ui.md) +* [Nginx](/reference/machine-learning/ootb-ml-jobs-nginx.md) +* [Security](/reference/machine-learning/ootb-ml-jobs-siem.md) +* [Uptime](/reference/machine-learning/ootb-ml-jobs-uptime.md) ::::{note} The configurations are only available if data exists that matches the queries specified in the manifest files. These recognizer queries are linked in the descriptions of the individual configurations. diff --git a/redirects.yml b/redirects.yml index 6c31e86c9c..f81cb02bc9 100644 --- a/redirects.yml +++ b/redirects.yml @@ -551,9 +551,36 @@ redirects: 'reference/data-analysis/kibana/canvas-functions.md': 'explore-analyze/visualize/canvas/canvas-function-reference.md' 'reference/data-analysis/kibana/tinymath-functions.md': 'explore-analyze/visualize/canvas/canvas-tinymath-functions.md' +# Related to data-analysis restructure - moved observability metrics to reference/observability + 'reference/data-analysis/observability/index.md': 'reference/observability/metrics-reference.md' + 'reference/data-analysis/observability/observability-host-metrics.md': 'reference/observability/observability-host-metrics.md' + 'reference/data-analysis/observability/observability-container-metrics.md': 'reference/observability/observability-container-metrics.md' + 'reference/data-analysis/observability/observability-kubernetes-pod-metrics.md': 'reference/observability/observability-kubernetes-pod-metrics.md' + 'reference/data-analysis/observability/observability-aws-metrics.md': 'reference/observability/observability-aws-metrics.md' + +# Renamed data-analysis to machine-learning + 'reference/data-analysis/index.md': 'reference/machine-learning/index.md' + 'reference/data-analysis/machine-learning/supplied-anomaly-detection-configurations.md': 'reference/machine-learning/supplied-anomaly-detection-configurations.md' + 'reference/data-analysis/machine-learning/machine-learning-functions.md': 'reference/machine-learning/machine-learning-functions.md' + 'reference/data-analysis/machine-learning/ml-count-functions.md': 'reference/machine-learning/ml-count-functions.md' + 'reference/data-analysis/machine-learning/ml-geo-functions.md': 'reference/machine-learning/ml-geo-functions.md' + 'reference/data-analysis/machine-learning/ml-info-functions.md': 'reference/machine-learning/ml-info-functions.md' + 'reference/data-analysis/machine-learning/ml-metric-functions.md': 'reference/machine-learning/ml-metric-functions.md' + 'reference/data-analysis/machine-learning/ml-rare-functions.md': 'reference/machine-learning/ml-rare-functions.md' + 'reference/data-analysis/machine-learning/ml-sum-functions.md': 'reference/machine-learning/ml-sum-functions.md' + 'reference/data-analysis/machine-learning/ml-time-functions.md': 'reference/machine-learning/ml-time-functions.md' + 'reference/data-analysis/machine-learning/ootb-ml-jobs-apache.md': 'reference/machine-learning/ootb-ml-jobs-apache.md' + 'reference/data-analysis/machine-learning/ootb-ml-jobs-apm.md': 'reference/machine-learning/ootb-ml-jobs-apm.md' + 'reference/data-analysis/machine-learning/ootb-ml-jobs-auditbeat.md': 'reference/machine-learning/ootb-ml-jobs-auditbeat.md' + 'reference/data-analysis/machine-learning/ootb-ml-jobs-logs-ui.md': 'reference/machine-learning/ootb-ml-jobs-logs-ui.md' + 'reference/data-analysis/machine-learning/ootb-ml-jobs-metricbeat.md': 'reference/machine-learning/ootb-ml-jobs-metricbeat.md' + 'reference/data-analysis/machine-learning/ootb-ml-jobs-metrics-ui.md': 'reference/machine-learning/ootb-ml-jobs-metrics-ui.md' + 'reference/data-analysis/machine-learning/ootb-ml-jobs-nginx.md': 'reference/machine-learning/ootb-ml-jobs-nginx.md' + 'reference/data-analysis/machine-learning/ootb-ml-jobs-siem.md': 'reference/machine-learning/ootb-ml-jobs-siem.md' + 'reference/data-analysis/machine-learning/ootb-ml-jobs-uptime.md': 'reference/machine-learning/ootb-ml-jobs-uptime.md' + # Remote cluster settings moved to reference: https://github.com/elastic/docs-content/issues/579 'deploy-manage/remote-clusters/remote-clusters-settings.md': 'elasticsearch://reference/elasticsearch/configuration-reference/remote-clusters.md' - diff --git a/reference/data-analysis/index.md b/reference/data-analysis/index.md deleted file mode 100644 index de6e9e0f49..0000000000 --- a/reference/data-analysis/index.md +++ /dev/null @@ -1,11 +0,0 @@ -# Data analysis - -This section contains reference information for data analysis features, including: - -* [Supplied {{anomaly-detect}} configurations](/reference/data-analysis/machine-learning/supplied-anomaly-detection-configurations.md) -* [Function reference for anomaly detection jobs](/reference/data-analysis/machine-learning/machine-learning-functions.md) -* [Metrics reference for the Infrastructure app](/reference/data-analysis/observability/index.md) -* [Text analysis components](elasticsearch://reference/text-analysis/index.md) -* [Aggregations](elasticsearch://reference/aggregations/index.md) - -Visit the [Explore and analyze](/explore-analyze/index.md) section to learn how to use the Elasticsearch platform to explore and analyze your data effectively. diff --git a/reference/data-analysis/observability/index.md b/reference/data-analysis/observability/index.md deleted file mode 100644 index 1a76670c34..0000000000 --- a/reference/data-analysis/observability/index.md +++ /dev/null @@ -1,22 +0,0 @@ ---- -mapped_pages: - - https://www.elastic.co/guide/en/observability/current/metrics-reference.html - - https://www.elastic.co/guide/en/serverless/current/observability-metrics-reference.html -products: - - id: observability - - id: cloud-serverless ---- - -# Metrics reference [metrics-reference] - -Learn about the key metrics displayed in the Infrastructure app and how they are calculated. - -* [Host metrics](/reference/data-analysis/observability/observability-host-metrics.md) -* [Container metrics](/reference/data-analysis/observability/observability-container-metrics.md) -* [Kubernetes pod metrics](/reference/data-analysis/observability/observability-kubernetes-pod-metrics.md) -* [AWS metrics](/reference/data-analysis/observability/observability-aws-metrics.md) - - - - - diff --git a/reference/data-analysis/toc.yml b/reference/data-analysis/toc.yml deleted file mode 100644 index 39315e9236..0000000000 --- a/reference/data-analysis/toc.yml +++ /dev/null @@ -1,28 +0,0 @@ -toc: - - file: index.md - - file: machine-learning/supplied-anomaly-detection-configurations.md - children: - - file: machine-learning/ootb-ml-jobs-apache.md - - file: machine-learning/ootb-ml-jobs-apm.md - - file: machine-learning/ootb-ml-jobs-auditbeat.md - - file: machine-learning/ootb-ml-jobs-logs-ui.md - - file: machine-learning/ootb-ml-jobs-metricbeat.md - - file: machine-learning/ootb-ml-jobs-metrics-ui.md - - file: machine-learning/ootb-ml-jobs-nginx.md - - file: machine-learning/ootb-ml-jobs-siem.md - - file: machine-learning/ootb-ml-jobs-uptime.md - - file: machine-learning/machine-learning-functions.md - children: - - file: machine-learning/ml-count-functions.md - - file: machine-learning/ml-geo-functions.md - - file: machine-learning/ml-info-functions.md - - file: machine-learning/ml-metric-functions.md - - file: machine-learning/ml-rare-functions.md - - file: machine-learning/ml-sum-functions.md - - file: machine-learning/ml-time-functions.md - - file: observability/index.md - children: - - file: observability/observability-host-metrics.md - - file: observability/observability-container-metrics.md - - file: observability/observability-kubernetes-pod-metrics.md - - file: observability/observability-aws-metrics.md \ No newline at end of file diff --git a/reference/index.md b/reference/index.md index 304ee150bd..e81dcdb087 100644 --- a/reference/index.md +++ b/reference/index.md @@ -23,41 +23,41 @@ Explore the reference documentation for [Elastic APIs]({{apis}}). | APM | • [APM Server](/solutions/observability/apm/apm-server/api.md)
• [Observability intake Serverless]({{intake-apis}})
| | {{ecloud}} | • [{{ech}}]({{cloud-apis}})
• [{{ecloud}} Serverless]({{cloud-serverless-apis}})
• [{{ece}}]({{ece-apis}})
• [{{eck}}](cloud-on-k8s://reference/api-docs.md)
• [{{ecloud}} billing]({{cloud-billing-apis}})
| -## Security +## Elasticsearch and index management -Access detailed reference documentation on field and object schemas as well as the different commands used to manage and troubleshoot Elastic Endpoint. +Customize your Elastic Stack setup with our configuration reference guides. Explore JVM settings, client documentation, Elasticsearch privileges, and index lifecycle actions to find the detailed information you need to configure your environment to your specific needs. -**Learn more in [Security](security/index.md)** +**Learn more in [Elasticsearch and index management](elasticsearch://reference/elasticsearch/index.md)** -## Observability +## Kibana -Access detailed reference documentation on field and object schemas as well as the Elastic Entity Model. +Visualize and analyze your data with Kibana. Configure advanced settings, explore plugins, and utilize command line tools to enhance your data insights. -**Learn more in [Observability](observability/index.md)** +**Learn more in [Kibana](kibana://reference/index.md)** -## Elasticsearch and index management +## Cloud -Customize your Elastic Stack setup with our configuration reference guides. Explore JVM settings, client documentation, Elasticsearch privileges, and index lifecycle actions to find the detailed information you need to configure your environment to your specific needs. +Leverage the power of the cloud with Elastic Cloud solutions. Explore Elastic Cloud on Kubernetes, Elastic Cloud Enterprise, and Elastic Cloud Hosted to scale your operations. -**Learn more in [Elasticsearch and index management](elasticsearch://reference/elasticsearch/index.md)** +**Learn more in [Cloud](cloud://reference/index.md)** -## Elastic Distributions of OpenTelemetry (EDOT) +## Elastic Security -Elastic Distributions of OpenTelemetry (EDOT) is an open-source ecosystem of OpenTelemetry distributions tailored to Elastic. They include a customized OpenTelemetry Collector and several OpenTelemetry Language SDKs. +Access detailed reference documentation on field and object schemas as well as the different commands used to manage and troubleshoot Elastic Endpoint. -**Learn more in [Elastic Distributions of OpenTelemetry](opentelemetry://reference/index.md)** +**Learn more in [Security](security/index.md)** -## Ingestion tools +## Elastic Observability -Streamline data ingestion with tools like Fleet and Elastic Agent, APM, and Beats. Explore processor references and Logstash plugins to efficiently manage your data flow. +Access detailed reference documentation on field and object schemas as well as the Elastic Entity Model. -**Learn more in [Ingestion tools](ingestion-tools/index.md)** +**Learn more in [Observability](observability/index.md)** -## Kibana +## Ingestion tools -Visualize and analyze your data with Kibana. Configure advanced settings, explore plugins, and utilize command line tools to enhance your data insights. +Streamline data ingestion with tools like Fleet and Elastic Agent, APM, Beats, and Elastic Distributions of OpenTelemetry. Explore processor references and Logstash plugins to efficiently manage your data flow. -**Learn more in [Kibana](kibana://reference/index.md)** +**Learn more in [Ingestion tools](ingestion-tools/index.md)** ## Elasticsearch plugins @@ -71,9 +71,9 @@ Master data querying with our comprehensive guides on QueryDSL, ES|QL, SQL, EQL, **Learn more in [Query languages](elasticsearch://reference/query-languages/index.md)** -## Scripting languages +## Painless scripting language -Access syntax references, function libraries, and best practices for Painless scripting. +Access syntax reference, function libraries, and best practices for Painless scripting. **Learn more in [Painless scripting](elasticsearch://reference/scripting-languages/painless/painless.md)** @@ -83,11 +83,11 @@ Standardize your data with ECS. Access logging libraries, field references, and **Learn more in [ECS](ecs://reference/index.md)** -## Data analysis +## Machine learning -Unlock insights with powerful data analysis tools. Utilize text analysis components, aggregations, and function references to derive meaningful conclusions from your data. +Explore reference content for Elastic machine learning features. -**Learn more in [Data analysis](data-analysis/index.md)** +**Learn more in [Machine learning](machine-learning/index.md)** ## Search UI library @@ -95,11 +95,4 @@ Explore reference content on the Search UI library and how you can develop fast, **Learn more in [Search UI](search-ui://reference/index.md)** -## Cloud - -Leverage the power of the cloud with Elastic Cloud solutions. Explore Elastic Cloud on Kubernetes, Elastic Cloud Enterprise, and Elastic Cloud Hosted to scale your operations. - -**Learn more in [Cloud](cloud://reference/index.md)** - - diff --git a/reference/machine-learning/index.md b/reference/machine-learning/index.md new file mode 100644 index 0000000000..bcfe6f9188 --- /dev/null +++ b/reference/machine-learning/index.md @@ -0,0 +1,8 @@ +# Machine learning + +This section contains reference information for the following machine learning features: + +* [{{kib}} {{anomaly-detect}} job wizards](/reference/machine-learning/supplied-anomaly-detection-configurations.md) +* [Function reference for anomaly detection jobs](/reference/machine-learning/machine-learning-functions.md) + +Visit the [Explore and analyze](/explore-analyze/machine-learning.md) section to learn how to use machine learning features. diff --git a/reference/data-analysis/machine-learning/machine-learning-functions.md b/reference/machine-learning/machine-learning-functions.md similarity index 78% rename from reference/data-analysis/machine-learning/machine-learning-functions.md rename to reference/machine-learning/machine-learning-functions.md index 05abe94282..e1e124211a 100644 --- a/reference/data-analysis/machine-learning/machine-learning-functions.md +++ b/reference/machine-learning/machine-learning-functions.md @@ -18,10 +18,10 @@ You can specify a `summary_count_field_name` with any function except `metric`. If your data is sparse, there may be gaps in the data which means you might have empty buckets. You might want to treat these as anomalies or you might want these gaps to be ignored. Your decision depends on your use case and what is important to you. It also depends on which functions you use. The `sum` and `count` functions are strongly affected by empty buckets. For this reason, there are `non_null_sum` and `non_zero_count` functions, which are tolerant to sparse data. These functions effectively ignore empty buckets. -* [Count functions](/reference/data-analysis/machine-learning/ml-count-functions.md) -* [Geographic functions](/reference/data-analysis/machine-learning/ml-geo-functions.md) -* [Information content functions](/reference/data-analysis/machine-learning/ml-info-functions.md) -* [Metric functions](/reference/data-analysis/machine-learning/ml-metric-functions.md) -* [Rare functions](/reference/data-analysis/machine-learning/ml-rare-functions.md) -* [Sum functions](/reference/data-analysis/machine-learning/ml-sum-functions.md) -* [Time functions](/reference/data-analysis/machine-learning/ml-time-functions.md) +* [Count functions](/reference/machine-learning/ml-count-functions.md) +* [Geographic functions](/reference/machine-learning/ml-geo-functions.md) +* [Information content functions](/reference/machine-learning/ml-info-functions.md) +* [Metric functions](/reference/machine-learning/ml-metric-functions.md) +* [Rare functions](/reference/machine-learning/ml-rare-functions.md) +* [Sum functions](/reference/machine-learning/ml-sum-functions.md) +* [Time functions](/reference/machine-learning/ml-time-functions.md) diff --git a/reference/data-analysis/machine-learning/ml-count-functions.md b/reference/machine-learning/ml-count-functions.md similarity index 97% rename from reference/data-analysis/machine-learning/ml-count-functions.md rename to reference/machine-learning/ml-count-functions.md index 5855e1824c..5d0605c33a 100644 --- a/reference/data-analysis/machine-learning/ml-count-functions.md +++ b/reference/machine-learning/ml-count-functions.md @@ -17,9 +17,9 @@ Use high-sided functions if you want to monitor unusually high event rates. Use The {{ml-features}} include the following count functions: -* [`count`, `high_count`, `low_count`](ml-count-functions.md#ml-count) -* [`non_zero_count`, `high_non_zero_count`, `low_non_zero_count`](ml-count-functions.md#ml-nonzero-count) -* [`distinct_count`, `high_distinct_count`, `low_distinct_count`](ml-count-functions.md#ml-distinct-count) +* [`count`, `high_count`, `low_count`](/reference/machine-learning/ml-count-functions.md#ml-count) +* [`non_zero_count`, `high_non_zero_count`, `low_non_zero_count`](/reference/machine-learning/ml-count-functions.md#ml-nonzero-count) +* [`distinct_count`, `high_distinct_count`, `low_distinct_count`](/reference/machine-learning/ml-count-functions.md#ml-distinct-count) ## Count, high_count, low_count [ml-count] diff --git a/reference/data-analysis/machine-learning/ml-geo-functions.md b/reference/machine-learning/ml-geo-functions.md similarity index 100% rename from reference/data-analysis/machine-learning/ml-geo-functions.md rename to reference/machine-learning/ml-geo-functions.md diff --git a/reference/data-analysis/machine-learning/ml-info-functions.md b/reference/machine-learning/ml-info-functions.md similarity index 100% rename from reference/data-analysis/machine-learning/ml-info-functions.md rename to reference/machine-learning/ml-info-functions.md diff --git a/reference/data-analysis/machine-learning/ml-metric-functions.md b/reference/machine-learning/ml-metric-functions.md similarity index 94% rename from reference/data-analysis/machine-learning/ml-metric-functions.md rename to reference/machine-learning/ml-metric-functions.md index ab273e1e59..28336f7161 100644 --- a/reference/data-analysis/machine-learning/ml-metric-functions.md +++ b/reference/machine-learning/ml-metric-functions.md @@ -11,12 +11,12 @@ The metric functions include functions such as mean, min and max. These values a The {{ml-features}} include the following metric functions: -* [`min`](ml-metric-functions.md#ml-metric-min) -* [`max`](ml-metric-functions.md#ml-metric-max) -* [`median`, `high_median`, `low_median`](ml-metric-functions.md#ml-metric-median) -* [`mean`, `high_mean`, `low_mean`](ml-metric-functions.md#ml-metric-mean) -* [`metric`](ml-metric-functions.md#ml-metric-metric) -* [`varp`, `high_varp`, `low_varp`](ml-metric-functions.md#ml-metric-varp) +* [`min`](/reference/machine-learning/ml-metric-functions.md#ml-metric-min) +* [`max`](/reference/machine-learning/ml-metric-functions.md#ml-metric-max) +* [`median`, `high_median`, `low_median`](/reference/machine-learning/ml-metric-functions.md#ml-metric-median) +* [`mean`, `high_mean`, `low_mean`](/reference/machine-learning/ml-metric-functions.md#ml-metric-mean) +* [`metric`](/reference/machine-learning/ml-metric-functions.md#ml-metric-metric) +* [`varp`, `high_varp`, `low_varp`](/reference/machine-learning/ml-metric-functions.md#ml-metric-varp) ::::{note} You cannot add rules with conditions to detectors that use the `metric` function. diff --git a/reference/data-analysis/machine-learning/ml-rare-functions.md b/reference/machine-learning/ml-rare-functions.md similarity index 97% rename from reference/data-analysis/machine-learning/ml-rare-functions.md rename to reference/machine-learning/ml-rare-functions.md index a5e5565775..30af26ad8c 100644 --- a/reference/data-analysis/machine-learning/ml-rare-functions.md +++ b/reference/machine-learning/ml-rare-functions.md @@ -23,8 +23,8 @@ The `rare` analysis detects anomalies according to the number of distinct rare v The {{ml-features}} include the following rare functions: -* [`rare`](ml-rare-functions.md#ml-rare) -* [`freq_rare`](ml-rare-functions.md#ml-freq-rare) +* [`rare`](/reference/machine-learning/ml-rare-functions.md#ml-rare) +* [`freq_rare`](/reference/machine-learning/ml-rare-functions.md#ml-freq-rare) ## Rare [ml-rare] diff --git a/reference/data-analysis/machine-learning/ml-sum-functions.md b/reference/machine-learning/ml-sum-functions.md similarity index 94% rename from reference/data-analysis/machine-learning/ml-sum-functions.md rename to reference/machine-learning/ml-sum-functions.md index 8a0f7afaec..81b58ca597 100644 --- a/reference/data-analysis/machine-learning/ml-sum-functions.md +++ b/reference/machine-learning/ml-sum-functions.md @@ -17,8 +17,8 @@ If your data is sparse, use `non_null_sum` functions. Buckets without values are The {{ml-features}} include the following sum functions: -* [`sum`, `high_sum`, `low_sum`](ml-sum-functions.md#ml-sum) -* [`non_null_sum`, `high_non_null_sum`, `low_non_null_sum`](ml-sum-functions.md#ml-nonnull-sum) +* [`sum`, `high_sum`, `low_sum`](/reference/machine-learning/ml-sum-functions.md#ml-sum) +* [`non_null_sum`, `high_non_null_sum`, `low_non_null_sum`](/reference/machine-learning/ml-sum-functions.md#ml-nonnull-sum) ## Sum, high_sum, low_sum [ml-sum] diff --git a/reference/data-analysis/machine-learning/ml-time-functions.md b/reference/machine-learning/ml-time-functions.md similarity index 96% rename from reference/data-analysis/machine-learning/ml-time-functions.md rename to reference/machine-learning/ml-time-functions.md index 283d3d6932..41c094a7f3 100644 --- a/reference/data-analysis/machine-learning/ml-time-functions.md +++ b/reference/machine-learning/ml-time-functions.md @@ -11,8 +11,8 @@ The time functions detect events that happen at unusual times, either of the day The {{ml-features}} include the following time functions: -* [`time_of_day`](ml-time-functions.md#ml-time-of-day) -* [`time_of_week`](ml-time-functions.md#ml-time-of-week) +* [`time_of_day`](/reference/machine-learning/ml-time-functions.md#ml-time-of-day) +* [`time_of_week`](/reference/machine-learning/ml-time-functions.md#ml-time-of-week) ::::{note} diff --git a/reference/data-analysis/machine-learning/ootb-ml-jobs-apache.md b/reference/machine-learning/ootb-ml-jobs-apache.md similarity index 97% rename from reference/data-analysis/machine-learning/ootb-ml-jobs-apache.md rename to reference/machine-learning/ootb-ml-jobs-apache.md index 56bad172f8..ae1d0b74ca 100644 --- a/reference/data-analysis/machine-learning/ootb-ml-jobs-apache.md +++ b/reference/machine-learning/ootb-ml-jobs-apache.md @@ -27,7 +27,7 @@ For more details, see the {{dfeed}} and job definitions in [GitHub](https://gith ## Apache access logs ({{filebeat}}) [apache-access-logs-filebeat] -These legacy {{anomaly-jobs}} find unusual activity in HTTP access logs. For the latest versions, install the Apache integration in {{fleet}}; see [Apache access logs](ootb-ml-jobs-apache.md#apache-access-logs). +These legacy {{anomaly-jobs}} find unusual activity in HTTP access logs. For the latest versions, install the Apache integration in {{fleet}}; see [Apache access logs](/reference/machine-learning/ootb-ml-jobs-apache.md#apache-access-logs). For more details, see the {{dfeed}} and job definitions in [GitHub](https://github.com/elastic/kibana/tree/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/apache_ecs/ml). diff --git a/reference/data-analysis/machine-learning/ootb-ml-jobs-apm.md b/reference/machine-learning/ootb-ml-jobs-apm.md similarity index 100% rename from reference/data-analysis/machine-learning/ootb-ml-jobs-apm.md rename to reference/machine-learning/ootb-ml-jobs-apm.md diff --git a/reference/data-analysis/machine-learning/ootb-ml-jobs-auditbeat.md b/reference/machine-learning/ootb-ml-jobs-auditbeat.md similarity index 100% rename from reference/data-analysis/machine-learning/ootb-ml-jobs-auditbeat.md rename to reference/machine-learning/ootb-ml-jobs-auditbeat.md diff --git a/reference/data-analysis/machine-learning/ootb-ml-jobs-logs-ui.md b/reference/machine-learning/ootb-ml-jobs-logs-ui.md similarity index 100% rename from reference/data-analysis/machine-learning/ootb-ml-jobs-logs-ui.md rename to reference/machine-learning/ootb-ml-jobs-logs-ui.md diff --git a/reference/data-analysis/machine-learning/ootb-ml-jobs-metricbeat.md b/reference/machine-learning/ootb-ml-jobs-metricbeat.md similarity index 100% rename from reference/data-analysis/machine-learning/ootb-ml-jobs-metricbeat.md rename to reference/machine-learning/ootb-ml-jobs-metricbeat.md diff --git a/reference/data-analysis/machine-learning/ootb-ml-jobs-metrics-ui.md b/reference/machine-learning/ootb-ml-jobs-metrics-ui.md similarity index 100% rename from reference/data-analysis/machine-learning/ootb-ml-jobs-metrics-ui.md rename to reference/machine-learning/ootb-ml-jobs-metrics-ui.md diff --git a/reference/data-analysis/machine-learning/ootb-ml-jobs-nginx.md b/reference/machine-learning/ootb-ml-jobs-nginx.md similarity index 97% rename from reference/data-analysis/machine-learning/ootb-ml-jobs-nginx.md rename to reference/machine-learning/ootb-ml-jobs-nginx.md index 5a24587be5..84209b6522 100644 --- a/reference/data-analysis/machine-learning/ootb-ml-jobs-nginx.md +++ b/reference/machine-learning/ootb-ml-jobs-nginx.md @@ -27,7 +27,7 @@ These jobs are available in {{kib}} only if data exists that matches the query s ## Nginx access logs ({{filebeat}}) [nginx-access-logs-filebeat] -These legacy {{anomaly-jobs}} find unusual activity in HTTP access logs. For the latest versions, install the Nginx integration in {{fleet}}; see [Nginx access logs](ootb-ml-jobs-nginx.md#nginx-access-logs). +These legacy {{anomaly-jobs}} find unusual activity in HTTP access logs. For the latest versions, install the Nginx integration in {{fleet}}; see [Nginx access logs](/reference/machine-learning/ootb-ml-jobs-nginx.md#nginx-access-logs). These jobs exist in {{kib}} only if data exists that matches the recognizer query specified in the [manifest file](https://github.com/elastic/kibana/blob/master/x-pack/platform/plugins/shared/ml/server/models/data_recognizer/modules/nginx_ecs/manifest.json). diff --git a/reference/data-analysis/machine-learning/ootb-ml-jobs-siem.md b/reference/machine-learning/ootb-ml-jobs-siem.md similarity index 100% rename from reference/data-analysis/machine-learning/ootb-ml-jobs-siem.md rename to reference/machine-learning/ootb-ml-jobs-siem.md diff --git a/reference/data-analysis/machine-learning/ootb-ml-jobs-uptime.md b/reference/machine-learning/ootb-ml-jobs-uptime.md similarity index 100% rename from reference/data-analysis/machine-learning/ootb-ml-jobs-uptime.md rename to reference/machine-learning/ootb-ml-jobs-uptime.md diff --git a/reference/data-analysis/machine-learning/supplied-anomaly-detection-configurations.md b/reference/machine-learning/supplied-anomaly-detection-configurations.md similarity index 64% rename from reference/data-analysis/machine-learning/supplied-anomaly-detection-configurations.md rename to reference/machine-learning/supplied-anomaly-detection-configurations.md index b43561bef1..14295b4c1c 100644 --- a/reference/data-analysis/machine-learning/supplied-anomaly-detection-configurations.md +++ b/reference/machine-learning/supplied-anomaly-detection-configurations.md @@ -1,25 +1,25 @@ --- -navigation_title: Supplied configurations +navigation_title: Kibana anomaly detection job wizards mapped_pages: - https://www.elastic.co/guide/en/machine-learning/current/ootb-ml-jobs.html products: - id: machine-learning --- -# Supplied {{anomaly-detect}} configurations [ootb-ml-jobs] +# {{kib}} {{anomaly-detect}} job wizards [ootb-ml-jobs] {{anomaly-jobs-cap}} contain the configuration information and metadata necessary to perform an analytics task. {{kib}} can recognize certain types of data and provide specialized wizards for that context. This page lists the categories of the {{anomaly-jobs}} that are ready to use via {{kib}} in **Machine learning**. Refer to [Create {{anomaly-jobs}}](/explore-analyze/machine-learning/anomaly-detection/ml-ad-run-jobs.md) to learn more about creating a job by using supplied configurations. Logs and Metrics supplied configurations are available and can be created via the related solution UI in {{kib}}. -* [Apache](/reference/data-analysis/machine-learning/ootb-ml-jobs-apache.md) -* [APM](/reference/data-analysis/machine-learning/ootb-ml-jobs-apm.md) -* [{{auditbeat}}](/reference/data-analysis/machine-learning/ootb-ml-jobs-auditbeat.md) -* [Logs](/reference/data-analysis/machine-learning/ootb-ml-jobs-logs-ui.md) -* [{{metricbeat}}](/reference/data-analysis/machine-learning/ootb-ml-jobs-metricbeat.md) -* [Metrics](/reference/data-analysis/machine-learning/ootb-ml-jobs-metrics-ui.md) -* [Nginx](/reference/data-analysis/machine-learning/ootb-ml-jobs-nginx.md) -* [Security](/reference/data-analysis/machine-learning/ootb-ml-jobs-siem.md) -* [Uptime](/reference/data-analysis/machine-learning/ootb-ml-jobs-uptime.md) +* [Apache](/reference/machine-learning/ootb-ml-jobs-apache.md) +* [APM](/reference/machine-learning/ootb-ml-jobs-apm.md) +* [{{auditbeat}}](/reference/machine-learning/ootb-ml-jobs-auditbeat.md) +* [Logs](/reference/machine-learning/ootb-ml-jobs-logs-ui.md) +* [{{metricbeat}}](/reference/machine-learning/ootb-ml-jobs-metricbeat.md) +* [Metrics](/reference/machine-learning/ootb-ml-jobs-metrics-ui.md) +* [Nginx](/reference/machine-learning/ootb-ml-jobs-nginx.md) +* [Security](/reference/machine-learning/ootb-ml-jobs-siem.md) +* [Uptime](/reference/machine-learning/ootb-ml-jobs-uptime.md) ::::{note} The configurations are only available if data exists that matches the queries specified in the manifest files. These recognizer queries are linked in the descriptions of the individual configurations. diff --git a/reference/machine-learning/toc.yml b/reference/machine-learning/toc.yml new file mode 100644 index 0000000000..ad0cd4d23a --- /dev/null +++ b/reference/machine-learning/toc.yml @@ -0,0 +1,22 @@ +toc: + - file: index.md + - file: supplied-anomaly-detection-configurations.md + children: + - file: ootb-ml-jobs-apache.md + - file: ootb-ml-jobs-apm.md + - file: ootb-ml-jobs-auditbeat.md + - file: ootb-ml-jobs-logs-ui.md + - file: ootb-ml-jobs-metricbeat.md + - file: ootb-ml-jobs-metrics-ui.md + - file: ootb-ml-jobs-nginx.md + - file: ootb-ml-jobs-siem.md + - file: ootb-ml-jobs-uptime.md + - file: machine-learning-functions.md + children: + - file: ml-count-functions.md + - file: ml-geo-functions.md + - file: ml-info-functions.md + - file: ml-metric-functions.md + - file: ml-rare-functions.md + - file: ml-sum-functions.md + - file: ml-time-functions.md \ No newline at end of file diff --git a/reference/observability/metrics-reference.md b/reference/observability/metrics-reference.md new file mode 100644 index 0000000000..d90da0506a --- /dev/null +++ b/reference/observability/metrics-reference.md @@ -0,0 +1,22 @@ +--- +mapped_pages: + - https://www.elastic.co/guide/en/observability/current/metrics-reference.html + - https://www.elastic.co/guide/en/serverless/current/observability-metrics-reference.html +products: + - id: observability + - id: cloud-serverless +--- + +# Infrastructure metrics reference [metrics-reference] + +Learn about the key metrics displayed in the [Infrastructure app](/solutions/observability/apm/infrastructure.md) and how they are calculated. + +* [Host metrics](/reference/observability/observability-host-metrics.md) +* [Container metrics](/reference/observability/observability-container-metrics.md) +* [Kubernetes pod metrics](/reference/observability/observability-kubernetes-pod-metrics.md) +* [AWS metrics](/reference/observability/observability-aws-metrics.md) + + + + + diff --git a/reference/data-analysis/observability/observability-aws-metrics.md b/reference/observability/observability-aws-metrics.md similarity index 100% rename from reference/data-analysis/observability/observability-aws-metrics.md rename to reference/observability/observability-aws-metrics.md diff --git a/reference/data-analysis/observability/observability-container-metrics.md b/reference/observability/observability-container-metrics.md similarity index 100% rename from reference/data-analysis/observability/observability-container-metrics.md rename to reference/observability/observability-container-metrics.md diff --git a/reference/data-analysis/observability/observability-host-metrics.md b/reference/observability/observability-host-metrics.md similarity index 100% rename from reference/data-analysis/observability/observability-host-metrics.md rename to reference/observability/observability-host-metrics.md diff --git a/reference/data-analysis/observability/observability-kubernetes-pod-metrics.md b/reference/observability/observability-kubernetes-pod-metrics.md similarity index 100% rename from reference/data-analysis/observability/observability-kubernetes-pod-metrics.md rename to reference/observability/observability-kubernetes-pod-metrics.md diff --git a/reference/observability/toc.yml b/reference/observability/toc.yml index c728690f3c..c0a157ad81 100644 --- a/reference/observability/toc.yml +++ b/reference/observability/toc.yml @@ -1,3 +1,9 @@ toc: - file: index.md - - file: fields-and-object-schemas.md \ No newline at end of file + - file: fields-and-object-schemas.md + - file: metrics-reference.md + children: + - file: observability-host-metrics.md + - file: observability-container-metrics.md + - file: observability-kubernetes-pod-metrics.md + - file: observability-aws-metrics.md \ No newline at end of file diff --git a/reference/toc.yml b/reference/toc.yml index f628472577..ff9b38de23 100644 --- a/reference/toc.yml +++ b/reference/toc.yml @@ -9,6 +9,6 @@ toc: - toc: apm # The next one should be child of ☝️ - toc: apm-agents - - toc: data-analysis + - toc: machine-learning - toc: glossary diff --git a/solutions/observability/infra-and-hosts.md b/solutions/observability/infra-and-hosts.md index bb4f2f8559..fdf0ad0392 100644 --- a/solutions/observability/infra-and-hosts.md +++ b/solutions/observability/infra-and-hosts.md @@ -24,4 +24,4 @@ Explore the topics in this section to learn how to observe and monitor hosts and | [Tutorial: Observe your Kubernetes deployments](/solutions/observability/infra-and-hosts/tutorial-observe-kubernetes-deployments.md) | Observe all layers of your application, including the orchestration software itself. | | [Tutorial: Observe your nginx instances](/solutions/observability/infra-and-hosts/tutorial-observe-nginx-instances.md) | Collect valuable metrics and logs from your nginx instances. | | [Troubleshooting](/troubleshoot/observability/troubleshooting-infrastructure-monitoring.md) | Troubleshoot common issues on your own or ask for help. | -| [Metrics reference](/reference/data-analysis/observability/index.md) | Learn about the key metrics displayed in the Infrastructure UI and how they are calculated. | \ No newline at end of file +| [Metrics reference](/reference/observability/metrics-reference.md) | Learn about the key metrics displayed in the Infrastructure UI and how they are calculated. | \ No newline at end of file diff --git a/solutions/observability/infra-and-hosts/analyze-compare-hosts.md b/solutions/observability/infra-and-hosts/analyze-compare-hosts.md index 114d1c25a7..1658716995 100644 --- a/solutions/observability/infra-and-hosts/analyze-compare-hosts.md +++ b/solutions/observability/infra-and-hosts/analyze-compare-hosts.md @@ -31,7 +31,7 @@ To access the **Hosts** page in: :screenshot: ::: -To learn more about the metrics shown on this page, refer to the [Metrics reference](/reference/data-analysis/observability/index.md) documentation. +To learn more about the metrics shown on this page, refer to the [Metrics reference](/reference/observability/metrics-reference.md) documentation. ::::{note} **Don’t see any metrics?** @@ -171,7 +171,7 @@ Without leaving the **Hosts** page, you can view enhanced metrics relating to ea {applies_to}`{stack: "ga 9.2", serverless: "ga"}` The host details overlay adapts according to the [selected schema](#host-schema-selector). When viewing host data collected using OpenTelemetry, you see the following differences: * Anomaly detection isn't available for OpenTelemetry hosts, so there is no **Anomalies** tab. -* The Lens charts use the [OpenTelemetry field calculation formulas](/reference/data-analysis/observability/observability-host-metrics.md#open-telemetry-host-metrics). +* The Lens charts use the [OpenTelemetry field calculation formulas](/reference/observability/observability-host-metrics.md#open-telemetry-host-metrics). ::::{tip} To expand the overlay and view more detail, click **Open as page** in the upper-right corner. diff --git a/solutions/observability/infra-and-hosts/analyze-infrastructure-host-metrics.md b/solutions/observability/infra-and-hosts/analyze-infrastructure-host-metrics.md index 9633424357..b62427afde 100644 --- a/solutions/observability/infra-and-hosts/analyze-infrastructure-host-metrics.md +++ b/solutions/observability/infra-and-hosts/analyze-infrastructure-host-metrics.md @@ -23,7 +23,7 @@ For more information, refer to the following links: * [Analyze and compare hosts](/solutions/observability/infra-and-hosts/analyze-compare-hosts.md): Use the **Hosts** page to get a metrics-driven view of your infrastructure backed by an easy-to-use interface called Lens. * [Detect metric anomalies](/solutions/observability/infra-and-hosts/detect-metric-anomalies.md): Detect and inspect memory usage and network traffic anomalies for hosts and Kubernetes pods. * [](/solutions/observability/infra-and-hosts/configure-settings.md): Learn how to configure infrastructure UI settings. -* [Metrics reference](/reference/data-analysis/observability/index.md): Learn about key metrics used for infrastructure monitoring. +* [Metrics reference](/reference/observability/metrics-reference.md): Learn about key metrics used for infrastructure monitoring. * [Infrastructure app fields](/reference/observability/fields-and-object-schemas.md): Learn about the fields required to display data in the Infrastructure UI. By default, the Infrastructure UI displays metrics from {{es}} indices that match the `metrics-*` and `metricbeat-*` index patterns. To learn how to change this behavior, refer to [Configure settings](/solutions/observability/infra-and-hosts/configure-settings.md). \ No newline at end of file diff --git a/solutions/observability/infra-and-hosts/explore-infrastructure-metrics-over-time.md b/solutions/observability/infra-and-hosts/explore-infrastructure-metrics-over-time.md index 5ece3441d3..5acdfe8df4 100644 --- a/solutions/observability/infra-and-hosts/explore-infrastructure-metrics-over-time.md +++ b/solutions/observability/infra-and-hosts/explore-infrastructure-metrics-over-time.md @@ -20,7 +20,7 @@ To open **Metrics Explorer**, find **Infrastructure** in the main menu or use th :screenshot: ::: -To learn more about the metrics shown on this page, refer to the [Metrics reference](/reference/data-analysis/observability/index.md) documentation. +To learn more about the metrics shown on this page, refer to the [Metrics reference](/reference/observability/metrics-reference.md) documentation. ::::{tip} If there are no metrics to display, {{kib}} prompts you to add a metrics integration. Click **Add a metrics integration** to get started. If you want to add more data in the future, click **Add data** from any page in the {{infrastructure-app}}. diff --git a/solutions/observability/infra-and-hosts/tutorial-observe-nginx-instances.md b/solutions/observability/infra-and-hosts/tutorial-observe-nginx-instances.md index 041593c11c..ccae44eb5e 100644 --- a/solutions/observability/infra-and-hosts/tutorial-observe-nginx-instances.md +++ b/solutions/observability/infra-and-hosts/tutorial-observe-nginx-instances.md @@ -255,19 +255,19 @@ The nginx ML module provides the following anomaly detection jobs: $$$horizontal$$$ Low request rates (`low_request_rate_nginx`) -: Uses the [`low_count`](/reference/data-analysis/machine-learning/ml-count-functions.md#ml-count) function to detect abnormally low request rates. Abnormally low request rates might indicate that network issues or other issues are preventing requests from reaching the server. +: Uses the [`low_count`](/reference/machine-learning/ml-count-functions.md#ml-count) function to detect abnormally low request rates. Abnormally low request rates might indicate that network issues or other issues are preventing requests from reaching the server. Unusual source IPs - high request rates (`source_ip_request_rate_nginx`) -: Uses the [`hight_count`](/reference/data-analysis/machine-learning/ml-count-functions.md#ml-count) function to detect abnormally high request rates from individual IP addresses. Many requests from a single IP or small group of IPs might indicate something malicious like a distributed denial of service (DDoS) attack where a large number of requests are sent to overwhelm the server and make it unavailable to users. +: Uses the [`hight_count`](/reference/machine-learning/ml-count-functions.md#ml-count) function to detect abnormally high request rates from individual IP addresses. Many requests from a single IP or small group of IPs might indicate something malicious like a distributed denial of service (DDoS) attack where a large number of requests are sent to overwhelm the server and make it unavailable to users. Unusual source IPs - high distinct count of URLs (`source_ip_url_count_nginx`) -: Uses the [`high_distinct_count`](/reference/data-analysis/machine-learning/ml-count-functions.md#ml-distinct-count) function to detect individual IP addresses accessing abnormally high numbers of unique URLs. A single IP accessing many unique URLs might indicate something malicious like web scraping or an attempt to find sensitive data or vulnerabilities. +: Uses the [`high_distinct_count`](/reference/machine-learning/ml-count-functions.md#ml-distinct-count) function to detect individual IP addresses accessing abnormally high numbers of unique URLs. A single IP accessing many unique URLs might indicate something malicious like web scraping or an attempt to find sensitive data or vulnerabilities. Unusual status code rates (`status_code_rate_nginx`) -: Uses the [`count`](/reference/data-analysis/machine-learning/ml-count-functions.md#ml-count) function to detect abnormal error status code rates. A high rate of status codes could indicate problems with broken links, bad URLs, or unauthorized access attempts. A high rate of status codes could also point to server issues like limited resources or bugs in your code. +: Uses the [`count`](/reference/machine-learning/ml-count-functions.md#ml-count) function to detect abnormal error status code rates. A high rate of status codes could indicate problems with broken links, bad URLs, or unauthorized access attempts. A high rate of status codes could also point to server issues like limited resources or bugs in your code. Unusual visitor rates (`visitor_rate_nginx`) -: Uses the [`non_zero_count`](/reference/data-analysis/machine-learning/ml-count-functions.md#ml-nonzero-count) function to detect abnormal visitor rates. High visitor rates could indicate something malicious like a DDoS attack. Low visitor rates could indicate issues with access to the server. +: Uses the [`non_zero_count`](/reference/machine-learning/ml-count-functions.md#ml-nonzero-count) function to detect abnormal visitor rates. High visitor rates could indicate something malicious like a DDoS attack. Low visitor rates could indicate issues with access to the server. ::::{note} These anomaly detection jobs are available when you have data that matches the query specified in the ML module manifest. Users not following this tutorial can refer to [nginx integration ML modules](https://docs.elastic.co/en/integrations/nginx#ml-modules) for more about the ML module manifest. diff --git a/solutions/observability/infra-and-hosts/understanding-no-results-found-message.md b/solutions/observability/infra-and-hosts/understanding-no-results-found-message.md index f7c09613c8..c7fcfa6ac2 100644 --- a/solutions/observability/infra-and-hosts/understanding-no-results-found-message.md +++ b/solutions/observability/infra-and-hosts/understanding-no-results-found-message.md @@ -11,7 +11,7 @@ products: # Understanding "no results found" message [observability-handle-no-results-found-message] -To correctly render visualizations in the Observability UI, all metrics used by the UI must be present in the collected data. For a description of these metrics, refer to [Metrics reference](/reference/data-analysis/observability/index.md). +To correctly render visualizations in the Observability UI, all metrics used by the UI must be present in the collected data. For a description of these metrics, refer to [Metrics reference](/reference/observability/metrics-reference.md). There are several reasons why metrics might be missing from the collected data: diff --git a/solutions/observability/infra-and-hosts/view-infrastructure-metrics-by-resource-type.md b/solutions/observability/infra-and-hosts/view-infrastructure-metrics-by-resource-type.md index 82f745ab1f..bfce569ea2 100644 --- a/solutions/observability/infra-and-hosts/view-infrastructure-metrics-by-resource-type.md +++ b/solutions/observability/infra-and-hosts/view-infrastructure-metrics-by-resource-type.md @@ -23,7 +23,7 @@ To open the **Infrastructure inventory** page in: :screenshot: ::: -To learn more about the metrics shown on this page, refer to the [Metrics reference](/reference/data-analysis/observability/index.md). +To learn more about the metrics shown on this page, refer to the [Metrics reference](/reference/observability/metrics-reference.md). ::::{note} **Don’t see any metrics?** diff --git a/solutions/security/advanced-entity-analytics/anomaly-detection.md b/solutions/security/advanced-entity-analytics/anomaly-detection.md index a39794600e..924bde4806 100644 --- a/solutions/security/advanced-entity-analytics/anomaly-detection.md +++ b/solutions/security/advanced-entity-analytics/anomaly-detection.md @@ -75,7 +75,7 @@ Or * You install one or more of the [Advanced Analytics integrations](/solutions/security/advanced-entity-analytics/behavioral-detection-use-cases.md#ml-integrations). -[](/reference/data-analysis/machine-learning/ootb-ml-jobs-siem.md) describes all available {{ml}} jobs and lists their requirements. For information on tuning anomaly results to reduce the number of false positives, see [Optimizing anomaly results](/solutions/security/advanced-entity-analytics/optimizing-anomaly-results.md). +[](/reference/machine-learning/ootb-ml-jobs-siem.md) describes all available {{ml}} jobs and lists their requirements. For information on tuning anomaly results to reduce the number of false positives, see [Optimizing anomaly results](/solutions/security/advanced-entity-analytics/optimizing-anomaly-results.md). ::::{note} Machine learning jobs look back and analyze two weeks of historical data prior to the time they are enabled. After jobs are enabled, they continuously analyze incoming data. When jobs are stopped and restarted within the two-week time frame, previously analyzed data is not processed again. diff --git a/solutions/security/advanced-entity-analytics/behavioral-detection-use-cases.md b/solutions/security/advanced-entity-analytics/behavioral-detection-use-cases.md index 6db629068d..13cbe6c9b4 100644 --- a/solutions/security/advanced-entity-analytics/behavioral-detection-use-cases.md +++ b/solutions/security/advanced-entity-analytics/behavioral-detection-use-cases.md @@ -39,5 +39,5 @@ Here’s a list of integrations for various behavioral detection use cases: * [Living off the Land Attack Detection](https://docs.elastic.co/en/integrations/problemchild) * [Network Beaconing Identification](https://docs.elastic.co/en/integrations/beaconing) -To learn more about {{ml}} jobs enabled by these integrations, refer to [](/reference/data-analysis/machine-learning/ootb-ml-jobs-siem.md). +To learn more about {{ml}} jobs enabled by these integrations, refer to [](/reference/machine-learning/ootb-ml-jobs-siem.md). diff --git a/solutions/security/advanced-entity-analytics/overview.md b/solutions/security/advanced-entity-analytics/overview.md index 1e794d4ebf..681b21d690 100644 --- a/solutions/security/advanced-entity-analytics/overview.md +++ b/solutions/security/advanced-entity-analytics/overview.md @@ -155,7 +155,7 @@ Interact with the table to filter data and view more details: Anomaly detection jobs identify suspicious or irregular behavior patterns. The **Anomalies** table displays the total number of anomalies identified by these prebuilt {{ml}} jobs (named in the **Anomaly name** column). :::{admonition} Requirements -To display anomaly results, you must [install and run](/explore-analyze/machine-learning/anomaly-detection/ml-ad-run-jobs.md) one or more [prebuilt anomaly detection jobs](/reference/data-analysis/machine-learning/ootb-ml-jobs-siem.md). You cannot add custom anomaly detection jobs to the **Entity analytics** page. +To display anomaly results, you must [install and run](/explore-analyze/machine-learning/anomaly-detection/ml-ad-run-jobs.md) one or more [prebuilt anomaly detection jobs](/reference/machine-learning/ootb-ml-jobs-siem.md). You cannot add custom anomaly detection jobs to the **Entity analytics** page. ::: diff --git a/solutions/security/dashboards/entity-analytics-dashboard.md b/solutions/security/dashboards/entity-analytics-dashboard.md index c2aea79785..1f78f890a5 100644 --- a/solutions/security/dashboards/entity-analytics-dashboard.md +++ b/solutions/security/dashboards/entity-analytics-dashboard.md @@ -155,7 +155,7 @@ Interact with the table to filter data and view more details: Anomaly detection jobs identify suspicious or irregular behavior patterns. The Anomalies table displays the total number of anomalies identified by these prebuilt {{ml}} jobs (named in the **Anomaly name** column). ::::{admonition} Requirements -To display anomaly results, you must [install and run](/explore-analyze/machine-learning/anomaly-detection/ml-ad-run-jobs.md) one or more [prebuilt anomaly detection jobs](/reference/data-analysis/machine-learning/ootb-ml-jobs-siem.md). You cannot add custom anomaly detection jobs to the Entity Analytics dashboard. +To display anomaly results, you must [install and run](/explore-analyze/machine-learning/anomaly-detection/ml-ad-run-jobs.md) one or more [prebuilt anomaly detection jobs](/reference/machine-learning/ootb-ml-jobs-siem.md). You cannot add custom anomaly detection jobs to the Entity Analytics dashboard. :::: diff --git a/troubleshoot/observability/troubleshooting-infrastructure-monitoring/understanding-no-results-found-message.md b/troubleshoot/observability/troubleshooting-infrastructure-monitoring/understanding-no-results-found-message.md index 5fd2e53511..a9d4f2a082 100644 --- a/troubleshoot/observability/troubleshooting-infrastructure-monitoring/understanding-no-results-found-message.md +++ b/troubleshoot/observability/troubleshooting-infrastructure-monitoring/understanding-no-results-found-message.md @@ -13,7 +13,7 @@ products: # Understand "no results found" message [handle-no-results-found-message] -To correctly render visualizations in the {{observability}} UI, all metrics used by the UI must be present in the collected data. For a description of these metrics, refer to [Metrics reference](/reference/data-analysis/observability/index.md). +To correctly render visualizations in the {{observability}} UI, all metrics used by the UI must be present in the collected data. For a description of these metrics, refer to [Metrics reference](/reference/observability/metrics-reference.md). There are several reasons why metrics might be missing from the collected data: