From f73c99c6a130553075785cef692c72e08ac07ae6 Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Wed, 13 Sep 2023 15:04:35 -0400 Subject: [PATCH 01/11] Update standalone input settings docs to match elastic-agent.yml --- .../inputs/input-configuration.asciidoc | 93 +++++++++++++------ 1 file changed, 65 insertions(+), 28 deletions(-) diff --git a/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc index 47d0a5e10..f60e787fc 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc @@ -7,38 +7,75 @@ The `inputs` section of the `elastic-agent.yml` file specifies how {agent} locates and processes input data. -By default {agent} collects system metrics, such as CPU, memory, network, and file system metrics, and sends them to the default output. For example, to define the datastream for `cpu` metrics, this is the configuration: +* <> +* <> + +[discrete] +[[elastic-agent-input-configuration-sample-metrics]] +== Sample metrics input configuration + +By default {agent} collects system metrics, such as CPU, memory, network, and file system metrics, and sends them to the default output. For example, to define datastreams for `cpu`, `memory`, `network` and `filesystem` metrics, this is the configuration: ["source","yaml"] ----------------------------------------------------------------------- -- id: unique-system-metrics-id <1> - type: system/metrics <2> - use_output: default <3> - meta: - package: <4> - name: system - version: 0.10.9 - data_stream: - namespace: default <5> +- type: system/metrics <1> + id: unique-system-metrics-id <2> + data_stream.namespace: default <3> + use_output: default <4> streams: - - data_stream: - dataset: system.cpu <6> - type: metrics <7> - metricsets: <8> - - cpu - period: 10s - cpu.metrics: - - percentages - - normalized_percentages + - metricsets: <5> + - cpu + data_stream.dataset: system.cpu <6> + - metricsets: + - memory + data_stream.dataset: system.memory + - metricsets: + - network + data_stream.dataset: system.network + - metricsets: + - filesystem + data_stream.dataset: system.filesystem ----------------------------------------------------------------------- -<1> A unique ID for the input. -<2> A generic type describing the data. -<3> The name of the `output` to use. If not specified, `default` will be used. -<4> Package specification. -<5> A user-defined namespace. -<6> A user-defined dataset. It can contain anything that makes sense to signify the source of the data. -<7> The type of the data stream. -<8> Enabled module metricsets. +<1> A generic type describing the data. +<2> A unique ID for the input. +<3> A user-defined namespace. +<4> The name of the `output` to use. If not specified, `default` will be used. +<5> The set of enabled module metricsets. + -In the {metricbeat-ref}/metricbeat-module-system.html[System module] there are several options. The `cpu` is just one of them. Its fields can be configured. \ No newline at end of file +Refer to the {metricbeat} {metricbeat-ref}/metricbeat-module-system.html[System module] for a list of available options. The metricset fields can be configured. +<6> A user-defined dataset. It can contain anything that makes sense to signify the source of the data. + +[discrete] +[[elastic-agent-input-configuration-sample-logs]] +== Sample log files input configuration + +To enable {agent} to collect log files, you can use a configuration like the following. + +["source","yaml"] +----------------------------------------------------------------------- +- type: filestream <1> + id: your-input-id <2> + streams: + - id: your-filestream-stream-id <3> + data_stream: <4> + dataset: generic + paths: + - /var/log/*.log +----------------------------------------------------------------------- + +<1> A generic type describing the data. +<2> A unique ID for the input. +<3> A unique ID for the data stream to track the state of the ingested files. +<4> The streams block is required only if multiple streams are used on the same input. Refer to the <> example to ingest a set of log files specified as an array. + +The input in this example harvests all files in the path `/var/log/*.log`, which +means that {beatname_uc} will harvest all files in the directory `/var/log/` +that end with `.log`. All patterns supported by +https://golang.org/pkg/path/filepath/#Glob[Go Glob] are also supported here. + +To fetch all files from a predefined level of subdirectories, use this pattern: +`/var/log/*/*.log`. This fetches all `.log` files from the subfolders of +`/var/log`. It does not fetch log files from the `/var/log` folder itself. +Currently it is not possible to recursively fetch all files in all +subdirectories of a directory. From 0e79411f805a155cdce0f3f99451f8396c0a2387 Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Wed, 13 Sep 2023 15:42:25 -0400 Subject: [PATCH 02/11] fixup --- .../configuration/inputs/input-configuration.asciidoc | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc index f60e787fc..611520446 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc @@ -69,13 +69,8 @@ To enable {agent} to collect log files, you can use a configuration like the fol <3> A unique ID for the data stream to track the state of the ingested files. <4> The streams block is required only if multiple streams are used on the same input. Refer to the <> example to ingest a set of log files specified as an array. -The input in this example harvests all files in the path `/var/log/*.log`, which -means that {beatname_uc} will harvest all files in the directory `/var/log/` -that end with `.log`. All patterns supported by -https://golang.org/pkg/path/filepath/#Glob[Go Glob] are also supported here. +The input in this example harvests all files in the path `/var/log/*.log`, that is, all logs in the directory `/var/log/` that end with `.log`. All patterns supported by https://golang.org/pkg/path/filepath/#Glob[Go Glob] are also supported here. To fetch all files from a predefined level of subdirectories, use this pattern: -`/var/log/*/*.log`. This fetches all `.log` files from the subfolders of -`/var/log`. It does not fetch log files from the `/var/log` folder itself. -Currently it is not possible to recursively fetch all files in all -subdirectories of a directory. +`/var/log/*/*.log`. This fetches all `.log` files from the subfolders of `/var/log`. It does not fetch log files from the `/var/log` folder itself. +Currently it is not possible to recursively fetch all files in all subdirectories of a directory. From 01ff9ece9c8b4ff28a582337261d24570eeedc7b Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Wed, 13 Sep 2023 17:12:52 -0400 Subject: [PATCH 03/11] Move simplfied input config to input page --- .../inputs/input-configuration.asciidoc | 24 +++++++++++++++++++ .../simplified-input-configuration.asciidoc | 24 ------------------- docs/en/ingest-management/index.asciidoc | 2 -- 3 files changed, 24 insertions(+), 26 deletions(-) delete mode 100644 docs/en/ingest-management/elastic-agent/configuration/inputs/simplified-input-configuration.asciidoc diff --git a/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc index 611520446..416067f56 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc @@ -9,6 +9,7 @@ The `inputs` section of the `elastic-agent.yml` file specifies how {agent} locat * <> * <> +* <> [discrete] [[elastic-agent-input-configuration-sample-metrics]] @@ -74,3 +75,26 @@ The input in this example harvests all files in the path `/var/log/*.log`, that To fetch all files from a predefined level of subdirectories, use this pattern: `/var/log/*/*.log`. This fetches all `.log` files from the subfolders of `/var/log`. It does not fetch log files from the `/var/log` folder itself. Currently it is not possible to recursively fetch all files in all subdirectories of a directory. + +[[elastic-agent-simplified-input-configuration]] += Simplified log ingestion + +There is a simplified option for ingesting log files with {agent}. +The simplest input configuration to ingest the file +`/var/log/my-application/log-file.log` is: + +["source","yaml"] +----------------------------------------------------------------------- +inputs: + - type: filestream <1> + id: unique-id-per-input <2> + paths: <3> + - /var/log/my-application/log-file.log +----------------------------------------------------------------------- + +<1> The input type must be `filestream`. +<2> A unique ID for the input. +<3> An array containing all log file paths. + +For other custom options to configure the input, refer to the +{filebeat-ref}/filebeat-input-filestream.html[filestream input] in the {filebeat} documentation. diff --git a/docs/en/ingest-management/elastic-agent/configuration/inputs/simplified-input-configuration.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/inputs/simplified-input-configuration.asciidoc deleted file mode 100644 index d7929e4fd..000000000 --- a/docs/en/ingest-management/elastic-agent/configuration/inputs/simplified-input-configuration.asciidoc +++ /dev/null @@ -1,24 +0,0 @@ -[[elastic-agent-simplified-input-configuration]] -= Simplified log ingestion - -include::{fleet-repo-dir}/standalone-note.asciidoc[] - -There is a simplified option for ingesting log files with {agent}. -The simplest input configuration to ingest the file -`/var/log/my-application/log-file.log` is: - -["source","yaml"] ------------------------------------------------------------------------ -inputs: - - type: filestream <1> - id: unique-id-per-input <2> - paths: <3> - - /var/log/my-application/log-file.log ------------------------------------------------------------------------ - -<1> The input type must be `filestream`. -<2> A unique ID for the input. -<3> An array containing all log file paths. - -For other custom options to configure the input, refer to the -{filebeat-ref}/filebeat-input-filestream.html[filestream input] in the {filebeat} documentation. \ No newline at end of file diff --git a/docs/en/ingest-management/index.asciidoc b/docs/en/ingest-management/index.asciidoc index 603d3938b..8843ceb7e 100644 --- a/docs/en/ingest-management/index.asciidoc +++ b/docs/en/ingest-management/index.asciidoc @@ -163,8 +163,6 @@ include::elastic-agent/configuration/structure-config-file.asciidoc[leveloffset= include::elastic-agent/configuration/inputs/input-configuration.asciidoc[leveloffset=+2] -include::elastic-agent/configuration/inputs/simplified-input-configuration.asciidoc[leveloffset=+3] - include::elastic-agent/elastic-agent-dynamic-inputs.asciidoc[leveloffset=+3] include::elastic-agent/configuration/providers/elastic-agent-providers.asciidoc[leveloffset=+2] From 18f6d1c38a402ad8415f48e2b6de7a4f10e3ee0d Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Wed, 13 Sep 2023 17:27:46 -0400 Subject: [PATCH 04/11] Re-add separate logging page --- .../inputs/input-configuration.asciidoc | 24 ------------------- .../simplified-input-configuration.asciidoc | 24 +++++++++++++++++++ docs/en/ingest-management/index.asciidoc | 2 ++ 3 files changed, 26 insertions(+), 24 deletions(-) create mode 100644 docs/en/ingest-management/elastic-agent/configuration/inputs/simplified-input-configuration.asciidoc diff --git a/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc index 416067f56..611520446 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc @@ -9,7 +9,6 @@ The `inputs` section of the `elastic-agent.yml` file specifies how {agent} locat * <> * <> -* <> [discrete] [[elastic-agent-input-configuration-sample-metrics]] @@ -75,26 +74,3 @@ The input in this example harvests all files in the path `/var/log/*.log`, that To fetch all files from a predefined level of subdirectories, use this pattern: `/var/log/*/*.log`. This fetches all `.log` files from the subfolders of `/var/log`. It does not fetch log files from the `/var/log` folder itself. Currently it is not possible to recursively fetch all files in all subdirectories of a directory. - -[[elastic-agent-simplified-input-configuration]] -= Simplified log ingestion - -There is a simplified option for ingesting log files with {agent}. -The simplest input configuration to ingest the file -`/var/log/my-application/log-file.log` is: - -["source","yaml"] ------------------------------------------------------------------------ -inputs: - - type: filestream <1> - id: unique-id-per-input <2> - paths: <3> - - /var/log/my-application/log-file.log ------------------------------------------------------------------------ - -<1> The input type must be `filestream`. -<2> A unique ID for the input. -<3> An array containing all log file paths. - -For other custom options to configure the input, refer to the -{filebeat-ref}/filebeat-input-filestream.html[filestream input] in the {filebeat} documentation. diff --git a/docs/en/ingest-management/elastic-agent/configuration/inputs/simplified-input-configuration.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/inputs/simplified-input-configuration.asciidoc new file mode 100644 index 000000000..d7929e4fd --- /dev/null +++ b/docs/en/ingest-management/elastic-agent/configuration/inputs/simplified-input-configuration.asciidoc @@ -0,0 +1,24 @@ +[[elastic-agent-simplified-input-configuration]] += Simplified log ingestion + +include::{fleet-repo-dir}/standalone-note.asciidoc[] + +There is a simplified option for ingesting log files with {agent}. +The simplest input configuration to ingest the file +`/var/log/my-application/log-file.log` is: + +["source","yaml"] +----------------------------------------------------------------------- +inputs: + - type: filestream <1> + id: unique-id-per-input <2> + paths: <3> + - /var/log/my-application/log-file.log +----------------------------------------------------------------------- + +<1> The input type must be `filestream`. +<2> A unique ID for the input. +<3> An array containing all log file paths. + +For other custom options to configure the input, refer to the +{filebeat-ref}/filebeat-input-filestream.html[filestream input] in the {filebeat} documentation. \ No newline at end of file diff --git a/docs/en/ingest-management/index.asciidoc b/docs/en/ingest-management/index.asciidoc index 8843ceb7e..603d3938b 100644 --- a/docs/en/ingest-management/index.asciidoc +++ b/docs/en/ingest-management/index.asciidoc @@ -163,6 +163,8 @@ include::elastic-agent/configuration/structure-config-file.asciidoc[leveloffset= include::elastic-agent/configuration/inputs/input-configuration.asciidoc[leveloffset=+2] +include::elastic-agent/configuration/inputs/simplified-input-configuration.asciidoc[leveloffset=+3] + include::elastic-agent/elastic-agent-dynamic-inputs.asciidoc[leveloffset=+3] include::elastic-agent/configuration/providers/elastic-agent-providers.asciidoc[leveloffset=+2] From 9898663ed514291e2b6fb4c041dba3ad242ccc71 Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Tue, 26 Sep 2023 18:22:26 -0400 Subject: [PATCH 05/11] Add input types tables --- .../elastic-agent-input-types.asciidoc | 252 ++++++++++++++++++ docs/en/ingest-management/index.asciidoc | 2 + 2 files changed, 254 insertions(+) create mode 100644 docs/en/ingest-management/elastic-agent/elastic-agent-input-types.asciidoc diff --git a/docs/en/ingest-management/elastic-agent/elastic-agent-input-types.asciidoc b/docs/en/ingest-management/elastic-agent/elastic-agent-input-types.asciidoc new file mode 100644 index 000000000..20579886b --- /dev/null +++ b/docs/en/ingest-management/elastic-agent/elastic-agent-input-types.asciidoc @@ -0,0 +1,252 @@ +[[elastic-agent-input-types]] += {agent} input types + +When you <> for standalone {agents}, the following values are supported for the input `type` parameter. + +* <> +* <> +* <> +* <> +* <> + +// Auditbeat +[[elastic-agent-input-types-auditbeat]] +.Audit the activities of users and processes on your systems +|=== +|Input type |Description |Learn more + +|`audit/auditd` +|Receives audit events from the Linux Audit Framework that is a part of the Linux kernel. +|{auditbeat-ref}/auditbeat-module-auditd.html[Auditd Module] ({auditbeat} docs) + +|`audit/file_integrity` +|Sends events when a file is changed (created, updated, or deleted) on disk. The events contain file metadata and hashes. +|{auditbeat-ref}/auditbeat-module-file_integrity.html[File Integrity Module] ({auditbeat} docs) + +|`audit/system` +|beta[] Collects various security related information about a system. All datasets send both periodic state information (e.g. all currently running processes) and real-time changes (e.g. when a new process starts or stops). +|{auditbeat-ref}/auditbeat-module-system.html[System Module] ({auditbeat} docs) + +|=== + +// Filebeat +[[elastic-agent-input-types-filebeat]] +.Forward and centralize log data +|=== +|Input type |Description |Learn more + +|`aws-cloudwatch` +|Stores log files +from Amazon Elastic Compute Cloud(EC2), AWS CloudTrail, Route53, and other sources. +|{filebeat-ref}/filebeat-input-aws-cloudwatch.html[AWS CloudWatch input] ({filebeat} docs) + +|`aws-s3` +|Retrieves logs from S3 objects that are pointed to by S3 notification events read from an SQS queue or directly polling list of S3 objects in an S3 bucket. +|{filebeat-ref}/filebeat-input-aws-s3.html[AWS S3 input] ({filebeat} docs) + +|`azure-blob-storage` +|Reads content from files stored in containers which reside on your Azure Cloud. +|{filebeat-ref}/filebeat-input-azure-blob-storage.html[Azure Blob Storage] ({filebeat} docs) + +|`azure-eventhub` +|Reads messages from an azure eventhub. +|{filebeat-ref}/filebeat-input-azure-eventhub.html[Azure eventhub input] ({filebeat} docs) + +|`cel` +|Reads messages from a file path or HTTP API with a variety of payloads using the https://opensource.google.com/projects/cel[Common Expression Language (CEL)] and the https://pkg.go.dev/github.com/elastic/mito/lib[mito] CEL extension libraries. +|{filebeat-ref}/filebeat-input-cel.html[Common Expression Language input] ({filebeat} docs) + +|`cloudbeat` +|? +|? + +|`cloudbeat/cis_k8s` +|CIS Kubernetes monitoring +|? + +|`cloudbeat/cis_aws` +|CIS AWS monitoring +|? + +|`cloudbeat/cis_eks` +|CIS Elastic Kubernetes monitoring +|? + +|`cloudbeat/cis_gcp` +|CIS GCP monitoring +|? + +|`cloudbeat/vuln_mgmt_aws` +|AWS Vulnerabilities management +|? + +|`cloudfoundry` +|Gets HTTP access logs, container logs and error logs from Cloud Foundry. +|{filebeat-ref}/filebeat-input-cloudfoundry.html[Cloud Foundry input] ({filebeat} docs) + +|`cometd` +|Streams the real-time events from a Salesforce generic subscription Push Topic. +|{filebeat-ref}/filebeat-input-cometd.html[CometD input] ({filebeat} docs) + +|`container` +|Reads containers log files. +|{filebeat-ref}/filebeat-input-container.html[Container input] ({filebeat} docs) + +|`docker` +|Reads Docker log files. +|? + +|`log/docker` +|Alias for `docker`. +|n/a + +|`entity-analytics` +|Collects identity assets, such as users, from external identity providers. +|{filebeat-ref}/filebeat-input-entity-analytics.html[Entity Analytics input] ({filebeat} docs) + +|`event/file` +|Alias for `log`. +|n/a + +|`event/tcp` +|Alias for `tcp`. +|n/a + +|`filestream` +|Reads lines from active log files. Replaces and imporoves on the `log` input. +|{filebeat-ref}/filebeat-input-filestream.html[filestream input] ({filebeat} docs) + +|`gcp-pubsub` +|Reads messages from a Google Cloud Pub/Sub topic subscription. +|{filebeat-ref}/filebeat-input-gcp-pubsub.html[GCP Pub/Sub input] ({filebeat} docs) + +|`gcs` +|beta[] Reads content from files stored in buckets which reside on your Google Cloud. +|{filebeat-ref}/filebeat-input-gcs.html[Google Cloud Storage input] ({filebeat} docs) + +|`http_endpoint` +|beta[] Initializes a listening HTTP server that collects incoming HTTP POST requests containing a JSON body. +|{filebeat-ref}/filebeat-input-http_endpoint.html[HTTP Endpoint input] ({filebeat} docs) + +|`httpjson` +|Read messages from an HTTP API with JSON payloads. +|{filebeat-ref}/filebeat-input-httpjson.html[HTTP JSON input] ({filebeat} docs) + +|`journald` +|beta[] A system service that collects and stores logging data. +|{filebeat-ref}/filebeat-input-journald.html[Journald input] ({filebeat} docs) + +|`kafka` +|Reads from topics in a Kafka cluster. +|{filebeat-ref}/filebeat-input-kafka.html[Kafka input] ({filebeat} docs) + +|`log` +|DEPRECATED: Please use the `filestream` input instead. +|n/a + +|`logfile` +|Alias for `log`. +|n/a + +|`log/redis_slowlog` +|Alias for `redis`. +|n/a + +|`log/syslog` +|Alias for `syslog`. +|n/a + +|`lumberjack` +|? +|? + +|`mqtt` +|Reads data transmitted using lightweight messaging protocol for small and mobile devices, optimized for high-latency or unreliable networks. +|{filebeat-ref}/filebeat-input-mqtt.html[MQTT input] ({filebeat} docs) + +|`netflow` +|Reads NetFlow and IPFIX exported flows and options records over UDP. +|{filebeat-ref}/filebeat-input-netflow.html[NetFlow input] ({filebeat} docs) + +|`o365audit` +|beta[] Retrieves audit messages from Office 365 and Azure AD activity logs. +|{filebeat-ref}/filebeat-input-o365audit.html[Office 365 Management Activity API input] ({filebeat} docs) + +|`osquery` +|Collects and decodes the result logs written by https://osquery.readthedocs.io/en/latest/introduction/using-osqueryd/[osqueryd] in the JSON format. +|? + +|`redis` +|beta[] Reads entries from Redis slowlogs. +|{filebeat-ref}/[Redis input] ({filebeat} docs) + +|`syslog` +|Reads Syslog events as specified by RFC 3164 and RFC 5424, over TCP, UDP, or a Unix stream socket. +|{filebeat-ref}/filebeat-input-syslog.html[Syslog input] ({filebeat} docs) + +|`tcp` +|Reads events over TCP. +|{filebeat-ref}/filebeat-input-tcp.html[TCP input] ({filebeat} docs) + +|`udp` +|Reads events over UDP. +|{filebeat-ref}/filebeat-input-udp.html[UDP input] ({filebeat} docs) + +|`unix` +|beta[] Reads events over a stream-oriented Unix domain socket. +|{filebeat-ref}/[Unix input] ({filebeat} docs) + +|`winlog` +|? +|? + +|=== + + +// Heartbeat +[[elastic-agent-input-types-heartbeat]] +.Periodically check the status of your services +|=== +|Input type |Description |Learn more + +|`synthetics/browser` +|? +|? + +|`synthetics/http` +|? +|? + +|`synthetics/icmp` +|? +|? + +|`synthetics/tcp` +|? +|? + +|=== + +// Metricbeat +[[elastic-agent-input-types-metricbeat]] +.Periodically collect metrics from operating systems and services running on your servers +|=== +|Input type |Description |Learn more + +|`beat/metrics` +|Collects metrics about any Beat or other software based on libbeat. +|{metricbeat-ref}/metricbeat-module-beat.html[Beat module] ({metricbeat} docs) + +|=== + +// Packetbeat +[[elastic-agent-input-types-packetbeat]] +.View network traffic between the servers of your network +|=== +|Input type |Description |Learn more + +|`packet` +|Sniffs the traffic between your servers, parses the application-level protocols on the fly, and correlates the messages into transactions. +|{packetbeat-ref}/packetbeat-overview.html[Packetbeat overview] ({packetbeat} docs) + +|=== \ No newline at end of file diff --git a/docs/en/ingest-management/index.asciidoc b/docs/en/ingest-management/index.asciidoc index 603d3938b..792e66dbc 100644 --- a/docs/en/ingest-management/index.asciidoc +++ b/docs/en/ingest-management/index.asciidoc @@ -165,6 +165,8 @@ include::elastic-agent/configuration/inputs/input-configuration.asciidoc[levelof include::elastic-agent/configuration/inputs/simplified-input-configuration.asciidoc[leveloffset=+3] +include::elastic-agent/elastic-agent-input-types.asciidoc[leveloffset=+3] + include::elastic-agent/elastic-agent-dynamic-inputs.asciidoc[leveloffset=+3] include::elastic-agent/configuration/providers/elastic-agent-providers.asciidoc[leveloffset=+2] From a1fdb82dcd133f785eee59444350a206a72afdde Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Wed, 27 Sep 2023 13:41:08 -0400 Subject: [PATCH 06/11] Add input types tables; address comments --- .../inputs/input-configuration.asciidoc | 6 +- .../configuration/inputs/input-types.asciidoc | 440 ++++++++++++++++++ .../elastic-agent-input-types.asciidoc | 232 ++++++++- docs/en/ingest-management/index.asciidoc | 2 +- 4 files changed, 654 insertions(+), 26 deletions(-) create mode 100644 docs/en/ingest-management/elastic-agent/configuration/inputs/input-types.asciidoc diff --git a/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc index 611520446..ef6e7025a 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc @@ -37,7 +37,7 @@ By default {agent} collects system metrics, such as CPU, memory, network, and fi data_stream.dataset: system.filesystem ----------------------------------------------------------------------- -<1> A generic type describing the data. +<1> The type of data to input. Refer to <> for the list of what's available. <2> A unique ID for the input. <3> A user-defined namespace. <4> The name of the `output` to use. If not specified, `default` will be used. @@ -64,10 +64,10 @@ To enable {agent} to collect log files, you can use a configuration like the fol - /var/log/*.log ----------------------------------------------------------------------- -<1> A generic type describing the data. +<1> The type of data to input. Refer to <> for the list of what's available. <2> A unique ID for the input. <3> A unique ID for the data stream to track the state of the ingested files. -<4> The streams block is required only if multiple streams are used on the same input. Refer to the <> example to ingest a set of log files specified as an array. +<4> The streams block is required only if multiple streams are used on the same input. Refer to the {filebeat} {filebeat-ref}/filebeat-input-filestream.html[filestream] documentation for a list of available options. Also, specifically for the `filestream` input type, refer to the <> for an example of ingesting a set of logs specified as an array. The input in this example harvests all files in the path `/var/log/*.log`, that is, all logs in the directory `/var/log/` that end with `.log`. All patterns supported by https://golang.org/pkg/path/filepath/#Glob[Go Glob] are also supported here. diff --git a/docs/en/ingest-management/elastic-agent/configuration/inputs/input-types.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/inputs/input-types.asciidoc new file mode 100644 index 000000000..ae5d53c03 --- /dev/null +++ b/docs/en/ingest-management/elastic-agent/configuration/inputs/input-types.asciidoc @@ -0,0 +1,440 @@ +[[elastic-agent-input-types]] += {agent} input types + +When you <> for standalone {agents}, the following values are supported for the input `type` parameter. + +Expand any section to view the available input types: + +// Auditbeat +[[elastic-agent-input-types-auditbeat]] +[%collapsible] +.Audit the activities of users and processes on your systems +==== + +|=== +|Input type |Description |Learn more + +|`audit/auditd` +|Receives audit events from the Linux Audit Framework that is a part of the Linux kernel. +|{auditbeat-ref}/auditbeat-module-auditd.html[Auditd Module] ({auditbeat} docs) + +|`audit/file_integrity` +|Sends events when a file is changed (created, updated, or deleted) on disk. The events contain file metadata and hashes. +|{auditbeat-ref}/auditbeat-module-file_integrity.html[File Integrity Module] ({auditbeat} docs) + +|`audit/system` +|beta[] Collects various security related information about a system. All datasets send both periodic state information (e.g. all currently running processes) and real-time changes (e.g. when a new process starts or stops). +|{auditbeat-ref}/auditbeat-module-system.html[System Module] ({auditbeat} docs) + +|=== + +==== + +// Metricbeat +[[elastic-agent-input-types-metricbeat]] +[%collapsible] +.Collect metrics from operating systems and services running on your servers +==== + +|=== +|Input type |Description |Learn more + +|`activemq/metrics` +|Periodically fetches JMX metrics from Apache ActiveMQ. +|{metricbeat-ref}/metricbeat-module-activemq.html[ActiveMQ module] ({metricbeat} docs) + +|`apache/metrics` +|Periodically fetches metrics from https://httpd.apache.org/[Apache HTTPD] servers. +|{metricbeat-ref}/metricbeat-module-apache.html[Apache module] ({metricbeat} docs) + +|`aws/metrics` +|Periodically fetches monitoring metrics from AWS CloudWatch using https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html[GetMetricData API] for AWS services. +|{metricbeat-ref}/metricbeat-module-aws.html[AWS module] ({metricbeat} docs) + +|`awsfargate/metrics` +|beta[] Retrieves various metadata, network metrics, and Docker stats about tasks and containers. +|{metricbeat-ref}/metricbeat-module-awsfargate.html[AWS Fargate module] ({metricbeat} docs) + +|`azure/metrics` +|Collects and aggregates Azure logs and metrics from a variety of sources into a common data platform where it can be used for analysis, visualization, and alerting. +|{metricbeat-ref}/metricbeat-module-azure.html[Azure module] ({metricbeat} docs) + +|`beat/metrics` +|Collects metrics about any Beat or other software based on libbeat. +|{metricbeat-ref}/metricbeat-module-beat.html[Beat module] ({metricbeat} docs) + +|`cloudfoundry/metrics` +|Connects to Cloud Foundry loggregator to gather container, counter, and value metrics into a common data platform where it can be used for analysis, visualization, and alerting. +|{metricbeat-ref}/metricbeat-module-cloudfoundry.html[Cloudfoundry module] ({metricbeat} docs) + +|`containerd/metrics` +|beta[] Collects cpu, memory and blkio statistics about running containers controlled by containerd runtime. +|{metricbeat-ref}/metricbeat-module-containerd.html[Containerd module] ({metricbeat} docs) + +|`docker/metrics` +|Fetches metrics from https://www.docker.com/[Docker] containers. +|{metricbeat-ref}/metricbeat-module-docker.html[Docker module] ({metricbeat} docs) + +|`elasticsearch/metrics` +|Collects metrics about {es}. +|{metricbeat-ref}/metricbeat-module-elasticsearch.html[Elasticsearch module] ({metricbeat} docs) + +|`enterprisesearch/metrics` +|Periodically fetches metrics and health information from Elastic {ents} instances using HTTP APIs. +|{metricbeat-ref}/metricbeat-module-enterprisesearch.html[{ents} module] ({metricbeat} docs) + +|`etcd/metrics` +|This module targets Etcd V2 and V3. When using V2, metrics are collected using https://coreos.com/etcd/docs/latest/v2/api.html[Etcd v2 API]. When using V3, metrics are retrieved from the `/metrics`` endpoint as intended for https://coreos.com/etcd/docs/latest/metrics.html[Etcd v3]. +|{metricbeat-ref}/metricbeat-module-etcd.html[Etcd module] ({metricbeat} docs) + +|`gcp/metrics` +|Periodically fetches monitoring metrics from Google Cloud Platform using https://cloud.google.com/monitoring/api/metrics_gcp[Stackdriver Monitoring API] for Google Cloud Platform services. +|{metricbeat-ref}/metricbeat-module-gcp.html[Google Cloud Platform module] ({metricbeat} docs) + +|`haproxy/metrics` +|Collects stats from http://www.haproxy.org/[HAProxy]. It supports collection from TCP sockets, UNIX sockets, or HTTP with or without basic authentication. +|{metricbeat-ref}/[HAProxy module] ({metricbeat} docs) + +|`http/metrics` +|Used to call arbitrary HTTP endpoints for which a dedicated Metricbeat module is not available. +|{metricbeat-ref}/metricbeat-module-http.html[HTTP module] ({metricbeat} docs) + +|`iis/metrics` +|Periodically retrieve IIS web server related metrics. +|{metricbeat-ref}/metricbeat-module-iis.html[IIS module] ({metricbeat} docs) + +|`jolokia/metrics` +|Collects metrics from https://jolokia.org/reference/html/agents.html[Jolokia agents] running on a target JMX server or dedicated proxy server. +|{metricbeat-ref}/metricbeat-module-jolokia.html[Jolokia module] ({metricbeat} docs) + +|`kafka/metrics` +|Collects metrics from the https://kafka.apache.org/intro[Apache Kafka] event streaming platform. +|{metricbeat-ref}/metricbeat-module-kafka.html[Kafka module] ({metricbeat} docs) + +|`kibana/metrics` +|Collects metrics about {Kibana}. +|{metricbeat-ref}/metricbeat-module-kibana.html[{kib} module] ({metricbeat} docs) + +|`kubernetes/metrics` +|As one of the main pieces provided for Kubernetes monitoring, this module is capable of fetching metrics from several components. +|{metricbeat-ref}/metricbeat-module-kubernetes.html[Kubernetes module] ({metricbeat} docs) + +|`linux/metrics` +|beta[] Reports on metrics exclusive to the Linux kernel and GNU/Linux OS. +|{metricbeat-ref}/metricbeat-module-linux.html[Linux module] ({metricbeat} docs) + +|`logstash/metrics` +|collects metrics about {ls}. +|{metricbeat-ref}/metricbeat-module-logstash.html[{ls} module] ({metricbeat} docs) + +|`memcached/metrics` +|Collects metrics about the https://memcached.org/[memcached] memory object caching system. +|{metricbeat-ref}/metricbeat-module-memcached.html[Memcached module] ({metricbeat} docs) + +|`mongodb/metrics` +|Periodically fetches metrics from https://www.mongodb.com/[MongoDB] servers. +|{metricbeat-ref}/metricbeat-module-mongodb.html[MongoDB module] ({metricbeat} docs) + +|`mssql/metrics` +|The https://www.microsoft.com/en-us/sql-server/sql-server-2017[Microsoft SQL 2017] Metricbeat module. It is still under active development to add new Metricsets and introduce enhancements. +|{metricbeat-ref}/metricbeat-module-mssql.html[MSSQL module] ({metricbeat} docs) + +|`mysql/metrics` +|Periodically fetches metrics from https://www.mysql.com/[MySQL] servers. +|{metricbeat-ref}/metricbeat-module-mysql.html[MySQL module] ({metricbeat} docs) + +|`nats/metrics` +|Uses the https://nats.io/documentation/managing_the_server/monitoring/[Nats monitoring server APIs] to collect metrics. +|{metricbeat-ref}/metricbeat-module-nats.html[NATS module] ({metricbeat} docs) + +|`nginx/metrics` +|Periodically fetches metrics from https://nginx.org/[Nginx] servers. +|{metricbeat-ref}/metricbeat-module-nginx.html[Nginx module] ({metricbeat} docs) + +|`oracle/metrics` +|The https://www.oracle.com/[Oracle] module for Metricbeat. It is under active development with feedback from the community. A single Metricset for Tablespace monitoring is added so the community can start gathering metrics from their nodes and contributing to the module. +|{metricbeat-ref}/metricbeat-module-oracle.html[Oracle module] ({metricbeat} docs) + +|`postgresql/metrics` +|Periodically fetches metrics from https://www.postgresql.org/[PostgreSQL] servers. +|{metricbeat-ref}/metricbeat-module-postgresql.html[PostgresSQL module] ({metricbeat} docs) + +|`prometheus/metrics` +|Periodically scrapes metrics from https://prometheus.io/docs/instrumenting/exporters/[Prometheus exporters]. +|{metricbeat-ref}/metricbeat-module-prometheus.html[Prometheus module] ({metricbeat} docs) + +|`rabbitmq/metrics` +|Uses the http://www.rabbitmq.com/management.html[HTTP API] created by the management plugin to collect RabbitMQ metrics. +|{metricbeat-ref}/metricbeat-module-rabbitmq.html[RabbitMQ module] ({metricbeat} docs) + +|`redis/metrics` +|Periodically fetches metrics from http://redis.io/[Redis] servers. +|{metricbeat-ref}/metricbeat-module-redis.html[Redis module] ({metricbeat} docs) + +|`sql/metrics` +|Allows you to execute custom queries against an SQL database and store the results in {es}. +|{metricbeat-ref}/metricbeat-module-sql.html[SQL module] ({metricbeat} docs) + +|`stan/metrics` +|Uses https://github.com/nats-io/nats-streaming-server/blob/master/server/monitor.go[STAN monitoring server APIs] to collect metrics. +|{metricbeat-ref}/metricbeat-module-stan.html[Stan module] ({metricbeat} docs) + +|`statsd/metrics` +|Spawns a UDP server and listens for metrics in StatsD compatible format. +|{metricbeat-ref}/metricbeat-module-statsd.html[Statsd module] ({metricbeat} docs) + +|`syncgateway/metrics` +|beta[] Monitor a Sync Gateway instance by using its REST API. +|{metricbeat-ref}/metricbeat-module-syncgateway.html[SyncGateway module] ({metricbeat} docs) + +|`system/metrics` +|Allows you to monitor your server metrics, including CPU, load, memory, network, processes, sockets, filesystem, fsstat, uptime, and more. +|{metricbeat-ref}/metricbeat-module-system.html[System module] ({metricbeat} docs) + +|`traefik/metrics` +|Periodically fetches metrics from a https://traefik.io/[Traefik] instance. +|{metricbeat-ref}/metricbeat-module-traefik.html[Traefik module] ({metricbeat} docs) + +|`uwsgi/metrics` +|By default, collects the uWSGI stats metricset, using https://uwsgi-docs.readthedocs.io/en/latest/StatsServer.html[StatsServer]. +|{metricbeat-ref}/metricbeat-module-uwsgi.html[uWSGI module] ({metricbeat} docs) + +|`vsphere/metrics` +|Uses the https://github.com/vmware/govmomi[Govmomi] library to collect metrics from any Vmware SDK URL (ESXi/VCenter). +|{metricbeat-ref}/metricbeat-module-vsphere.html[vSphere module] ({metricbeat} docs) + +|`windows/metrics` +|Collects metrics from Windows systems. +|{metricbeat-ref}/metricbeat-module-windows.html[Windows module] ({metricbeat} docs) + +|`zookeeper/metrics` +|Fetches statistics from the ZooKeeper service. +|{metricbeat-ref}/metricbeat-module-zookeeper.html[ZooKeeper module] ({metricbeat} docs) + +|=== + +==== + +// Filebeat +[[elastic-agent-input-types-filebeat]] +[%collapsible] +.Forward and centralize log data +==== + +|=== +|Input type |Description |Learn more + +|`aws-cloudwatch` +|Stores log files +from Amazon Elastic Compute Cloud(EC2), AWS CloudTrail, Route53, and other sources. +|{filebeat-ref}/filebeat-input-aws-cloudwatch.html[AWS CloudWatch input] ({filebeat} docs) + +|`aws-s3` +|Retrieves logs from S3 objects that are pointed to by S3 notification events read from an SQS queue or directly polling list of S3 objects in an S3 bucket. +|{filebeat-ref}/filebeat-input-aws-s3.html[AWS S3 input] ({filebeat} docs) + +|`azure-blob-storage` +|Reads content from files stored in containers which reside on your Azure Cloud. +|{filebeat-ref}/filebeat-input-azure-blob-storage.html[Azure Blob Storage] ({filebeat} docs) + +|`azure-eventhub` +|Reads messages from an azure eventhub. +|{filebeat-ref}/filebeat-input-azure-eventhub.html[Azure eventhub input] ({filebeat} docs) + +|`cel` +|Reads messages from a file path or HTTP API with a variety of payloads using the https://opensource.google.com/projects/cel[Common Expression Language (CEL)] and the https://pkg.go.dev/github.com/elastic/mito/lib[mito] CEL extension libraries. +|{filebeat-ref}/filebeat-input-cel.html[Common Expression Language input] ({filebeat} docs) + +|`cloudbeat` +|- +|- + +|`cloudbeat/cis_k8s` +|CIS Kubernetes monitoring +|- + +|`cloudbeat/cis_aws` +|CIS AWS monitoring +|- + +|`cloudbeat/cis_eks` +|CIS Elastic Kubernetes monitoring +|- + +|`cloudbeat/cis_gcp` +|CIS GCP monitoring +|- + +|`cloudbeat/vuln_mgmt_aws` +|AWS Vulnerabilities management +|- + +|`cloudfoundry` +|Gets HTTP access logs, container logs and error logs from Cloud Foundry. +|{filebeat-ref}/filebeat-input-cloudfoundry.html[Cloud Foundry input] ({filebeat} docs) + +|`cometd` +|Streams the real-time events from a Salesforce generic subscription Push Topic. +|{filebeat-ref}/filebeat-input-cometd.html[CometD input] ({filebeat} docs) + +|`container` +|Reads containers log files. +|{filebeat-ref}/filebeat-input-container.html[Container input] ({filebeat} docs) + +|`docker` +|Reads Docker log files. +|- + +|`log/docker` +|Alias for `docker`. +|n/a + +|`entity-analytics` +|Collects identity assets, such as users, from external identity providers. +|{filebeat-ref}/filebeat-input-entity-analytics.html[Entity Analytics input] ({filebeat} docs) + +|`event/file` +|Alias for `log`. +|n/a + +|`event/tcp` +|Alias for `tcp`. +|n/a + +|`filestream` +|Reads lines from active log files. Replaces and imporoves on the `log` input. +|{filebeat-ref}/filebeat-input-filestream.html[filestream input] ({filebeat} docs) + +|`gcp-pubsub` +|Reads messages from a Google Cloud Pub/Sub topic subscription. +|{filebeat-ref}/filebeat-input-gcp-pubsub.html[GCP Pub/Sub input] ({filebeat} docs) + +|`gcs` +|beta[] Reads content from files stored in buckets which reside on your Google Cloud. +|{filebeat-ref}/filebeat-input-gcs.html[Google Cloud Storage input] ({filebeat} docs) + +|`http_endpoint` +|beta[] Initializes a listening HTTP server that collects incoming HTTP POST requests containing a JSON body. +|{filebeat-ref}/filebeat-input-http_endpoint.html[HTTP Endpoint input] ({filebeat} docs) + +|`httpjson` +|Read messages from an HTTP API with JSON payloads. +|{filebeat-ref}/filebeat-input-httpjson.html[HTTP JSON input] ({filebeat} docs) + +|`journald` +|beta[] A system service that collects and stores logging data. +|{filebeat-ref}/filebeat-input-journald.html[Journald input] ({filebeat} docs) + +|`kafka` +|Reads from topics in a Kafka cluster. +|{filebeat-ref}/filebeat-input-kafka.html[Kafka input] ({filebeat} docs) + +|`log` +|DEPRECATED: Please use the `filestream` input instead. +|n/a + +|`logfile` +|Alias for `log`. +|n/a + +|`log/redis_slowlog` +|Alias for `redis`. +|n/a + +|`log/syslog` +|Alias for `syslog`. +|n/a + +|`lumberjack` +|- +|- + +|`mqtt` +|Reads data transmitted using lightweight messaging protocol for small and mobile devices, optimized for high-latency or unreliable networks. +|{filebeat-ref}/filebeat-input-mqtt.html[MQTT input] ({filebeat} docs) + +|`netflow` +|Reads NetFlow and IPFIX exported flows and options records over UDP. +|{filebeat-ref}/filebeat-input-netflow.html[NetFlow input] ({filebeat} docs) + +|`o365audit` +|beta[] Retrieves audit messages from Office 365 and Azure AD activity logs. +|{filebeat-ref}/filebeat-input-o365audit.html[Office 365 Management Activity API input] ({filebeat} docs) + +|`osquery` +|Collects and decodes the result logs written by https://osquery.readthedocs.io/en/latest/introduction/using-osqueryd/[osqueryd] in the JSON format. +| - + +|`redis` +|beta[] Reads entries from Redis slowlogs. +|{filebeat-ref}/[Redis input] ({filebeat} docs) + +|`syslog` +|Reads Syslog events as specified by RFC 3164 and RFC 5424, over TCP, UDP, or a Unix stream socket. +|{filebeat-ref}/filebeat-input-syslog.html[Syslog input] ({filebeat} docs) + +|`tcp` +|Reads events over TCP. +|{filebeat-ref}/filebeat-input-tcp.html[TCP input] ({filebeat} docs) + +|`udp` +|Reads events over UDP. +|{filebeat-ref}/filebeat-input-udp.html[UDP input] ({filebeat} docs) + +|`unix` +|beta[] Reads events over a stream-oriented Unix domain socket. +|{filebeat-ref}/[Unix input] ({filebeat} docs) + +|`winlog` +|Reads from one or more event logs using Windows APIs, filters the events based on user-configured criteria, then sends the event data to the configured outputs ({es} or {ls}). +|{winlogbeat-ref}[Winlogbeat Overview] ({winlogbeat} docs) + +|=== + +==== + +// Heartbeat +[[elastic-agent-input-types-heartbeat]] +[%collapsible] +.Monitor the status of your services +==== + +|=== +|Input type |Description |Learn more + +|`synthetics/browser` +|- +|- + +|`synthetics/http` +|Connect via HTTP and optionally verify that the host returns the expected response. +|{heartbeat-ref}/monitor-http-options.html[HTTP options] ({heartbeat} docs) + +|`synthetics/icmp` +|Use ICMP (v4 and v6) Echo Requests to check the configured hosts. +|{heartbeat-ref}/monitor-icmp-options.html[ICMP options] ({heartbeat} docs) + +|`synthetics/tcp` +|Connect via TCP and optionally verify the endpoint by sending and/or receiving a custom payload. +|{heartbeat-ref}/monitor-tcp-options.html[TCP options] ({heartbeat} docs) + +|=== + +==== + +// Packetbeat +[[elastic-agent-input-types-packetbeat]] +[%collapsible] +.View network traffic between the servers of your network +==== + +|=== +|Input type |Description |Learn more + +|`packet` +|Sniffs the traffic between your servers, parses the application-level protocols on the fly, and correlates the messages into transactions. +|{packetbeat-ref}/packetbeat-overview.html[Packetbeat overview] ({packetbeat} docs) + +|=== + +==== \ No newline at end of file diff --git a/docs/en/ingest-management/elastic-agent/elastic-agent-input-types.asciidoc b/docs/en/ingest-management/elastic-agent/elastic-agent-input-types.asciidoc index 20579886b..0aa55a22b 100644 --- a/docs/en/ingest-management/elastic-agent/elastic-agent-input-types.asciidoc +++ b/docs/en/ingest-management/elastic-agent/elastic-agent-input-types.asciidoc @@ -3,15 +3,14 @@ When you <> for standalone {agents}, the following values are supported for the input `type` parameter. -* <> -* <> -* <> -* <> -* <> +Expand any section to view the available input types: // Auditbeat [[elastic-agent-input-types-auditbeat]] +[%collapsible] .Audit the activities of users and processes on your systems +==== + |=== |Input type |Description |Learn more @@ -29,9 +28,199 @@ When you <> for standalone { |=== +==== + +// Metricbeat +[[elastic-agent-input-types-metricbeat]] +[%collapsible] +.Collect metrics from operating systems and services running on your servers +==== + +|=== +|Input type |Description |Learn more + +|`activemq/metrics` +|Periodically fetches JMX metrics from Apache ActiveMQ. +|{metricbeat-ref}/metricbeat-module-activemq.html[ActiveMQ module] ({metricbeat} docs) + +|`apache/metrics` +|Periodically fetches metrics from https://httpd.apache.org/[Apache HTTPD] servers. +|{metricbeat-ref}/metricbeat-module-apache.html[Apache module] ({metricbeat} docs) + +|`aws/metrics` +|Periodically fetches monitoring metrics from AWS CloudWatch using https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html[GetMetricData API] for AWS services. +|{metricbeat-ref}/metricbeat-module-aws.html[AWS module] ({metricbeat} docs) + +|`awsfargate/metrics` +|beta[] Retrieves various metadata, network metrics, and Docker stats about tasks and containers. +|{metricbeat-ref}/metricbeat-module-awsfargate.html[AWS Fargate module] ({metricbeat} docs) + +|`azure/metrics` +|Collects and aggregates Azure logs and metrics from a variety of sources into a common data platform where it can be used for analysis, visualization, and alerting. +|{metricbeat-ref}/metricbeat-module-azure.html[Azure module] ({metricbeat} docs) + +|`beat/metrics` +|Collects metrics about any Beat or other software based on libbeat. +|{metricbeat-ref}/metricbeat-module-beat.html[Beat module] ({metricbeat} docs) + +|`cloudfoundry/metrics` +|Connects to Cloud Foundry loggregator to gather container, counter, and value metrics into a common data platform where it can be used for analysis, visualization, and alerting. +|{metricbeat-ref}/metricbeat-module-cloudfoundry.html[Cloudfoundry module] ({metricbeat} docs) + +|`containerd/metrics` +|beta[] Collects cpu, memory and blkio statistics about running containers controlled by containerd runtime. +|{metricbeat-ref}/metricbeat-module-containerd.html[Containerd module] ({metricbeat} docs) + +|`docker/metrics` +|Fetches metrics from https://www.docker.com/[Docker] containers. +|{metricbeat-ref}/metricbeat-module-docker.html[Docker module] ({metricbeat} docs) + +|`elasticsearch/metrics` +|Collects metrics about {es}. +|{metricbeat-ref}/metricbeat-module-elasticsearch.html[Elasticsearch module] ({metricbeat} docs) + +|`enterprisesearch/metrics` +|Periodically fetches metrics and health information from Elastic {ents} instances using HTTP APIs. +|{metricbeat-ref}/metricbeat-module-enterprisesearch.html[{ents} module] ({metricbeat} docs) + +|`etcd/metrics` +|This module targets Etcd V2 and V3. When using V2, metrics are collected using https://coreos.com/etcd/docs/latest/v2/api.html[Etcd v2 API]. When using V3, metrics are retrieved from the `/metrics`` endpoint as intended for https://coreos.com/etcd/docs/latest/metrics.html[Etcd v3]. +|{metricbeat-ref}/metricbeat-module-etcd.html[Etcd module] ({metricbeat} docs) + +|`gcp/metrics` +|Periodically fetches monitoring metrics from Google Cloud Platform using https://cloud.google.com/monitoring/api/metrics_gcp[Stackdriver Monitoring API] for Google Cloud Platform services. +|{metricbeat-ref}/metricbeat-module-gcp.html[Google Cloud Platform module] ({metricbeat} docs) + +|`haproxy/metrics` +|Collects stats from http://www.haproxy.org/[HAProxy]. It supports collection from TCP sockets, UNIX sockets, or HTTP with or without basic authentication. +|{metricbeat-ref}/[HAProxy module] ({metricbeat} docs) + +|`http/metrics` +|Used to call arbitrary HTTP endpoints for which a dedicated Metricbeat module is not available. +|{metricbeat-ref}/metricbeat-module-http.html[HTTP module] ({metricbeat} docs) + +|`iis/metrics` +|Periodically retrieve IIS web server related metrics. +|{metricbeat-ref}/metricbeat-module-iis.html[IIS module] ({metricbeat} docs) + +|`jolokia/metrics` +|Collects metrics from https://jolokia.org/reference/html/agents.html[Jolokia agents] running on a target JMX server or dedicated proxy server. +|{metricbeat-ref}/metricbeat-module-jolokia.html[Jolokia module] ({metricbeat} docs) + +|`kafka/metrics` +|Collects metrics from the https://kafka.apache.org/intro[Apache Kafka] event streaming platform. +|{metricbeat-ref}/metricbeat-module-kafka.html[Kafka module] ({metricbeat} docs) + +|`kibana/metrics` +|Collects metrics about {Kibana}. +|{metricbeat-ref}/metricbeat-module-kibana.html[{kib} module] ({metricbeat} docs) + +|`kubernetes/metrics` +|As one of the main pieces provided for Kubernetes monitoring, this module is capable of fetching metrics from several components. +|{metricbeat-ref}/metricbeat-module-kubernetes.html[Kubernetes module] ({metricbeat} docs) + +|`linux/metrics` +|beta[] Reports on metrics exclusive to the Linux kernel and GNU/Linux OS. +|{metricbeat-ref}/metricbeat-module-linux.html[Linux module] ({metricbeat} docs) + +|`logstash/metrics` +|collects metrics about {ls}. +|{metricbeat-ref}/metricbeat-module-logstash.html[{ls} module] ({metricbeat} docs) + +|`memcached/metrics` +|Collects metrics about the https://memcached.org/[memcached] memory object caching system. +|{metricbeat-ref}/metricbeat-module-memcached.html[Memcached module] ({metricbeat} docs) + +|`mongodb/metrics` +|Periodically fetches metrics from https://www.mongodb.com/[MongoDB] servers. +|{metricbeat-ref}/metricbeat-module-mongodb.html[MongoDB module] ({metricbeat} docs) + +|`mssql/metrics` +|The https://www.microsoft.com/en-us/sql-server/sql-server-2017[Microsoft SQL 2017] Metricbeat module. It is still under active development to add new Metricsets and introduce enhancements. +|{metricbeat-ref}/metricbeat-module-mssql.html[MSSQL module] ({metricbeat} docs) + +|`mysql/metrics` +|Periodically fetches metrics from https://www.mysql.com/[MySQL] servers. +|{metricbeat-ref}/metricbeat-module-mysql.html[MySQL module] ({metricbeat} docs) + +|`nats/metrics` +|Uses the https://nats.io/documentation/managing_the_server/monitoring/[Nats monitoring server APIs] to collect metrics. +|{metricbeat-ref}/metricbeat-module-nats.html[NATS module] ({metricbeat} docs) + +|`nginx/metrics` +|Periodically fetches metrics from https://nginx.org/[Nginx] servers. +|{metricbeat-ref}/metricbeat-module-nginx.html[Nginx module] ({metricbeat} docs) + +|`oracle/metrics` +|The https://www.oracle.com/[Oracle] module for Metricbeat. It is under active development with feedback from the community. A single Metricset for Tablespace monitoring is added so the community can start gathering metrics from their nodes and contributing to the module. +|{metricbeat-ref}/metricbeat-module-oracle.html[Oracle module] ({metricbeat} docs) + +|`postgresql/metrics` +|Periodically fetches metrics from https://www.postgresql.org/[PostgreSQL] servers. +|{metricbeat-ref}/metricbeat-module-postgresql.html[PostgresSQL module] ({metricbeat} docs) + +|`prometheus/metrics` +|Periodically scrapes metrics from https://prometheus.io/docs/instrumenting/exporters/[Prometheus exporters]. +|{metricbeat-ref}/metricbeat-module-prometheus.html[Prometheus module] ({metricbeat} docs) + +|`rabbitmq/metrics` +|Uses the http://www.rabbitmq.com/management.html[HTTP API] created by the management plugin to collect RabbitMQ metrics. +|{metricbeat-ref}/metricbeat-module-rabbitmq.html[RabbitMQ module] ({metricbeat} docs) + +|`redis/metrics` +|Periodically fetches metrics from http://redis.io/[Redis] servers. +|{metricbeat-ref}/metricbeat-module-redis.html[Redis module] ({metricbeat} docs) + +|`sql/metrics` +|Allows you to execute custom queries against an SQL database and store the results in {es}. +|{metricbeat-ref}/metricbeat-module-sql.html[SQL module] ({metricbeat} docs) + +|`stan/metrics` +|Uses https://github.com/nats-io/nats-streaming-server/blob/master/server/monitor.go[STAN monitoring server APIs] to collect metrics. +|{metricbeat-ref}/metricbeat-module-stan.html[Stan module] ({metricbeat} docs) + +|`statsd/metrics` +|Spawns a UDP server and listens for metrics in StatsD compatible format. +|{metricbeat-ref}/metricbeat-module-statsd.html[Statsd module] ({metricbeat} docs) + +|`syncgateway/metrics` +|beta[] Monitor a Sync Gateway instance by using its REST API. +|{metricbeat-ref}/metricbeat-module-syncgateway.html[SyncGateway module] ({metricbeat} docs) + +|`system/metrics` +|Allows you to monitor your server metrics, including CPU, load, memory, network, processes, sockets, filesystem, fsstat, uptime, and more. +|{metricbeat-ref}/metricbeat-module-system.html[System module] ({metricbeat} docs) + +|`traefik/metrics` +|Periodically fetches metrics from a https://traefik.io/[Traefik] instance. +|{metricbeat-ref}/metricbeat-module-traefik.html[Traefik module] ({metricbeat} docs) + +|`uwsgi/metrics` +|By default, collects the uWSGI stats metricset, using https://uwsgi-docs.readthedocs.io/en/latest/StatsServer.html[StatsServer]. +|{metricbeat-ref}/metricbeat-module-uwsgi.html[uWSGI module] ({metricbeat} docs) + +|`vsphere/metrics` +|Uses the https://github.com/vmware/govmomi[Govmomi] library to collect metrics from any Vmware SDK URL (ESXi/VCenter). +|{metricbeat-ref}/metricbeat-module-vsphere.html[vSphere module] ({metricbeat} docs) + +|`windows/metrics` +|Collects metrics from Windows systems. +|{metricbeat-ref}/metricbeat-module-windows.html[Windows module] ({metricbeat} docs) + +|`zookeeper/metrics` +|Fetches statistics from the ZooKeeper service. +|{metricbeat-ref}/metricbeat-module-zookeeper.html[ZooKeeper module] ({metricbeat} docs) + +|=== + +==== + // Filebeat [[elastic-agent-input-types-filebeat]] +[%collapsible] .Forward and centralize log data +==== + |=== |Input type |Description |Learn more @@ -94,7 +283,7 @@ from Amazon Elastic Compute Cloud(EC2), AWS CloudTrail, Route53, and other sourc |`docker` |Reads Docker log files. -|? +|? otast |`log/docker` |Alias for `docker`. @@ -174,7 +363,7 @@ from Amazon Elastic Compute Cloud(EC2), AWS CloudTrail, Route53, and other sourc |`osquery` |Collects and decodes the result logs written by https://osquery.readthedocs.io/en/latest/introduction/using-osqueryd/[osqueryd] in the JSON format. -|? +|{filebeat-ref}/filebeat-module-osquery.html[Osquery module] ({filebeat} docs) |`redis` |beta[] Reads entries from Redis slowlogs. @@ -197,15 +386,19 @@ from Amazon Elastic Compute Cloud(EC2), AWS CloudTrail, Route53, and other sourc |{filebeat-ref}/[Unix input] ({filebeat} docs) |`winlog` -|? -|? +|Reads from one or more event logs using Windows APIs, filters the events based on user-configured criteria, then sends the event data to the configured outputs ({es} or {ls}). +|{winlogbeat-ref}[Winlogbeat Overview] ({winlogbeat} docs) |=== +==== // Heartbeat [[elastic-agent-input-types-heartbeat]] -.Periodically check the status of your services +[%collapsible] +.Monitor the status of your services +==== + |=== |Input type |Description |Learn more @@ -227,21 +420,14 @@ from Amazon Elastic Compute Cloud(EC2), AWS CloudTrail, Route53, and other sourc |=== -// Metricbeat -[[elastic-agent-input-types-metricbeat]] -.Periodically collect metrics from operating systems and services running on your servers -|=== -|Input type |Description |Learn more - -|`beat/metrics` -|Collects metrics about any Beat or other software based on libbeat. -|{metricbeat-ref}/metricbeat-module-beat.html[Beat module] ({metricbeat} docs) - -|=== +==== // Packetbeat [[elastic-agent-input-types-packetbeat]] +[%collapsible] .View network traffic between the servers of your network +==== + |=== |Input type |Description |Learn more @@ -249,4 +435,6 @@ from Amazon Elastic Compute Cloud(EC2), AWS CloudTrail, Route53, and other sourc |Sniffs the traffic between your servers, parses the application-level protocols on the fly, and correlates the messages into transactions. |{packetbeat-ref}/packetbeat-overview.html[Packetbeat overview] ({packetbeat} docs) -|=== \ No newline at end of file +|=== + +==== \ No newline at end of file diff --git a/docs/en/ingest-management/index.asciidoc b/docs/en/ingest-management/index.asciidoc index 792e66dbc..32cf1d8f7 100644 --- a/docs/en/ingest-management/index.asciidoc +++ b/docs/en/ingest-management/index.asciidoc @@ -165,7 +165,7 @@ include::elastic-agent/configuration/inputs/input-configuration.asciidoc[levelof include::elastic-agent/configuration/inputs/simplified-input-configuration.asciidoc[leveloffset=+3] -include::elastic-agent/elastic-agent-input-types.asciidoc[leveloffset=+3] +include::elastic-agent/configuration/inputs/input-types.asciidoc[leveloffset=+3] include::elastic-agent/elastic-agent-dynamic-inputs.asciidoc[leveloffset=+3] From 5a5cd94baca8579c78f72c81a346b930b4acd6fe Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Wed, 27 Sep 2023 15:10:31 -0400 Subject: [PATCH 07/11] touchup --- .../elastic-agent/configuration/inputs/input-types.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/ingest-management/elastic-agent/configuration/inputs/input-types.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/inputs/input-types.asciidoc index ae5d53c03..d0c89b96b 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/inputs/input-types.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/inputs/input-types.asciidoc @@ -3,7 +3,7 @@ When you <> for standalone {agents}, the following values are supported for the input `type` parameter. -Expand any section to view the available input types: +*Expand any section to view the available input types:* // Auditbeat [[elastic-agent-input-types-auditbeat]] From 5208f390884f66ccc654b73172be9d4e9607d37c Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Thu, 5 Oct 2023 16:59:58 -0400 Subject: [PATCH 08/11] Remove list of agent inputs; I'll merge them as a separate PR --- .../inputs/input-configuration.asciidoc | 6 +- .../configuration/inputs/input-types.asciidoc | 440 ------------------ .../elastic-agent-input-types.asciidoc | 440 ------------------ docs/en/ingest-management/index.asciidoc | 2 - 4 files changed, 4 insertions(+), 884 deletions(-) delete mode 100644 docs/en/ingest-management/elastic-agent/configuration/inputs/input-types.asciidoc delete mode 100644 docs/en/ingest-management/elastic-agent/elastic-agent-input-types.asciidoc diff --git a/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc index ef6e7025a..2938507b0 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc @@ -37,7 +37,8 @@ By default {agent} collects system metrics, such as CPU, memory, network, and fi data_stream.dataset: system.filesystem ----------------------------------------------------------------------- -<1> The type of data to input. Refer to <> for the list of what's available. +<1> The name of the input. +//Refer to <> for the list of what's available. <2> A unique ID for the input. <3> A user-defined namespace. <4> The name of the `output` to use. If not specified, `default` will be used. @@ -64,7 +65,8 @@ To enable {agent} to collect log files, you can use a configuration like the fol - /var/log/*.log ----------------------------------------------------------------------- -<1> The type of data to input. Refer to <> for the list of what's available. +<1> The name of the input. +//Refer to <> for the list of what's available. <2> A unique ID for the input. <3> A unique ID for the data stream to track the state of the ingested files. <4> The streams block is required only if multiple streams are used on the same input. Refer to the {filebeat} {filebeat-ref}/filebeat-input-filestream.html[filestream] documentation for a list of available options. Also, specifically for the `filestream` input type, refer to the <> for an example of ingesting a set of logs specified as an array. diff --git a/docs/en/ingest-management/elastic-agent/configuration/inputs/input-types.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/inputs/input-types.asciidoc deleted file mode 100644 index d0c89b96b..000000000 --- a/docs/en/ingest-management/elastic-agent/configuration/inputs/input-types.asciidoc +++ /dev/null @@ -1,440 +0,0 @@ -[[elastic-agent-input-types]] -= {agent} input types - -When you <> for standalone {agents}, the following values are supported for the input `type` parameter. - -*Expand any section to view the available input types:* - -// Auditbeat -[[elastic-agent-input-types-auditbeat]] -[%collapsible] -.Audit the activities of users and processes on your systems -==== - -|=== -|Input type |Description |Learn more - -|`audit/auditd` -|Receives audit events from the Linux Audit Framework that is a part of the Linux kernel. -|{auditbeat-ref}/auditbeat-module-auditd.html[Auditd Module] ({auditbeat} docs) - -|`audit/file_integrity` -|Sends events when a file is changed (created, updated, or deleted) on disk. The events contain file metadata and hashes. -|{auditbeat-ref}/auditbeat-module-file_integrity.html[File Integrity Module] ({auditbeat} docs) - -|`audit/system` -|beta[] Collects various security related information about a system. All datasets send both periodic state information (e.g. all currently running processes) and real-time changes (e.g. when a new process starts or stops). -|{auditbeat-ref}/auditbeat-module-system.html[System Module] ({auditbeat} docs) - -|=== - -==== - -// Metricbeat -[[elastic-agent-input-types-metricbeat]] -[%collapsible] -.Collect metrics from operating systems and services running on your servers -==== - -|=== -|Input type |Description |Learn more - -|`activemq/metrics` -|Periodically fetches JMX metrics from Apache ActiveMQ. -|{metricbeat-ref}/metricbeat-module-activemq.html[ActiveMQ module] ({metricbeat} docs) - -|`apache/metrics` -|Periodically fetches metrics from https://httpd.apache.org/[Apache HTTPD] servers. -|{metricbeat-ref}/metricbeat-module-apache.html[Apache module] ({metricbeat} docs) - -|`aws/metrics` -|Periodically fetches monitoring metrics from AWS CloudWatch using https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html[GetMetricData API] for AWS services. -|{metricbeat-ref}/metricbeat-module-aws.html[AWS module] ({metricbeat} docs) - -|`awsfargate/metrics` -|beta[] Retrieves various metadata, network metrics, and Docker stats about tasks and containers. -|{metricbeat-ref}/metricbeat-module-awsfargate.html[AWS Fargate module] ({metricbeat} docs) - -|`azure/metrics` -|Collects and aggregates Azure logs and metrics from a variety of sources into a common data platform where it can be used for analysis, visualization, and alerting. -|{metricbeat-ref}/metricbeat-module-azure.html[Azure module] ({metricbeat} docs) - -|`beat/metrics` -|Collects metrics about any Beat or other software based on libbeat. -|{metricbeat-ref}/metricbeat-module-beat.html[Beat module] ({metricbeat} docs) - -|`cloudfoundry/metrics` -|Connects to Cloud Foundry loggregator to gather container, counter, and value metrics into a common data platform where it can be used for analysis, visualization, and alerting. -|{metricbeat-ref}/metricbeat-module-cloudfoundry.html[Cloudfoundry module] ({metricbeat} docs) - -|`containerd/metrics` -|beta[] Collects cpu, memory and blkio statistics about running containers controlled by containerd runtime. -|{metricbeat-ref}/metricbeat-module-containerd.html[Containerd module] ({metricbeat} docs) - -|`docker/metrics` -|Fetches metrics from https://www.docker.com/[Docker] containers. -|{metricbeat-ref}/metricbeat-module-docker.html[Docker module] ({metricbeat} docs) - -|`elasticsearch/metrics` -|Collects metrics about {es}. -|{metricbeat-ref}/metricbeat-module-elasticsearch.html[Elasticsearch module] ({metricbeat} docs) - -|`enterprisesearch/metrics` -|Periodically fetches metrics and health information from Elastic {ents} instances using HTTP APIs. -|{metricbeat-ref}/metricbeat-module-enterprisesearch.html[{ents} module] ({metricbeat} docs) - -|`etcd/metrics` -|This module targets Etcd V2 and V3. When using V2, metrics are collected using https://coreos.com/etcd/docs/latest/v2/api.html[Etcd v2 API]. When using V3, metrics are retrieved from the `/metrics`` endpoint as intended for https://coreos.com/etcd/docs/latest/metrics.html[Etcd v3]. -|{metricbeat-ref}/metricbeat-module-etcd.html[Etcd module] ({metricbeat} docs) - -|`gcp/metrics` -|Periodically fetches monitoring metrics from Google Cloud Platform using https://cloud.google.com/monitoring/api/metrics_gcp[Stackdriver Monitoring API] for Google Cloud Platform services. -|{metricbeat-ref}/metricbeat-module-gcp.html[Google Cloud Platform module] ({metricbeat} docs) - -|`haproxy/metrics` -|Collects stats from http://www.haproxy.org/[HAProxy]. It supports collection from TCP sockets, UNIX sockets, or HTTP with or without basic authentication. -|{metricbeat-ref}/[HAProxy module] ({metricbeat} docs) - -|`http/metrics` -|Used to call arbitrary HTTP endpoints for which a dedicated Metricbeat module is not available. -|{metricbeat-ref}/metricbeat-module-http.html[HTTP module] ({metricbeat} docs) - -|`iis/metrics` -|Periodically retrieve IIS web server related metrics. -|{metricbeat-ref}/metricbeat-module-iis.html[IIS module] ({metricbeat} docs) - -|`jolokia/metrics` -|Collects metrics from https://jolokia.org/reference/html/agents.html[Jolokia agents] running on a target JMX server or dedicated proxy server. -|{metricbeat-ref}/metricbeat-module-jolokia.html[Jolokia module] ({metricbeat} docs) - -|`kafka/metrics` -|Collects metrics from the https://kafka.apache.org/intro[Apache Kafka] event streaming platform. -|{metricbeat-ref}/metricbeat-module-kafka.html[Kafka module] ({metricbeat} docs) - -|`kibana/metrics` -|Collects metrics about {Kibana}. -|{metricbeat-ref}/metricbeat-module-kibana.html[{kib} module] ({metricbeat} docs) - -|`kubernetes/metrics` -|As one of the main pieces provided for Kubernetes monitoring, this module is capable of fetching metrics from several components. -|{metricbeat-ref}/metricbeat-module-kubernetes.html[Kubernetes module] ({metricbeat} docs) - -|`linux/metrics` -|beta[] Reports on metrics exclusive to the Linux kernel and GNU/Linux OS. -|{metricbeat-ref}/metricbeat-module-linux.html[Linux module] ({metricbeat} docs) - -|`logstash/metrics` -|collects metrics about {ls}. -|{metricbeat-ref}/metricbeat-module-logstash.html[{ls} module] ({metricbeat} docs) - -|`memcached/metrics` -|Collects metrics about the https://memcached.org/[memcached] memory object caching system. -|{metricbeat-ref}/metricbeat-module-memcached.html[Memcached module] ({metricbeat} docs) - -|`mongodb/metrics` -|Periodically fetches metrics from https://www.mongodb.com/[MongoDB] servers. -|{metricbeat-ref}/metricbeat-module-mongodb.html[MongoDB module] ({metricbeat} docs) - -|`mssql/metrics` -|The https://www.microsoft.com/en-us/sql-server/sql-server-2017[Microsoft SQL 2017] Metricbeat module. It is still under active development to add new Metricsets and introduce enhancements. -|{metricbeat-ref}/metricbeat-module-mssql.html[MSSQL module] ({metricbeat} docs) - -|`mysql/metrics` -|Periodically fetches metrics from https://www.mysql.com/[MySQL] servers. -|{metricbeat-ref}/metricbeat-module-mysql.html[MySQL module] ({metricbeat} docs) - -|`nats/metrics` -|Uses the https://nats.io/documentation/managing_the_server/monitoring/[Nats monitoring server APIs] to collect metrics. -|{metricbeat-ref}/metricbeat-module-nats.html[NATS module] ({metricbeat} docs) - -|`nginx/metrics` -|Periodically fetches metrics from https://nginx.org/[Nginx] servers. -|{metricbeat-ref}/metricbeat-module-nginx.html[Nginx module] ({metricbeat} docs) - -|`oracle/metrics` -|The https://www.oracle.com/[Oracle] module for Metricbeat. It is under active development with feedback from the community. A single Metricset for Tablespace monitoring is added so the community can start gathering metrics from their nodes and contributing to the module. -|{metricbeat-ref}/metricbeat-module-oracle.html[Oracle module] ({metricbeat} docs) - -|`postgresql/metrics` -|Periodically fetches metrics from https://www.postgresql.org/[PostgreSQL] servers. -|{metricbeat-ref}/metricbeat-module-postgresql.html[PostgresSQL module] ({metricbeat} docs) - -|`prometheus/metrics` -|Periodically scrapes metrics from https://prometheus.io/docs/instrumenting/exporters/[Prometheus exporters]. -|{metricbeat-ref}/metricbeat-module-prometheus.html[Prometheus module] ({metricbeat} docs) - -|`rabbitmq/metrics` -|Uses the http://www.rabbitmq.com/management.html[HTTP API] created by the management plugin to collect RabbitMQ metrics. -|{metricbeat-ref}/metricbeat-module-rabbitmq.html[RabbitMQ module] ({metricbeat} docs) - -|`redis/metrics` -|Periodically fetches metrics from http://redis.io/[Redis] servers. -|{metricbeat-ref}/metricbeat-module-redis.html[Redis module] ({metricbeat} docs) - -|`sql/metrics` -|Allows you to execute custom queries against an SQL database and store the results in {es}. -|{metricbeat-ref}/metricbeat-module-sql.html[SQL module] ({metricbeat} docs) - -|`stan/metrics` -|Uses https://github.com/nats-io/nats-streaming-server/blob/master/server/monitor.go[STAN monitoring server APIs] to collect metrics. -|{metricbeat-ref}/metricbeat-module-stan.html[Stan module] ({metricbeat} docs) - -|`statsd/metrics` -|Spawns a UDP server and listens for metrics in StatsD compatible format. -|{metricbeat-ref}/metricbeat-module-statsd.html[Statsd module] ({metricbeat} docs) - -|`syncgateway/metrics` -|beta[] Monitor a Sync Gateway instance by using its REST API. -|{metricbeat-ref}/metricbeat-module-syncgateway.html[SyncGateway module] ({metricbeat} docs) - -|`system/metrics` -|Allows you to monitor your server metrics, including CPU, load, memory, network, processes, sockets, filesystem, fsstat, uptime, and more. -|{metricbeat-ref}/metricbeat-module-system.html[System module] ({metricbeat} docs) - -|`traefik/metrics` -|Periodically fetches metrics from a https://traefik.io/[Traefik] instance. -|{metricbeat-ref}/metricbeat-module-traefik.html[Traefik module] ({metricbeat} docs) - -|`uwsgi/metrics` -|By default, collects the uWSGI stats metricset, using https://uwsgi-docs.readthedocs.io/en/latest/StatsServer.html[StatsServer]. -|{metricbeat-ref}/metricbeat-module-uwsgi.html[uWSGI module] ({metricbeat} docs) - -|`vsphere/metrics` -|Uses the https://github.com/vmware/govmomi[Govmomi] library to collect metrics from any Vmware SDK URL (ESXi/VCenter). -|{metricbeat-ref}/metricbeat-module-vsphere.html[vSphere module] ({metricbeat} docs) - -|`windows/metrics` -|Collects metrics from Windows systems. -|{metricbeat-ref}/metricbeat-module-windows.html[Windows module] ({metricbeat} docs) - -|`zookeeper/metrics` -|Fetches statistics from the ZooKeeper service. -|{metricbeat-ref}/metricbeat-module-zookeeper.html[ZooKeeper module] ({metricbeat} docs) - -|=== - -==== - -// Filebeat -[[elastic-agent-input-types-filebeat]] -[%collapsible] -.Forward and centralize log data -==== - -|=== -|Input type |Description |Learn more - -|`aws-cloudwatch` -|Stores log files -from Amazon Elastic Compute Cloud(EC2), AWS CloudTrail, Route53, and other sources. -|{filebeat-ref}/filebeat-input-aws-cloudwatch.html[AWS CloudWatch input] ({filebeat} docs) - -|`aws-s3` -|Retrieves logs from S3 objects that are pointed to by S3 notification events read from an SQS queue or directly polling list of S3 objects in an S3 bucket. -|{filebeat-ref}/filebeat-input-aws-s3.html[AWS S3 input] ({filebeat} docs) - -|`azure-blob-storage` -|Reads content from files stored in containers which reside on your Azure Cloud. -|{filebeat-ref}/filebeat-input-azure-blob-storage.html[Azure Blob Storage] ({filebeat} docs) - -|`azure-eventhub` -|Reads messages from an azure eventhub. -|{filebeat-ref}/filebeat-input-azure-eventhub.html[Azure eventhub input] ({filebeat} docs) - -|`cel` -|Reads messages from a file path or HTTP API with a variety of payloads using the https://opensource.google.com/projects/cel[Common Expression Language (CEL)] and the https://pkg.go.dev/github.com/elastic/mito/lib[mito] CEL extension libraries. -|{filebeat-ref}/filebeat-input-cel.html[Common Expression Language input] ({filebeat} docs) - -|`cloudbeat` -|- -|- - -|`cloudbeat/cis_k8s` -|CIS Kubernetes monitoring -|- - -|`cloudbeat/cis_aws` -|CIS AWS monitoring -|- - -|`cloudbeat/cis_eks` -|CIS Elastic Kubernetes monitoring -|- - -|`cloudbeat/cis_gcp` -|CIS GCP monitoring -|- - -|`cloudbeat/vuln_mgmt_aws` -|AWS Vulnerabilities management -|- - -|`cloudfoundry` -|Gets HTTP access logs, container logs and error logs from Cloud Foundry. -|{filebeat-ref}/filebeat-input-cloudfoundry.html[Cloud Foundry input] ({filebeat} docs) - -|`cometd` -|Streams the real-time events from a Salesforce generic subscription Push Topic. -|{filebeat-ref}/filebeat-input-cometd.html[CometD input] ({filebeat} docs) - -|`container` -|Reads containers log files. -|{filebeat-ref}/filebeat-input-container.html[Container input] ({filebeat} docs) - -|`docker` -|Reads Docker log files. -|- - -|`log/docker` -|Alias for `docker`. -|n/a - -|`entity-analytics` -|Collects identity assets, such as users, from external identity providers. -|{filebeat-ref}/filebeat-input-entity-analytics.html[Entity Analytics input] ({filebeat} docs) - -|`event/file` -|Alias for `log`. -|n/a - -|`event/tcp` -|Alias for `tcp`. -|n/a - -|`filestream` -|Reads lines from active log files. Replaces and imporoves on the `log` input. -|{filebeat-ref}/filebeat-input-filestream.html[filestream input] ({filebeat} docs) - -|`gcp-pubsub` -|Reads messages from a Google Cloud Pub/Sub topic subscription. -|{filebeat-ref}/filebeat-input-gcp-pubsub.html[GCP Pub/Sub input] ({filebeat} docs) - -|`gcs` -|beta[] Reads content from files stored in buckets which reside on your Google Cloud. -|{filebeat-ref}/filebeat-input-gcs.html[Google Cloud Storage input] ({filebeat} docs) - -|`http_endpoint` -|beta[] Initializes a listening HTTP server that collects incoming HTTP POST requests containing a JSON body. -|{filebeat-ref}/filebeat-input-http_endpoint.html[HTTP Endpoint input] ({filebeat} docs) - -|`httpjson` -|Read messages from an HTTP API with JSON payloads. -|{filebeat-ref}/filebeat-input-httpjson.html[HTTP JSON input] ({filebeat} docs) - -|`journald` -|beta[] A system service that collects and stores logging data. -|{filebeat-ref}/filebeat-input-journald.html[Journald input] ({filebeat} docs) - -|`kafka` -|Reads from topics in a Kafka cluster. -|{filebeat-ref}/filebeat-input-kafka.html[Kafka input] ({filebeat} docs) - -|`log` -|DEPRECATED: Please use the `filestream` input instead. -|n/a - -|`logfile` -|Alias for `log`. -|n/a - -|`log/redis_slowlog` -|Alias for `redis`. -|n/a - -|`log/syslog` -|Alias for `syslog`. -|n/a - -|`lumberjack` -|- -|- - -|`mqtt` -|Reads data transmitted using lightweight messaging protocol for small and mobile devices, optimized for high-latency or unreliable networks. -|{filebeat-ref}/filebeat-input-mqtt.html[MQTT input] ({filebeat} docs) - -|`netflow` -|Reads NetFlow and IPFIX exported flows and options records over UDP. -|{filebeat-ref}/filebeat-input-netflow.html[NetFlow input] ({filebeat} docs) - -|`o365audit` -|beta[] Retrieves audit messages from Office 365 and Azure AD activity logs. -|{filebeat-ref}/filebeat-input-o365audit.html[Office 365 Management Activity API input] ({filebeat} docs) - -|`osquery` -|Collects and decodes the result logs written by https://osquery.readthedocs.io/en/latest/introduction/using-osqueryd/[osqueryd] in the JSON format. -| - - -|`redis` -|beta[] Reads entries from Redis slowlogs. -|{filebeat-ref}/[Redis input] ({filebeat} docs) - -|`syslog` -|Reads Syslog events as specified by RFC 3164 and RFC 5424, over TCP, UDP, or a Unix stream socket. -|{filebeat-ref}/filebeat-input-syslog.html[Syslog input] ({filebeat} docs) - -|`tcp` -|Reads events over TCP. -|{filebeat-ref}/filebeat-input-tcp.html[TCP input] ({filebeat} docs) - -|`udp` -|Reads events over UDP. -|{filebeat-ref}/filebeat-input-udp.html[UDP input] ({filebeat} docs) - -|`unix` -|beta[] Reads events over a stream-oriented Unix domain socket. -|{filebeat-ref}/[Unix input] ({filebeat} docs) - -|`winlog` -|Reads from one or more event logs using Windows APIs, filters the events based on user-configured criteria, then sends the event data to the configured outputs ({es} or {ls}). -|{winlogbeat-ref}[Winlogbeat Overview] ({winlogbeat} docs) - -|=== - -==== - -// Heartbeat -[[elastic-agent-input-types-heartbeat]] -[%collapsible] -.Monitor the status of your services -==== - -|=== -|Input type |Description |Learn more - -|`synthetics/browser` -|- -|- - -|`synthetics/http` -|Connect via HTTP and optionally verify that the host returns the expected response. -|{heartbeat-ref}/monitor-http-options.html[HTTP options] ({heartbeat} docs) - -|`synthetics/icmp` -|Use ICMP (v4 and v6) Echo Requests to check the configured hosts. -|{heartbeat-ref}/monitor-icmp-options.html[ICMP options] ({heartbeat} docs) - -|`synthetics/tcp` -|Connect via TCP and optionally verify the endpoint by sending and/or receiving a custom payload. -|{heartbeat-ref}/monitor-tcp-options.html[TCP options] ({heartbeat} docs) - -|=== - -==== - -// Packetbeat -[[elastic-agent-input-types-packetbeat]] -[%collapsible] -.View network traffic between the servers of your network -==== - -|=== -|Input type |Description |Learn more - -|`packet` -|Sniffs the traffic between your servers, parses the application-level protocols on the fly, and correlates the messages into transactions. -|{packetbeat-ref}/packetbeat-overview.html[Packetbeat overview] ({packetbeat} docs) - -|=== - -==== \ No newline at end of file diff --git a/docs/en/ingest-management/elastic-agent/elastic-agent-input-types.asciidoc b/docs/en/ingest-management/elastic-agent/elastic-agent-input-types.asciidoc deleted file mode 100644 index 0aa55a22b..000000000 --- a/docs/en/ingest-management/elastic-agent/elastic-agent-input-types.asciidoc +++ /dev/null @@ -1,440 +0,0 @@ -[[elastic-agent-input-types]] -= {agent} input types - -When you <> for standalone {agents}, the following values are supported for the input `type` parameter. - -Expand any section to view the available input types: - -// Auditbeat -[[elastic-agent-input-types-auditbeat]] -[%collapsible] -.Audit the activities of users and processes on your systems -==== - -|=== -|Input type |Description |Learn more - -|`audit/auditd` -|Receives audit events from the Linux Audit Framework that is a part of the Linux kernel. -|{auditbeat-ref}/auditbeat-module-auditd.html[Auditd Module] ({auditbeat} docs) - -|`audit/file_integrity` -|Sends events when a file is changed (created, updated, or deleted) on disk. The events contain file metadata and hashes. -|{auditbeat-ref}/auditbeat-module-file_integrity.html[File Integrity Module] ({auditbeat} docs) - -|`audit/system` -|beta[] Collects various security related information about a system. All datasets send both periodic state information (e.g. all currently running processes) and real-time changes (e.g. when a new process starts or stops). -|{auditbeat-ref}/auditbeat-module-system.html[System Module] ({auditbeat} docs) - -|=== - -==== - -// Metricbeat -[[elastic-agent-input-types-metricbeat]] -[%collapsible] -.Collect metrics from operating systems and services running on your servers -==== - -|=== -|Input type |Description |Learn more - -|`activemq/metrics` -|Periodically fetches JMX metrics from Apache ActiveMQ. -|{metricbeat-ref}/metricbeat-module-activemq.html[ActiveMQ module] ({metricbeat} docs) - -|`apache/metrics` -|Periodically fetches metrics from https://httpd.apache.org/[Apache HTTPD] servers. -|{metricbeat-ref}/metricbeat-module-apache.html[Apache module] ({metricbeat} docs) - -|`aws/metrics` -|Periodically fetches monitoring metrics from AWS CloudWatch using https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricData.html[GetMetricData API] for AWS services. -|{metricbeat-ref}/metricbeat-module-aws.html[AWS module] ({metricbeat} docs) - -|`awsfargate/metrics` -|beta[] Retrieves various metadata, network metrics, and Docker stats about tasks and containers. -|{metricbeat-ref}/metricbeat-module-awsfargate.html[AWS Fargate module] ({metricbeat} docs) - -|`azure/metrics` -|Collects and aggregates Azure logs and metrics from a variety of sources into a common data platform where it can be used for analysis, visualization, and alerting. -|{metricbeat-ref}/metricbeat-module-azure.html[Azure module] ({metricbeat} docs) - -|`beat/metrics` -|Collects metrics about any Beat or other software based on libbeat. -|{metricbeat-ref}/metricbeat-module-beat.html[Beat module] ({metricbeat} docs) - -|`cloudfoundry/metrics` -|Connects to Cloud Foundry loggregator to gather container, counter, and value metrics into a common data platform where it can be used for analysis, visualization, and alerting. -|{metricbeat-ref}/metricbeat-module-cloudfoundry.html[Cloudfoundry module] ({metricbeat} docs) - -|`containerd/metrics` -|beta[] Collects cpu, memory and blkio statistics about running containers controlled by containerd runtime. -|{metricbeat-ref}/metricbeat-module-containerd.html[Containerd module] ({metricbeat} docs) - -|`docker/metrics` -|Fetches metrics from https://www.docker.com/[Docker] containers. -|{metricbeat-ref}/metricbeat-module-docker.html[Docker module] ({metricbeat} docs) - -|`elasticsearch/metrics` -|Collects metrics about {es}. -|{metricbeat-ref}/metricbeat-module-elasticsearch.html[Elasticsearch module] ({metricbeat} docs) - -|`enterprisesearch/metrics` -|Periodically fetches metrics and health information from Elastic {ents} instances using HTTP APIs. -|{metricbeat-ref}/metricbeat-module-enterprisesearch.html[{ents} module] ({metricbeat} docs) - -|`etcd/metrics` -|This module targets Etcd V2 and V3. When using V2, metrics are collected using https://coreos.com/etcd/docs/latest/v2/api.html[Etcd v2 API]. When using V3, metrics are retrieved from the `/metrics`` endpoint as intended for https://coreos.com/etcd/docs/latest/metrics.html[Etcd v3]. -|{metricbeat-ref}/metricbeat-module-etcd.html[Etcd module] ({metricbeat} docs) - -|`gcp/metrics` -|Periodically fetches monitoring metrics from Google Cloud Platform using https://cloud.google.com/monitoring/api/metrics_gcp[Stackdriver Monitoring API] for Google Cloud Platform services. -|{metricbeat-ref}/metricbeat-module-gcp.html[Google Cloud Platform module] ({metricbeat} docs) - -|`haproxy/metrics` -|Collects stats from http://www.haproxy.org/[HAProxy]. It supports collection from TCP sockets, UNIX sockets, or HTTP with or without basic authentication. -|{metricbeat-ref}/[HAProxy module] ({metricbeat} docs) - -|`http/metrics` -|Used to call arbitrary HTTP endpoints for which a dedicated Metricbeat module is not available. -|{metricbeat-ref}/metricbeat-module-http.html[HTTP module] ({metricbeat} docs) - -|`iis/metrics` -|Periodically retrieve IIS web server related metrics. -|{metricbeat-ref}/metricbeat-module-iis.html[IIS module] ({metricbeat} docs) - -|`jolokia/metrics` -|Collects metrics from https://jolokia.org/reference/html/agents.html[Jolokia agents] running on a target JMX server or dedicated proxy server. -|{metricbeat-ref}/metricbeat-module-jolokia.html[Jolokia module] ({metricbeat} docs) - -|`kafka/metrics` -|Collects metrics from the https://kafka.apache.org/intro[Apache Kafka] event streaming platform. -|{metricbeat-ref}/metricbeat-module-kafka.html[Kafka module] ({metricbeat} docs) - -|`kibana/metrics` -|Collects metrics about {Kibana}. -|{metricbeat-ref}/metricbeat-module-kibana.html[{kib} module] ({metricbeat} docs) - -|`kubernetes/metrics` -|As one of the main pieces provided for Kubernetes monitoring, this module is capable of fetching metrics from several components. -|{metricbeat-ref}/metricbeat-module-kubernetes.html[Kubernetes module] ({metricbeat} docs) - -|`linux/metrics` -|beta[] Reports on metrics exclusive to the Linux kernel and GNU/Linux OS. -|{metricbeat-ref}/metricbeat-module-linux.html[Linux module] ({metricbeat} docs) - -|`logstash/metrics` -|collects metrics about {ls}. -|{metricbeat-ref}/metricbeat-module-logstash.html[{ls} module] ({metricbeat} docs) - -|`memcached/metrics` -|Collects metrics about the https://memcached.org/[memcached] memory object caching system. -|{metricbeat-ref}/metricbeat-module-memcached.html[Memcached module] ({metricbeat} docs) - -|`mongodb/metrics` -|Periodically fetches metrics from https://www.mongodb.com/[MongoDB] servers. -|{metricbeat-ref}/metricbeat-module-mongodb.html[MongoDB module] ({metricbeat} docs) - -|`mssql/metrics` -|The https://www.microsoft.com/en-us/sql-server/sql-server-2017[Microsoft SQL 2017] Metricbeat module. It is still under active development to add new Metricsets and introduce enhancements. -|{metricbeat-ref}/metricbeat-module-mssql.html[MSSQL module] ({metricbeat} docs) - -|`mysql/metrics` -|Periodically fetches metrics from https://www.mysql.com/[MySQL] servers. -|{metricbeat-ref}/metricbeat-module-mysql.html[MySQL module] ({metricbeat} docs) - -|`nats/metrics` -|Uses the https://nats.io/documentation/managing_the_server/monitoring/[Nats monitoring server APIs] to collect metrics. -|{metricbeat-ref}/metricbeat-module-nats.html[NATS module] ({metricbeat} docs) - -|`nginx/metrics` -|Periodically fetches metrics from https://nginx.org/[Nginx] servers. -|{metricbeat-ref}/metricbeat-module-nginx.html[Nginx module] ({metricbeat} docs) - -|`oracle/metrics` -|The https://www.oracle.com/[Oracle] module for Metricbeat. It is under active development with feedback from the community. A single Metricset for Tablespace monitoring is added so the community can start gathering metrics from their nodes and contributing to the module. -|{metricbeat-ref}/metricbeat-module-oracle.html[Oracle module] ({metricbeat} docs) - -|`postgresql/metrics` -|Periodically fetches metrics from https://www.postgresql.org/[PostgreSQL] servers. -|{metricbeat-ref}/metricbeat-module-postgresql.html[PostgresSQL module] ({metricbeat} docs) - -|`prometheus/metrics` -|Periodically scrapes metrics from https://prometheus.io/docs/instrumenting/exporters/[Prometheus exporters]. -|{metricbeat-ref}/metricbeat-module-prometheus.html[Prometheus module] ({metricbeat} docs) - -|`rabbitmq/metrics` -|Uses the http://www.rabbitmq.com/management.html[HTTP API] created by the management plugin to collect RabbitMQ metrics. -|{metricbeat-ref}/metricbeat-module-rabbitmq.html[RabbitMQ module] ({metricbeat} docs) - -|`redis/metrics` -|Periodically fetches metrics from http://redis.io/[Redis] servers. -|{metricbeat-ref}/metricbeat-module-redis.html[Redis module] ({metricbeat} docs) - -|`sql/metrics` -|Allows you to execute custom queries against an SQL database and store the results in {es}. -|{metricbeat-ref}/metricbeat-module-sql.html[SQL module] ({metricbeat} docs) - -|`stan/metrics` -|Uses https://github.com/nats-io/nats-streaming-server/blob/master/server/monitor.go[STAN monitoring server APIs] to collect metrics. -|{metricbeat-ref}/metricbeat-module-stan.html[Stan module] ({metricbeat} docs) - -|`statsd/metrics` -|Spawns a UDP server and listens for metrics in StatsD compatible format. -|{metricbeat-ref}/metricbeat-module-statsd.html[Statsd module] ({metricbeat} docs) - -|`syncgateway/metrics` -|beta[] Monitor a Sync Gateway instance by using its REST API. -|{metricbeat-ref}/metricbeat-module-syncgateway.html[SyncGateway module] ({metricbeat} docs) - -|`system/metrics` -|Allows you to monitor your server metrics, including CPU, load, memory, network, processes, sockets, filesystem, fsstat, uptime, and more. -|{metricbeat-ref}/metricbeat-module-system.html[System module] ({metricbeat} docs) - -|`traefik/metrics` -|Periodically fetches metrics from a https://traefik.io/[Traefik] instance. -|{metricbeat-ref}/metricbeat-module-traefik.html[Traefik module] ({metricbeat} docs) - -|`uwsgi/metrics` -|By default, collects the uWSGI stats metricset, using https://uwsgi-docs.readthedocs.io/en/latest/StatsServer.html[StatsServer]. -|{metricbeat-ref}/metricbeat-module-uwsgi.html[uWSGI module] ({metricbeat} docs) - -|`vsphere/metrics` -|Uses the https://github.com/vmware/govmomi[Govmomi] library to collect metrics from any Vmware SDK URL (ESXi/VCenter). -|{metricbeat-ref}/metricbeat-module-vsphere.html[vSphere module] ({metricbeat} docs) - -|`windows/metrics` -|Collects metrics from Windows systems. -|{metricbeat-ref}/metricbeat-module-windows.html[Windows module] ({metricbeat} docs) - -|`zookeeper/metrics` -|Fetches statistics from the ZooKeeper service. -|{metricbeat-ref}/metricbeat-module-zookeeper.html[ZooKeeper module] ({metricbeat} docs) - -|=== - -==== - -// Filebeat -[[elastic-agent-input-types-filebeat]] -[%collapsible] -.Forward and centralize log data -==== - -|=== -|Input type |Description |Learn more - -|`aws-cloudwatch` -|Stores log files -from Amazon Elastic Compute Cloud(EC2), AWS CloudTrail, Route53, and other sources. -|{filebeat-ref}/filebeat-input-aws-cloudwatch.html[AWS CloudWatch input] ({filebeat} docs) - -|`aws-s3` -|Retrieves logs from S3 objects that are pointed to by S3 notification events read from an SQS queue or directly polling list of S3 objects in an S3 bucket. -|{filebeat-ref}/filebeat-input-aws-s3.html[AWS S3 input] ({filebeat} docs) - -|`azure-blob-storage` -|Reads content from files stored in containers which reside on your Azure Cloud. -|{filebeat-ref}/filebeat-input-azure-blob-storage.html[Azure Blob Storage] ({filebeat} docs) - -|`azure-eventhub` -|Reads messages from an azure eventhub. -|{filebeat-ref}/filebeat-input-azure-eventhub.html[Azure eventhub input] ({filebeat} docs) - -|`cel` -|Reads messages from a file path or HTTP API with a variety of payloads using the https://opensource.google.com/projects/cel[Common Expression Language (CEL)] and the https://pkg.go.dev/github.com/elastic/mito/lib[mito] CEL extension libraries. -|{filebeat-ref}/filebeat-input-cel.html[Common Expression Language input] ({filebeat} docs) - -|`cloudbeat` -|? -|? - -|`cloudbeat/cis_k8s` -|CIS Kubernetes monitoring -|? - -|`cloudbeat/cis_aws` -|CIS AWS monitoring -|? - -|`cloudbeat/cis_eks` -|CIS Elastic Kubernetes monitoring -|? - -|`cloudbeat/cis_gcp` -|CIS GCP monitoring -|? - -|`cloudbeat/vuln_mgmt_aws` -|AWS Vulnerabilities management -|? - -|`cloudfoundry` -|Gets HTTP access logs, container logs and error logs from Cloud Foundry. -|{filebeat-ref}/filebeat-input-cloudfoundry.html[Cloud Foundry input] ({filebeat} docs) - -|`cometd` -|Streams the real-time events from a Salesforce generic subscription Push Topic. -|{filebeat-ref}/filebeat-input-cometd.html[CometD input] ({filebeat} docs) - -|`container` -|Reads containers log files. -|{filebeat-ref}/filebeat-input-container.html[Container input] ({filebeat} docs) - -|`docker` -|Reads Docker log files. -|? otast - -|`log/docker` -|Alias for `docker`. -|n/a - -|`entity-analytics` -|Collects identity assets, such as users, from external identity providers. -|{filebeat-ref}/filebeat-input-entity-analytics.html[Entity Analytics input] ({filebeat} docs) - -|`event/file` -|Alias for `log`. -|n/a - -|`event/tcp` -|Alias for `tcp`. -|n/a - -|`filestream` -|Reads lines from active log files. Replaces and imporoves on the `log` input. -|{filebeat-ref}/filebeat-input-filestream.html[filestream input] ({filebeat} docs) - -|`gcp-pubsub` -|Reads messages from a Google Cloud Pub/Sub topic subscription. -|{filebeat-ref}/filebeat-input-gcp-pubsub.html[GCP Pub/Sub input] ({filebeat} docs) - -|`gcs` -|beta[] Reads content from files stored in buckets which reside on your Google Cloud. -|{filebeat-ref}/filebeat-input-gcs.html[Google Cloud Storage input] ({filebeat} docs) - -|`http_endpoint` -|beta[] Initializes a listening HTTP server that collects incoming HTTP POST requests containing a JSON body. -|{filebeat-ref}/filebeat-input-http_endpoint.html[HTTP Endpoint input] ({filebeat} docs) - -|`httpjson` -|Read messages from an HTTP API with JSON payloads. -|{filebeat-ref}/filebeat-input-httpjson.html[HTTP JSON input] ({filebeat} docs) - -|`journald` -|beta[] A system service that collects and stores logging data. -|{filebeat-ref}/filebeat-input-journald.html[Journald input] ({filebeat} docs) - -|`kafka` -|Reads from topics in a Kafka cluster. -|{filebeat-ref}/filebeat-input-kafka.html[Kafka input] ({filebeat} docs) - -|`log` -|DEPRECATED: Please use the `filestream` input instead. -|n/a - -|`logfile` -|Alias for `log`. -|n/a - -|`log/redis_slowlog` -|Alias for `redis`. -|n/a - -|`log/syslog` -|Alias for `syslog`. -|n/a - -|`lumberjack` -|? -|? - -|`mqtt` -|Reads data transmitted using lightweight messaging protocol for small and mobile devices, optimized for high-latency or unreliable networks. -|{filebeat-ref}/filebeat-input-mqtt.html[MQTT input] ({filebeat} docs) - -|`netflow` -|Reads NetFlow and IPFIX exported flows and options records over UDP. -|{filebeat-ref}/filebeat-input-netflow.html[NetFlow input] ({filebeat} docs) - -|`o365audit` -|beta[] Retrieves audit messages from Office 365 and Azure AD activity logs. -|{filebeat-ref}/filebeat-input-o365audit.html[Office 365 Management Activity API input] ({filebeat} docs) - -|`osquery` -|Collects and decodes the result logs written by https://osquery.readthedocs.io/en/latest/introduction/using-osqueryd/[osqueryd] in the JSON format. -|{filebeat-ref}/filebeat-module-osquery.html[Osquery module] ({filebeat} docs) - -|`redis` -|beta[] Reads entries from Redis slowlogs. -|{filebeat-ref}/[Redis input] ({filebeat} docs) - -|`syslog` -|Reads Syslog events as specified by RFC 3164 and RFC 5424, over TCP, UDP, or a Unix stream socket. -|{filebeat-ref}/filebeat-input-syslog.html[Syslog input] ({filebeat} docs) - -|`tcp` -|Reads events over TCP. -|{filebeat-ref}/filebeat-input-tcp.html[TCP input] ({filebeat} docs) - -|`udp` -|Reads events over UDP. -|{filebeat-ref}/filebeat-input-udp.html[UDP input] ({filebeat} docs) - -|`unix` -|beta[] Reads events over a stream-oriented Unix domain socket. -|{filebeat-ref}/[Unix input] ({filebeat} docs) - -|`winlog` -|Reads from one or more event logs using Windows APIs, filters the events based on user-configured criteria, then sends the event data to the configured outputs ({es} or {ls}). -|{winlogbeat-ref}[Winlogbeat Overview] ({winlogbeat} docs) - -|=== - -==== - -// Heartbeat -[[elastic-agent-input-types-heartbeat]] -[%collapsible] -.Monitor the status of your services -==== - -|=== -|Input type |Description |Learn more - -|`synthetics/browser` -|? -|? - -|`synthetics/http` -|? -|? - -|`synthetics/icmp` -|? -|? - -|`synthetics/tcp` -|? -|? - -|=== - -==== - -// Packetbeat -[[elastic-agent-input-types-packetbeat]] -[%collapsible] -.View network traffic between the servers of your network -==== - -|=== -|Input type |Description |Learn more - -|`packet` -|Sniffs the traffic between your servers, parses the application-level protocols on the fly, and correlates the messages into transactions. -|{packetbeat-ref}/packetbeat-overview.html[Packetbeat overview] ({packetbeat} docs) - -|=== - -==== \ No newline at end of file diff --git a/docs/en/ingest-management/index.asciidoc b/docs/en/ingest-management/index.asciidoc index 32cf1d8f7..603d3938b 100644 --- a/docs/en/ingest-management/index.asciidoc +++ b/docs/en/ingest-management/index.asciidoc @@ -165,8 +165,6 @@ include::elastic-agent/configuration/inputs/input-configuration.asciidoc[levelof include::elastic-agent/configuration/inputs/simplified-input-configuration.asciidoc[leveloffset=+3] -include::elastic-agent/configuration/inputs/input-types.asciidoc[leveloffset=+3] - include::elastic-agent/elastic-agent-dynamic-inputs.asciidoc[leveloffset=+3] include::elastic-agent/configuration/providers/elastic-agent-providers.asciidoc[leveloffset=+2] From a53a0da329d17712c71a9ed8a310b020ea7898fc Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Tue, 28 Nov 2023 11:13:33 -0500 Subject: [PATCH 09/11] Rebuild preview page From 987ed3831847862dfd90c18d34fde9de75713010 Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Tue, 28 Nov 2023 11:38:32 -0500 Subject: [PATCH 10/11] Re-add inputs list link --- .../configuration/inputs/input-configuration.asciidoc | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc b/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc index e49122a23..6fd6e5733 100644 --- a/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc +++ b/docs/en/ingest-management/elastic-agent/configuration/inputs/input-configuration.asciidoc @@ -64,8 +64,7 @@ To enable {agent} to collect log files, you can use a configuration like the fol - /var/log/*.log ----------------------------------------------------------------------- -<1> The name of the input. -//Refer to <> for the list of what's available. +<1> The name of the input. Refer to <> for the list of what's available. <2> A unique ID for the input. <3> A unique ID for the data stream to track the state of the ingested files. <4> The streams block is required only if multiple streams are used on the same input. Refer to the {filebeat} {filebeat-ref}/filebeat-input-filestream.html[filestream] documentation for a list of available options. Also, specifically for the `filestream` input type, refer to the <> for an example of ingesting a set of logs specified as an array. From ba9ae528f50d26421368770aa1203a351fde3038 Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Mon, 15 Jan 2024 13:28:22 -0500 Subject: [PATCH 11/11] Rebuild