From 0b7165a33f1636fbc7ee9f61d021acdbcf53b17d Mon Sep 17 00:00:00 2001 From: megastef Date: Mon, 10 Feb 2020 17:10:26 +0100 Subject: [PATCH 1/3] add LA plugins --- docs/logagent/input-filter-containerd.md | 48 +++++++++++++++ docs/logagent/input-kubernetes-audit.md | 65 +++++++++++++++++++++ docs/logagent/input-kubernetes-events.md | 57 ++++++++++++++++++ docs/logagent/input-plugin-dockerlogs.md | 2 +- docs/logagent/installation-docker.md | 18 ++++-- docs/logagent/output-filter-kubernetes.md | 3 +- docs/logagent/output-filter-removefields.md | 4 ++ docs/logagent/output-plugin-http.md | 38 ++++++++++++ docs/logagent/plugins.md | 11 +++- mkdocs.yml | 5 +- 10 files changed, 241 insertions(+), 10 deletions(-) create mode 100644 docs/logagent/input-filter-containerd.md create mode 100644 docs/logagent/input-kubernetes-audit.md create mode 100644 docs/logagent/input-kubernetes-events.md create mode 100644 docs/logagent/output-plugin-http.md diff --git a/docs/logagent/input-filter-containerd.md b/docs/logagent/input-filter-containerd.md new file mode 100644 index 000000000..ae3bc7de5 --- /dev/null +++ b/docs/logagent/input-filter-containerd.md @@ -0,0 +1,48 @@ +title: Parsing cri-o log format, add Kubernetes context to container logs +description: Logagent features modular logging architecture framework where each input or output module is implemented as a plugin, and loaded on demand as declared in the configuration file. Input filters process raw input from input plugins before log events get parsed + +## Input Filter: Kubernetes cri-o / containerd + +Parsing cri-o containerd log format. Use the file input plugin to read log files. + +## Configuration + +Note: the plugin can be enabled via command line option `--k8sContainerd`. + +Add the following section to the Logagent configuration file: + +``` +inputFilter: + # parse containerd log format, add pod info to log context + - module: input-filter-k8s-containerd +``` + +The following example collects container log files, parses cri-o format format, parses container logs with default log patterns, and adds Kubernetes meta-data via Kubernetes API before it ships logs to Sematext Cloud: + +```yaml +input: + # make sure files include your cri-o log container folder + files: + - '/var/log/containers/*.log' + +inputFilter: + # parse containerd log format, add pod info to log context + - module: input-filter-k8s-containerd + +outputFilter: + # add k8s metadata via k8s API + - module: k8s-enrichment + +output: + elasticsearch: + module: elasticsearch + url: https://logsene-receiver.sematext.com + index: YOUR_LOGS_TOKEN + +``` + + +Run Logagent: +``` +logagent --config crio.yml +``` diff --git a/docs/logagent/input-kubernetes-audit.md b/docs/logagent/input-kubernetes-audit.md new file mode 100644 index 000000000..b0f31a1d5 --- /dev/null +++ b/docs/logagent/input-kubernetes-audit.md @@ -0,0 +1,65 @@ +title: Logagent input plugin for Kubernetes audit logs +description: Logagent features modular logging architecture framework where each input or output module is implemented as a plugin. Logagent can receive Kubernetes audit logs via http. + +## Input Plugin: Kubernetes Audit Logs + +Input plugin to receive Kubernetes audit logs via http. + +Features: + +- parse bulk messages + +Applications: + +- centralize Kubernetes audit logs +- act as webhook to receive Kubernetes audit logs +- index Kubernetes audit logs in Elasticsearch or Sematext Cloud +- create alerts on Kubernetes audit logs + + +Requirements: + +- Configure Kubernetes to send audit logs via webhook + +### Configuration + +```yaml + +# Receive Kubernetes Audit events via HTTP server +input: + kubernetesAudit: + module: input-kubernetes-audit + # server listens to a port + port: 9091 + # dynamic index setting by posting audit logs to /indexName/ URL + useIndexFromUrlPath: true + # number of extra processes to fork as web server workers + worker: 0 + tags: + receiver: logagent_kubernetes_audit + +output: + # view events on console during test setups + stdout: yaml + # ship audit logs to Sematext Cloud + elasticsearch: + module: elasticsearch + url: https://logsene-receiver.sematext.com + index: YOUR_LOGS_TOKEN + + +``` + +Start Logagent + +``` +logagent --config kubernetes-audit.yml +``` + + +Note, you can use the command line argument `--k8sAudit portNumber` to activate the plugin via logagent command. The following command would listen to Kubernetes events on TCP port 9091 and dumps the events in YAML format to console. + +``` +logagent --k8sAudit 9091 --yaml +``` + diff --git a/docs/logagent/input-kubernetes-events.md b/docs/logagent/input-kubernetes-events.md new file mode 100644 index 000000000..d3f4cfa5c --- /dev/null +++ b/docs/logagent/input-kubernetes-events.md @@ -0,0 +1,57 @@ +title: Logagent input plugin for Kubernetes Events +description: Logagent features modular logging architecture framework where each input or output module is implemented as a plugin. Logagent collects Kubernetes event logs. + +## Input Plugin: Kubernetes Event Collection + +Input plugin to collect Kubernetes events via API. + +Features: + +- parse bulk messages + +Applications: + +- centralize Kubernetes events +- index Kubernetes events in Elasticsearch or Sematext Cloud +- create alerts on Kubernetes events +- create analytics and monitor event logs + +Requirements: + +- A working `kubectl` config +- Or run Loagent in a pod, role bindings to access k8s API pods, events, and namespaces + +### Configuration + +```yaml + +# Receive Kubernetes events via HTTP server +input: + kubernetesEvents: + module: input-kubernetes-events + +output: + # view events on console during test setups + stdout: yaml + # ship events to Sematext Cloud + elasticsearch: + module: elasticsearch + url: https://logsene-receiver.sematext.com + index: YOUR_LOGS_TOKEN + + +``` + +Start Logagent + +``` +logagent --config kubernetes-events.yml +``` + + +Note, you can use the command line argument `--k8sEvents` to activate the plugin via logagent command. The following command would listen to Kubernetes events on TCP port 9091 and dumps the events in YAML format to console. + +``` +logagent --k8sEvents --yaml +``` + diff --git a/docs/logagent/input-plugin-dockerlogs.md b/docs/logagent/input-plugin-dockerlogs.md index e62106c8d..b6dc45e01 100644 --- a/docs/logagent/input-plugin-dockerlogs.md +++ b/docs/logagent/input-plugin-dockerlogs.md @@ -10,7 +10,7 @@ Features: - collect all container logs - discovers new containers - all logs are tagged with container id/name and image name -- disable container logs by setting LOGSENE_ENBALED=false label/env variable +- disable log collection for a container by setting LOGS_ENBALED=false label/env variable - enrich logs with with labels for later use in docker-enrichment plugin Applications: diff --git a/docs/logagent/installation-docker.md b/docs/logagent/installation-docker.md index 9d1642c4c..e5549c8ad 100644 --- a/docs/logagent/installation-docker.md +++ b/docs/logagent/installation-docker.md @@ -484,21 +484,29 @@ The component for detecting and parsing log messages — [logagent-js](http://se ### Log Routing -Routing logs from different containers to separate Sematext Cloud Logs Apps can be configured via docker labels (or environment variables e.g. on Kubernetes). Simply tag a container with the label (or environment variable) ```LOGSENE_TOKEN=YOUR_LOGSENE_TOKEN```. +Routing logs from different containers to separate Sematext Cloud Logs Apps can be configured via docker labels (or environment variables e.g. on Kubernetes). Simply tag a container with the label (or environment variable) ```LOGS_TOKEN=YOUR_LOGS_TOKEN```. Logagent inspects the containers for this label and ships the logs to the specified Logs App. +The following container environment variables and labels are supported: +- `LOGS_TOKEN=` - logs token for the container +- `LOGS_ENABLED=` - switch log collection for the container on or off. Note, the default value is configurable in Logagent configuration via the setting LOGSENE_ENABLED_DEFAULT. +- `LOGS_RECEIVER_URL=` - set multiple log destinations. The URL should include the token or index of an Elasticsearch API endpoint. E.g. `https://logsene-receiver.sematext.com/your_logs_token` + +The Kubernetes pod annotations `sematext.com/logs-token=`, `sematext.com/logs-enabled=`, and`sematext.com/logs-receiver-urls=` are equivalent. + + __Example:__ The following command will start Nginx webserver and logs for this container will be shipped to the related Logs App. ``` -docker run --label LOGSENE_TOKEN=REPLACE_WITH_YOUR_LOGS_TOKEN -p 80:80 nginx +docker run --label LOGS_TOKEN=REPLACE_WITH_YOUR_LOGS_TOKEN -p 80:80 nginx # or use environment variable on Kubernetes (no support for Docker labels) -# docker run -e LOGSENE_TOKEN=REPLACE_WITH_YOUR_LOG_TOKEN -p 80:80 nginx +# docker run -e LOGS_TOKEN=REPLACE_WITH_YOUR_LOG_TOKEN -p 80:80 nginx ``` -All other container logs will be shipped to the Logs App specified in the docker run command for ```sematext/logagent``` with the environment variable ```LOGSENE_TOKEN```. +All other container logs will be shipped to the Logs App specified in the docker run command for ```sematext/logagent``` with the environment variable ```LOGS_TOKEN```. -By default, all logs from all containers are collected and sent to Sematext Cloud/Elasticsearch. You can change this default by setting the ```LOGSENE_ENABLED_DEFAULT=false``` label for the Logagent container. This default can be overridden, on each container, through the ```LOGSENE_ENABLED``` label. +By default, all logs from all containers are collected and sent to Sematext Cloud/Elasticsearch. You can change this default by setting the ```LOGS_ENABLED_DEFAULT=false``` label for the Logagent container. This default can be overridden, on each container, through the ```LOGS_ENABLED``` label. Please refer to [Docker Log Management & Enrichment](https://sematext.com/blog/2017/05/15/docker-log-management-enrichment/) for further details. diff --git a/docs/logagent/output-filter-kubernetes.md b/docs/logagent/output-filter-kubernetes.md index 658a8b4a6..c242bff41 100644 --- a/docs/logagent/output-filter-kubernetes.md +++ b/docs/logagent/output-filter-kubernetes.md @@ -11,12 +11,13 @@ Features: - Evaluates pod annotations for Sematext Cloud: - `sematext/logs-token=YOUR_LOGS_TOKEN` to set the log index - `sematext/logs-enabled=` to switch logging per pod on or off + - `sematext/logs-receiver-url=https://logsene-receiver.sematext.com/token` set Elasticsearch API endpoint for log ingestion - `sematext/logs-remove-fields=` to remove specific log fields from pod logs Applications: - Log routing based on Kubernetes annotations -- Use Kuberntes API to get correct Kubernetes metadata +- Use Kubernetes API to get correct Kubernetes metadata ### Configuration diff --git a/docs/logagent/output-filter-removefields.md b/docs/logagent/output-filter-removefields.md index 1fed88864..8d721c672 100644 --- a/docs/logagent/output-filter-removefields.md +++ b/docs/logagent/output-filter-removefields.md @@ -35,6 +35,10 @@ outputFilter: fields: - user - client_ip + # json-path expressions are supported for nested fields + # See: https://jsonpath.com/ online evaluator + # - /request/header + # - ../body ``` diff --git a/docs/logagent/output-plugin-http.md b/docs/logagent/output-plugin-http.md new file mode 100644 index 000000000..ca5a01464 --- /dev/null +++ b/docs/logagent/output-plugin-http.md @@ -0,0 +1,38 @@ +title: Ship logs via http or https +description: Sends log messages as http or https post request + +## Output Plugin: HTTP + +Post logs to web services via `http` or `https`. + +Supported formats: +- ldjson - line delimited json + +### Configuration + +``` + output: + module: output-http + url: http://localhost:8080/events + format: ldjson + # maximum number of events to buffer + # 1 - each event creates a separate http request + # >1 - multiple events in each http request + maxBufferSize: 1 + # flush interval in seconds + flushInterval: 1 + # add tags before logs are shipped + tags: + token: SPM_TOKEN + role: backend + host: myServerName + filter: + field: logSource + match: sensor.* + ``` + +Start Logagent + +``` +logagent --config http.yaml +``` \ No newline at end of file diff --git a/docs/logagent/plugins.md b/docs/logagent/plugins.md index 5a0f6a89b..9d9ea81a8 100644 --- a/docs/logagent/plugins.md +++ b/docs/logagent/plugins.md @@ -1,4 +1,4 @@ -title: Logagent plugins +!title: Logagent plugins description: Logagent features modular logging architecture framework where each input or output module is implemented as a plugin, and loaded on demand as declared in the configuration file. It is used with elasticsearch, syslog, gelf, cassandra, mysql, postgres, mqtt, log anonymization, apache kafka, and more Logagent features a modular architecture. Each input or output module is implemented as a plugin for the Logagent framework. Plugins are loaded on demand as declared in the configuration file. @@ -8,6 +8,8 @@ Logagent features a modular architecture. Each input or output module is impleme | [stdin (default)](input-plugin-stdin) | input | Reads from standard input | | [files](input-plugin-files) | input | Watching and tailing files | | [docker-logs](input-plugin-dockerlogs) | input | Collection of Docker container logs | +| [input-kubernetes-events](input-kubernetes-events) | input | Collection of Kubernetes events | +| [input-kubernetes-audit](input-kubernetes-audit) | input | Receive Kubernetes audit logs via http / webhook | | [logagent-input-windows-events](https://www.npmjs.com/package/logagent-input-windows-events) | input | Collect Windows Events. Available as separate npm package | | [logagent-input-elasticsearch-stats](https://www.npmjs.com/package/logagent-input-elasticsearch-stats) | input | Monitoring of Elasticsearch metrics. Available as separate npm package | | [syslog](input-plugin-syslog.md) | input | Receive Syslog messages via UDP | @@ -29,8 +31,8 @@ Logagent features a modular architecture. Each input or output module is impleme | [logagent-apple-location](logagent-apple-location) | input | Tracking of GPS positions from Apple devices via "find-my-iphone" API | | [logagent-novasds](logagent-novasds) | input | Read PM10 and PM2.5 values from Nova SDS011 dust sensor (USB to serial interface) | | [grep](input-filter-grep) | Processor / input filter | Filters text with regular expressions before parsing | +| [input-filter-k8s-containerd](input-filter-containerd.md) | Processor / input filter | Parsing cri-o log format and add Kubernetes context to container logs | | [sql](output-filter-sql) | Processor / output filter | Transforms and aggregates parsed messages with SQL statements | -| [access-watch](output-filter-accesswatch) | Processor / output filter | Enriches web server logs with robot detection and traffic intelligence | | [aes-encrypt-fields](output-filter-aesencryptfields) | Processor / output filter | Encrypt field values with AES before any output happens | | [hash-fields](output-filter-hashfields) | Processor / output filter | Hashing of field values before any output happens | | [ip-truncate-fields](output-filter-iptruncatefields) | Processor / output filter | Replaces the last block of IPv4 and IPv6 address fields with "0" to anonymize IP addresses | @@ -48,9 +50,14 @@ Logagent features a modular architecture. Each input or output module is impleme | [output-files](output-plugin-files) | output | Stores parsed messages files. Log rotation and dynamic file name generation are supported. | | [output-clickhouse](output-plugin-clickhouse) | output | Sends parsed messages to Yandex ClickHouse DB | | [logagent-output-kafka](output-plugin-kafka) | output | Sends parsed messages to Apache Kafka topics. 3rd party module. 3rd party module. | +| [output-http](output-plugin-clickhouse) | output | Sends parsed messages via HTTP or HTTPS | | [slack-webhook](output-plugin-slack) | output | Sends parsed messages to Slack chat. Should be combined with SQL filter plugin or filter function to define alert criterias. | | [@sematext/logagent-nodejs-monitor](https://www.npmjs.com/package/@sematext/logagent-nodejs-monitor) | other | Monitors server and nodejs metrics of the Logagent process using [spm-agent-nodejs](https://www.npmjs.com/package/spm-agent-nodejs) | +## Find plugins on NPM + +Developers of 3rd party plugins publish logagent plugins in the NPM registry. +Simply search for [logagent](https://www.npmjs.com/search?q=logagent) to discover more plugins. ## For Developers: How Logagent plugins work diff --git a/mkdocs.yml b/mkdocs.yml index c986bdb0f..4507deb5a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -199,6 +199,7 @@ pages: - Input Filter: - Grep input filter: logagent/input-filter-grep.md - Grok input filter: logagent/input-filter-grok.md + - Kubernetes cri-o logs: logagent/input-filter-containerd.md - Input Plugins: - Standard input: logagent/input-plugin-stdin.md - File input: logagent/input-plugin-files.md @@ -222,8 +223,9 @@ pages: - Apple device location: logagent/logagent-apple-location.md - Nova dust sensor: logagent/logagent-novasds.md - Cloud Foundry input: logagent/input-plugin-cloudfoundry.md + - Kubernetes Events: logagent/input-kubernetes-events.md + - Kubernetes Audit Logs: logagent/input-kubernetes-audit.md - Output Filter: - - Access Watch output filter: logagent/output-filter-accesswatch.md - SQL output filter: logagent/output-filter-sql.md - AES encrypt fields: logagent/output-filter-aesencryptfields.md - Hash fields: logagent/output-filter-hashfields.md @@ -246,6 +248,7 @@ pages: - ClickHouse DB: logagent/output-plugin-clickhouse.md - Slack Webhook: logagent/output-plugin-slack.md - Prometheus Alertmanager: logagent/output-plugin-prometheus-alertmanager.md + - HTTP post: logagent/output-plugin-http.md - How-To: - Nginx Ingress Log Shipping: logagent/how-to-nginx-ingress-log-shipping.md - GDPR Personal Data In Web Server Logs: logagent/how-to-gdpr_web_logs.md From 638918bf421e335bd241e4ce25f82ca10423140e Mon Sep 17 00:00:00 2001 From: Adnan Rahic Date: Thu, 13 Feb 2020 13:34:38 +0100 Subject: [PATCH 2/3] fix broken links for agents --- docs/agents/index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/agents/index.md b/docs/agents/index.md index 10e11bb51..4e3fee2af 100644 --- a/docs/agents/index.md +++ b/docs/agents/index.md @@ -4,7 +4,7 @@ description: Sematext has several different Agents available for you to monitor To start monitoring your infrastructure you need to set up the appropriate Sematext monitoring agent. You can choose from: - - [Sematext Agent](../sematext-agent/), a lightweight, blazing + - [Sematext Agent](./sematext-agent/), a lightweight, blazing fast Go-based Monitoring Agent with a tiny footprint for both infrastructure and containers. It also collects metrics for various [integrations](../integration) using App Agents. App Agents can also @@ -12,5 +12,5 @@ appropriate Sematext monitoring agent. You can choose from: [on demand profiling](../monitoring/on-demand-profiling). - [Node.js-based App Agent](./node-agent), which can [monitor - Apache](../integration/apache) and [Nginx](../integration/nginx) - (including [Nginx Plus](../integration/nginxplus)). + Apache](../integration/apache), [Nginx](../integration/nginx) + (including [Nginx Plus](../integration/nginxplus)), [Express.js](../integration/express.js), and [Node.js](../integration/node.js) of course. From 0b978767db9990456dd58bf8c9708a4f89bf346b Mon Sep 17 00:00:00 2001 From: megastef Date: Fri, 14 Feb 2020 10:46:58 +0100 Subject: [PATCH 3/3] changes after review --- docs/logagent/input-filter-containerd.md | 4 +-- docs/logagent/input-kubernetes-audit.md | 39 ++++++++++++------------ docs/logagent/output-plugin-http.md | 17 ++++------- docs/logagent/plugins.md | 19 +++++++----- 4 files changed, 39 insertions(+), 40 deletions(-) diff --git a/docs/logagent/input-filter-containerd.md b/docs/logagent/input-filter-containerd.md index ae3bc7de5..68d35ea1c 100644 --- a/docs/logagent/input-filter-containerd.md +++ b/docs/logagent/input-filter-containerd.md @@ -7,7 +7,7 @@ Parsing cri-o containerd log format. Use the file input plugin to read log files ## Configuration -Note: the plugin can be enabled via command line option `--k8sContainerd`. +Note: The plugin can be enabled via command line option `--k8sContainerd`. Add the following section to the Logagent configuration file: @@ -17,7 +17,7 @@ inputFilter: - module: input-filter-k8s-containerd ``` -The following example collects container log files, parses cri-o format format, parses container logs with default log patterns, and adds Kubernetes meta-data via Kubernetes API before it ships logs to Sematext Cloud: +The following example collects container log files, parses cri-o format, parses container logs with default log patterns, and adds Kubernetes meta-data via Kubernetes API before it ships logs to Sematext Cloud: ```yaml input: diff --git a/docs/logagent/input-kubernetes-audit.md b/docs/logagent/input-kubernetes-audit.md index b0f31a1d5..f99644536 100644 --- a/docs/logagent/input-kubernetes-audit.md +++ b/docs/logagent/input-kubernetes-audit.md @@ -1,9 +1,9 @@ -title: Logagent input plugin for Kubernetes audit logs -description: Logagent features modular logging architecture framework where each input or output module is implemented as a plugin. Logagent can receive Kubernetes audit logs via http. +title: Logagent input plugin for Kubernetes Audit logs +description: Logagent features modular logging architecture framework where each input or output module is implemented as a plugin. Logagent can receive Kubernetes Audit logs via http. ## Input Plugin: Kubernetes Audit Logs -Input plugin to receive Kubernetes audit logs via http. +Input plugin to receive Kubernetes Audit logs via HTTP. Features: @@ -11,27 +11,27 @@ Features: Applications: -- centralize Kubernetes audit logs -- act as webhook to receive Kubernetes audit logs -- index Kubernetes audit logs in Elasticsearch or Sematext Cloud -- create alerts on Kubernetes audit logs +- centralize Kubernetes Audit logs +- act as webhook to receive Kubernetes Audit logs +- index Kubernetes Audit logs in Elasticsearch or Sematext Cloud +- create alerts on Kubernetes Audit logs Requirements: -- Configure Kubernetes to send audit logs via webhook +- configure Kubernetes to send Audit logs via webhook ### Configuration ```yaml -# Receive Kubernetes Audit events via HTTP server +# Receive Kubernetes Audit logs via HTTP server input: - kubernetesAudit: + kubernetesAudit: module: input-kubernetes-audit # server listens to a port port: 9091 - # dynamic index setting by posting audit logs to /indexName/ URL + # dynamic index setting by posting Audit logs to /indexName/ URL useIndexFromUrlPath: true # number of extra processes to fork as web server workers worker: 0 @@ -39,14 +39,13 @@ input: receiver: logagent_kubernetes_audit output: - # view events on console during test setups - stdout: yaml - # ship audit logs to Sematext Cloud - elasticsearch: - module: elasticsearch - url: https://logsene-receiver.sematext.com - index: YOUR_LOGS_TOKEN - + # view events on console during test setups + stdout: yaml + # ship Audit logs to Sematext Cloud + elasticsearch: + module: elasticsearch + url: https://logsene-receiver.sematext.com + index: YOUR_LOGS_TOKEN ``` @@ -57,7 +56,7 @@ logagent --config kubernetes-audit.yml ``` -Note, you can use the command line argument `--k8sAudit portNumber` to activate the plugin via logagent command. The following command would listen to Kubernetes events on TCP port 9091 and dumps the events in YAML format to console. +Note: You can use the command line argument `--k8sAudit portNumber` to activate the plugin via the `logagent` command. The following command listens on TCP port 9091 for Kubernetes logs and dumps the logs in YAML format to the console. ``` logagent --k8sAudit 9091 --yaml diff --git a/docs/logagent/output-plugin-http.md b/docs/logagent/output-plugin-http.md index ca5a01464..162fa3831 100644 --- a/docs/logagent/output-plugin-http.md +++ b/docs/logagent/output-plugin-http.md @@ -1,12 +1,12 @@ -title: Ship logs via http or https -description: Sends log messages as http or https post request +title: Ship logs via HTTP or HTTPS +description: Sends log messages as HTTP or HTTPS post request ## Output Plugin: HTTP -Post logs to web services via `http` or `https`. +Post logs to web services via `HTTP` or `HTTPS`. Supported formats: -- ldjson - line delimited json +- ldjson - line delimited json ### Configuration @@ -15,17 +15,12 @@ Supported formats: module: output-http url: http://localhost:8080/events format: ldjson - # maximum number of events to buffer + # maximum number of events per request # 1 - each event creates a separate http request # >1 - multiple events in each http request maxBufferSize: 1 # flush interval in seconds - flushInterval: 1 - # add tags before logs are shipped - tags: - token: SPM_TOKEN - role: backend - host: myServerName + flushInterval: 5 filter: field: logSource match: sensor.* diff --git a/docs/logagent/plugins.md b/docs/logagent/plugins.md index 9d9ea81a8..047dbbc0b 100644 --- a/docs/logagent/plugins.md +++ b/docs/logagent/plugins.md @@ -1,4 +1,4 @@ -!title: Logagent plugins +title: Logagent plugins description: Logagent features modular logging architecture framework where each input or output module is implemented as a plugin, and loaded on demand as declared in the configuration file. It is used with elasticsearch, syslog, gelf, cassandra, mysql, postgres, mqtt, log anonymization, apache kafka, and more Logagent features a modular architecture. Each input or output module is implemented as a plugin for the Logagent framework. Plugins are loaded on demand as declared in the configuration file. @@ -54,21 +54,25 @@ Logagent features a modular architecture. Each input or output module is impleme | [slack-webhook](output-plugin-slack) | output | Sends parsed messages to Slack chat. Should be combined with SQL filter plugin or filter function to define alert criterias. | | [@sematext/logagent-nodejs-monitor](https://www.npmjs.com/package/@sematext/logagent-nodejs-monitor) | other | Monitors server and nodejs metrics of the Logagent process using [spm-agent-nodejs](https://www.npmjs.com/package/spm-agent-nodejs) | -## Find plugins on NPM +## Find plugins on npm -Developers of 3rd party plugins publish logagent plugins in the NPM registry. +Developers of 3rd party plugins publish logagent plugins in the [npm registry](https://npmjs.org). Simply search for [logagent](https://www.npmjs.com/search?q=logagent) to discover more plugins. ## For Developers: How Logagent plugins work -- Logagent checks the configuration file for properties with a "module" key for the nodejs module name. External plugins need to be installed via npm. +- Logagent checks the configuration file for properties with a "module" key for the nodejs module name. + External plugins need to be installed via npm. - Plugins are initialized with the Logagent configuration (from command line arguments + configuration file) and the event emitter for Logagent. Plugins should provide a start and stop method. - Input plugins read data from a data source and emit events to the Logagent event emitter. - These events have the identifier "data.raw" and 2 parameters: - - data - data read from a data source + These events have the identifier `data.raw` and 2 parameters: + - data - a string containing a text line, read from a data source - context - an object with meta data e.g. {sourceName: '/var/log/httpd/access.log'} The "context" helps other plugins to process the data correctly, e.g. to handle multiple open files. -- Output plugins listen to "data.parsed" events and store or forward the data to the target. + In some cases, input plugins create strcutured data, and it makes no sense to process the data with text bases input-filters and Logagent parser. Input plugins can emit a `data.object` event, and only output-filters and output plugins will process such events with the following parameters: + - data - a JavaScript object e.g. `{message: 'hello', severity: 'info'}` + - context - an object with meta data e.g. {sourceName: '/var/log/httpd/access.log'} +- Output plugins listen to `data.parsed` events and store or forward the data to the target. ### Examples @@ -182,6 +186,7 @@ __Example Output Plugin (stdout)__ 'use strict' var prettyjson = require('prettyjson') var safeStringify = require('fast-safe-stringify') + function OutputStdout (config, eventEmitter) { this.config = config this.eventEmitter = eventEmitter