Merge pull request #174 from sematext/logagent_updates_2

Add Logagent plugins
sematext · Feb 14, 2020 · 4913dd3 · 4913dd3
2 parents 638918b + 0b97876
commit 4913dd3
Show file tree

Hide file tree

Showing 10 changed files with 246 additions and 14 deletions.
diff --git a/docs/logagent/input-filter-containerd.md b/docs/logagent/input-filter-containerd.md
@@ -0,0 +1,48 @@
+title: Parsing cri-o log format, add Kubernetes context to container logs 
+description: Logagent features modular logging architecture framework where each input or output module is implemented as a plugin, and loaded on demand as declared in the configuration file. Input filters process raw input from input plugins before log events get parsed 
+
+## Input Filter: Kubernetes cri-o / containerd  
+
+Parsing cri-o containerd log format. Use the file input plugin to read log files. 
+
+## Configuration 
+
+Note: The plugin can be enabled via command line option `--k8sContainerd`.
+
+Add the following section to the Logagent configuration file:  
+
+```
+inputFilter:
+  # parse containerd log format, add pod info to log context 
+  - module: input-filter-k8s-containerd
+```
+
+The following example collects container log files, parses cri-o format, parses container logs with default log patterns, and adds Kubernetes meta-data via Kubernetes API before it ships logs to Sematext Cloud: 
+
+```yaml
+input: 
+  # make sure files include your cri-o log container folder 
+  files:
+    - '/var/log/containers/*.log'
+
+inputFilter:
+  # parse containerd log format, add pod info to log context 
+  - module: input-filter-k8s-containerd
+
+outputFilter:
+  # add k8s metadata via k8s API 
+  - module: k8s-enrichment 
+
+output:
+  elasticsearch:
+    module: elasticsearch
+    url: https://logsene-receiver.sematext.com
+    index: YOUR_LOGS_TOKEN
+
+```
+
+
+Run Logagent: 
+```
+logagent --config crio.yml 
+```
diff --git a/docs/logagent/input-kubernetes-audit.md b/docs/logagent/input-kubernetes-audit.md
@@ -0,0 +1,64 @@
+title: Logagent input plugin for Kubernetes Audit logs
+description: Logagent features modular logging architecture framework where each input or output module is implemented as a plugin. Logagent can receive Kubernetes Audit logs via http. 
+
+## Input Plugin: Kubernetes Audit Logs
+
+Input plugin to receive Kubernetes Audit logs via HTTP.
+
+Features:
+
+- parse bulk messages
+
+Applications:
+
+- centralize Kubernetes Audit logs
+- act as webhook to receive Kubernetes Audit logs
+- index Kubernetes Audit logs in Elasticsearch or Sematext Cloud
+- create alerts on Kubernetes Audit logs
+
+
+Requirements: 
+
+- configure Kubernetes to send Audit logs via webhook 
+
+### Configuration
+
+```yaml
+
+# Receive Kubernetes Audit logs via HTTP server
+input:
+  kubernetesAudit:
+    module: input-kubernetes-audit
+    # server listens to a port 
+    port: 9091
+    # dynamic index setting by posting Audit logs to /indexName/ URL 
+    useIndexFromUrlPath: true
+    # number of extra processes to fork as web server workers
+    worker: 0
+    tags:
+      receiver: logagent_kubernetes_audit
+
+output:
+  # view events on console during test setups
+  stdout: yaml
+  # ship Audit logs to Sematext Cloud
+  elasticsearch:
+    module: elasticsearch
+    url: https://logsene-receiver.sematext.com
+    index: YOUR_LOGS_TOKEN
+
+```
+
+Start Logagent
+
+```
+logagent --config kubernetes-audit.yml
+```
+
+
+Note: You can use the command line argument `--k8sAudit portNumber` to activate the plugin via the `logagent` command. The following command listens on TCP port 9091 for Kubernetes logs and dumps the logs in YAML format to the console.
+
+```
+logagent --k8sAudit 9091 --yaml
+```
+
diff --git a/docs/logagent/input-kubernetes-events.md b/docs/logagent/input-kubernetes-events.md
@@ -0,0 +1,57 @@
+title: Logagent input plugin for Kubernetes Events
+description: Logagent features modular logging architecture framework where each input or output module is implemented as a plugin. Logagent collects Kubernetes event logs.  
+
+## Input Plugin: Kubernetes Event Collection
+
+Input plugin to collect Kubernetes events via API.
+
+Features:
+
+- parse bulk messages
+
+Applications:
+
+- centralize Kubernetes events
+- index Kubernetes events in Elasticsearch or Sematext Cloud
+- create alerts on Kubernetes events
+- create analytics and monitor event logs 
+
+Requirements: 
+
+- A working `kubectl` config
+- Or run Loagent in a pod, role bindings to access k8s API pods, events, and namespaces 
+
+### Configuration
+
+```yaml
+
+# Receive Kubernetes events via HTTP server
+input:
+   kubernetesEvents:
+    module: input-kubernetes-events
+
+output:
+    # view events on console during test setups
+    stdout: yaml
+    # ship events to Sematext Cloud
+    elasticsearch:
+      module: elasticsearch
+      url: https://logsene-receiver.sematext.com
+      index: YOUR_LOGS_TOKEN
+
+
+```
+
+Start Logagent
+
+```
+logagent --config kubernetes-events.yml
+```
+
+
+Note, you can use the command line argument `--k8sEvents` to activate the plugin via logagent command. The following command would listen to Kubernetes events on TCP port 9091 and dumps the events in YAML format to console. 
+
+```
+logagent --k8sEvents --yaml
+```
+
diff --git a/docs/logagent/input-plugin-dockerlogs.md b/docs/logagent/input-plugin-dockerlogs.md
@@ -10,7 +10,7 @@ Features:
 - collect all container logs
 - discovers new containers
 - all logs are tagged with container id/name and image name
-- disable container logs by setting LOGSENE_ENBALED=false label/env variable
+- disable log collection for a container by setting LOGS_ENBALED=false label/env variable
 - enrich logs with with labels for later use in docker-enrichment plugin 
 
 Applications:

diff --git a/docs/logagent/installation-docker.md b/docs/logagent/installation-docker.md
@@ -484,21 +484,31 @@ The component for detecting and parsing log messages — [logagent-js](http://se
 
 ### Log Routing
 
+Routing logs from different containers to separate Sematext Cloud Logs Apps can be configured via docker labels (or environment variables e.g. on Kubernetes). Simply tag a container with the label (or environment variable) ```LOGS_TOKEN=YOUR_LOGS_TOKEN```. 
+Logagent inspects the containers for this label and ships the logs to the specified Logs App. 
+
+The following container environment variables and labels are supported:
+- `LOGS_TOKEN=<YOUR_LOGS_TOKEN>` - logs token for the container
+- `LOGS_ENABLED=<true|false>` - switch log collection for the container on or off. Note, the default value is configurable in Logagent configuration via the setting LOGSENE_ENABLED_DEFAULT.  
+- `LOGS_RECEIVER_URL=<URL, URL, URL>` - set multiple log destinations. The URL should include the token or index of an Elasticsearch API endpoint. E.g. `https://logsene-receiver.sematext.com/your_logs_token`
+
+The Kubernetes pod annotations `sematext.com/logs-token=<YOUR_LOGS_TOKEN>`, `sematext.com/logs-enabled=<true|false>`, and`sematext.com/logs-receiver-urls=<url,url,url>` are equivalent.  
+
+
+__Example:__ 
+The following command will start Nginx webserver and logs for this container will be shipped to the related Logs App. 
 Routing logs from different containers to separate Sematext Cloud Logs Apps can be configured via docker labels (or environment variables e.g. on Kubernetes). Simply tag a container with the label (or environment variable) ```LOGSENE_TOKEN=YOUR_LOGSENE_TOKEN```.
 Logagent inspects the containers for this label and ships the logs to the specified Logs App.
 
-__Example:__
-The following command will start Nginx webserver and logs for this container will be shipped to the related Logs App.
-
 ```
-docker run --label LOGSENE_TOKEN=REPLACE_WITH_YOUR_LOGS_TOKEN -p 80:80 nginx
+docker run --label LOGS_TOKEN=REPLACE_WITH_YOUR_LOGS_TOKEN -p 80:80 nginx
 # or use environment variable on Kubernetes (no support for Docker labels)
-# docker run -e LOGSENE_TOKEN=REPLACE_WITH_YOUR_LOG_TOKEN -p 80:80 nginx
+# docker run -e LOGS_TOKEN=REPLACE_WITH_YOUR_LOG_TOKEN -p 80:80 nginx
 ```
 
-All other container logs will be shipped to the Logs App specified in the docker run command for ```sematext/logagent``` with the environment variable ```LOGSENE_TOKEN```.
+All other container logs will be shipped to the Logs App specified in the docker run command for ```sematext/logagent``` with the environment variable ```LOGS_TOKEN```.
 
-By default, all logs from all containers are collected and sent to Sematext Cloud/Elasticsearch. You can change this default by setting the ```LOGSENE_ENABLED_DEFAULT=false``` label for the Logagent container. This default can be overridden, on each container, through the ```LOGSENE_ENABLED``` label.
+By default, all logs from all containers are collected and sent to Sematext Cloud/Elasticsearch. You can change this default by setting the ```LOGS_ENABLED_DEFAULT=false``` label for the Logagent container. This default can be overridden, on each container, through the ```LOGS_ENABLED``` label.
 
 Please refer to [Docker Log Management & Enrichment](https://sematext.com/blog/2017/05/15/docker-log-management-enrichment/) for further details.
 

diff --git a/docs/logagent/output-filter-kubernetes.md b/docs/logagent/output-filter-kubernetes.md
@@ -11,6 +11,7 @@ Features:
 - Evaluates pod annotations for Sematext Cloud: 
   -  `sematext/logs-token=YOUR_LOGS_TOKEN` to set the log index
   -  `sematext/logs-enabled=<true|false>` to switch logging per pod on or off
+  -  `sematext/logs-receiver-url=https://logsene-receiver.sematext.com/token` set Elasticsearch API endpoint for log ingestion 
    - `sematext/logs-remove-fields=<field list>` to remove specific log fields from pod logs 
 
 Applications:

diff --git a/docs/logagent/output-filter-removefields.md b/docs/logagent/output-filter-removefields.md
@@ -35,6 +35,10 @@ outputFilter:
     fields:
       - user
       - client_ip
+      # json-path expressions are supported for nested fields
+      # See: https://jsonpath.com/ online evaluator
+      # - /request/header
+      # - ../body
 
 ```
 

diff --git a/docs/logagent/output-plugin-http.md b/docs/logagent/output-plugin-http.md
@@ -0,0 +1,33 @@
+title: Ship logs via HTTP or HTTPS
+description: Sends log messages as HTTP or HTTPS post request
+
+## Output Plugin: HTTP
+
+Post logs to web services via `HTTP` or `HTTPS`. 
+
+Supported formats: 
+- ldjson - line delimited json
+
+### Configuration
+
+```
+ output:
+    module: output-http
+    url: http://localhost:8080/events
+    format: ldjson
+    # maximum number of events per request
+    # 1 - each event creates a separate http request
+    # >1 - multiple events in each http request
+    maxBufferSize: 1
+    # flush interval in seconds
+    flushInterval: 5
+    filter:
+      field: logSource
+      match: sensor.*
+ ```
+
+Start Logagent
+
+```
+logagent --config http.yaml
+```
diff --git a/docs/logagent/plugins.md b/docs/logagent/plugins.md
@@ -8,6 +8,8 @@ Logagent features a modular architecture. Each input or output module is impleme
 | [stdin (default)](input-plugin-stdin)                       | input   | Reads from standard input                                                          |
 | [files](input-plugin-files)           | input   | Watching and tailing files                                                                               |
 | [docker-logs](input-plugin-dockerlogs)           | input   | Collection of Docker container logs                                                           |
+| [input-kubernetes-events](input-kubernetes-events)           | input   | Collection of Kubernetes events                                                         |
+| [input-kubernetes-audit](input-kubernetes-audit)           | input   | Receive Kubernetes audit logs via http / webhook                                    |
 | [logagent-input-windows-events](https://www.npmjs.com/package/logagent-input-windows-events) | input  | Collect Windows Events. Available as separate npm package |
 | [logagent-input-elasticsearch-stats](https://www.npmjs.com/package/logagent-input-elasticsearch-stats) | input | Monitoring of Elasticsearch metrics. Available as separate npm package |
 | [syslog](input-plugin-syslog.md)      | input | Receive Syslog messages via UDP |
@@ -29,8 +31,8 @@ Logagent features a modular architecture. Each input or output module is impleme
 | [logagent-apple-location](logagent-apple-location)         | input          | Tracking of GPS positions from Apple devices via "find-my-iphone" API |
 | [logagent-novasds](logagent-novasds)         | input          | Read PM10 and PM2.5 values from Nova SDS011 dust sensor (USB to serial interface) |
 | [grep](input-filter-grep) | Processor / input filter  | Filters text with regular expressions before parsing                                                     |
+| [input-filter-k8s-containerd](input-filter-containerd.md) | Processor / input filter  | Parsing cri-o log format and add Kubernetes context to container logs  |
 | [sql](output-filter-sql)  | Processor / output filter | Transforms and aggregates parsed messages with SQL statements                                            |
-| [access-watch](output-filter-accesswatch) | Processor / output filter | Enriches web server logs with robot detection and traffic intelligence                                   |
 | [aes-encrypt-fields](output-filter-aesencryptfields) | Processor / output filter | Encrypt field values with AES before any output happens |
 | [hash-fields](output-filter-hashfields) | Processor / output filter | Hashing of field values before any output happens |
 | [ip-truncate-fields](output-filter-iptruncatefields) | Processor / output filter | Replaces the last block of IPv4 and IPv6 address fields with "0" to anonymize IP addresses |
@@ -48,20 +50,29 @@ Logagent features a modular architecture. Each input or output module is impleme
 | [output-files](output-plugin-files)     | output                    | Stores parsed messages files. Log rotation and dynamic file name generation are supported.                                                                  |
 | [output-clickhouse](output-plugin-clickhouse)       | output                   | Sends parsed messages to Yandex ClickHouse DB |
 | [logagent-output-kafka](output-plugin-kafka)       | output                   | Sends parsed messages to Apache Kafka topics. 3rd party module. 3rd party module.                                                             |
+| [output-http](output-plugin-clickhouse)       | output                   | Sends parsed messages via HTTP or HTTPS |
 | [slack-webhook](output-plugin-slack)      | output                    | Sends parsed messages to Slack chat. Should be combined with SQL filter plugin or filter function to define alert criterias. |
 | [@sematext/logagent-nodejs-monitor](https://www.npmjs.com/package/@sematext/logagent-nodejs-monitor) | other | Monitors server and  nodejs metrics of the Logagent process using [spm-agent-nodejs](https://www.npmjs.com/package/spm-agent-nodejs) |
 
+## Find plugins on npm 
+
+Developers of 3rd party plugins publish logagent plugins in the [npm registry](https://npmjs.org). 
+Simply search for [logagent](https://www.npmjs.com/search?q=logagent) to discover more plugins. 
 
 ## For Developers: How Logagent plugins work 
 
-- Logagent checks the configuration file for properties with a "module" key for the nodejs module name. External plugins need to be installed via npm. 
+- Logagent checks the configuration file for properties with a "module" key for the nodejs module name. 
+  External plugins need to be installed via npm. 
 - Plugins are initialized with the Logagent configuration (from command line arguments + configuration file) and the event emitter for Logagent. Plugins should provide a start and stop method.
 - Input plugins read data from a data source and emit events to the Logagent event emitter.
-  These events have the identifier "data.raw" and 2 parameters: 
-  - data - data read from a data source 
+  These events have the identifier `data.raw` and 2 parameters: 
+  - data - a string containing a text line, read from a data source 
   - context - an object with meta data e.g. {sourceName: '/var/log/httpd/access.log'}
     The "context" helps other plugins to process the data correctly, e.g. to handle multiple open files. 
-- Output plugins listen to "data.parsed" events and store or forward the data to the target. 
+  In some cases, input plugins create strcutured data, and it makes no sense to process the data with text bases input-filters and Logagent parser. Input plugins can emit a `data.object` event, and only output-filters and output plugins will process such events with the following parameters: 
+  - data - a JavaScript object e.g. `{message: 'hello', severity: 'info'}`
+  - context - an object with meta data e.g. {sourceName: '/var/log/httpd/access.log'}
+- Output plugins listen to `data.parsed` events and store or forward the data to the target. 
 
 ### Examples 
 
@@ -175,6 +186,7 @@ __Example Output Plugin (stdout)__
 'use strict'
 var prettyjson = require('prettyjson')
 var safeStringify = require('fast-safe-stringify')
+
 function OutputStdout (config, eventEmitter) {
   this.config = config
   this.eventEmitter = eventEmitter

diff --git a/mkdocs.yml b/mkdocs.yml
@@ -201,6 +201,7 @@ pages:
           - Input Filter:
             - Grep input filter: logagent/input-filter-grep.md
             - Grok input filter: logagent/input-filter-grok.md
+            - Kubernetes cri-o logs: logagent/input-filter-containerd.md
           - Input Plugins:
             - Standard input: logagent/input-plugin-stdin.md
             - File input: logagent/input-plugin-files.md
@@ -224,8 +225,9 @@ pages:
             - Apple device location: logagent/logagent-apple-location.md
             - Nova dust sensor: logagent/logagent-novasds.md
             - Cloud Foundry input: logagent/input-plugin-cloudfoundry.md
+            - Kubernetes Events: logagent/input-kubernetes-events.md
+            - Kubernetes Audit Logs: logagent/input-kubernetes-audit.md
           - Output Filter:
-            - Access Watch output filter: logagent/output-filter-accesswatch.md
             - SQL output filter: logagent/output-filter-sql.md
             - AES encrypt fields: logagent/output-filter-aesencryptfields.md
             - Hash fields: logagent/output-filter-hashfields.md
@@ -248,6 +250,7 @@ pages:
             - ClickHouse DB: logagent/output-plugin-clickhouse.md
             - Slack Webhook: logagent/output-plugin-slack.md
             - Prometheus Alertmanager: logagent/output-plugin-prometheus-alertmanager.md
+            - HTTP post: logagent/output-plugin-http.md
         - How-To:
             - Nginx Ingress Log Shipping: logagent/how-to-nginx-ingress-log-shipping.md
             - GDPR Personal Data In Web Server Logs: logagent/how-to-gdpr_web_logs.md