Merge branch 'master' into megastef-logagent-updates

docs/agents/index.md

@@ -4,13 +4,13 @@ description: Sematext has several different Agents available for you to monitor
 To start monitoring your infrastructure you need to set up the
 appropriate Sematext monitoring agent. You can choose from:
 
-  - [Sematext Agent](../sematext-agent/), a lightweight, blazing
+  - [Sematext Agent](./sematext-agent/), a lightweight, blazing
     fast Go-based Monitoring Agent with a tiny footprint for both
     infrastructure and containers. It also collects metrics for various
     [integrations](../integration) using App Agents. App Agents can also
     instrument JVM-based apps to collect transaction traces and perform
     [on demand profiling](../monitoring/on-demand-profiling).
 
   - [Node.js-based App Agent](./node-agent), which can [monitor
-    Apache](../integration/apache) and [Nginx](../integration/nginx)
-    (including [Nginx Plus](../integration/nginxplus)).
+    Apache](../integration/apache), [Nginx](../integration/nginx)
+    (including [Nginx Plus](../integration/nginxplus)), [Express.js](../integration/express.js), and [Node.js](../integration/node.js) of course.

docs/logagent/input-filter-containerd.md

@@ -0,0 +1,48 @@
+title: Parsing cri-o log format, add Kubernetes context to container logs 
+description: Logagent features modular logging architecture framework where each input or output module is implemented as a plugin, and loaded on demand as declared in the configuration file. Input filters process raw input from input plugins before log events get parsed 
+
+## Input Filter: Kubernetes cri-o / containerd  
+
+Parsing cri-o containerd log format. Use the file input plugin to read log files. 
+
+## Configuration 
+
+Note: The plugin can be enabled via command line option `--k8sContainerd`.
+
+Add the following section to the Logagent configuration file:  
+
+```
+inputFilter:
+  # parse containerd log format, add pod info to log context 
+  - module: input-filter-k8s-containerd
+```
+
+The following example collects container log files, parses cri-o format, parses container logs with default log patterns, and adds Kubernetes meta-data via Kubernetes API before it ships logs to Sematext Cloud: 
+
+```yaml
+input: 
+  # make sure files include your cri-o log container folder 
+  files:
+    - '/var/log/containers/*.log'
+
+inputFilter:
+  # parse containerd log format, add pod info to log context 
+  - module: input-filter-k8s-containerd
+
+outputFilter:
+  # add k8s metadata via k8s API 
+  - module: k8s-enrichment 
+
+output:
+  elasticsearch:
+    module: elasticsearch
+    url: https://logsene-receiver.sematext.com
+    index: YOUR_LOGS_TOKEN
+
+```
+
+
+Run Logagent: 
+```
+logagent --config crio.yml 
+```

docs/logagent/input-kubernetes-audit.md

@@ -0,0 +1,64 @@
+title: Logagent input plugin for Kubernetes Audit logs
+description: Logagent features modular logging architecture framework where each input or output module is implemented as a plugin. Logagent can receive Kubernetes Audit logs via http. 
+
+## Input Plugin: Kubernetes Audit Logs
+
+Input plugin to receive Kubernetes Audit logs via HTTP.
+
+Features:
+
+- parse bulk messages
+
+Applications:
+
+- centralize Kubernetes Audit logs
+- act as webhook to receive Kubernetes Audit logs
+- index Kubernetes Audit logs in Elasticsearch or Sematext Cloud
+- create alerts on Kubernetes Audit logs
+
+
+Requirements: 
+
+- configure Kubernetes to send Audit logs via webhook 
+
+### Configuration
+
+```yaml
+
+# Receive Kubernetes Audit logs via HTTP server
+input:
+  kubernetesAudit:
+    module: input-kubernetes-audit
+    # server listens to a port 
+    port: 9091
+    # dynamic index setting by posting Audit logs to /indexName/ URL 
+    useIndexFromUrlPath: true
+    # number of extra processes to fork as web server workers
+    worker: 0
+    tags:
+      receiver: logagent_kubernetes_audit
+
+output:
+  # view events on console during test setups
+  stdout: yaml
+  # ship Audit logs to Sematext Cloud
+  elasticsearch:
+    module: elasticsearch
+    url: https://logsene-receiver.sematext.com
+    index: YOUR_LOGS_TOKEN
+
+```
+
+Start Logagent
+
+```
+logagent --config kubernetes-audit.yml
+```
+
+
+Note: You can use the command line argument `--k8sAudit portNumber` to activate the plugin via the `logagent` command. The following command listens on TCP port 9091 for Kubernetes logs and dumps the logs in YAML format to the console.
+
+```
+logagent --k8sAudit 9091 --yaml
+```
+

docs/logagent/input-kubernetes-events.md

@@ -0,0 +1,57 @@
+title: Logagent input plugin for Kubernetes Events
+description: Logagent features modular logging architecture framework where each input or output module is implemented as a plugin. Logagent collects Kubernetes event logs.  
+
+## Input Plugin: Kubernetes Event Collection
+
+Input plugin to collect Kubernetes events via API.
+
+Features:
+
+- parse bulk messages
+
+Applications:
+
+- centralize Kubernetes events
+- index Kubernetes events in Elasticsearch or Sematext Cloud
+- create alerts on Kubernetes events
+- create analytics and monitor event logs 
+
+Requirements: 
+
+- A working `kubectl` config
+- Or run Loagent in a pod, role bindings to access k8s API pods, events, and namespaces 
+
+### Configuration
+
+```yaml
+
+# Receive Kubernetes events via HTTP server
+input:
+   kubernetesEvents:
+    module: input-kubernetes-events
+
+output:
+    # view events on console during test setups
+    stdout: yaml
+    # ship events to Sematext Cloud
+    elasticsearch:
+      module: elasticsearch
+      url: https://logsene-receiver.sematext.com
+      index: YOUR_LOGS_TOKEN
+
+
+```
+
+Start Logagent
+
+```
+logagent --config kubernetes-events.yml
+```
+
+
+Note, you can use the command line argument `--k8sEvents` to activate the plugin via logagent command. The following command would listen to Kubernetes events on TCP port 9091 and dumps the events in YAML format to console. 
+
+```
+logagent --k8sEvents --yaml
+```
+

docs/logagent/input-plugin-dockerlogs.md

@@ -10,7 +10,7 @@ Features:
 - collect all container logs
 - discovers new containers
 - all logs are tagged with container id/name and image name
-- disable container logs by setting LOGSENE_ENBALED=false label/env variable
+- disable log collection for a container by setting LOGS_ENBALED=false label/env variable
 - enrich logs with with labels for later use in docker-enrichment plugin 
 
 Applications:

docs/logagent/installation-docker.md

@@ -497,6 +497,8 @@ The Kubernetes pod annotations `sematext.com/logs-token=<YOUR_LOGS_TOKEN>`, `sem
 
 __Example:__ 
 The following command will start Nginx webserver and logs for this container will be shipped to the related Logs App. 
+Routing logs from different containers to separate Sematext Cloud Logs Apps can be configured via docker labels (or environment variables e.g. on Kubernetes). Simply tag a container with the label (or environment variable) ```LOGSENE_TOKEN=YOUR_LOGSENE_TOKEN```.
+Logagent inspects the containers for this label and ships the logs to the specified Logs App.
 
 ```
 docker run --label LOGS_TOKEN=REPLACE_WITH_YOUR_LOGS_TOKEN -p 80:80 nginx

docs/logagent/output-filter-removefields.md

@@ -35,6 +35,10 @@ outputFilter:
     fields:
       - user
       - client_ip
+      # json-path expressions are supported for nested fields
+      # See: https://jsonpath.com/ online evaluator
+      # - /request/header
+      # - ../body
 
 ```
 

docs/logagent/output-plugin-http.md

@@ -0,0 +1,33 @@
+title: Ship logs via HTTP or HTTPS
+description: Sends log messages as HTTP or HTTPS post request
+
+## Output Plugin: HTTP
+
+Post logs to web services via `HTTP` or `HTTPS`. 
+
+Supported formats: 
+- ldjson - line delimited json
+
+### Configuration
+
+```
+ output:
+    module: output-http
+    url: http://localhost:8080/events
+    format: ldjson
+    # maximum number of events per request
+    # 1 - each event creates a separate http request
+    # >1 - multiple events in each http request
+    maxBufferSize: 1
+    # flush interval in seconds
+    flushInterval: 5
+    filter:
+      field: logSource
+      match: sensor.*
+ ```
+
+Start Logagent
+
+```
+logagent --config http.yaml
+```

docs/logagent/plugins.md

@@ -8,6 +8,8 @@ Logagent features a modular architecture. Each input or output module is impleme
 | [stdin (default)](input-plugin-stdin)                       | input   | Reads from standard input                                                          |
 | [files](input-plugin-files)           | input   | Watching and tailing files                                                                               |
 | [docker-logs](input-plugin-dockerlogs)           | input   | Collection of Docker container logs                                                           |
+| [input-kubernetes-events](input-kubernetes-events)           | input   | Collection of Kubernetes events                                                         |
+| [input-kubernetes-audit](input-kubernetes-audit)           | input   | Receive Kubernetes audit logs via http / webhook                                    |
 | [logagent-input-windows-events](https://www.npmjs.com/package/logagent-input-windows-events) | input  | Collect Windows Events. Available as separate npm package |
 | [logagent-input-elasticsearch-stats](https://www.npmjs.com/package/logagent-input-elasticsearch-stats) | input | Monitoring of Elasticsearch metrics. Available as separate npm package |
 | [syslog](input-plugin-syslog.md)      | input | Receive Syslog messages via UDP |
@@ -29,8 +31,8 @@ Logagent features a modular architecture. Each input or output module is impleme
 | [logagent-apple-location](logagent-apple-location)         | input          | Tracking of GPS positions from Apple devices via "find-my-iphone" API |
 | [logagent-novasds](logagent-novasds)         | input          | Read PM10 and PM2.5 values from Nova SDS011 dust sensor (USB to serial interface) |
 | [grep](input-filter-grep) | Processor / input filter  | Filters text with regular expressions before parsing                                                     |
+| [input-filter-k8s-containerd](input-filter-containerd.md) | Processor / input filter  | Parsing cri-o log format and add Kubernetes context to container logs  |
 | [sql](output-filter-sql)  | Processor / output filter | Transforms and aggregates parsed messages with SQL statements                                            |
-| [access-watch](output-filter-accesswatch) | Processor / output filter | Enriches web server logs with robot detection and traffic intelligence                                   |
 | [aes-encrypt-fields](output-filter-aesencryptfields) | Processor / output filter | Encrypt field values with AES before any output happens |
 | [hash-fields](output-filter-hashfields) | Processor / output filter | Hashing of field values before any output happens |
 | [ip-truncate-fields](output-filter-iptruncatefields) | Processor / output filter | Replaces the last block of IPv4 and IPv6 address fields with "0" to anonymize IP addresses |
@@ -48,20 +50,29 @@ Logagent features a modular architecture. Each input or output module is impleme
 | [output-files](output-plugin-files)     | output                    | Stores parsed messages files. Log rotation and dynamic file name generation are supported.                                                                  |
 | [output-clickhouse](output-plugin-clickhouse)       | output                   | Sends parsed messages to Yandex ClickHouse DB |
 | [logagent-output-kafka](output-plugin-kafka)       | output                   | Sends parsed messages to Apache Kafka topics. 3rd party module. 3rd party module.                                                             |
+| [output-http](output-plugin-clickhouse)       | output                   | Sends parsed messages via HTTP or HTTPS |
 | [slack-webhook](output-plugin-slack)      | output                    | Sends parsed messages to Slack chat. Should be combined with SQL filter plugin or filter function to define alert criterias. |
 | [@sematext/logagent-nodejs-monitor](https://www.npmjs.com/package/@sematext/logagent-nodejs-monitor) | other | Monitors server and  nodejs metrics of the Logagent process using [spm-agent-nodejs](https://www.npmjs.com/package/spm-agent-nodejs) |
 
+## Find plugins on npm 
+
+Developers of 3rd party plugins publish logagent plugins in the [npm registry](https://npmjs.org). 
+Simply search for [logagent](https://www.npmjs.com/search?q=logagent) to discover more plugins. 
 
 ## For Developers: How Logagent plugins work 
 
-- Logagent checks the configuration file for properties with a "module" key for the nodejs module name. External plugins need to be installed via npm. 
+- Logagent checks the configuration file for properties with a "module" key for the nodejs module name. 
+  External plugins need to be installed via npm. 
 - Plugins are initialized with the Logagent configuration (from command line arguments + configuration file) and the event emitter for Logagent. Plugins should provide a start and stop method.
 - Input plugins read data from a data source and emit events to the Logagent event emitter.
-  These events have the identifier "data.raw" and 2 parameters: 
-  - data - data read from a data source 
+  These events have the identifier `data.raw` and 2 parameters: 
+  - data - a string containing a text line, read from a data source 
   - context - an object with meta data e.g. {sourceName: '/var/log/httpd/access.log'}
     The "context" helps other plugins to process the data correctly, e.g. to handle multiple open files. 
-- Output plugins listen to "data.parsed" events and store or forward the data to the target. 
+  In some cases, input plugins create strcutured data, and it makes no sense to process the data with text bases input-filters and Logagent parser. Input plugins can emit a `data.object` event, and only output-filters and output plugins will process such events with the following parameters: 
+  - data - a JavaScript object e.g. `{message: 'hello', severity: 'info'}`
+  - context - an object with meta data e.g. {sourceName: '/var/log/httpd/access.log'}
+- Output plugins listen to `data.parsed` events and store or forward the data to the target. 
 
 ### Examples 
 
@@ -175,6 +186,7 @@ __Example Output Plugin (stdout)__
 'use strict'
 var prettyjson = require('prettyjson')
 var safeStringify = require('fast-safe-stringify')
+
 function OutputStdout (config, eventEmitter) {
   this.config = config
   this.eventEmitter = eventEmitter

mkdocs.yml

@@ -201,6 +201,7 @@ pages:
           - Input Filter:
             - Grep input filter: logagent/input-filter-grep.md
             - Grok input filter: logagent/input-filter-grok.md
+            - Kubernetes cri-o logs: logagent/input-filter-containerd.md
           - Input Plugins:
             - Standard input: logagent/input-plugin-stdin.md
             - File input: logagent/input-plugin-files.md
@@ -224,8 +225,9 @@ pages:
             - Apple device location: logagent/logagent-apple-location.md
             - Nova dust sensor: logagent/logagent-novasds.md
             - Cloud Foundry input: logagent/input-plugin-cloudfoundry.md
+            - Kubernetes Events: logagent/input-kubernetes-events.md
+            - Kubernetes Audit Logs: logagent/input-kubernetes-audit.md
           - Output Filter:
-            - Access Watch output filter: logagent/output-filter-accesswatch.md
             - SQL output filter: logagent/output-filter-sql.md
             - AES encrypt fields: logagent/output-filter-aesencryptfields.md
             - Hash fields: logagent/output-filter-hashfields.md
@@ -248,6 +250,7 @@ pages:
             - ClickHouse DB: logagent/output-plugin-clickhouse.md
             - Slack Webhook: logagent/output-plugin-slack.md
             - Prometheus Alertmanager: logagent/output-plugin-prometheus-alertmanager.md
+            - HTTP post: logagent/output-plugin-http.md
         - How-To:
             - Nginx Ingress Log Shipping: logagent/how-to-nginx-ingress-log-shipping.md
             - GDPR Personal Data In Web Server Logs: logagent/how-to-gdpr_web_logs.md