Merge branch 'master' of github.com:fluent/fluent-bit-docs

Author: edsiper

SUMMARY.md

@@ -48,6 +48,7 @@
   * [Collectd](input/collectd.md)
   * [CPU Usage](input/cpu.md)
   * [Disk Usage](input/disk.md)
+  * [Docker](input/docker.md)
   * [Dummy](input/dummy.md)
   * [Exec](input/exec.md)
   * [Forward](input/forward.md)
@@ -61,6 +62,7 @@
   * [Random](input/random.md)
   * [Serial Interface](input/serial.md)
   * [Standard Input](input/stdin.md)
+  * [StatsD Input](input/statsd.md)
   * [Syslog](input/syslog.md)
   * [Systemd](input/systemd.md)
   * [Tail](input/tail.md)
@@ -74,6 +76,7 @@
   * [Logfmt Parser](parser/logfmt.md)
   * [Decoders](parser/decoder.md)
 * [Filter Plugins](filter/README.md)
+  * [AWS](filter/aws.md)
   * [Grep](filter/grep.md)
   * [Kubernetes](filter/kubernetes.md)
   * [Lua](filter/lua.md)

configuration/variables.md

@@ -1,18 +1,18 @@
 # Configuration Variables
 
-Fluent Bit support the usage of environment variables in any value associated to a key when using a configuration file.
+Fluent Bit supports the usage of environment variables in any value associated to a key when using a configuration file.
 
 The variables are case sensitive and can be used in the following format:
 
 ```text
 ${MY_VARIABLE}
 ```
 
-When Fluent Bit starts, the configuration reader will detect any request for ${MY\_VARIABLE} and will try to resolve it value.
+When Fluent Bit starts, the configuration reader will detect any request for `${MY_VARIABLE}` and will try to resolve its value.
 
 ## Example
 
-Create the following configuration file \(_fluent-bit.conf_\):
+Create the following configuration file (`fluent-bit.conf`):
 
 ```text
 [SERVICE]
@@ -35,7 +35,7 @@ Open a terminal and set the environment variable:
 $ export MY_OUTPUT=stdout
 ```
 
-> The above command set the 'stdout' value to the variable MY\_OUTPUT.
+> The above command set the 'stdout' value to the variable `MY_OUTPUT`.
 
 Run Fluent Bit with the recently created configuration file:
 

filter/README.md

@@ -4,6 +4,7 @@ The _filter plugins_ allows to **alter** the incoming data generated by the _inp
 
 | name | title | description |
 | :--- | :--- | :--- |
+| [aws](aws.md) | AWS | Enrich logs with AWS Metadata. |
 | [grep](grep.md) | Grep | Match or exclude specific records by patterns. |
 | [kubernetes](kubernetes.md) | Kubernetes | Enrich logs with Kubernetes Metadata. |
 | [lua](lua.md) | Lua | Filter records using Lua Scripts. |
@@ -15,4 +16,3 @@ The _filter plugins_ allows to **alter** the incoming data generated by the _inp
 | [modify](modify.md) | Modify | Modifications to record. |
 
 In order to let a Filter be applied over some data, the **Match** rule must exists and it must match the Tag for the incoming data.
-

filter/aws.md

@@ -0,0 +1,42 @@
+# AWS
+
+The _AWS Filter_ Enriches logs with AWS Metadata. Currently the plugin adds the EC2 instance ID and availability zone to log records. To use this plugin, you must be running in EC2 and have the [instance metadata service enabled](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html).
+
+## Configuration Parameters
+
+The plugin supports the following configuration parameters:
+
+| Key | Value Format | Description |
+| :--- | :--- | :--- |
+| imds_version | VERSION | Specify which version of the instance metadata service to use. Valid values are 'v1' or 'v2'; 'v2' is the default. |
+
+Note: *If you run Fluent Bit in a container, you may have to use instance metadata v1.* The plugin behaves the same regardless of which version is used.
+
+## Usage
+
+### Command Line
+
+```
+$ bin/fluent-bit -i dummy -F aws -m '*' -o stdout
+
+[2020/01/17 07:57:17] [ info] [engine] started (pid=32744)
+[0] dummy.0: [1579247838.000171227, {"message"=>"dummy", "az"=>"us-west-2b", "ec2_instance_id"=>"i-06bc83dbc2ac2fdf8"}]
+[1] dummy.0: [1579247839.000125097, {"message"=>"dummy", "az"=>"us-west-2b", "ec2_instance_id"=>"i-06bc87dbc2ac3fdf8"}]
+```
+
+### Configuration File
+
+```
+[INPUT]
+    Name dummy
+    Tag dummy
+
+[FILTER]
+    Name aws
+    Match *
+    imds_version v1
+
+[OUTPUT]
+    Name stdout
+    Match *
+```

input/README.md

@@ -7,6 +7,7 @@ The _input plugins_ defines the source from where [Fluent Bit](http://fluentbit.
 | [collectd](collectd.md) | Collectd | Listen for UDP packets from Collectd. |
 | [cpu](cpu.md) | CPU Usage | measure total CPU usage of the system. |
 | [disk](disk.md) | Disk Usage | measure Disk I/Os. |
+| [docker](docker.md) | Docker | collect Docker container metrics. |
 | [dummy](dummy.md) | Dummy | generate dummy event. |
 | [exec](exec.md) | Exec | executes external program and collects event logs. |
 | [forward](forward.md) | Forward | Fluentd forward protocol. |
@@ -20,6 +21,7 @@ The _input plugins_ defines the source from where [Fluent Bit](http://fluentbit.
 | [random](random.md) | Random | Generate Random samples. |
 | [serial](serial.md) | Serial Interface | read data information from the serial interface. |
 | [stdin](stdin.md) | Standard Input | read data from the standard input. |
+| [statsd](statsd.md) | StatsD | receive metrics in StatsD protocol. |
 | [syslog](syslog.md) | Syslog | read syslog messages from a Unix socket. |
 | [systemd](systemd.md) | Systemd | read logs from Systemd/Journald. |
 | [tail](tail.md) | Tail | Tail log files |

input/collectd.md

@@ -5,15 +5,15 @@ The **collectd** input plugin allows you to receive datagrams from collectd.
 Content:
 
 * [Configuration Parameters](collectd.md#config)
-* [Configuration Parameters](collectd.md#config_example)
+* [Configuration Examples](collectd.md#config_example)
 
 ## Configuration Parameters {#config}
 
 The plugin supports the following configuration parameters:
 
 | Key      | Description                     | Default                      |
 | :------- | :------------------------------ | :--------------------------- |
-| Address  | Set the address to listen to    | 0.0.0.0                      |
+| Listen   | Set the address to listen to    | 0.0.0.0                      |
 | Port     | Set the port to listen to       | 25826                        |
 | TypesDB  | Set the data specification file | /usr/share/collectd/types.db |
 
@@ -24,7 +24,7 @@ Here is a basic configuration example.
 ```python
 [INPUT]
     Name         collectd
-    Address      0.0.0.0
+    Listen       0.0.0.0
     Port         25826
     TypesDB      /usr/share/collectd/types.db,/etc/collectd/custom.db
 

input/docker.md

@@ -0,0 +1,43 @@
+# Docker
+
+The **docker** input plugin allows you to collect Docker container metrics
+like memory usage and CPU consumption.
+
+Content:
+
+* [Configuration Parameters](docker.md#config)
+* [Configuration Parameters](docker.md#config_example)
+
+## Configuration Parameters {#config}
+
+The plugin supports the following configuration parameters:
+
+| Key           | Description                                     | Default  |
+| :------------ | :---------------------------------------------- | :--------|
+| Interval\_Sec | Polling interval in seconds                     | 1        |
+| Include       | A space-separated list of containers to include |          |
+| Exclude       | A space-separated list of containers to exclude |          |
+
+If you set neither `Include` nor `Exclude`, the plugin will try to get
+metrics from _all_ the running containers.
+
+## Configuration Examples {#config_example}
+
+Here is an example configuration that collects metrics from two docker
+instances (`6bab19c3a0f9` and `14159be4ca2c`).
+
+```python
+[INPUT]
+    Name         docker
+    Include      6bab19c3a0f9 14159be4ca2c
+
+[OUTPUT]
+    Name   stdout
+    Match  *
+```
+
+This configuration will produce records like below.
+
+```
+[1] docker.0: [1571994772.00555745, {"id"=>"6bab19c3a0f9", "name"=>"postgresql", "cpu_used"=>172102435, "mem_used"=>5693400, "mem_limit"=>4294963200}]
+```

input/exec.md

@@ -12,6 +12,7 @@ The plugin supports the following configuration parameters:
 | Parser | Specify the name of a parser to interpret the entry as a structured message. |
 | Interval\_Sec | Polling interval \(seconds\). |
 | Interval\_NSec | Polling interval \(nanosecond\). |
+| Buf\_Size | Size of the buffer (check [unit sizes](https://docs.fluentbit.io/manual/configuration/unit_sizes) for allowed values) |
 
 ## Getting Started
 
@@ -47,6 +48,7 @@ In your main configuration file append the following _Input_ & _Output_ sections
     Command       ls /var/log
     Interval_Sec  1
     Interval_NSec 0
+    Buf_Size      8mb
 
 [OUTPUT]
     Name   stdout

input/statsd.md

@@ -0,0 +1,42 @@
+# StatsD
+
+The **statsd** input plugin allows you to receive metrics via StatsD protocol.
+
+Content:
+
+* [Configuration Parameters](statsd.md#config)
+* [Configuration Examples](statsd.md#config_example)
+
+## Configuration Parameters {#config}
+
+The plugin supports the following configuration parameters:
+
+| Key      | Description                              | Default |
+| :------- | :--------------------------------------- | :------ |
+| Listen   | Listener network interface.              | 0.0.0.0 |
+| Port     | UDP port where listening for connections | 8125    |
+
+## Configuration Examples {#config_example}
+
+Here is a configuration example.
+
+```python
+[INPUT]
+    Name   statsd
+    Listen 0.0.0.0
+    Port   8125
+
+[OUTPUT]
+    Name   stdout
+    Match  *
+```
+
+Now you can input metrics through the UDP port as follows.
+
+  $ echo "click:10|c|@0.1" | nc -q0 -u 127.0.0.1 8125
+  $ echo "active:99|g" | nc -q0 -u 127.0.0.1 8125
+
+Fluent Bit will produce the following records:
+
+    [0] statsd.0: [1574905088.971380537, {"type"=>"counter", "bucket"=>"click", "value"=>10.000000, "sample_rate"=>0.100000}]
+    [0] statsd.0: [1574905141.863344517, {"type"=>"gauge", "bucket"=>"active", "value"=>99.000000, "incremental"=>0}]

input/syslog.md

@@ -14,8 +14,8 @@ The plugin supports the following configuration parameters:
 | Path | If _Mode_ is set to _unix\_tcp_ or _unix\_udp_, set the absolute path to the Unix socket file. |  |
 | Unix_Perm | If _Mode_ is set to _unix\_tcp_ or _unix\_udp_, set the permission of the Unix socket file. | 0644 |
 | Parser | Specify an alternative parser for the message. By default, the plugin uses the parser _syslog-rfc3164_. If your syslog messages have fractional seconds set this Parser value to _syslog-rfc5424_ instead. |  |
-| Buffer\_Chunk\_Size | By default the buffer to store the incoming Syslog messages, do not allocate the maximum memory allowed, instead it allocate memory when is required. The rounds of allocations are set by _Buffer\_Chunk\_Size_ in KB. If not set, _Buffer\_Chunk\_Size_ is equal to 32 \(32KB\). Read considerations below when using _udp_ or _unix\_udp_ mode. |  |
-| Buffer\_Max_Size | Specify the maximum buffer size in KB to receive a Syslog message. If not set, the default size will be the value of _Buffer\_Chunk\_Size_. |  |
+| Buffer\_Chunk\_Size | By default the buffer to store the incoming Syslog messages, do not allocate the maximum memory allowed, instead it allocate memory when is required. The rounds of allocations are set by _Buffer\_Chunk\_Size_. If not set, _Buffer\_Chunk\_Size_ is equal to 32000 bytes \(32KB\). Read considerations below when using _udp_ or _unix\_udp_ mode. |  |
+| Buffer\_Max_Size | Specify the maximum buffer size to receive a Syslog message. If not set, the default size will be the value of _Buffer\_Chunk\_Size_. |  |
 
 ### Considerations
 
@@ -50,8 +50,8 @@ In your main configuration file append the following _Input_ & _Output_ sections
 [INPUT]
     Name                syslog
     Path                /tmp/in_syslog
-    Buffer_Chunk_Size   32
-    Buffer_Max_Size     64
+    Buffer_Chunk_Size   32000
+    Buffer_Max_Size     64000
 
 [OUTPUT]
     Name   stdout