pipeline: input: migrate docs

Signed-off-by: Eduardo Silva <eduardo@treasure-data.com>

Author: edsiper

SUMMARY.md

@@ -62,7 +62,7 @@
 * [Inputs](pipeline/inputs/README.md)
   * [Collectd](pipeline/inputs/collectd.md)
   * [CPU Metrics](pipeline/inputs/cpu-metrics.md)
-  * [Disk I/O Metrics](pipeline/inputs/disk-i-o-metrics.md)
+  * [Disk I/O Metrics](pipeline/inputs/disk-io-metrics.md)
   * [Dummy](pipeline/inputs/dummy.md)
   * [Exec](pipeline/inputs/exec.md)
   * [Forward](pipeline/inputs/forward.md)
@@ -71,7 +71,7 @@
   * [Kernel Logs](pipeline/inputs/kernel-logs.md)
   * [Memory Metrics](pipeline/inputs/memory-metrics.md)
   * [MQTT](pipeline/inputs/mqtt.md)
-  * [Network I/O Metrics](pipeline/inputs/network-i-o-metrics.md)
+  * [Network I/O Metrics](pipeline/inputs/network-io-metrics.md)
   * [Process](pipeline/inputs/process.md)
   * [Random](pipeline/inputs/random.md)
   * [Serial Interface](pipeline/inputs/serial-interface.md)

pipeline/inputs/collectd.md

@@ -1,6 +1,40 @@
----
-description: example...
----
-
 # Collectd
 
+The **collectd** input plugin allows you to receive datagrams from collectd.
+
+Content:
+
+* [Configuration Parameters](collectd.md#config)
+* [Configuration Examples](collectd.md#config_example)
+
+## Configuration Parameters {#config}
+
+The plugin supports the following configuration parameters:
+
+| Key      | Description                     | Default                      |
+| :------- | :------------------------------ | :--------------------------- |
+| Listen   | Set the address to listen to    | 0.0.0.0                      |
+| Port     | Set the port to listen to       | 25826                        |
+| TypesDB  | Set the data specification file | /usr/share/collectd/types.db |
+
+## Configuration Examples {#config_example}
+
+Here is a basic configuration example.
+
+```python
+[INPUT]
+    Name         collectd
+    Listen       0.0.0.0
+    Port         25826
+    TypesDB      /usr/share/collectd/types.db,/etc/collectd/custom.db
+
+[OUTPUT]
+    Name   stdout
+    Match  *
+```
+
+With this configuration, fluent-bit listens to `0.0.0.0:25826`, and
+outputs incoming datagram packets to stdout.
+
+You must set the same types.db files that your collectd server uses.
+Otherwise, fluent-bit may not be able to interpret the payload properly.

pipeline/inputs/cpu-metrics.md

@@ -1,2 +1,64 @@
-# CPU Metrics
+# CPU Usage
+
+The **cpu** input plugin, measures the CPU usage of a process or the whole system by default (considering per CPU core). It reports values in percentage unit for every interval of time set. At the moment this plugin is only available for Linux.
+
+The following tables describes the information generated by the plugin. The keys below represent the data used by the overall system, all values associated to the keys are in a percentage unit \(0 to 100%\):
+
+| key | description |
+| :--- | :--- |
+| cpu\_p | CPU usage of the overall system, this value is the summatory of time spent on user and kernel space. The result takes in consideration the numbers of CPU cores in the system. |
+| user\_p | CPU usage in User mode, for short it means the CPU usage by user space programs. The result of this value takes in consideration the numbers of CPU cores in the system. |
+| system\_p | CPU usage in Kernel mode, for short it means the CPU usage by the Kernel. The result of this value takes in consideration the numbers of CPU cores in the system. |
+
+In addition to the keys reported in the above table, a similar content is created **per** CPU core. The cores are listed from _0_ to _N_ as the Kernel reports:
+
+| key | description |
+| :--- | :--- |
+| cpu**N**.p\_cpu | Represents the total CPU usage by core **N**. |
+| cpu**N**.p\_user | Total CPU spent in user mode or user space programs associated to this core. |
+| cpu**N**.p\_system | Total CPU spent in system or kernel mode associated to this core. |
+
+## Configuration Parameters
+
+The plugin supports the following configuration parameters:
+
+| Key | Description | Default |
+| :--- | :--- | ---- |
+| Interval\_Sec | Polling interval in seconds | 1 |
+| Interval\_NSec | Polling interval in nanoseconds | 0 |
+| PID | Specify the ID (PID) of a running process in the system. By default the plugin monitors the whole system but if this option is set, it will only monitor the given process ID. |  |
+
+## Getting Started
+
+In order to get the statistics of the CPU usage of your system, you can run the plugin from the command line or through the configuration file:
+
+### Command Line
+
+```bash
+$ build/bin/fluent-bit -i cpu -t my_cpu -o stdout -m '*'
+Fluent-Bit v1.x.x
+Copyright (C) Treasure Data
+
+[2019/09/02 10:46:29] [ info] starting engine
+[0] [1452185189, {"cpu_p"=>7.00, "user_p"=>5.00, "system_p"=>2.00, "cpu0.p_cpu"=>10.00, "cpu0.p_user"=>8.00, "cpu0.p_system"=>2.00, "cpu1.p_cpu"=>6.00, "cpu1.p_user"=>4.00, "cpu1.p_system"=>2.00}]
+[1] [1452185190, {"cpu_p"=>6.50, "user_p"=>5.00, "system_p"=>1.50, "cpu0.p_cpu"=>6.00, "cpu0.p_user"=>5.00, "cpu0.p_system"=>1.00, "cpu1.p_cpu"=>7.00, "cpu1.p_user"=>5.00, "cpu1.p_system"=>2.00}]
+[2] [1452185191, {"cpu_p"=>7.50, "user_p"=>5.00, "system_p"=>2.50, "cpu0.p_cpu"=>7.00, "cpu0.p_user"=>3.00, "cpu0.p_system"=>4.00, "cpu1.p_cpu"=>6.00, "cpu1.p_user"=>6.00, "cpu1.p_system"=>0.00}]
+[3] [1452185192, {"cpu_p"=>4.50, "user_p"=>3.50, "system_p"=>1.00, "cpu0.p_cpu"=>6.00, "cpu0.p_user"=>5.00, "cpu0.p_system"=>1.00, "cpu1.p_cpu"=>5.00, "cpu1.p_user"=>3.00, "cpu1.p_system"=>2.00}]
+```
+
+As described above, the CPU input plugin gathers the overall usage every one second and flushed the information to the output on the fifth second. On this example we used the **stdout** plugin to demonstrate the output records. In a real use-case you may want to flush this information to some central aggregator such as [Fluentd](http://fluentd.org) or [Elasticsearch](http://elastic.co).
+
+### Configuration File
+
+In your main configuration file append the following _Input_ & _Output_ sections:
+
+```python
+[INPUT]
+    Name cpu
+    Tag  my_cpu
+
+[OUTPUT]
+    Name  stdout
+    Match *
+```
 

pipeline/inputs/disk-i-o-metrics.md

@@ -0,0 +1,51 @@
+# Disk Throughput
+
+The **disk** input plugin, gathers the information about the disk throughput of the running system every certain interval of time and reports them.
+
+## Configuration Parameters
+
+The plugin supports the following configuration parameters:
+
+| Key | Description |
+| :--- | :--- |
+| Interval\_Sec | Polling interval \(seconds\).  default: 1 |
+| Interval\_NSec | Polling interval \(nanosecond\). default: 0 |
+| Dev\_Name | Device name to limit the target. \(e.g. sda\). If not set, _in\_disk_ gathers information from all of disks and partitions. |
+
+## Getting Started
+
+In order to get disk usage from your system, you can run the plugin from the command line or through the configuration file:
+
+### Command Line
+
+```bash
+$ fluent-bit -i disk -o stdout
+Fluent-Bit v0.11.0
+Copyright (C) Treasure Data
+
+[2017/01/28 16:58:16] [ info] [engine] started
+[0] disk.0: [1485590297, {"read_size"=>0, "write_size"=>0}]
+[1] disk.0: [1485590298, {"read_size"=>0, "write_size"=>0}]
+[2] disk.0: [1485590299, {"read_size"=>0, "write_size"=>0}]
+[3] disk.0: [1485590300, {"read_size"=>0, "write_size"=>11997184}]
+```
+
+### Configuration File
+
+In your main configuration file append the following _Input_ & _Output_ sections:
+
+```python
+[INPUT]
+    Name          disk
+    Tag           disk
+    Interval_Sec  1
+    Interval_NSec 0
+[OUTPUT]
+    Name   stdout
+    Match  *
+```
+
+Note: Total interval \(sec\) = Interval\_Sec + \(Interval\_Nsec / 1000000000\).
+
+e.g. 1.5s = 1s + 500000000ns
+

pipeline/inputs/disk-io-metrics.md

@@ -1,2 +1,45 @@
 # Dummy
 
+The **dummy** input plugin, generates dummy events. It is useful for testing, debugging, benchmarking and getting started with Fluent Bit.
+
+## Configuration Parameters
+
+The plugin supports the following configuration parameters:
+
+## Getting Started
+
+You can run the plugin from the command line or through the configuration file:
+
+| Key | Description |
+| :--- | :--- |
+| Dummy | Dummy JSON record. Default: `{"message":"dummy"}` |
+| Rate | Events number generated per second. Default: 1 |
+
+### Command Line
+
+```bash
+$ fluent-bit -i dummy -o stdout
+Fluent-Bit v0.12.0
+Copyright (C) Treasure Data
+
+[2017/07/06 21:55:29] [ info] [engine] started
+[0] dummy.0: [1499345730.015265366, {"message"=>"dummy"}]
+[1] dummy.0: [1499345731.002371371, {"message"=>"dummy"}]
+[2] dummy.0: [1499345732.000267932, {"message"=>"dummy"}]
+[3] dummy.0: [1499345733.000757746, {"message"=>"dummy"}]
+```
+
+### Configuration File
+
+In your main configuration file append the following _Input_ & _Output_ sections:
+
+```python
+[INPUT]
+    Name   dummy
+    Tag    dummy.log
+
+[OUTPUT]
+    Name   stdout
+    Match  *
+```
+

pipeline/inputs/dummy.md

@@ -1,2 +1,57 @@
 # Exec
 
+The **exec** input plugin, allows to execute external program and collects event logs.
+
+## Configuration Parameters
+
+The plugin supports the following configuration parameters:
+
+| Key | Description |
+| :--- | :--- |
+| Command | The command to execute. |
+| Parser | Specify the name of a parser to interpret the entry as a structured message. |
+| Interval\_Sec | Polling interval \(seconds\). |
+| Interval\_NSec | Polling interval \(nanosecond\). |
+| Buf\_Size | Size of the buffer (check [unit sizes](https://docs.fluentbit.io/manual/configuration/unit_sizes) for allowed values) |
+
+## Getting Started
+
+You can run the plugin from the command line or through the configuration file:
+
+### Command Line
+
+The following example will read events from the output of _ls_.
+
+```bash
+$ fluent-bit -i exec -p 'command=ls /var/log' -o stdout
+Fluent-Bit v0.13.0
+Copyright (C) Treasure Data
+
+[2018/03/21 17:46:49] [ info] [engine] started
+[0] exec.0: [1521622010.013470159, {"exec"=>"ConsoleKit"}]
+[1] exec.0: [1521622010.013490313, {"exec"=>"Xorg.0.log"}]
+[2] exec.0: [1521622010.013492079, {"exec"=>"Xorg.0.log.old"}]
+[3] exec.0: [1521622010.013493443, {"exec"=>"anaconda.ifcfg.log"}]
+[4] exec.0: [1521622010.013494707, {"exec"=>"anaconda.log"}]
+[5] exec.0: [1521622010.013496016, {"exec"=>"anaconda.program.log"}]
+[6] exec.0: [1521622010.013497225, {"exec"=>"anaconda.storage.log"}]
+```
+
+### Configuration File
+
+In your main configuration file append the following _Input_ & _Output_ sections:
+
+```python
+[INPUT]
+    Name          exec
+    Tag           exec_ls
+    Command       ls /var/log
+    Interval_Sec  1
+    Interval_NSec 0
+    Buf_Size      8mb
+
+[OUTPUT]
+    Name   stdout
+    Match  *
+```
+

pipeline/inputs/exec.md

@@ -1,2 +1,72 @@
 # Forward
 
+_Forward_ is the protocol used by [Fluent Bit](http://fluentbit.io) and [Fluentd](http://www.fluentd.org) to route messages between peers. This plugin implements the input service to listen for Forward messages.
+
+## Configuration Parameters
+
+The plugin supports the following configuration parameters:
+
+| Key | Description | Default |
+| :--- | :--- | :--- |
+| Listen | Listener network interface. | 0.0.0.0 |
+| Port | TCP port to listen for incoming connections. | 24224 |
+| Buffer\_Max\_Size | Specify the maximum buffer memory size used to receive a Forward message. The value must be according to the [Unit Size](../configuration/unit_sizes.md) specification. | _Buffer\_Chunk\_Size_ |
+| Buffer\_Chunk\_Size | By default the buffer to store the incoming Forward messages, do not allocate the maximum memory allowed, instead it allocate memory when is required. The rounds of allocations are set by _Buffer\_Chunk\_Size_. The value must be according to the [Unit Size](../configuration/unit_sizes.md) specification. | 32KB |
+
+## Getting Started
+
+In order to receive Forward messages, you can run the plugin from the command line or through the configuration file:
+
+### Command Line
+
+From the command line you can let Fluent Bit listen for _Forward_ messages with the following options:
+
+```bash
+$ fluent-bit -i forward -o stdout
+```
+
+By default the service will listen an all interfaces \(0.0.0.0\) through TCP port 24224, optionally you can change this directly, e.g:
+
+```bash
+$ fluent-bit -i forward://192.168.3.2:9090 -o stdout
+```
+
+In the example the Forward messages will only arrive through network interface under 192.168.3.2 address and TCP Port 9090.
+
+### Configuration File
+
+In your main configuration file append the following _Input_ & _Output_ sections:
+
+```python
+[INPUT]
+    Name              forward
+    Listen            0.0.0.0
+    Port              24224
+    Buffer_Chunk_Size 32KB
+    Buffer_Max_Size   64KB
+
+[OUTPUT]
+    Name   stdout
+    Match  *
+```
+
+## Testing
+
+Once Fluent Bit is running, you can send some messages using the _fluent-cat_ tool \(this tool is provided by [Fluentd](http://www.fluentd.org):
+
+```bash
+$ echo '{"key 1": 123456789, "key 2": "abcdefg"}' | fluent-cat my_tag
+```
+
+In [Fluent Bit](http://fluentbit.io) we should see the following output:
+
+```bash
+$ bin/fluent-bit -i forward -o stdout
+Fluent-Bit v0.9.0
+Copyright (C) Treasure Data
+
+[2016/10/07 21:49:40] [ info] [engine] started
+[2016/10/07 21:49:40] [ info] [in_fw] binding 0.0.0.0:24224
+[0] my_tag: [1475898594, {"key 1"=>123456789, "key 2"=>"abcdefg"}]
+```
+