GitBook: [1.6] 19 pages and one asset modified

Author: edsiper

.gitbook/assets/logo_documentation_1.6.png

@@ -2,9 +2,9 @@
 description: High Performance Logs Processor
 ---
 
-# Fluent Bit v1.5 Documentation
+# Fluent Bit v1.6 Documentation
 
-![](imgs/logo_documentation_1.6.png)
+![](.gitbook/assets/logo_documentation_1.6.png)
 
 [Fluent Bit](http://fluentbit.io) is a Fast and Lightweight Log Processor, Stream Processor and Forwarder for Linux, OSX, Windows and BSD family operating systems. It has been made with a strong focus on performance to allow the collection of events from different sources without complexity.
 

README.md

@@ -1,6 +1,6 @@
 # Table of contents
 
-* [Fluent Bit v1.5 Documentation](README.md)
+* [Fluent Bit v1.6 Documentation](README.md)
 
 ## About
 
@@ -159,3 +159,4 @@
 * [Ingest Records Manually](development/ingest-records-manually.md)
 * [Golang Output Plugins](development/golang-output-plugins.md)
 * [Developer guide for beginners on contributing to Fluent Bit](development/developer-guide.md)
+

SUMMARY.md

@@ -4,27 +4,27 @@ description: A full feature set to access content of your records
 
 # Record Accessor
 
-Fluent Bit works internally with structured records and it can be composed of an unlimited number of keys and values. Values can be anything like a number, string, array, or a map. 
+Fluent Bit works internally with structured records and it can be composed of an unlimited number of keys and values. Values can be anything like a number, string, array, or a map.
 
-Having a way to select a specific part of the record is critical for certain core functionalities or plugins, this feature is called _Record Accessor._ 
+Having a way to select a specific part of the record is critical for certain core functionalities or plugins, this feature is called _Record Accessor._
 
 > consider Record Accessor a simple grammar to specify record content and other miscellaneus values.
 
-### Format
+## Format
 
 A _record accessor_ rule starts with the character `$`. Using the structured content above as an example the following table describes how to access a record:
 
 ```javascript
 {
-	"log": "some message",
-	"stream": "stdout",
-	"labels": {
-		"color": "blue",
-		"unset": null,
-		"project": {
-			"env": "production"
-		}
-	}
+    "log": "some message",
+    "stream": "stdout",
+    "labels": {
+        "color": "blue",
+        "unset": null,
+        "project": {
+            "env": "production"
+        }
+    }
 }
 ```
 
@@ -38,9 +38,9 @@ The following table describe some accessing rules and the expected returned valu
 | $labels\['unset'\] | null |
 | $labels\['undefined'\] |  |
 
-If the accessor key does not exist in the record like the last example `$labels['undefined']` , the operation is simply omitted, no exception will occur.  
+If the accessor key does not exist in the record like the last example `$labels['undefined']` , the operation is simply omitted, no exception will occur.
 
-### Usage Example
+## Usage Example
 
 The feature is enabled on a per plugin basis, not all plugins enable this feature. As an example consider a configuration that aims to filter records using [grep](../../pipeline/filters/grep.md) that only matches where labels have a color blue:
 

administration/configuring-fluent-bit/record-accessor.md

@@ -27,7 +27,7 @@ The following **output** plugins can take advantage of the TLS feature:
 * [BigQuery](../pipeline/outputs/bigquery.md)
 * [Datadog](../pipeline/outputs/datadog.md)
 * [Elasticsearch](../pipeline/outputs/elasticsearch.md)
-* [Forward]()
+* [Forward](security.md)
 * [GELF](../pipeline/outputs/gelf.md)
 * [HTTP](../pipeline/outputs/http.md)
 * [InfluxDB](../pipeline/outputs/influxdb.md)
@@ -93,3 +93,4 @@ Fluent Bit supports [TLS server name indication](https://en.wikipedia.org/wiki/S
     tls.ca_file /etc/certs/fluent.crt
     tls.vhost   fluent.example.com
 ```
+

administration/security.md

@@ -102,18 +102,18 @@ When deploying Fluent Bit to Kubernetes, there are three log files that you need
 
 `C:\k\kubelet.err.log`
 
- * This is the error log file from kubelet daemon running on host.
- * You will need to retain this file for future troubleshooting (to debug deployment failures etc.)
+* This is the error log file from kubelet daemon running on host.
+* You will need to retain this file for future troubleshooting \(to debug deployment failures etc.\)
 
 `C:\var\log\containers\<pod>_<namespace>_<container>-<docker>.log`
 
- * This is the main log file you need to watch. Configure Fluent Bit to follow this file.
- * It is actually a symlink to the Docker log file in `C:\ProgramData\`, with some additional metadata on its file name.
+* This is the main log file you need to watch. Configure Fluent Bit to follow this file.
+* It is actually a symlink to the Docker log file in `C:\ProgramData\`, with some additional metadata on its file name.
 
 `C:\ProgramData\Docker\containers\<docker>\<docker>.log`
 
- * This is the log file produced by Docker.
- * Normally you don't directly read from this file, but you need to make sure that this file is visible from Fluent Bit.
+* This is the log file produced by Docker.
+* Normally you don't directly read from this file, but you need to make sure that this file is visible from Fluent Bit.
 
 Typically, your deployment yaml contains the following volume configuration.
 
@@ -185,17 +185,18 @@ parsers.conf: |
 
 ### Mitigate unstable network on Windows pods
 
-Windows pods often lack working DNS immediately after boot ([#78479](https://github.com/kubernetes/kubernetes/issues/78479)). To mitigate this issue, `filter_kubernetes` provides a built-in mechanism to wait until the network starts up:
+Windows pods often lack working DNS immediately after boot \([\#78479](https://github.com/kubernetes/kubernetes/issues/78479)\). To mitigate this issue, `filter_kubernetes` provides a built-in mechanism to wait until the network starts up:
 
- * `DNS_Retries` - Retries N times until the network start working (6)
- * `DNS_Wait_Time` - Lookup interval between network status checks (30)
+* `DNS_Retries` - Retries N times until the network start working \(6\)
+* `DNS_Wait_Time` - Lookup interval between network status checks \(30\)
 
-By default, Fluent Bit waits for 3 minutes (30 seconds x 6 times). If it's not enough for you, tweak the configuration as follows.
+By default, Fluent Bit waits for 3 minutes \(30 seconds x 6 times\). If it's not enough for you, tweak the configuration as follows.
 
-```
+```text
 [filter]
     Name kubernetes
     ...
     DNS_Retries 10
     DNS_Wait_Time 30
 ```
+

installation/kubernetes.md

@@ -112,7 +112,7 @@ The _input plugins_ provides certain features to gather information from a speci
 | [FLB\_IN\_DISK](../../pipeline/inputs/disk-io-metrics.md) | Enable Disk I/O Metrics input plugin | On |
 | [FLB\_IN\_DOCKER](../docker.md) | Enable Docker metrics input plugin | On |
 | [FLB\_IN\_EXEC](../../pipeline/inputs/exec.md) | Enable Exec input plugin | On |
-| [FLB\_IN\_FORWARD]() | Enable Forward input plugin | On |
+| [FLB\_IN\_FORWARD](build-and-install.md) | Enable Forward input plugin | On |
 | [FLB\_IN\_HEAD](../../pipeline/inputs/head.md) | Enable Head input plugin | On |
 | [FLB\_IN\_HEALTH](../../pipeline/inputs/health.md) | Enable Health input plugin | On |
 | [FLB\_IN\_KMSG](../../pipeline/inputs/kernel-logs.md) | Enable Kernel log input plugin | On |
@@ -182,3 +182,4 @@ The _output plugins_ gives the capacity to flush the information to some externa
 | [FLB\_OUT\_STDOUT](build-and-install.md) | Enable STDOUT output plugin | On |
 | FLB\_OUT\_TCP | Enable TCP/TLS output plugin | On |
 | [FLB\_OUT\_TD](../../pipeline/outputs/treasure-data.md) | Enable [Treasure Data](http://www.treasuredata.com) output plugin | On |
+

installation/sources/build-and-install.md

@@ -11,7 +11,7 @@ The plugin supports the following configuration parameters:
 | imds\_version | Specify which version of the instance metadata service to use. Valid values are 'v1' or 'v2'. | v2 |
 | az | The [availability zone](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html); for example, "us-east-1a". | true |
 | ec2\_instance\_id | The EC2 instance ID. | true |
-| ec2\_instance\_type  | The EC2 instance type. | false |
+| ec2\_instance\_type | The EC2 instance type. | false |
 | private\_ip | The EC2 instance private ip. | false |
 | ami\_id | The EC2 instance image id. | false |
 | account\_id | The account ID for current EC2 instance. | false |
@@ -53,4 +53,5 @@ $ bin/fluent-bit -c /PATH_TO_CONF_FILE/fluent-bit.conf
 [OUTPUT]
     Name stdout
     Match *
-```
+```
+

pipeline/filters/aws-metadata.md

@@ -1,41 +1,44 @@
 # Tensorflow
 
-_Tensorflow Filter_ allows running Machine Learning inference tasks on the records of data coming from
-input plugins or stream processor. This filter uses [Tensorflow Lite](https://www.tensorflow.org/lite/)
-as the inference engine, and **requires Tensorflow Lite shared library to be present during build and at runtime**.
+## Tensorflow
 
-Tensorflow Lite is a lightweight open-source deep learning framework that is used for mobile and IoT applications. Tensorflow Lite only handles inference (not training), therefore, it loads pre-trained models (`.tflite` files) that are converted into Tensorflow Lite format (`FlatBuffer`). You can read more on converting Tensorflow models [here](https://www.tensorflow.org/lite/convert)
+_Tensorflow Filter_ allows running Machine Learning inference tasks on the records of data coming from input plugins or stream processor. This filter uses [Tensorflow Lite](https://www.tensorflow.org/lite/) as the inference engine, and **requires Tensorflow Lite shared library to be present during build and at runtime**.
 
-## Configuration Parameters
+Tensorflow Lite is a lightweight open-source deep learning framework that is used for mobile and IoT applications. Tensorflow Lite only handles inference \(not training\), therefore, it loads pre-trained models \(`.tflite` files\) that are converted into Tensorflow Lite format \(`FlatBuffer`\). You can read more on converting Tensorflow models [here](https://www.tensorflow.org/lite/convert)
+
+### Configuration Parameters
 
 The plugin supports the following configuration parameters:
 
 | Key | Description | Default |
 | :--- | :--- | :--- |
-| input_field | Specify the name of the field in the record to apply inference on. |  |
-| model_file  | Path to the model file (`.tflite`) to be loaded by Tensorflow Lite. | |
-| include_input_fields | Include all input filed in filter's output | True |
-| normalization_value | Divide input values to normalization_value | |
+| input\_field | Specify the name of the field in the record to apply inference on. |  |
+| model\_file | Path to the model file \(`.tflite`\) to be loaded by Tensorflow Lite. |  |
+| include\_input\_fields | Include all input filed in filter's output | True |
+| normalization\_value | Divide input values to normalization\_value |  |
 
-## Creating Tensorflow Lite shared library
+### Creating Tensorflow Lite shared library
 
 Clone [Tensorflow repository](https://github.com/tensorflow/tensorflow), install bazel package manager, and run the following command in order to create the shared library:
+
 ```bash
 $ bazel build -c opt //tensorflow/lite/c:tensorflowlite_c  # see https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/c
 ```
-The script creates the shared library `bazel-bin/tensorflow/lite/c/libtensorflowlite_c.so`. You need to copy the library to a location (such as `/usr/lib`) that can be used by Fluent Bit.
 
-## Building Fluent Bit with Tensorflow filter plugin
+The script creates the shared library `bazel-bin/tensorflow/lite/c/libtensorflowlite_c.so`. You need to copy the library to a location \(such as `/usr/lib`\) that can be used by Fluent Bit.
+
+### Building Fluent Bit with Tensorflow filter plugin
+
+Tensorflow filter plugin is disabled by default. You need to build Fluent Bit with Tensorflow plugin enabled. In addition, it requires access to Tensorflow Lite header files to compile. Therefore, you also need to pass the address of the Tensorflow source code on your machine to the [build script](https://github.com/fluent/fluent-bit#build-from-scratch):
 
-Tensorflow filter plugin is disabled by default. You need to build Fluent Bit with Tensorflow plugin enabled. In addition, it requires access to Tensorflow Lite header files to compile.
-Therefore, you also need to pass the address of the Tensorflow source code on your machine to the [build script](https://github.com/fluent/fluent-bit#build-from-scratch):
 ```bash
 cmake -DFLB_FILTER_TENSORFLOW=On -DTensorflow_DIR=<AddressOfTensorflowSourceCode> ...
 ```
 
-### Command line
+#### Command line
 
 If Tensorflow plugin initializes correctly, it reports successful creation of the interpreter, and prints a summary of model's input/output types and dimensions.
+
 ```bash
 $ bin/fluent-bit -i mqtt -p 'tag=mqtt.data' -F tensorflow -m '*' -p 'input_field=image' -p 'model_file=/home/user/model.tflite' -p 'include_input_fields=false' -p 'normalization_value=255' -o stdout
 [2020/08/04 20:00:00] [ info] Tensorflow Lite interpreter created!
@@ -45,7 +48,7 @@ $ bin/fluent-bit -i mqtt -p 'tag=mqtt.data' -F tensorflow -m '*' -p 'input_field
 [2020/08/04 20:00:00] [ info] [tensorflow] type: FLOAT32  dimensions: {1, 2}
 ```
 
-### Configuration File
+#### Configuration File
 
 ```text
 [SERVICE]
@@ -70,7 +73,8 @@ $ bin/fluent-bit -i mqtt -p 'tag=mqtt.data' -F tensorflow -m '*' -p 'input_field
     Match *
 ```
 
-# Limitations
+## Limitations
 
 1. Currently supports single-input models
 2. Uses Tensorflow 2.3 header files
+

pipeline/filters/tensorflow.md

@@ -10,8 +10,8 @@ The plugin supports the following configuration parameters:
 | :--- | :--- | :--- |
 | Listen | Listener network interface. | 0.0.0.0 |
 | Port | TCP port to listen for incoming connections. | 24224 |
-| Buffer\_Max\_Size | Specify the maximum buffer memory size used to receive a Forward message. The value must be according to the [Unit Size](../administration/configuring-fluent-bit/unit-sizes.md) specification. | _Buffer\_Chunk\_Size_ |
-| Buffer\_Chunk\_Size | By default the buffer to store the incoming Forward messages, do not allocate the maximum memory allowed, instead it allocate memory when is required. The rounds of allocations are set by _Buffer\_Chunk\_Size_. The value must be according to the [Unit Size](../administration/configuring-fluent-bit/unit-sizes.md) specification. | 32KB |
+| Buffer\_Max\_Size | Specify the maximum buffer memory size used to receive a Forward message. The value must be according to the [Unit Size](https://github.com/fluent/fluent-bit-docs/tree/07822c841d38702ae88b58bf3b3c07c9f1e35434/pipeline/administration/configuring-fluent-bit/unit-sizes.md) specification. | _Buffer\_Chunk\_Size_ |
+| Buffer\_Chunk\_Size | By default the buffer to store the incoming Forward messages, do not allocate the maximum memory allowed, instead it allocate memory when is required. The rounds of allocations are set by _Buffer\_Chunk\_Size_. The value must be according to the [Unit Size](https://github.com/fluent/fluent-bit-docs/tree/07822c841d38702ae88b58bf3b3c07c9f1e35434/pipeline/administration/configuring-fluent-bit/unit-sizes.md) specification. | 32KB |
 
 ## Getting Started
 
@@ -69,3 +69,4 @@ Copyright (C) Treasure Data
 [2016/10/07 21:49:40] [ info] [in_fw] binding 0.0.0.0:24224
 [0] my_tag: [1475898594, {"key 1"=>123456789, "key 2"=>"abcdefg"}]
 ```
+

pipeline/inputs/forward.md

@@ -117,3 +117,4 @@ By default SQLite client tool do not format the columns in a human read-way, so
 File rotation is properly handled, including logrotate's _copytruncate_ mode.
 
 Note that the `Path` patterns **cannot** match the rotated files. Otherwise, the rotated file would be read again and lead to duplicate records.
+

pipeline/inputs/tail.md

@@ -23,7 +23,7 @@ We expose different configuration properties. The following table lists all the
 | account\_name | Azure Storage account name. This configuration property is mandatory |  |
 | shared\_key | Specify the Azure Storage Shared Key to authenticate against the service. This configuration property is mandatory. |  |
 | container\_name | Name of the container that will contain the blobs. This configuration property is mandatory |  |
-| blob\_type | Specify the desired blob type. Fluent Bit supports `appendblob` and `blockblob`.  | appendblob |
+| blob\_type | Specify the desired blob type. Fluent Bit supports `appendblob` and `blockblob`. | appendblob |
 | auto\_create\_container | If `container_name` does not exist in the remote service, enabling this option will handle the exception and auto-create the container. | on |
 | path | Optional path to store your blobs. If your blob name is `myblob`, you can specify sub-directories where to store it using path, so setting path to `/logs/kubernetes` will store your blob in `/logs/kubernetes/myblob`. |  |
 | emulator\_mode | If you want to send data to an Azure emulator service like [Azurite](https://github.com/Azure/Azurite), enable this option so the plugin will format the requests to the expected format. | off |