Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion concepts/buffering.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: Performance and Data Safety

# Buffering

When [Fluent Bit](https://fluentbit.io) process data, it uses the system memory \(heap\) as a primary and temporal place to store the record logs before they get delivered, on this private memory area the records are processed.
When [Fluent Bit](https://fluentbit.io) processes data, it uses the system memory \(heap\) as a primary and temporal place to store the record logs before they get delivered, on this private memory area the records are processed.

Buffering refers to the ability to store the records somewhere, and while they are processed and delivered, still be able to store more. Buffering in memory is the fastest mechanism, but there are certain scenarios where the mechanism requires special strategies to deal with [backpressure](../administration/backpressure.md), data safety or reduce memory consumption by the service in constraint environments.

Expand Down
2 changes: 1 addition & 1 deletion concepts/data-pipeline/buffer.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ The `buffer` phase already contains the data in an immutable state, meaning, no
![](../../.gitbook/assets/logging_pipeline_buffer%20%281%29.png)

{% hint style="info" %}
Note that buffered data is not longer a raw text, instead it's in Fluent Bit internal binary representation.
Note that buffered data is not raw text, it's in Fluent Bit's internal binary representation.
{% endhint %}

Fluent Bit offers a buffering mechanism in the file system that acts as a _backup system_ to avoid data loss in case of system failures.
Expand Down
2 changes: 1 addition & 1 deletion concepts/key-concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ Source events can have or not have a structure. A structure defines a set of _ke
At a low level both are just an array of bytes, but the Structured message defines _keys_ and _values_, having a structure helps to implement faster operations on data modifications.

{% hint style="info" %}
Fluent Bit **always** handle every Event message as a structured message. For performance reasons, we use a binary serialization data format called [MessagePack](https://msgpack.org/).
Fluent Bit **always** handles every Event message as a structured message. For performance reasons, we use a binary serialization data format called [MessagePack](https://msgpack.org/).

Consider [MessagePack](https://msgpack.org/) as a binary version of JSON on steroids.
{% endhint %}
Expand Down
7 changes: 4 additions & 3 deletions installation/upgrade-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,16 @@ For more details about changes on each release please refer to the [Official Rel

The migration from v1.4 to v1.5 is pretty straightforward.

- If you enabled ```keepalive``` mode in your configuration, note that this configuration property has been renamed to ```net.keepalive```. Now all Network I/O is keepalive default enabled, to learn more about this and other associated configuration properties read the [Networking Administration](https://docs.fluentbit.io/manual/administration/networking#tcp-keepalive) section.
- If you enabled `keepalive` mode in your configuration, note that this configuration property has been renamed to `net.keepalive`. Now all Network I/O keepalive is enabled by default, to learn more about this and other associated configuration properties read the [Networking Administration](https://docs.fluentbit.io/manual/administration/networking#tcp-keepalive) section.
- If you use the Elasticsearch output plugin, note the default value of `type` [changed from `flb_type` to `_doc`](https://github.com/fluent/fluent-bit/commit/04ed3d8104ca8a2f491453777ae6e38e5377817e#diff-c9ae115d3acaceac5efb949edbb21196). Many versions of Elasticsearch will tolerate this, but ES v5.6 through v6.1 require a type *without* a leading underscore. See the [Elasticsearch output plugin documentation FAQ entry](https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch#faq-underscore) for more.

## Fluent Bit v1.4

If you are migrating from Fluent Bit v1.3, there are not breaking changes. Just new exciting features to enjoy :\)
If you are migrating from Fluent Bit v1.3, there are no breaking changes. Just new exciting features to enjoy :\)

## Fluent Bit v1.3

If you are migrating from Fluent Bit v1.2 to v1.3, there are not breaking changes. If you are upgrading from an older version please review the incremental changes below.
If you are migrating from Fluent Bit v1.2 to v1.3, there are no breaking changes. If you are upgrading from an older version please review the incremental changes below.

## Fluent Bit v1.2

Expand Down
23 changes: 20 additions & 3 deletions pipeline/outputs/elasticsearch.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,9 +122,26 @@ If you see an error message like below, you'll need to fix your configuration to

For details, please read [the official blog post on that issue](https://www.elastic.co/guide/en/elasticsearch/reference/6.7/removal-of-types.html).

### Fluent Bit + Amazon Elasticsearch Service
### Elasticsearch rejects requests saying "Document mapping type name can't start with '\_'" <a id="faq-underscore"></a>

Amazon ElasticSearch Service adds an extra security layer where HTTP requests must be signed with AWS Sigv4. Fluent Bit v1.5 introduced full support for Amazon ElasticSearch Service with IAM Authentication.
Fluent Bit v1.5 changed the default mapping type from `flb_type` to `_doc`, which matches the recommendation from Elasticsearch from version 6.2 forwards \([see commit with rationale](https://github.com/fluent/fluent-bit/commit/04ed3d8104ca8a2f491453777ae6e38e5377817e#diff-c9ae115d3acaceac5efb949edbb21196)\). This doesn't work in Elasticsearch versions 5.6 through 6.1 \([see Elasticsearch discussion and fix](https://discuss.elastic.co/t/cant-use-doc-as-type-despite-it-being-declared-the-preferred-method/113837/9)\). Ensure you set an explicit map (such as `doc` or `flb_type`) in the configuration, as seen on the last line:

```text
[OUTPUT]
Name es
Match *
Host vpc-test-domain-ke7thhzoo7jawsrhmm6mb7ite7y.us-west-2.es.amazonaws.com
Port 443
Index my_index
AWS_Auth On
AWS_Region us-west-2
tls On
Type doc
```

### Fluent Bit + Amazon Elasticsearch Service <a id="#aws-es"></a>

The Amazon ElasticSearch Service adds an extra security layer where HTTP requests must be signed with AWS Sigv4. Fluent Bit v1.5 introduced full support for Amazon ElasticSearch Service with IAM Authentication.

Fluent Bit supports sourcing AWS credentials from any of the standard sources \(for example, an [Amazon EKS IAM Role for a Service Account](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html)\).

Expand All @@ -143,5 +160,5 @@ Example configuration:
tls On
```

Notice that the `Port` is set to `443`, and that `tls` is enabled.
Notice that the `Port` is set to `443`, `tls` is enabled, and `AWS_Region` is set.