diff --git a/concepts/buffering.md b/concepts/buffering.md index fecef2e36..e2e6c7526 100644 --- a/concepts/buffering.md +++ b/concepts/buffering.md @@ -4,7 +4,7 @@ description: Performance and Data Safety # Buffering -When [Fluent Bit](https://fluentbit.io) process data, it uses the system memory \(heap\) as a primary and temporal place to store the record logs before they get delivered, on this private memory area the records are processed. +When [Fluent Bit](https://fluentbit.io) processes data, it uses the system memory \(heap\) as a primary and temporal place to store the record logs before they get delivered, on this private memory area the records are processed. Buffering refers to the ability to store the records somewhere, and while they are processed and delivered, still be able to store more. Buffering in memory is the fastest mechanism, but there are certain scenarios where the mechanism requires special strategies to deal with [backpressure](../administration/backpressure.md), data safety or reduce memory consumption by the service in constraint environments. diff --git a/concepts/data-pipeline/buffer.md b/concepts/data-pipeline/buffer.md index 7d612b7ba..d9b3b5eda 100644 --- a/concepts/data-pipeline/buffer.md +++ b/concepts/data-pipeline/buffer.md @@ -11,7 +11,7 @@ The `buffer` phase already contains the data in an immutable state, meaning, no ![](../../.gitbook/assets/logging_pipeline_buffer%20%281%29.png) {% hint style="info" %} -Note that buffered data is not longer a raw text, instead it's in Fluent Bit internal binary representation. +Note that buffered data is not raw text, it's in Fluent Bit's internal binary representation. {% endhint %} Fluent Bit offers a buffering mechanism in the file system that acts as a _backup system_ to avoid data loss in case of system failures. diff --git a/concepts/key-concepts.md b/concepts/key-concepts.md index 3acdef57e..6d7ee635b 100644 --- a/concepts/key-concepts.md +++ b/concepts/key-concepts.md @@ -103,7 +103,7 @@ Source events can have or not have a structure. A structure defines a set of _ke At a low level both are just an array of bytes, but the Structured message defines _keys_ and _values_, having a structure helps to implement faster operations on data modifications. {% hint style="info" %} -Fluent Bit **always** handle every Event message as a structured message. For performance reasons, we use a binary serialization data format called [MessagePack](https://msgpack.org/). +Fluent Bit **always** handles every Event message as a structured message. For performance reasons, we use a binary serialization data format called [MessagePack](https://msgpack.org/). Consider [MessagePack](https://msgpack.org/) as a binary version of JSON on steroids. {% endhint %} diff --git a/installation/upgrade-notes.md b/installation/upgrade-notes.md index a61ecfab4..b21048ef8 100644 --- a/installation/upgrade-notes.md +++ b/installation/upgrade-notes.md @@ -8,15 +8,16 @@ For more details about changes on each release please refer to the [Official Rel The migration from v1.4 to v1.5 is pretty straightforward. -- If you enabled ```keepalive``` mode in your configuration, note that this configuration property has been renamed to ```net.keepalive```. Now all Network I/O is keepalive default enabled, to learn more about this and other associated configuration properties read the [Networking Administration](https://docs.fluentbit.io/manual/administration/networking#tcp-keepalive) section. +- If you enabled `keepalive` mode in your configuration, note that this configuration property has been renamed to `net.keepalive`. Now all Network I/O keepalive is enabled by default, to learn more about this and other associated configuration properties read the [Networking Administration](https://docs.fluentbit.io/manual/administration/networking#tcp-keepalive) section. +- If you use the Elasticsearch output plugin, note the default value of `type` [changed from `flb_type` to `_doc`](https://github.com/fluent/fluent-bit/commit/04ed3d8104ca8a2f491453777ae6e38e5377817e#diff-c9ae115d3acaceac5efb949edbb21196). Many versions of Elasticsearch will tolerate this, but ES v5.6 through v6.1 require a type *without* a leading underscore. See the [Elasticsearch output plugin documentation FAQ entry](https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch#faq-underscore) for more. ## Fluent Bit v1.4 -If you are migrating from Fluent Bit v1.3, there are not breaking changes. Just new exciting features to enjoy :\) +If you are migrating from Fluent Bit v1.3, there are no breaking changes. Just new exciting features to enjoy :\) ## Fluent Bit v1.3 -If you are migrating from Fluent Bit v1.2 to v1.3, there are not breaking changes. If you are upgrading from an older version please review the incremental changes below. +If you are migrating from Fluent Bit v1.2 to v1.3, there are no breaking changes. If you are upgrading from an older version please review the incremental changes below. ## Fluent Bit v1.2 diff --git a/pipeline/outputs/elasticsearch.md b/pipeline/outputs/elasticsearch.md index ed37e3c5d..903552c59 100644 --- a/pipeline/outputs/elasticsearch.md +++ b/pipeline/outputs/elasticsearch.md @@ -122,9 +122,26 @@ If you see an error message like below, you'll need to fix your configuration to For details, please read [the official blog post on that issue](https://www.elastic.co/guide/en/elasticsearch/reference/6.7/removal-of-types.html). -### Fluent Bit + Amazon Elasticsearch Service +### Elasticsearch rejects requests saying "Document mapping type name can't start with '\_'" -Amazon ElasticSearch Service adds an extra security layer where HTTP requests must be signed with AWS Sigv4. Fluent Bit v1.5 introduced full support for Amazon ElasticSearch Service with IAM Authentication. +Fluent Bit v1.5 changed the default mapping type from `flb_type` to `_doc`, which matches the recommendation from Elasticsearch from version 6.2 forwards \([see commit with rationale](https://github.com/fluent/fluent-bit/commit/04ed3d8104ca8a2f491453777ae6e38e5377817e#diff-c9ae115d3acaceac5efb949edbb21196)\). This doesn't work in Elasticsearch versions 5.6 through 6.1 \([see Elasticsearch discussion and fix](https://discuss.elastic.co/t/cant-use-doc-as-type-despite-it-being-declared-the-preferred-method/113837/9)\). Ensure you set an explicit map (such as `doc` or `flb_type`) in the configuration, as seen on the last line: + +```text +[OUTPUT] + Name es + Match * + Host vpc-test-domain-ke7thhzoo7jawsrhmm6mb7ite7y.us-west-2.es.amazonaws.com + Port 443 + Index my_index + AWS_Auth On + AWS_Region us-west-2 + tls On + Type doc +``` + +### Fluent Bit + Amazon Elasticsearch Service + +The Amazon ElasticSearch Service adds an extra security layer where HTTP requests must be signed with AWS Sigv4. Fluent Bit v1.5 introduced full support for Amazon ElasticSearch Service with IAM Authentication. Fluent Bit supports sourcing AWS credentials from any of the standard sources \(for example, an [Amazon EKS IAM Role for a Service Account](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html)\). @@ -143,5 +160,5 @@ Example configuration: tls On ``` -Notice that the `Port` is set to `443`, and that `tls` is enabled. +Notice that the `Port` is set to `443`, `tls` is enabled, and `AWS_Region` is set.