From b34194d2f711f53eecef4bd2aff920023a395863 Mon Sep 17 00:00:00 2001 From: ted Date: Wed, 15 Jul 2020 12:37:11 -0700 Subject: [PATCH 1/4] add docs around v1.5 out_es change to ES type; ce - in v1.5, the default `type` was changed, which can cause problems. Highlighted this in the upgrade notes and added a 'FAQ' to the `out_es` page. - since v1.5 introduces aws auth support, added more text to highlight that `AWS\_Region` is required. If not, it dies at runtime. Seems like it could be inferred, but at least the docs reflect it a little better. - minor copyedits throughout. Signed-off-by: ted --- concepts/buffering.md | 2 +- concepts/data-pipeline/buffer.md | 2 +- concepts/key-concepts.md | 2 +- installation/upgrade-notes.md | 5 +++-- pipeline/outputs/elasticsearch.md | 23 ++++++++++++++++++++--- 5 files changed, 26 insertions(+), 8 deletions(-) diff --git a/concepts/buffering.md b/concepts/buffering.md index fecef2e36..e2e6c7526 100644 --- a/concepts/buffering.md +++ b/concepts/buffering.md @@ -4,7 +4,7 @@ description: Performance and Data Safety # Buffering -When [Fluent Bit](https://fluentbit.io) process data, it uses the system memory \(heap\) as a primary and temporal place to store the record logs before they get delivered, on this private memory area the records are processed. +When [Fluent Bit](https://fluentbit.io) processes data, it uses the system memory \(heap\) as a primary and temporal place to store the record logs before they get delivered, on this private memory area the records are processed. Buffering refers to the ability to store the records somewhere, and while they are processed and delivered, still be able to store more. Buffering in memory is the fastest mechanism, but there are certain scenarios where the mechanism requires special strategies to deal with [backpressure](../administration/backpressure.md), data safety or reduce memory consumption by the service in constraint environments. diff --git a/concepts/data-pipeline/buffer.md b/concepts/data-pipeline/buffer.md index 7d612b7ba..d9b3b5eda 100644 --- a/concepts/data-pipeline/buffer.md +++ b/concepts/data-pipeline/buffer.md @@ -11,7 +11,7 @@ The `buffer` phase already contains the data in an immutable state, meaning, no ![](../../.gitbook/assets/logging_pipeline_buffer%20%281%29.png) {% hint style="info" %} -Note that buffered data is not longer a raw text, instead it's in Fluent Bit internal binary representation. +Note that buffered data is not raw text, it's in Fluent Bit's internal binary representation. {% endhint %} Fluent Bit offers a buffering mechanism in the file system that acts as a _backup system_ to avoid data loss in case of system failures. diff --git a/concepts/key-concepts.md b/concepts/key-concepts.md index 3acdef57e..6d7ee635b 100644 --- a/concepts/key-concepts.md +++ b/concepts/key-concepts.md @@ -103,7 +103,7 @@ Source events can have or not have a structure. A structure defines a set of _ke At a low level both are just an array of bytes, but the Structured message defines _keys_ and _values_, having a structure helps to implement faster operations on data modifications. {% hint style="info" %} -Fluent Bit **always** handle every Event message as a structured message. For performance reasons, we use a binary serialization data format called [MessagePack](https://msgpack.org/). +Fluent Bit **always** handles every Event message as a structured message. For performance reasons, we use a binary serialization data format called [MessagePack](https://msgpack.org/). Consider [MessagePack](https://msgpack.org/) as a binary version of JSON on steroids. {% endhint %} diff --git a/installation/upgrade-notes.md b/installation/upgrade-notes.md index a61ecfab4..480e5b2d2 100644 --- a/installation/upgrade-notes.md +++ b/installation/upgrade-notes.md @@ -9,14 +9,15 @@ For more details about changes on each release please refer to the [Official Rel The migration from v1.4 to v1.5 is pretty straightforward. - If you enabled ```keepalive``` mode in your configuration, note that this configuration property has been renamed to ```net.keepalive```. Now all Network I/O is keepalive default enabled, to learn more about this and other associated configuration properties read the [Networking Administration](https://docs.fluentbit.io/manual/administration/networking#tcp-keepalive) section. +- If you use the Elasticsearch output plugin, note the default value of ```type``` [changed from `flb\_type` to `\_doc`](https://github.com/fluent/fluent-bit/commit/04ed3d8104ca8a2f491453777ae6e38e5377817e#diff-c9ae115d3acaceac5efb949edbb21196). Many versions of Elasticsearch will tolerate this, but ES v5.6 through v6.1 require a type *without* a leading underscore. [See Elasticsearch output plugin documentation](pipeline/outputs/elasticsearch.md) for more. ## Fluent Bit v1.4 -If you are migrating from Fluent Bit v1.3, there are not breaking changes. Just new exciting features to enjoy :\) +If you are migrating from Fluent Bit v1.3, there are no breaking changes. Just new exciting features to enjoy :\) ## Fluent Bit v1.3 -If you are migrating from Fluent Bit v1.2 to v1.3, there are not breaking changes. If you are upgrading from an older version please review the incremental changes below. +If you are migrating from Fluent Bit v1.2 to v1.3, there are no breaking changes. If you are upgrading from an older version please review the incremental changes below. ## Fluent Bit v1.2 diff --git a/pipeline/outputs/elasticsearch.md b/pipeline/outputs/elasticsearch.md index ed37e3c5d..903552c59 100644 --- a/pipeline/outputs/elasticsearch.md +++ b/pipeline/outputs/elasticsearch.md @@ -122,9 +122,26 @@ If you see an error message like below, you'll need to fix your configuration to For details, please read [the official blog post on that issue](https://www.elastic.co/guide/en/elasticsearch/reference/6.7/removal-of-types.html). -### Fluent Bit + Amazon Elasticsearch Service +### Elasticsearch rejects requests saying "Document mapping type name can't start with '\_'" -Amazon ElasticSearch Service adds an extra security layer where HTTP requests must be signed with AWS Sigv4. Fluent Bit v1.5 introduced full support for Amazon ElasticSearch Service with IAM Authentication. +Fluent Bit v1.5 changed the default mapping type from `flb_type` to `_doc`, which matches the recommendation from Elasticsearch from version 6.2 forwards \([see commit with rationale](https://github.com/fluent/fluent-bit/commit/04ed3d8104ca8a2f491453777ae6e38e5377817e#diff-c9ae115d3acaceac5efb949edbb21196)\). This doesn't work in Elasticsearch versions 5.6 through 6.1 \([see Elasticsearch discussion and fix](https://discuss.elastic.co/t/cant-use-doc-as-type-despite-it-being-declared-the-preferred-method/113837/9)\). Ensure you set an explicit map (such as `doc` or `flb_type`) in the configuration, as seen on the last line: + +```text +[OUTPUT] + Name es + Match * + Host vpc-test-domain-ke7thhzoo7jawsrhmm6mb7ite7y.us-west-2.es.amazonaws.com + Port 443 + Index my_index + AWS_Auth On + AWS_Region us-west-2 + tls On + Type doc +``` + +### Fluent Bit + Amazon Elasticsearch Service + +The Amazon ElasticSearch Service adds an extra security layer where HTTP requests must be signed with AWS Sigv4. Fluent Bit v1.5 introduced full support for Amazon ElasticSearch Service with IAM Authentication. Fluent Bit supports sourcing AWS credentials from any of the standard sources \(for example, an [Amazon EKS IAM Role for a Service Account](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html)\). @@ -143,5 +160,5 @@ Example configuration: tls On ``` -Notice that the `Port` is set to `443`, and that `tls` is enabled. +Notice that the `Port` is set to `443`, `tls` is enabled, and `AWS_Region` is set. From 914173d6579577db98efb36d9ff68f2c7a6b560a Mon Sep 17 00:00:00 2001 From: ted Date: Wed, 15 Jul 2020 12:51:47 -0700 Subject: [PATCH 2/4] single ticks Signed-off-by: ted --- installation/upgrade-notes.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/installation/upgrade-notes.md b/installation/upgrade-notes.md index 480e5b2d2..050284b5f 100644 --- a/installation/upgrade-notes.md +++ b/installation/upgrade-notes.md @@ -8,8 +8,8 @@ For more details about changes on each release please refer to the [Official Rel The migration from v1.4 to v1.5 is pretty straightforward. -- If you enabled ```keepalive``` mode in your configuration, note that this configuration property has been renamed to ```net.keepalive```. Now all Network I/O is keepalive default enabled, to learn more about this and other associated configuration properties read the [Networking Administration](https://docs.fluentbit.io/manual/administration/networking#tcp-keepalive) section. -- If you use the Elasticsearch output plugin, note the default value of ```type``` [changed from `flb\_type` to `\_doc`](https://github.com/fluent/fluent-bit/commit/04ed3d8104ca8a2f491453777ae6e38e5377817e#diff-c9ae115d3acaceac5efb949edbb21196). Many versions of Elasticsearch will tolerate this, but ES v5.6 through v6.1 require a type *without* a leading underscore. [See Elasticsearch output plugin documentation](pipeline/outputs/elasticsearch.md) for more. +- If you enabled `keepalive` mode in your configuration, note that this configuration property has been renamed to `net.keepalive`. Now all Network I/O keepalive is enabled by default, to learn more about this and other associated configuration properties read the [Networking Administration](https://docs.fluentbit.io/manual/administration/networking#tcp-keepalive) section. +- If you use the Elasticsearch output plugin, note the default value of `type` [changed from `flb\_type` to `\_doc`](https://github.com/fluent/fluent-bit/commit/04ed3d8104ca8a2f491453777ae6e38e5377817e#diff-c9ae115d3acaceac5efb949edbb21196). Many versions of Elasticsearch will tolerate this, but ES v5.6 through v6.1 require a type *without* a leading underscore. [See Elasticsearch output plugin documentation](pipeline/outputs/elasticsearch.md) for more. ## Fluent Bit v1.4 From 28d2ff60405c069de5a6b8362a919781cc3e594d Mon Sep 17 00:00:00 2001 From: ted Date: Wed, 15 Jul 2020 12:54:42 -0700 Subject: [PATCH 3/4] remove escapes, which don't work properly here Signed-off-by: ted --- installation/upgrade-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/installation/upgrade-notes.md b/installation/upgrade-notes.md index 050284b5f..da5461218 100644 --- a/installation/upgrade-notes.md +++ b/installation/upgrade-notes.md @@ -9,7 +9,7 @@ For more details about changes on each release please refer to the [Official Rel The migration from v1.4 to v1.5 is pretty straightforward. - If you enabled `keepalive` mode in your configuration, note that this configuration property has been renamed to `net.keepalive`. Now all Network I/O keepalive is enabled by default, to learn more about this and other associated configuration properties read the [Networking Administration](https://docs.fluentbit.io/manual/administration/networking#tcp-keepalive) section. -- If you use the Elasticsearch output plugin, note the default value of `type` [changed from `flb\_type` to `\_doc`](https://github.com/fluent/fluent-bit/commit/04ed3d8104ca8a2f491453777ae6e38e5377817e#diff-c9ae115d3acaceac5efb949edbb21196). Many versions of Elasticsearch will tolerate this, but ES v5.6 through v6.1 require a type *without* a leading underscore. [See Elasticsearch output plugin documentation](pipeline/outputs/elasticsearch.md) for more. +- If you use the Elasticsearch output plugin, note the default value of `type` [changed from `flb_type` to `_doc`](https://github.com/fluent/fluent-bit/commit/04ed3d8104ca8a2f491453777ae6e38e5377817e#diff-c9ae115d3acaceac5efb949edbb21196). Many versions of Elasticsearch will tolerate this, but ES v5.6 through v6.1 require a type *without* a leading underscore. [See Elasticsearch output plugin documentation](pipeline/outputs/elasticsearch.md) for more. ## Fluent Bit v1.4 From 9bb1fa52a6eb6a34ded3f2c9f57b1f46f43a9aeb Mon Sep 17 00:00:00 2001 From: ted Date: Wed, 15 Jul 2020 12:59:03 -0700 Subject: [PATCH 4/4] fix broken link; try to match style of previous entry for link text. Signed-off-by: ted --- installation/upgrade-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/installation/upgrade-notes.md b/installation/upgrade-notes.md index da5461218..b21048ef8 100644 --- a/installation/upgrade-notes.md +++ b/installation/upgrade-notes.md @@ -9,7 +9,7 @@ For more details about changes on each release please refer to the [Official Rel The migration from v1.4 to v1.5 is pretty straightforward. - If you enabled `keepalive` mode in your configuration, note that this configuration property has been renamed to `net.keepalive`. Now all Network I/O keepalive is enabled by default, to learn more about this and other associated configuration properties read the [Networking Administration](https://docs.fluentbit.io/manual/administration/networking#tcp-keepalive) section. -- If you use the Elasticsearch output plugin, note the default value of `type` [changed from `flb_type` to `_doc`](https://github.com/fluent/fluent-bit/commit/04ed3d8104ca8a2f491453777ae6e38e5377817e#diff-c9ae115d3acaceac5efb949edbb21196). Many versions of Elasticsearch will tolerate this, but ES v5.6 through v6.1 require a type *without* a leading underscore. [See Elasticsearch output plugin documentation](pipeline/outputs/elasticsearch.md) for more. +- If you use the Elasticsearch output plugin, note the default value of `type` [changed from `flb_type` to `_doc`](https://github.com/fluent/fluent-bit/commit/04ed3d8104ca8a2f491453777ae6e38e5377817e#diff-c9ae115d3acaceac5efb949edbb21196). Many versions of Elasticsearch will tolerate this, but ES v5.6 through v6.1 require a type *without* a leading underscore. See the [Elasticsearch output plugin documentation FAQ entry](https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch#faq-underscore) for more. ## Fluent Bit v1.4