Support for fluent-bit 2.+

[cdancy]

Just wondering if there is a time frame when fluent-bit 2.+ will be supported?

Thanks ahead of time,
Chris

[PettitWesley]

Fluent Bit 2.0 is on our roadmap. We plan to do testing and release some time early next year.

Can you share the features you are most excited about in 2.0 and which use cases it can solve for you that our current 1.9 based releases can not?

[PettitWesley]

Another reason to delay is that we need to have a 1.0 release that we can have customers rely on for sometime, at least for the entirety of Q1 2023. Currently we have bug reports with our latest 1.0 release, so we need more time to get a stable 1.0 series release before I even feel comfortable making our latest a 2.0 release.

[py4chen]

Can you share the features you are most excited about in 2.0 and which use cases it can solve for you that our current 1.9 based releases can not?

Hi, it looks like several input plugins could take advantage of TLS features starting from 2.x. It's a security feature I'm looking forward to for log aggregation.
Ref: https://docs.fluentbit.io/manual/administration/security

Thanks.

[marinoborges]

Troubleshooting with the new TAP functionality, the new Internal Metrics and the new threaded mechanism with performance gain are very desirable features on 2.0.

[avanish-vaghela]

I am interested in the OpenTelemetry output plugin as we are utilising the OpenTelemetry Collector. It was released in FluentBit version 2.0.2. As a workaround, we have currently implemented the Fluent Forward Receiver.

[albertoCrego]

I am interested in the extra Prometheus metrics (Internal Metrics) to monitor buffer, storage, etc.
It's deployed on v2.0.0

[cdancy]

Input plugins, at least those coming from fluentbit, now have the threaded option.

https://github.com/fluent/fluent-bit/releases/tag/v2.0.0

[lrivera-verimatrix]

I am interested in the OTEL plugin that comes with fluentbit v2.0 and beyond. Any idea if this will be a Q2 or Q3 release?

[PettitWesley]

Until AWS supports this, one option is to use the upstream image: https://hub.docker.com/r/fluent/fluent-bit/

[pureiboi]

with suggested upstream image, does that means we need to forgo the AWS features?

namely

Amazon CloudWatch Logs
Amazon Kinesis Streams
Amazon Kinesis Firehose

[PettitWesley]

@pureiboi the AWS go plugins are not present in the upstream image, but the C plugins are: https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#aws-go-plugins-vs-aws-core-c-plugins

[YannickZ]

Until AWS supports this, one option is to use the upstream image: https://hub.docker.com/r/fluent/fluent-bit/

hey @PettitWesley, i had several issues using the open source image in ECS. Not only to get the logs through between the containers but also to get the logs hydrated with the ECS metadata.

[lubingfeng]

@YannickZ for your issue with open source image, I am wondering if you have submitted an issue with the community on https://github.com/fluent/fluent-bit/issues?

[PettitWesley]

We now have two issues where users are posting upgrade requests: #646

The original intention had been for this issue to be the sole one for that purpose.

[IskanderNovena]

We want to export metrics to AWS Managed Service for Prometheus using the prometheus_remote_write output, which requires the AWS specific settings to be available. This requires FluentBit version 2.1.x or higher.

[IskanderNovena]

We now have two issues where users are posting upgrade requests: #646

The original intention had been for this issue to be the sole one for that purpose.

Perhaps close the other one as a duplicate?

[PettitWesley]

New request: #706

[marevers]

As per #706 :

Fluent Bit 2.1 has a new parameter for the grep filter plugin: Logical_Op. With that parameter it is possible to include multiple regex matchers in a single filter and choose what logical operation should be use to combine these matchers. More information: Fluent Bit Manual - Grep

E.g.

[FILTER]
    Name       grep
    Match      *
    Regex      log foo
    Regex      log bar
    Logical_Op OR

It would be nice if aws-for-fluent-bit would support this also in the future.

[PettitWesley]

Seems elasticsearch input was added in 2.x: #708

[PettitWesley]

Another request for [FILTER] with uuid_key

[tmakamure]

Fluentbit 2.1.0 Introduces an new filter called logs_to_metrics. This filter allows for custom prometheus metrics to be generated based on the contents of the application logs that fluentbit processes e.g

[FILTER]
    name               log_to_metrics
    match              dummy.log*
    tag                test_metric
    metric_mode        counter
    metric_name        count_all_error_dummy_messages
    metric_description This metric counts dummy messages with a log level of error
    regex log_level ERROR
    label_field application_name

It would be great if this feature can also be supported in the near future releases, as there is great use cases for it.

This plugin works well with the prometheus exporter output plugin (https://docs.fluentbit.io/manual/pipeline/outputs/prometheus-exporter) which can then be scraped for these metrics.

Links:

• Fluentbit: https://docs.fluentbit.io/manual/pipeline/filters/log_to_metrics
• Github: https://github.com/fluent/fluent-bit/tree/v2.1.0/plugins/filter_log_to_metrics

[PettitWesley]

@tmakamure I agree the logs to metrics filter is a cool feature. Can you provide more details on how you plan to use it? Which output will you send the metrics to? Right now, in AWS, the only output in FLB which can send metrics I think is prometheus remote write.

[tmakamure]

@tmakamure I agree the logs to metrics filter is a cool feature. Can you provide more details on how you plan to use it? Which output will you send the metrics to? Right now, in AWS, the only output in FLB which can send metrics I think is prometheus remote write.

@PettitWesley This would work with the prometheus exporter output plugin: https://docs.fluentbit.io/manual/pipeline/outputs/prometheus-exporter. This output plugin would need to be supported too inorder to fit the usecase.

To provide more context as to how we intend to use this - We have workloads running on multiple kubernetes clusters in AWS EKS, and we we use aws-for-fluentbit as our log processor of choice, outputting to cloudwatch among other outputs. The application logs conform to a specific format, and we would like to generate prometheus metrics based on the various aspects of the log's contents e.g when there are errors for example, for a specific application, we would like to create a counter metric that we export to Prometheus, from where we can calculate the error rate and trigger an alert if the error rate exceeds a certain threshold.

An example fluentbit configuration could look like:

[INPUT]
Tag examle_tag
# Input ingesting the logs

[PARSER]
Format Regex
Name example_regex_parser
# Custom regex parsers to structure the logs, and labels fields of interest accordingly
# e.g creates new fields called log_level and app_name 

[FILTER]
Name parser-filter
Match example_tag
Key_Name log
Parser example_regex_parser

[FILTER]
name log_to_metrics      # The log_to_metrics filter in question
match   example_tag   # The tag set on the INPUT plugin above
tag    custom_prometheus_metric
metric_mode counter
metric_name error_counter
regex log_level ERROR      # Checks a field called log_level in the record, and if it contains the word ERROR
label_field  app_name  # adds a label to the metric based on the value of a key "app_name" in the record

[OUTPUT]
Name  prometheus_exporter
Match custom_prometheus_metric   # The tag from the log_to_metrics filter
Host  0.0.0.0
Port  2021

[tmakamure]

Fluent Bit 2.0 is on our roadmap. We plan to do testing and release some time early next year.

Can you share the features you are most excited about in 2.0 and which use cases it can solve for you that our current 1.9 based releases can not?

Hi @PettitWesley , any update as to when support for fluentbit 2.x can be suported? Is it something we can expect in the next 2-3 months? Not sure if there is a link to the roadmap somewhere?

[fermani]

In our case, we want to parse multiline logs with the regex /pattern/m option, a feature which was added to Fluent Bit v2.0.6.

• Upstream PR: regex: support /patttern/option fluent/fluent-bit#6257
• Docs: https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/multiline-parsing#get-structured-data-from-multiline-message

[ziqingyuan]

Hi @PettitWesley! I am also interested in the OpenTelemetry output plugin feature. There's a mismatch between the log structure for fluentd and otel. It would be awesome if we could directly utilize that output plugin instead of resorting to a compromised solution that requires a lot of additional configuration with the Fluent Forward Receiver and the Transform Processor. Thank you so much!

[fmardini]

Another important feature is the support for compression in the elastic output plugin.

Any updates on when this will land?

[daljitsharma]

Another reason is support for Kubernetes Events input that ingest kube-events

[pmody-te]

Has this project/task been abandoned ?

[FT-JMendyk]

@PettitWesley Could you provide a statement regarding the plan for fluent-bit 2.+ Over 18 months ago, in December 2022 you have stated:

Fluent Bit 2.0 is on our roadmap. We plan to do testing and release some time early next year.

Can you share the features you are most excited about in 2.0 and which use cases it can solve for you that our current 1.9 based releases can not?

There's been over a year since the "early next year" time period. I believe that an update is well due 🙏

[cdancy]

@FT-JMendyk we've recently moved to vanilla fluentbit (latest and greatest) and away from aws-for-fluent-bit and it all "just worked" with no issues to speak of.

[jpage-godaddy]

Any updates on this? The lack of Logical_Op being available for grep filters with us being stuck on Fluent Bit 1.9.10 is very limiting. @cdancy, I am interested, if you still have the details, on how you did the switch to vanilla Fluent Bit.

[cdancy]

@jpage-godaddy the switch more/less "just worked". The only thing I remember that stuck out was upgrades breaking due to the name on the helm resources being aws-for-fluent-bit. We had to carry that over but all things considered it was a small price to pay to start using the OOTB fluentbit.