-
Notifications
You must be signed in to change notification settings - Fork 133
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for fluent-bit 2.+ #494
Comments
Fluent Bit 2.0 is on our roadmap. We plan to do testing and release some time early next year. Can you share the features you are most excited about in 2.0 and which use cases it can solve for you that our current 1.9 based releases can not? |
Another reason to delay is that we need to have a 1.0 release that we can have customers rely on for sometime, at least for the entirety of Q1 2023. Currently we have bug reports with our latest 1.0 release, so we need more time to get a stable 1.0 series release before I even feel comfortable making our latest a 2.0 release. |
Hi, it looks like several input plugins could take advantage of TLS features starting from 2.x. It's a security feature I'm looking forward to for log aggregation. Thanks. |
Troubleshooting with the new TAP functionality, the new Internal Metrics and the new threaded mechanism with performance gain are very desirable features on 2.0. |
I am interested in the OpenTelemetry output plugin as we are utilising the OpenTelemetry Collector. It was released in FluentBit version 2.0.2. As a workaround, we have currently implemented the Fluent Forward Receiver. |
I am interested in the extra Prometheus metrics (Internal Metrics) to monitor buffer, storage, etc. |
Input plugins, at least those coming from fluentbit, now have the |
I am interested in the OTEL plugin that comes with fluentbit v2.0 and beyond. Any idea if this will be a Q2 or Q3 release? |
Until AWS supports this, one option is to use the upstream image: https://hub.docker.com/r/fluent/fluent-bit/ |
with suggested upstream image, does that means we need to forgo the AWS features? namely Amazon CloudWatch Logs |
@pureiboi the AWS go plugins are not present in the upstream image, but the C plugins are: https://github.com/aws/aws-for-fluent-bit/blob/mainline/troubleshooting/debugging.md#aws-go-plugins-vs-aws-core-c-plugins |
hey @PettitWesley, i had several issues using the open source image in ECS. Not only to get the logs through between the containers but also to get the logs hydrated with the ECS metadata. |
@YannickZ for your issue with open source image, I am wondering if you have submitted an issue with the community on https://github.com/fluent/fluent-bit/issues? |
We now have two issues where users are posting upgrade requests: #646 The original intention had been for this issue to be the sole one for that purpose. |
We want to export metrics to AWS Managed Service for Prometheus using the prometheus_remote_write output, which requires the AWS specific settings to be available. This requires FluentBit version 2.1.x or higher. |
Perhaps close the other one as a duplicate? |
New request: #706 |
As per #706 : Fluent Bit 2.1 has a new parameter for the grep filter plugin: E.g.
It would be nice if aws-for-fluent-bit would support this also in the future. |
Seems elasticsearch input was added in 2.x: #708 |
Another request for |
Fluentbit 2.1.0 Introduces an new filter called logs_to_metrics. This filter allows for custom prometheus metrics to be generated based on the contents of the application logs that fluentbit processes e.g
It would be great if this feature can also be supported in the near future releases, as there is great use cases for it. This plugin works well with the prometheus exporter output plugin (https://docs.fluentbit.io/manual/pipeline/outputs/prometheus-exporter) which can then be scraped for these metrics. Links: |
@tmakamure I agree the logs to metrics filter is a cool feature. Can you provide more details on how you plan to use it? Which output will you send the metrics to? Right now, in AWS, the only output in FLB which can send metrics I think is prometheus remote write. |
@PettitWesley This would work with the prometheus exporter output plugin: https://docs.fluentbit.io/manual/pipeline/outputs/prometheus-exporter. This output plugin would need to be supported too inorder to fit the usecase. To provide more context as to how we intend to use this - We have workloads running on multiple kubernetes clusters in AWS EKS, and we we use An example fluentbit configuration could look like:
|
Hi @PettitWesley , any update as to when support for fluentbit 2.x can be suported? Is it something we can expect in the next 2-3 months? Not sure if there is a link to the roadmap somewhere? |
In our case, we want to parse multiline logs with the regex /pattern/m option, a feature which was added to Fluent Bit v2.0.6. |
Hi @PettitWesley! I am also interested in the OpenTelemetry output plugin feature. There's a mismatch between the log structure for fluentd and otel. It would be awesome if we could directly utilize that output plugin instead of resorting to a compromised solution that requires a lot of additional configuration with the Fluent Forward Receiver and the Transform Processor. Thank you so much! |
Another important feature is the support for compression in the elastic output plugin. Any updates on when this will land? |
Another reason is support for Kubernetes Events input that ingest kube-events |
Has this project/task been abandoned ? |
@PettitWesley Could you provide a statement regarding the plan for fluent-bit 2.+ Over 18 months ago, in December 2022 you have stated:
There's been over a year since the "early next year" time period. I believe that an update is well due 🙏 |
@FT-JMendyk we've recently moved to vanilla fluentbit (latest and greatest) and away from aws-for-fluent-bit and it all "just worked" with no issues to speak of. |
Any updates on this? The lack of |
@jpage-godaddy the switch more/less "just worked". The only thing I remember that stuck out was upgrades breaking due to the name on the helm resources being |
Just wondering if there is a time frame when fluent-bit 2.+ will be supported?
Thanks ahead of time,
Chris
The text was updated successfully, but these errors were encountered: