-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AWS Metadata Filter Plugin for EC2/ECS Metadata #1780
Comments
I have plans to use fluentbit to do a lot of log aggregation and I'd love to be able to decorate logs with information from EC2 metadata like account number, instance ID, region, IAM role, etc. |
@PettitWesley I have some familiarity with the AWS-o-sphere. If you need/want help on this one, let me know. |
@nigels-com To be honest, I don't think I'll personally contribute much more for this plugin than the initial implementation in #1795. I need a simple "getting started with Fluent Bit dev" task to give to a co-worker, and I was thinking this would be ideal. We should start work on expanding it at the beginning of February. That being said, if you or anyone else want this plugin expanded sooner, please go ahead and write the code for it- I can come up with a different task to use to train my coworker. Let me also introduce myself a bit- I'm the creator of AWS for Fluent Bit, the AWS distribution of Fluent Bit. As well as FireLens, the ECS integration with Fluentd/Bit. If you'd like to help out with my effort to better support AWS natively in Fluent Bit, consider reviewing the code in #1852- which is the first in a series of PRs to add standard AWS credential providers to Fluent Bit. |
Also... @nigels-com, are you one of the other "official" maintainers? Eduardo said that he recently added 2 folks. |
Hello @PettitWesley I'm not an official maintainer, but I hang around here a bit. |
Using V1 vs V2 of instance metadata will have to be a config option. V2 does not work for many users who will Fluent Bit in a container (i.e. most). I'm going to make V2 the default. If a user does not configure the plugin to use the correct version, it'll freeze Fluent Bit (because the request to retrieve metadata is synchronous/blocking). |
The next step for this will be to add support for ECS Metadata. One request I have gotten is to support optionally adding docker labels. |
After https://aws.amazon.com/about-aws/whats-new/2022/01/instance-tags-amazon-ec2-instance-metadata-service/ |
@PettitWesley Any updates on this? We are trying to set up Fluent Bit as a daemon service on ECS. Because of this, there is no Firelens and we lose the ability to decorate our logs with ECS metadata. Sure there are ways around it by modifying application code but we would prefer something global just like how Firelens does it. |
@PrayagS I just began working on the design for the filter this week itself. I can't promise any timeline and I don't know exactly when it will be out, but the goal/hope is to have a full ECS EC2 Daemon Support and Tutorial out for Fluent Bit by the end of the spring. |
@PettitWesley That's great to hear. Looking forward to trying it out. |
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the |
I've mostly got it working: https://github.com/PettitWesley/fluent-bit/tree/filter-ecs Hopefully I will get this released in one of the next 1.9 versions. |
PR which adds support for EC2 tags to the AWS Filter: #6065 |
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the |
This issue was closed because it has been stalled for 5 days with no activity. |
I am considering building an 'AWS Metadata' plugin. It'd query the EC2 and ECS metadata services and add useful metadata to log records. For EC2, it'd use the new IMDSv2, since it is more secure if you're running applications on your instance that are exposed on the public internet.
Basically, replicate the functionality of these Fluentd plugins:
EC2 metadata will be prioritized and implemented first.
Do you think this feature would be useful? Please plus one this issue.
Do you have requests for what it will support or how it will work? Please comment.
The text was updated successfully, but these errors were encountered: