IMDSv2 Support and IMDSv1 Breaking Compatibility in 2.21.0: causes Fluent Bit to hang

[matthewfala]

aws-for-fluent-bit release note for 2.21.0:

IMDSv2 Support

This release introduces IMDSv2 support BUT breaks backwards compatibility for IMDSv1.
Instances that rely on IMDS for security credentials must set EC2's instance-metadata-option http-put-response-hop-limit to 2. It should be noted that if ECS is used without a taskRoleArn, IMDS will be used to retrieve the instances role and each ECS instance must have the instance-metadata-option http-put-response-hop-limit to 2.

A patch is being worked on to allow IMDSv1 to be used as a fallback option.

Setting Hop Limit

The AWS cli can be used to set http-put-response-hop-limit to 2 via the following:

aws ec2 modify-instance-metadata-options \
     --instance-id <my-instance-id> \
     --http-put-response-hop-limit 2

The AWS cli can also be used to require IMDSv2 for security purposes:

aws ec2 modify-instance-metadata-options \
     --instance-id <my-instance-id> \
     --http-tokens required \
     --http-endpoint enabled

[lado936]

WTF, i am debugging like 1 day why my logs are not showing up in Elasticsearch and after googling like 4 hours came up with this breaking changes, don't get me wrong but this kind of changes must be notified at least in AWS notifications bar, as a lot of companies rely on this functionality and would be like 5 minute fix and not 1 day checking everything to find out that logs are not showing up for more then a week.

[matthewfala]

Thank you for your feedback. As a team we will try harder to prevent breaking backwards compatibility in the future.

Please make sure to not deploy “latest” directly to prod without going through a test stage or consider using a specific aws-for-fluent-bit version tag and reviewing the release notes on migrating to a new image version. That's why we put a notice in the release notes for 2.21.0 warning of the change. Defaulting to IMDSv2 over IMDSv1 is the general best practice/requirement for AWS tools; unfortunately, that leads to this hang problem right now due to limitations in Fluent Bit's core networking library. I am actively working on fixing that limitation.

[matthewfala]

This issue is resolved in aws-for-fluent-bit version 2.21.2.
Graceful IMDSv2 failure and fallback to IMDSv1 is now supported.

v2.21.2 behavior:
If IMDSv2 is not available due to the hops-limit problem, a warning will be logged, and fallback to IMDSv1 will be attempted. If IMDSv1 is not available, a warning will be logged and credential acquisition will fail and be retried at a later time.

On IMDSv1 fallback, a single network error will occur:

[error] [src/flb_network.c:222 errno=9] Bad file descriptor
...
[ info][imds] to use IMDSv2, set --http-put-response-limit to 2
[ warn][imds] falling back on IMDSv1

If you would like to continue using IMDSv1, which is not recommended by AWS, please ignore this error message.
Otherwise, if you would like to start using IMDSv2, please set http-put-response-hop-limit to 2

Please see:
https://github.com/aws/aws-for-fluent-bit/releases/tag/v2.21.2