Performance Degradation because of EC2 Metadata Client

[Benmartin92]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone though Developer Guide and API reference
• I've searched for previous similar issues and didn't find any solution

Describe the bug
We were experiencing very slow speeds after upgrading to version v1.8.24. I am aware that the same issue has already been reported in #1440, however I think this still needs to be addressed somehow.

I can confirm that setting AWS_EC2_METADATA_DISABLED=true indeed solves the issue, however we are supporting several authentication methods including EC2 Instance Profile credentials (https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html, method 6) that obtains the credentials through EC2 Metadata Service which is in this case would be blocked. Therefore, we cannot disable this service.

I also understand that setting either AWS_DEFAULT_REGION or AWS_REGION prevents ClientConfiguration constructor from reaching out to the metadata service but these are global settings and for applications that may access multiple buckets in different regions one configuration might not be doable.

Moreover, these are new environment variables introduced in v1.8 so I suppose the region handling are implemented without utilizing these variables in many applications. For me it seems to be a hack to temporarily set AWS_DEFAULT_REGION or AWS_REGION to some region just to avoid a potentially slow EC2 metadata call.

I think it would be nice to not reaching out to the EC2 metadata client to obtain the region just because EC2 metadata is enabled, maybe make it configurable trough a different environment variable?

Thanks,
Ben

SDK version number
1.8.24

Platform/OS/Hardware/Device
Windows Server version 1607

To Reproduce (observed behavior)
Just create an S3Client with a default-constructed ClientConfiguration instance.

[KaibaLopez]

Hi @Benmartin92 ,
Yea... you're right, the ec2 metadata call and the workaround are not ideal and we are planning on changing that.

I do appreciate you taking the time to open the issue though, while it is a kind of "work in progress" thing that's already in our backlog, it helps to have it explained in a concise manner and I think you did it perfectly.

[TimLand]

mark

[adame11]

We are implementing the somewhat hacky workaround of setting the environment variable AWS_EC2_METADATA_DISABLED=true before instantiating a ClientConfiguration and removing it or setting to false afterwards, but we are wondering if another permanent fix is forthcoming? And secondly, is there anything you could think of that would explain why the performance issue happens on some not all non-ec2 instance machines (PCs that are naive to any environment variables or .aws\config files)?

We stumbled across this issue after experience the performance degradation described above on SOME (but not all) of our machines. Again, any idea on what could explain why just some?

1. All of our machines do not have any AWS_DEFAULT_REGION nor AWS_REGION environment variables nor any .aws\config files anywhere on their file system.
2. By experiment, the "expensive" ec2 metadata lookup does NOT appear to happen (using sysinternals process monitor app procmon64 to verify this) on some of our PCs
3. For example, the performance slowdown did not occur on bare metal Windows 10 desktops. You can see the failure to find an .aws\config file but then no attempt to query 169.254.169.254 (the ec2 metadata service)
4. However, the performance slowdown does notably happen on, for example, a bare metal Windows Server 2019 that we tried. Also, the slowdown happens reliably on our Windows 10 and Windows 11 esxi hosted virtual machines: first there is a failed attempt to find an %USERPROFILE%.aws\config file, next there is a failed attempt to query 169.254.169.254 (the ec2 metadata service), which tries a few times and fails (hence the slowness). Also, although we have no way of verifying whether the ec2 metadata service call is happening or not, the slowdown does not appear to happen on Mac OS nor Ubuntu vms? Is this all just coincidence?

I looked at the GetCurrentRegion method after the GetEC2MetadataClient but didn't see anything there that could explain why some machines are "fast" and some machines experience the expensive/slow ec2 metadata lookup. Ideas are welcome!

Suggested Fix

Would a reasonable fix for this be to offer a new overload of the ClientConfiguration constructor that takes a default region string as a parameter? In that case, don't bother looking at environment variables nor doing the slow ec2 metadata lookup. We have the same needs as the OP here: need to access buckets in potentially different regions from a single machine/application.

[jmklix]

We have added an optional cmake flag that will turn off IMDS (edit: only disables IMDS v1 calls. Not v2) calls: DISABLE_INTERNAL_IMDSV1_CALLS. This should fix your problems but please let me know if you have any more questions. Commit with changes: 3fefb1c

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.