Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: added configuration variables to handle EC2 metadata service #2385

Merged
merged 4 commits into from
Apr 12, 2024

Conversation

mightyshazam
Copy link
Contributor

Description

Rusoto and the AWS SDK have different behavior when interacting with the ec2 metadata service. Despite the lower default timeout in the AWS SDK, it still takes longer to abandon the IMDS credential provider. This PR add a setting AWS_EC2_METADATA_DISABLED to disable the provider. It can be provided as an environment variable or isolated to the s3 storage settings. Additionally, I added AWS_EC2_METADATA_TIMEOUT to represent the timeout for the imds credential provider in milliseconds with a default value of 100.
This does not completely eliminate the regression, but it provides opt-in/opt-out options to mitigate the issues in exceptional cases.

Related Issue(s)

Documentation

Rust SDK Environment Variables

@github-actions github-actions bot added the binding/rust Issues for the Rust crate label Apr 4, 2024
@echai58
Copy link

echai58 commented Apr 9, 2024

Hi, bump on reviewing this 🙂 , this is blocking my ability to upgrade past 0.16.1 which has some important bug fixes/features.

@echai58
Copy link

echai58 commented Apr 11, 2024

@mightyshazam Hi, I tried running against a local build with this commit, and it seems like passing "AWS_EC2_METADATA_DISABLED": "true" via the storage_options still raises the failed to load region from IMDS err warning and slow reads, whereas setting the environment variable prevents this behavior.

@rtyler rtyler merged commit 64b3e54 into delta-io:main Apr 12, 2024
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
binding/rust Issues for the Rust crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Very slow s3 connection after 0.16.1
3 participants