Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fetching IMDSv2 token should have timeout for read #104244

Closed
ywangd opened this issue Jan 11, 2024 · 1 comment · Fixed by #104253 or #104407
Closed

Fetching IMDSv2 token should have timeout for read #104244

ywangd opened this issue Jan 11, 2024 · 1 comment · Fixed by #104253 or #104407
Labels
:Distributed/Discovery-Plugins Anything related to our integration plugins with EC2, GCP and Azure >enhancement Team:Distributed Meta label for distributed team

Comments

@ywangd
Copy link
Member

ywangd commented Jan 11, 2024

We configure connectTimeout for fetching token from IMDSv2's token API.

elasticsearch/plugins/discovery-ec2/src/main/java/org/elasticsearch/discovery/ec2/AwsEc2Utils.java

Line 42 in 1e1d151

urlConnection.setConnectTimeout(CONNECT_TIMEOUT);

However, we do not have timeout for reading from the connection. Therefore, the read can get stuck* indefinitely

elasticsearch/plugins/discovery-ec2/src/main/java/org/elasticsearch/discovery/ec2/AwsEc2Utils.java

Line 49 in 1e1d151

var in = urlConnection.getInputStream();

Since we rely on error or empty respnose from the token API to fallback to IMDSv1 (#84410), stucking at reading prevents the fallback mechanism to work.

[*] One reason for read to stuck is that the request needs more than 1 hop https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-metadata-v2-how-it-works.html
@ywangd ywangd added >enhancement :Distributed/Discovery-Plugins Anything related to our integration plugins with EC2, GCP and Azure labels Jan 11, 2024
@elasticsearchmachine elasticsearchmachine added the Team:Distributed Meta label for distributed team label Jan 11, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@arteam arteam mentioned this issue Jan 11, 2024
Merged
arteam added a commit that referenced this issue Jan 16, 2024
@arteam
Set read timeout for fetching IMDSv2 token (#104253)
063313f 
Use the timeout set by AWS_METADATA_SERVICE_TIMEOUT environment variable both as connect and read timeout analogous to the AWS SDK.

See https://docs.aws.amazon.com/sdkref/latest/guide/feature-ec2-instance-metadata.html

Resolves #104244
arteam added a commit to arteam/elasticsearch that referenced this issue Jan 16, 2024
@arteam
Set read timeout for fetching IMDSv2 token
7a41726 
Use the timeout set by AWS_METADATA_SERVICE_TIMEOUT environment variable both as connect and read timeout analogous to the AWS SDK.

See https://docs.aws.amazon.com/sdkref/latest/guide/feature-ec2-instance-metadata.html

Resolves elastic#104244
@arteam arteam mentioned this issue Jan 16, 2024
Merged
costin pushed a commit to costin/elasticsearch that referenced this issue Jan 17, 2024
@arteam @costin
Set read timeout for fetching IMDSv2 token (elastic#104253)
251f1ce 
Use the timeout set by AWS_METADATA_SERVICE_TIMEOUT environment variable both as connect and read timeout analogous to the AWS SDK.

See https://docs.aws.amazon.com/sdkref/latest/guide/feature-ec2-instance-metadata.html

Resolves elastic#104244
jedrazb pushed a commit to jedrazb/elasticsearch that referenced this issue Jan 17, 2024
@arteam @jedrazb
Set read timeout for fetching IMDSv2 token (elastic#104253)
57af9be 
Use the timeout set by AWS_METADATA_SERVICE_TIMEOUT environment variable both as connect and read timeout analogous to the AWS SDK.

See https://docs.aws.amazon.com/sdkref/latest/guide/feature-ec2-instance-metadata.html

Resolves elastic#104244
arteam added a commit that referenced this issue Jan 17, 2024
@arteam
Set read timeout for fetching IMDSv2 token (#104407)
0acff18 
Resubmit of #104397 without setting AWS_METADATA_SERVICE_TIMEOUT randomly in the build.

```
    Use the timeout set by AWS_METADATA_SERVICE_TIMEOUT environment variable both as connect and read timeout analogous to the AWS SDK.
    See https://docs.aws.amazon.com/sdkref/latest/guide/feature-ec2-instance-metadata.html
```

Resolves #104244
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:Distributed/Discovery-Plugins Anything related to our integration plugins with EC2, GCP and Azure >enhancement Team:Distributed Meta label for distributed team
Projects
None yet
Milestone
No milestone
2 participants
@ywangd @elasticsearchmachine