Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pluto: retry on empty private-dns-name from EC2 #3364

Merged
merged 1 commit into from
Sep 11, 2023

Conversation

etungsten
Copy link
Contributor

Issue number:

Resolves #3363

Description of changes:

    pluto: retry on empty private-dns-name from EC2
    
    Use fibonacci backoff on requests to EC2 for fetching the private DNS
    name of the instance. Retry on both API failures and on when the private
    DNS name is empty.

Testing done:
Instance comes up fine. Directly calling pluto works:
On an instance where hostname is resource-based in the subnet:

bash-5.1# pluto private-dns-name
"i-088c99da0374c92a5.us-west-2.compute.internal"

On an instance where hostname is IP based:

bash-5.1# pluto private-dns-name
"ip-192-168-21-104.us-west-2.compute.internal"

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

Copy link
Contributor

@stmcginnis stmcginnis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@yeazelm yeazelm self-requested a review August 25, 2023 21:14
@etungsten etungsten marked this pull request as draft August 25, 2023 21:35
@etungsten
Copy link
Contributor Author

Re-evaluating approach since there is potentially a valid combination of VPC configuration and kubelet/aws-iam-authenticator configurations where a node can still join a cluster where private DNS name is empty.

Use fibonacci backoff on requests to EC2 for fetching the private DNS
name of the instance. Retry on both API failures and on when the private
DNS name is empty.
@etungsten etungsten marked this pull request as ready for review September 11, 2023 20:44
@bcressey
Copy link
Contributor

Re-evaluating approach since there is potentially a valid combination of VPC configuration and kubelet/aws-iam-authenticator configurations where a node can still join a cluster where private DNS name is empty.

@etungsten did you determine that no such valid combination exists or would ever have worked on Bottlerocket previously? That's my current understanding.

@etungsten
Copy link
Contributor Author

There is a very fringe case where a user can potentially set up a bootstrap container to set their own hostname-override setting after pluto has set the hostname-override setting to "". After merging this, to maintain that same workflow, a user would just have to set "" or (a random string) for the hostname-override setting in userdata and let their bootstrap container do the overwrite,

@etungsten etungsten merged commit d5e821a into bottlerocket-os:develop Sep 11, 2023
46 of 48 checks passed
@etungsten etungsten deleted the retry-empty-dns-name branch September 11, 2023 23:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add retries in pluto to handle eventual-consistent EC2 private DNS names
4 participants