Repository navigation
Replies: 1 comment
|
Nice writeup. One thing to be aware of for the proposal: Bottlerocket already has Until something like that exists, the pattern I'd use keeps all the logic off the node, which also covers your "broken userdata" case (a node that never comes up can't signal anything, and the hook just times out):
Compared with a host container, that means no IAM permissions for That said, a built-in |
Uh oh!
There was an error while loading. Please reload this page.
Hello!
I would like to create a set of static self-managed AWS EKS Nodes using an ASG and LT that uses the Bottlerocket AMI. One thing I've noticed is that when an ASG Instance Refresh occurs the new instances are placed InService before the EKS Nodes are "Ready". This is problematic because the Instance Refresh could replace all of the instances with new ones that never become "Ready" (e.g., due to a misconfigured UserData script).
My first thought was to setup custom healthchecks for the instances. By using Bottlerocket bootstrap containers I was planning on programmatically setting the instance health to "Unhealthy" with a non-zero health check grace period on the ASG. The custom healthcheck would poll the Node status until the "Ready" condition was "True"; only then would it set the instance health to "Healthy". This doesn't actually work in practice because the orchestrator, kubelet, doesn't start until the bootstrap containers complete.
Another option I considered was ASG Lifecycle Hooks. A Host container could continually poll the Node status endpoint on bootstrap and invoke
CompleteLifecycleActiononce the Node is "Ready". This only needs to run once during start-up/scale-out. Host containers run through the lifetime of the instance and restart if they terminate.I had an idea that I would appreciate feedback on. The thought is for Bottlerocket to introduce a new property
settings.autoscaling.should-signal, (similar tosettings.cloudformation.should-signal) that invokesCompleteLifecycleActionif the orchestrator is ready, where readiness is defined by predefined (or custom) orchestrator healthchecking logic (e.g., Read node conditions and check if "Ready").Some benefits to more advanced orchestrator healthchecks could include:
settings.cloudformation.should-signalto determine if a signal should be sentsettings.autoscaling.should-waitproperty was added to support ASG Warm Pools in an orchestrator agnostic way. Since lifecycle state is available via the instance metadata service, this same challenge could maybe be solved by using Lifecycle hooks. Bottlerocket could check if the instance is in aPending:Waitstate and only then start the kubelet. Once the healthchecker returns that the kubelet is healthy it could invokeCompleteLifecycleAction.Warmed:Pending:Waitstate would not join the cluster.settings.autoscaling.should-signalis enabled.Something else I noticed is that if
InstanceReusePolicyis set toReuseOnScaleInthekubelet.serviceis not stopped, nor would thewarm-pool-wait.servicerun again on scale out. This is documented on the MNG docs so maybe this is a known issue: https://docs.aws.amazon.com/eks/latest/userguide/warm-pools-managed-node-groups.htmlAll reactions