Availability on managed node groups

[rverma-jm]

Our team is considering moving from self-managed autoscaling groups to managed node groups for worker.
Wondering if bottlerocket would be available on day 1 (public release) under managed node group or will it take some time. If its former, we probably want to halt that exercise :).

[jhaynes]

I can't comment on timeline specifics, but I can assure you that we're working closely with EKS on integrating Bottlerocket into their managed node groups feature.

Out of curiosity: why do you prefer managed node groups over using Bottlerocket's native update feature and our operator?

[rverma-jm]

I think the point behind managed nodegroup is more about smooth transition between kubernetes updates. Instances in a managed node group use the latest version of the Amazon EKS-optimized Amazon Linux 2 AMI for its cluster's Kubernetes version.
It suppose to auto-update coredns,kube-proxy and aws-node too, but I am not pretty confident about that.

Some important requirements form amazon docs.

1. Amazon EKS follows the shared responsibility model for CVEs and security patches on managed node groups
2. If your managed node group encounters a health issue, Amazon EKS returns an error message to help you to diagnose the issue
3. Amazon EKS automatically drains nodes using the Kubernetes API during terminations or updates. Updates respect the pod disruption budgets that you set for your pods.

I think bottlerocket's native update feature is pretty cool, but how would it manages the updates between api-server's version update, i think its likely kubelet version upgrade, but could be other things too.

[nuvanti]

Any updates? Would really love to see bottlerocket on EKS managed groups natively supported.

[jhaynes]

We're actively working on MNG support and hoping to have support soon. Out of curiosity, @nuvanti are you interested in MNG support via node replacement or via Bottlerocket's in-place update strategy? I ask because we're excited to deliver both, but expect the former to be supported prior to the latter.

[basert]

We are also looking forward to replace our current managed node groups with Bottlerocket. We are currently deploying the node groups using terraform aws_eks_node_group with a custom launch template to add MAP tags to the EC2 machines. Is this going to be supported? Currently AWS API rejects launch templates with custom AMI when creating a MNG.

Regarding the update strategy: I guess "node replacement" is similar to the current behavior? The launch template is updated with the new AMI version and the autoscaling group replaces nodes by running an instance refresh?

[przemolb]

@jhaynes from the application point of view there is no difference between node replacement and Bottlerocket's in-place update - both require apps restart, am I right or not ?

[MartinEmrich]

Apparently eksctl supports Bottlerocket for some time... I just tried it and it seems to work well. It even supports custom settings, like:

managedNodeGroups:
  - name: myNodeGroup
    instanceType: t3.large
    amiFamily: Bottlerocket
    bottlerocket:
      settings:
        kernel:
          sysctl:
            vm.max_map_count: "262144"
...

As this issue is still open: Where's the trap I missed so far? ;)

[zmrow]

Just for reference, this is is the open issue for Bottlerocket MNG support. Follow that issue for the the latest status! As mentioned Bottlerocket can be used for MNG today using custom launch templates. Native support for MNG will be available roughly end of September.

@przemolb Re: node replacement vs. update: Under the hood there's a fundamental difference between node replacement and update as the former shuts down a host and starts a new one and the latter reboots the existing host. From the application point of view, yes, the application will stop and start with both methods, but I would expect that node replacement may take a little bit more time since containers need to be pulled again, etc.

[przemolb]

@zmrow Thanks for explanation. Just thinking that from the stability point of view (all in k8s) it might be more safe to go for node replacement: force migration to the new node pool, if there is something wrong move them back to the previous. In case of in-place upgrades (Bottlerocket ?) I've heard there is a way to roll it back but does it work with EKS ?

[cbgbt]

In the case of the update operator, you’d need to intervene using the instructions in the Bottlerocket README to rollback individual nodes. You’d also need to apply a setting locking the Bottlerocket instance to the known-working version (or disable the update operator) to prevent another update.

We’re currently working on an iteration to the architecture of the update operator which will allow it to perform and expose information about node health changes as a result of updates. We’ll have the operator cease performing updates if nodes are deemed unhealthy on reboot; however, we would still need something like the settings operator proposed in #873 to facilitate fleet-wide rollbacks.

[przemolb]

Let me ask this way: let's say I have two alternative options (versions below invented by me for explanation purposes):

1. migrate off old node group (bottlerocket ver 2.0) by creating a new node group (bottlerocket ver 3.0) and enforcing pods restart onto the new node group
2. upgrade existing node group in-place (bottlerocket 2.0 -> 3.0)

Is there any advantage of 2 over 1 ?

[cPu1]

Apparently eksctl supports Bottlerocket for some time... I just tried it and it seems to work well. It even supports custom settings, like:

managedNodeGroups:
  - name: myNodeGroup
    instanceType: t3.large
    amiFamily: Bottlerocket
    bottlerocket:
      settings:
        kernel:
          sysctl:
            vm.max_map_count: "262144"
...

As this issue is still open: Where's the trap I missed so far? ;)

@MartinEmrich, eksctl supports it today by supplying a launch template with a custom bootstrap script to the Managed Nodegroups API. Unlike EKS AL2, the MNG API does not recognise Bottlerocket as an AMI type. We have opened an issue to track native support for Bottlerocket in MNG. As noted in that issue, this would just be a behind-the-scenes implementation change as the UX for creating Bottlerocket MNG will remain unchanged. Although we will have to add a new feature when Bottlerocket in MNG supports in-place upgrades.

[kdaula]

EKS Managed Node Groups support Bottlerocket.
further read: https://aws.amazon.com/blogs/containers/amazon-eks-adds-native-support-for-bottlerocket-in-managed-node-groups/