[EKS] Custom AMI for managed worker nodes

[cshivashankar]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
As of date, managed nodes can only run only on predefined options of AMI.
I see only 2 options at present. Provide option of choosing custom AMI in managed nodes.

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now?
A custom AMI, based out of "https://github.com/awslabs/amazon-eks-ami " is being used in EKS cluster which I am managing. Even though there are little changes from the "aws-eks-ami" those changes are critical.
This can be overcome by configuring the nodes after they are alive and present in cluster.But this adds overhead of initial configuration and periodic checks for new nodes.
What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem.
Its not possible to move to managed node groups.

Are you currently working around this issue?
Yes, I am not using managed node groups.

Additional context
None

Attachments
None

[Pratima]

We would love to have a BYO AMI feature for EKS (similar to ECS). We need to get our compute CIS and docker benchmark compliant. AND we would need to get SSM connected to the AMI in order to access the instances.

[tabern]

This should be solved by #585

[cshivashankar]

Thanks @tabern , Any ETA for shipping this feature?

[mikestef9]

Hi @cshivashankar and @Pratima

Do you build your custom AMIs based off the EKS Optimized AMI template?
https://github.com/awslabs/amazon-eks-ami/blob/master/files/bootstrap.sh

How would you expect worker node bootstrapping to work?
Today, EKS Managed Nodes set EC2 user data that includes a command to call the bootstrap script with cluster name and certificate authority

This is required for a node to join the cluster. Would you want managed nodes to continue to include this user data and merge with any other user data you have? Or you would rather have full control over what user data is passed in that is need for your custom AMI based worker node to join the cluster?

[cshivashankar]

Hi @mikestef9 ,

Yes I do consume files based on EKS optimized AMI template.
User data is used for joining the cluster .

I would say managed nodes should continue including the basic user data so that basic functionalities like joining the cluster are still automated.
However Either through additional user data or some option of running custom configurations will be helpful. It might be installing required templates and customization or it could be some other configuration in the server itself.
An oversimplified example will be to install the Zabbix agent and connect to the central Zabbix server at boot or maybe a customized daemon configs, however, there might be other complicated cases.

[bneelima84]

Related #596

[MeghanaSrinath]

We also have a case wherein we would need an option to specify the AMI for the managed worker nodes. So basically, we use the base AMIs, encrypt them with KMS encryption (also encrypt the volumes that gets attached to the instances) and then specify this encrypted image (on the fly) for the managed nodes that we bring up in EKS. Our application requires the encryption at all the stages. Hence it'd be great if an option to specify the AMIs can be provided while bringing up managed worker nodes in AWS EKS.

[MeghanaSrinath]

@mikestef9 Is there any ETA on this feature?