Control instance userdata under provisioner field

[amitde69]

Tell us about your request
Currently we are working with cluster autoscaler in production, our only blocker from taking Karpenter to production is controlling the instance userdata when creating the provisioner resource.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
It seems that the current implementation does not support custom userdata unless you’re using a custom launch-template.
Using a launch-template is not optimal because that defeats some of the purposes of Karpenter in terms of instance types/securitygroup/az balancing.

Are you currently working around this issue?
Currently its a blocker for us for using the tool
In production.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[ellistarn]

Hey @amitde69, we'd love to understand what about userdata you need. It's a challenging space that intermingles kubelet configuration and instance configuration. The more clear you can be about the use case, the better can find a solution that works cleanly with all use cases.

[amitde69]

we are running on AWS and we want to avoid using AutoScalingGroups / Launch Templates entirely.

we are using chef/custom scripts to bootstrap all EKS nodes to install some prerequisites before joining the nodes into the cluster, so we have custom userdata configure for all the EC2 EKS nodes.

if we can configure the userdata in Karpenter objects, it will allow us to not relay on AutoScalingGroups / Launch Templates entirely and let Karpenter do all the thinking.

[ellistarn]

Is it possible to install these prerequisites using a daemonset with root permissions?

[amitde69]

Not really because we are installing SSM agent on the nodes at start up, if the node fails to join the cluster then we wont have SSM access to debug it
but thats just one example, eventually decoupling Karpenter from the launch template sounds like the best option to me, to be able to configure all the instance options in the provisioner and let karpeneter to the rest for you.

[adjain131995]

Is there any plan of integrating the feature of providing user data in the provisions themselves instead of going ahead with the Launch templates?
This will solve multiple github issues and save us a lot of hassle of going via launch templates.

[ellistarn]

https://karpenter.sh/v0.13.1/aws/user-data/