-
Notifications
You must be signed in to change notification settings - Fork 515
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Instances on k8s 1.26 image don't show up in SSM #3544
Comments
Thanks for the report @Shershebnev ! Are any errors reported on the instance via the console using "Get system log", "Get instance screenshot", or "EC2 serial console"? I suspect that some additional user data and/or roles may be needed for the instance. I'm looking back over the changelogs to confirm this suspicion and explain the differences between k8s versions. |
Log is empty, but screenshot shows some encryption error
i-0ce93f121a3bf8b3a.log This turned into quite a long post, sorry about that. In a nutshell:
Hope this is helpful :) |
Related to #3525 (comment) I think we might need to add in EC2 Describe Images access to the IAM Role policies attached in https://github.com/aws-samples/containers-blog-maelstrom/blob/ee8e18c0bb170f625b86a59dfc0605e9c98cdee3/bottlerocket-images-cache/ebs-snapshot-instance.yaml#L44. For example, I have
as the policy. This might be the missing piece. Can you try this and see if it resolves the issues with 1.26 coming up? If so, we can try and get this other repo updated to cover this permissions addition. |
I've tried with |
Sounds great! Glad we got you sorted! |
Probably related to #3525
I'm trying to follow this article https://aws.amazon.com/blogs/containers/reduce-container-startup-time-on-amazon-eks-with-bottlerocket-data-volume/ (code - https://github.com/aws-samples/containers-blog-maelstrom/blob/main/bottlerocket-images-cache/snapshot.sh), they have aws-k8s-1.24 image there as default -
/aws/service/bottlerocket/aws-k8s-1.24/x86_64/latest/image_id
, however given that my EKS cluster is on 1.26, I've tried to change the bottlerocket image to 1.26 respectively. With 1.24 everything works fine, instance appears in SSM almost immediately, but when switching to 1.26 it just never shows up in SSM (I've added also an instance name into CloudFormation stack as I've seen in some old issue that instances without names are skipped by SSM). I've also tried 1.25 - works fine, but 1.27 also never shows upImage I'm using:
bottlerocket-aws-k8s-1.26-x86_64-v1.15.1-264e294c
(latest for 1.26 currently)bottlerocket-aws-k8s-1.26-x86_64-v1.14.3-764e37e4
(tried an older one as well)What I expected to happen:
Instance shows up in SSM
What actually happened:
It never shows up
How to reproduce the problem:
Original CF stack - https://github.com/aws-samples/containers-blog-maelstrom/blob/main/bottlerocket-images-cache/ebs-snapshot-instance.yaml
CF stack with my modifications (instance name and ebs volume size increased)
When switching
1.26
to1.24
or1.25
everything works fine, but1.27
also doesn't show upJust in case, I've also tried cpu-only instances as well as NVidia version of the image
Since my purpose is to just use the data volume for docker images, should I just use 1.24/1.25 images?
The text was updated successfully, but these errors were encountered: