Cluster autoscaler addon permission problem

[ftoresan]

2. What kops version are you running? use kops version
   1.8.0-beta.1

3. What Kubernetes version are you running? use kubectl version
   1.8.3

4. What cloud provider are you using?
   AWS

5. What commands did you execute (Please provide cluster manifest kops get --name my.example.com, if available) and what happened after commands executed?
   Installed the cluster add-on while having insufficient CPU on current nodes.

6. What you expected to happen:
   A new node to be created.

7. How can we to reproduce it (as minimally and precisely as possible):
   Create a cluster with the default instance group having min nodes = 2 and max nodes = 6. Apply same configuration to the autoscaler addon while having an insufficient CPU to schedule current pods.

The instance group applied the desired number to the ASG correcty, and the addon failed to do so. In its log an error was logged saying that the EC2 role had insufficient permission to set the desired capacity.

My workaround: I added the permission in the IAM role without the condition that checks if the "KubernetesCluster" tag with the cluster name is present. It seems that the auto scaler (maybe because it is running in a pod?) does not match the condition, while the instance group does.

[chrislovecnm]

What IAM perms did you add?

[chrislovecnm]

/cc @KashifSaadat

[chrislovecnm]

@arun-gupta you got an IAM expert that can comment? Are limiting the master role for the autoscaler to only have access to ASG's that are tagged with KubernetesCluster, and this is not working.

[chrislovecnm]

If someone wants to comment out this code https://github.com/kubernetes/kops/blob/master/pkg/model/iam/iam_builder.go#L664 and just use the regular resource.

Resource: resource,

That would fix it. I do not have autoscaler setup, so testing would be long for me.

[ftoresan]

@chrislovecnm I added the "autoscaling:SetDesiredCapacity". This permission was already added in the policy kopsK8sASMasterPermsTaggedResources, but it was not having effect, probably due to the condition I mentioned: "ec2:ResourceTag/KubernetesCluster": "platform.k8s.local".

I added it to the policy kopsK8sASMasterPermsAllResources, that originally has only read-only permissions.

[chrislovecnm]

/area security

Can you PR? I will review and git it into the 1.8 kops release

[KashifSaadat]

Hi @ftoresan, thanks for raising an issue!

According to the AWS Auto Scaling IAM Documentation, we should actually be using autoscaling:ResourceTag/key instead of the ec2: key.

Please would you be able to test this case and raise a PR? The relevant locations to update code (and tests) are:

• https://github.com/kubernetes/kops/blob/master/pkg/model/iam/iam_builder.go#L664
• https://github.com/kubernetes/kops/tree/master/pkg/model/iam/tests/ (the files here ending in _strict.json)

[ftoresan]

Ok @KashifSaadat , I'll make the change and create the PR, thanks for the reply!

[KashifSaadat]

That would be great, thank you! Let me know if you have any questions. :)