Improvements around setting up IAM (also for multiple clusters) #573

StevenACoffman · 2019-02-23T20:16:03Z

Currently, when I launch a cluster using eksctl, I need to have the access rights to create IAM policies, profiles, and roles.

I believe it is possible to have an administrator create these beforehand and then allow users without IAM creation privileges to launch clusters using these pre-existing policies. It would be nice if this was better documented.

Our use case is that we have a pre-existing vpc, and we want developers to be able to spin up an ephemeral eks clusters inside it, but not be able to accidentally modify IAM permissions in a way that leaves us vulnerable.

Related: #508

StevenACoffman · 2019-02-23T20:23:39Z

The example in #508 shows:

apiVersion: eksctl.io/v1alpha4
kind: ClusterConfig
metadata:
  name: test-cluster-c-1
  region: eu-north-1

vpc:
  securityGroup: "sg-0f2ae54eb340e8191"
  sharedNodeSecurityGroup: "sg-02a47cee779b317a7"
  subnets:
    Private:
      eu-north-1c:
          id: "subnet-065fe8f12d0910d06"
          cidr: "10.1.128.0/19"
      eu-north-1b:
          id: "subnet-06ebc3649c1321fc5"
          cidr: "10.1.96.0/19"

iam: 
  serviceRoleARN: "arn:aws:iam::123:role/eksctl-test-cluster-a-3-cluster-ServiceRole-5YEWP7CFA24K"

nodeGroups:
  - name: ng2-private
    instanceType: m5.large
    desiredCapacity: 1
    privateNetworking: true
    securityGroups:
      withShared: true
      withLocal: false
      attachIDs: [sg-0b85ff315ea644478]
    iam:
      instanceProfileARN: "arn:aws:iam::123:instance-profile/eksctl-test-cluster-a-3-nodegroup-ng2-private-NodeInstanceProfile-Y4YKHLNINMXC"
      instanceRoleARN: "arn:aws:iam::123:role/eksctl-test-cluster-a-3-nodegroup-NodeInstanceRole-DNGMQTQHQHBJ"

After I create an eks cluster using eksctl, I can see the service role created has attached these managed policies:

AmazonEKSClusterPolicy
AmazonEKSServicePolicy

And these two inline policies:

PolicyNLB
PolicyCloudWatchMetrics

PolicyNLB has this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "elasticloadbalancing:*",
                "ec2:CreateSecurityGroup",
                "ec2:Describe*"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

while PolicyCloudWatchMetrics has this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "cloudwatch:PutMetricData"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

It has a trust policy of:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "eks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Creating a role that matches these would presumably work for the serviceRoleARN

StevenACoffman · 2019-02-23T20:30:06Z

The eksctl created nodegroup IAM role gets these managed policies:

AmazonEKSWorkerNodePolicy
AmazonEC2ContainerRegistryPowerUser or AmazonEC2ContainerRegistryReadOnly (depending on imageBuilder setting in config file.
AmazonEKS_CNI_Policy

It has three inline policies (cluster specific names truncated), depending on autoScaler and externalDNS settings:

PolicyAutoScaling
PolicyExternalDNSChangeSet
PolicyExternalDNSHostedZones

PolicyAutoScaling:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeLaunchConfigurations",
                "autoscaling:DescribeTags",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

PolicyExternalDNSChangeSet has:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "route53:ChangeResourceRecordSets"
            ],
            "Resource": "arn:aws:route53:::hostedzone/*",
            "Effect": "Allow"
        }
    ]
}

PolicyExternalDNSHostedZones has:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "route53:ListHostedZones",
                "route53:ListResourceRecordSets"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

Presumably a role that was created in advance with these could be set for the instanceRoleARN setting.

StevenACoffman · 2019-02-23T20:43:50Z

So AFAICT, when you create a role, you get a default instance profile with the same name, so if you created a role as above, I think you can do this:

    iam:
      instanceProfileARN: "arn:aws:iam::111111:instance-profile/eks-nodes-base-role"
      instanceRoleARN: "arn:aws:iam::111111:role/eks-nodes-base-role"

I am not as acquainted with instance profiles, so if that doesn't work, I'm documenting my eksctl created instance profile while truncating the cluster specific names/ARNs, so presumably a similar, permanent, shared instance profile could be created for re-use by specifying the instanceProfileARN to match it in the config file.

$ aws iam get-instance-profile --instance-profile-name NodeInstanceProfile
{
    "InstanceProfile": {
        "Path": "/",
        "InstanceProfileName": "NodeInstanceProfile",
        "InstanceProfileId": "AIPAJYEFNR6WC4XFCA4HG",
        "Arn": "arn:aws:iam::11111:instance-profile/NodeInstanceProfile",
        "CreateDate": "2019-02-23T18:06:53Z",
        "Roles": [
            {
                "Path": "/",
                "RoleName": "NodeInstanceRole",
                "RoleId": "AROAJUB5X2XIGHLBUHRGE",
                "Arn": "arn:aws:iam::111111111:role/NodeInstanceRole",
                "CreateDate": "2019-02-23T18:06:39Z",
                "AssumeRolePolicyDocument": {
                    "Version": "2012-10-17",
                    "Statement": [
                        {
                            "Effect": "Allow",
                            "Principal": {
                                "Service": "ec2.amazonaws.com"
                            },
                            "Action": "sts:AssumeRole"
                        }
                    ]
                }
            }
        ]
    }
}

StevenACoffman · 2019-02-23T21:32:33Z

So if someone is trying to automate this in a bash script, while there are some minor additional inline policy privileges to add, this looks like it will work pretty well with slight modification

StevenACoffman · 2019-02-23T22:50:41Z

Here's what worked for me for pre-existing shared vpc and pre-existing, shared iam roles:

apiVersion: eksctl.io/v1alpha4
kind: ClusterConfig
metadata:
  name: my-test
  region: us-east-1
vpc:
  id: "vpc-11111"
  cidr: "152.28.0.0/16"
  subnets:
    private:
      us-east-1d:
          id: "subnet-1111"
          cidr: "152.28.152.0/21"
      us-east-1c:
          id: "subnet-11112"
          cidr: "152.28.144.0/21"
      us-east-1a:
          id: "subnet-11113"
          cidr: "152.28.136.0/21"
iam: 
  serviceRoleARN: "arn:aws:iam::11111:role/eks-base-service-role"
nodeGroups:
  - name: my-test-m5-private
    labels: {pool: my-test-m5-private}
    instanceType: m5.large
    desiredCapacity: 3
    minSize: 1
    maxSize: 15
    volumeSize: 50
    volumeType: gp2
    iam:
      instanceProfileARN: "arn:aws:iam::11111:instance-profile/eks-nodes-base-role"
      instanceRoleARN: "arn:aws:iam::1111:role/eks-nodes-base-role"
    privateNetworking: true
    securityGroups:
      withShared: true
      withLocal: true
      attachIDs: ['sg-11111', 'sg-11112']
    allowSSH: true
    sshPublicKeyName: 'my-instance-key'
    tags:
      'environment:basedomain': 'example.org'

errordeveloper · 2019-02-25T09:12:19Z

xref #122 #204

mumoshu · 2019-03-06T00:40:06Z

Hey!

Thanks for your efforts. This definitely helps users who wants to create cluster without huge IAM permission(that allows privilege escalation).

I'm going a bit off topic, but for simplicity, I guess we should be able to omit instanceRoleARN when instanceProfileARN is specified:

    iam:
      instanceProfileARN: "arn:aws:iam::11111:instance-profile/eks-nodes-base-role"

If it doesn't work, I'd just enhance eksctl to automatically detect the corresponding roleARN for the specified profile.

mumoshu · 2019-03-06T00:43:16Z

Also - I once considered if it would be nice to enhance eksctl to allow creating an instance profile according to the cluster config, something like:

$ eksctl utils create-instance-profile -f cluster.yaml --nodegroup my-test-m5-private
# Creates `eksctl-test-cluster-a-3-nodegroup-ng2-private-NodeInstanceProfile-Y4YKHLNINMXC` and prints the ARN of it

errordeveloper · 2019-03-06T00:56:48Z

Yusuke, I think we should consider 'eksctl create iam --cinfig-file' really, but the question is which stack it would be part of, what do you think? Maybe it should a standalone shared IAM stack that multiple clusters can import? It is kind of similar with VPC.

…

On Wed, 6 Mar 2019, 12:43 am KUOKA Yusuke, ***@***.***> wrote: Also - I once considered if it would be nice to enhance eksctl to allow creating an instance profile according to the cluster config, something like: $ eksctl utils create-instance-profile -f cluster.yaml --nodegroup my-test-m5-private # Creates `eksctl-test-cluster-a-3-nodegroup-ng2-private-NodeInstanceProfile-Y4YKHLNINMXC` and prints the ARN of it — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#573 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPWS81occNmfWObd4auBBGxFnsCFrMQks5vTw8lgaJpZM4bOI1J> .

mumoshu · 2019-03-06T03:15:47Z

Maybe it should a standalone shared IAM stack that multiple clusters can import?

A shared IAM stack per region? Sounds good!

Would it implies that we'd have a dedicated config for IAM?

# prof.yaml
apiVersion: eksctl.io/v1alpha4
kind: InstanceProfileConfig
metadata:
  name: my-test
  region: us-east-1
withAddonPolicies:
  imageBuilder: true
  autoScaler: true

Also, I was thinking if it can be a good starting point to just provide eksctl utils print-iam-policy --config-file prof.yaml? So that the user can use it to print the iam policy json that can be consumed by awscli, terraform, and cloudformation.

It is kind of similar with VPC.

Could be!

Quick thoughts - Maybe we could have a dedicated VPCConfig object that can be consumed by eksctl create vpc --config-file or eksctl utils print-vpc-stack-template --config-file. Would it share the stack with the iam related things, or not? I have no strong opinion on it yet.

kalbir · 2019-12-03T17:05:29Z

Just had a user conversation at reinvent about this 👍

pwsiegel · 2019-12-24T16:30:09Z

Our use case is that we have a pre-existing vpc, and we want developers to be able to spin up an ephemeral eks clusters inside it, but not be able to accidentally modify IAM permissions in a way that leaves us vulnerable.

I am also in this exact position, and I've been going back and forth with my AWS admin for days trying to figure out how I can have permissions to launch clusters but not hurt anything else. So this thread has been really helpful, and I offer a 👍 if there are features under consideration which would make it easier.

aclevername · 2021-09-21T13:35:31Z

Closing due to staleness. Feel free to re-open 😄

Push amazonlinux target to Docker Hub

StevenACoffman mentioned this issue Feb 23, 2019

more examples of config files #508

Closed

errordeveloper added kind/docs User documentation area/config-file area/aws-iam labels Feb 25, 2019

martina-if added the priority/important-longterm Important over the long term, but may not be currently staffed and/or may require multiple releases label Sep 15, 2020

michaelbeaumont changed the title ~~Document IAM so teams without IAM writes can still launch clusters~~ Improvements around setting up IAM (also for multiple clusters) Nov 16, 2020

aclevername closed this as completed Sep 21, 2021

torredil pushed a commit to torredil/eksctl that referenced this issue May 20, 2022

Merge pull request eksctl-io#573 from wongma7/al2

e7a3e94

Push amazonlinux target to Docker Hub

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improvements around setting up IAM (also for multiple clusters) #573

Improvements around setting up IAM (also for multiple clusters) #573

StevenACoffman commented Feb 23, 2019 •

edited

Loading

StevenACoffman commented Feb 23, 2019 •

edited

Loading

StevenACoffman commented Feb 23, 2019 •

edited

Loading

StevenACoffman commented Feb 23, 2019 •

edited

Loading

StevenACoffman commented Feb 23, 2019 •

edited

Loading

StevenACoffman commented Feb 23, 2019 •

edited

Loading

errordeveloper commented Feb 25, 2019

mumoshu commented Mar 6, 2019

mumoshu commented Mar 6, 2019

errordeveloper commented Mar 6, 2019 via email

mumoshu commented Mar 6, 2019

kalbir commented Dec 3, 2019

pwsiegel commented Dec 24, 2019

aclevername commented Sep 21, 2021

Improvements around setting up IAM (also for multiple clusters) #573

Improvements around setting up IAM (also for multiple clusters) #573

Comments

StevenACoffman commented Feb 23, 2019 • edited Loading

StevenACoffman commented Feb 23, 2019 • edited Loading

StevenACoffman commented Feb 23, 2019 • edited Loading

StevenACoffman commented Feb 23, 2019 • edited Loading

StevenACoffman commented Feb 23, 2019 • edited Loading

StevenACoffman commented Feb 23, 2019 • edited Loading

errordeveloper commented Feb 25, 2019

mumoshu commented Mar 6, 2019

mumoshu commented Mar 6, 2019

errordeveloper commented Mar 6, 2019 via email

mumoshu commented Mar 6, 2019

kalbir commented Dec 3, 2019

pwsiegel commented Dec 24, 2019

aclevername commented Sep 21, 2021

StevenACoffman commented Feb 23, 2019 •

edited

Loading

StevenACoffman commented Feb 23, 2019 •

edited

Loading

StevenACoffman commented Feb 23, 2019 •

edited

Loading

StevenACoffman commented Feb 23, 2019 •

edited

Loading

StevenACoffman commented Feb 23, 2019 •

edited

Loading

StevenACoffman commented Feb 23, 2019 •

edited

Loading