Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update the cross-account example with working policy #9019

Merged
merged 3 commits into from
May 1, 2020
Merged

Update the cross-account example with working policy #9019

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ccb0276
Update the cross-account example with working policy
Apr 28, 2020
e9114d8
Update docs/state.md
olemarkus Apr 30, 2020
df81a76
Update docs/state.md
olemarkus Apr 30, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 12 additions & 7 deletions docs/state.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,11 +103,10 @@ The state store can easily be moved to a different s3 bucket. The steps for a si

Repeat for each cluster needing to be moved.

#### Cross Account State-store (AWS S3)
#### Cross Account State-store

There are situations in which the entity executing kops to create the cluster is not in the same account as the owner of the state store bucket. In this case, you must explicitly grant the permission: `s3:getBucketLocation` to the ARN that is running kops.

You can use the following policy to guide your implementation:
Many enterprises prefer to run many AWS accounts. In these setups, having a shared cross-account S3 bucket for state may make inventory and management easier.
Consider the S3 bucket living in Account B and the kops cluster living in Account A. In order to achieve this, you first need to let Account A access the s3 bucket. This is done by adding the following _bucket policy_ on the S3 bucket:

```
{
Expand All @@ -117,20 +116,26 @@ You can use the following policy to guide your implementation:
{
"Sid": "123",
"Action": [
"s3:GetBucketLocation"
"s3:*"
],
"Effect": "Allow",
"Resource": "arn:aws:s3:::state-store-bucket",
"Resource": [
"arn:aws:s3:::<state-store-bucket>",
"arn:aws:s3:::<state-store-bucket>/*",
}
"Principal": {
"AWS": [
"arn:aws:iam::123456789:user/kopsuser"
"arn:aws:iam::<remote-account>:root"
olemarkus marked this conversation as resolved.
Show resolved Hide resolved
]
}
}
]
}
```

Kops will then use that bucket as if it was in the remote account, including creating appropriate IAM policies that limits nodes from doing bad things.
Note that any user/role with full S3 access will be able to delete any cluster from the state store, but may not delete any instances or other things outside of S3.
Copy link
Member

@zetaab zetaab Apr 30, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are using policy like this

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowRootAndHomeListingOfCompanyBucket",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::<state bucket>",
            "Condition": {
                "StringEquals": {
                    "s3:delimiter": "/",
                    "s3:prefix": ""
                }
            }
        },
        {
            "Sid": "AllowListingOfUserFolder",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::<state bucket>",
            "Condition": {
                "StringLike": {
                    "s3:prefix": "${aws:username}/*"
                }
            }
        },
        {
            "Sid": "AllowAllS3ActionsInUserFolder",
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::<state bucket>/${aws:username}/*"
        }
    ]
}

So what we do:

  1. create new IAM group and attach that policy there (this needs to be done only once)
  2. When we need new cluster:
    • create new IAM user which matches to clustername, example: mycluster.k8s.local
    • attach IAM group to user
    • use IAM user credentials to authenticate against S3 bucket (env vars S3_ACCESS_KEY_ID and S3_SECRET_ACCESS_KEY needs to be defined)

After that each cluster has access to only own state store folder. It makes it possible to also have "root" account which has access to all clusters if needed for ops purposes, but note that all modifying kops commands should be executed using those IAM user credentials.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is in addition to the content in the PR, right? So the group is done once per account, and the IAM user is once per cluser? You share this user among everyone that can operate that particular cluster?


## Digital Ocean (do://)

DigitalOcean storage is configured as a flavor of a S3 store.
Expand Down