Add External Policies (AWS managed policy attachments) #7837

matt0x6F · 2019-10-28T21:37:33Z

This feature allows Managed Policies to be attached to Instance Group Roles. It will manage the full lifecycle of all Managed Policies attached to master, node, and bastion by first adding new managed policies followed by removing unspecified policies (policies not present in the config) in order to avoid service disruption.

k8s-ci-robot · 2019-10-28T21:37:36Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: login-issues@jira.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2019-10-28T21:37:40Z

Welcome @mattouille!

It looks like this is your first PR to kubernetes/kops 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kops has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2019-10-28T21:37:41Z

Hi @mattouille. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

matt0x6F · 2019-10-28T22:23:33Z

I signed it

chrisz100 · 2019-11-15T18:57:38Z

/ok-to-test

matt0x6F · 2019-11-25T19:15:30Z

I'm having a hard time figuring out how a call to AWS is resulting in a nil pointer dereference:

		response, err := cloud.IAM().ListAttachedRolePolicies(request)
		if awsErr, ok := err.(awserr.Error); ok {
			if awsErr.Code() == "NoSuchEntity" {
				return nil, nil
			}

			return nil, fmt.Errorf("error getting policies for role: %v", err)
		}

Any help would be appreciated!

matt0x6F · 2019-11-25T23:47:05Z

I did get my above question figured out. I didn't realize it was throwing a nil pointer dereference because of a missing mock method. That's all fixed now.

matt0x6F · 2019-11-26T18:56:36Z

/test pull-kops-e2e-kubernetes-aws

rifelpet · 2019-12-20T17:52:41Z

As mentioned on the call, this is the integration test that you could update to test this functionality:

https://github.com/kubernetes/kops/tree/master/tests/integration/update_cluster/complex

Add uses of the new field to the in-*.yaml files and the new tf resources to the .tf file.

I also noticed that RenderCloudFormation wasnt updated, if its simple enough we should add cloudformation support for this feature but if not we should at least document that its not supported in cloudformation.

johngmyers · 2019-12-23T17:22:32Z

/test pull-kops-verify-staticcheck

matt0x6F · 2019-12-26T23:50:32Z

As mentioned on the call, this is the integration test that you could update to test this functionality:

https://github.com/kubernetes/kops/tree/master/tests/integration/update_cluster/complex

Add uses of the new field to the in-*.yaml files and the new tf resources to the .tf file.

I also noticed that RenderCloudFormation wasnt updated, if its simple enough we should add cloudformation support for this feature but if not we should at least document that its not supported in cloudformation.

I will start working on those tests. Unfortunately I couldn't come up with a good way to do this in CloudFormation -- someone else might have a better idea of how to do it. I'll make it clear in the docs that this feature is not supported by CloudFormation.

rifelpet · 2019-12-27T01:18:41Z

Yeah I think cloudformation support might be a bit more involved. With terraform its just a different resource type (aws_iam_role_policy vs aws_iam_role_policy_attachment) but looking at the CloudFormation docs there isnt a dedicated resource for the attachment of a policy to a role, instead it is the ManagedPolicyArns property of the Role resource so we may need to modify the iamrole task instead.

kops/upup/pkg/fi/cloudup/awstasks/iamrole.go

Lines 220 to 223 in 8436a3e

    
           type cloudformationIAMRole struct { 
        
           	RoleName                 *string `json:"RoleName"` 
        
           	AssumeRolePolicyDocument map[string]interface{} 
        
           }

but I'm not a big fan of having logic split over different tasks depending on the target :(

one short term option in addition to documenting the limitation would be to return an error in RenderCloudFormation if someone is trying to use managed policies with target = cloudformation

matt0x6F · 2020-01-03T01:26:52Z

@rifelpet I think I may be doing something wrong and require a nudge in the right direction.

I'm getting this error:
--- FAIL: TestComplex (5.62s) integration_test.go:284: error running update cluster "complex.example.com": error closing target: error writing terraform data to output: error formatting HCL: At 247:3: Unknown token: 247:3 IDENT role

Originally I thought this was because I made some HCLv2 references, but even since changing them back the error has been consistent.

The "role" token it's referring to is required under aws_iam_role_policy_attachment.

mikesplain · 2020-01-03T17:38:08Z

I wonder if this could be a formatting issue @mattouille, see terraform-aws-modules/terraform-aws-vpc#267

I don't see in my looking but figured I'd post here too.

justinsb · 2020-01-05T15:50:55Z

docs/iam_roles.md

+    - aws:arn:iam:123456789000:policy:test-policy
+```
+
+Managed Policy attachments are treated declaritively. Any policies declared will be attached to the role, any policies not specified will be detached _after_ new policies are attached.


I'm a little confused as to whether this replaces the built-in kops policies; I think the answer is "no", but I think a sentence that explicitly spells this out would be great.

pkg/apis/kops/cluster.go

pkg/model/iam.go

pkg/resources/aws/aws.go

justinsb · 2020-01-05T16:02:30Z

upup/pkg/fi/cloudup/awstasks/iamrolepolicy.go

 	cloud := c.Cloud.(awsup.AWSCloud)

+	// Handle policy overrides
+	if e.PolicyOverrides != nil {


Maybe this should be a separate task (e.g. IAMRolePolicyAttachements) - or we should raise an error if we specify PolicyOverrides and PolicyDocument on the same task.

Creating a new task is a trickier thing normally, so I'm happy to do that separately...

I actually did try! I couldn't quite get it figured out.

justinsb · 2020-01-05T16:03:13Z

upup/pkg/fi/cloudup/awstasks/iamrolepolicy.go

+			return nil, fmt.Errorf("error getting policies for role: %v", err)
+		}
+
+		policies := make([]string, 0)


Nit: Normally it's fine just to do var policies []string - unless you're making a distinction between the empty list and nil (which is typically a bad idea, though I'm sure I've done it!)

justinsb · 2020-01-05T16:05:03Z

upup/pkg/fi/cloudup/awstasks/iamrolepolicy.go

@@ -121,6 +158,51 @@ func (_ *IAMRolePolicy) RenderAWS(t *awsup.AWSAPITarget, a, e, changes *IAMRoleP
 		return fmt.Errorf("error rendering PolicyDocument: %v", err)
 	}

+	// Handles the full lifecycle of Policy Overrides
+	if e.Managed {


I guess Managed is more "ManageAttachments"?

I was using this so I didn't have to check if the slice was nil and then check it's length. It just looked cleaner.

upup/pkg/fi/cloudup/awstasks/iamrolepolicy.go

justinsb · 2020-01-05T16:44:48Z

Some nits / thoughts - mostly about naming & structure. Figuring out the API field names are blockers because we can't change them, the task structure is internal so we can iterate on those and thus they aren't blockers.

So two blocking questions:

name of the field e.g. - externalPolicies
should we put this onto each IG - trade off making users repeat themselves for more power (policies per IG) and maybe simplicity (no need to think about roles, just directly specify it)

On the error, I ran make bazel-test (though go test should produce the same output) and we did log the invalid HCL. I scanned through the iam stuff and spotted this one:

E0105 16:16:49.191522   16052 hcl_printer.go:127] 251   
E0105 16:16:49.191532   16052 hcl_printer.go:127] 252   resource "aws_iam_role_policy_attachment" "node-policyoverride" {
E0105 16:16:49.191541   16052 hcl_printer.go:127] 253     name = 
E0105 16:16:49.191551   16052 hcl_printer.go:127] 254     role = "${aws_iam_role.nodes-complex-example-com.name}"
E0105 16:16:49.191560   16052 hcl_printer.go:127] 255     policy = 
E0105 16:16:49.191568   16052 hcl_printer.go:127] 256     policy_arn = "aws:arn:iam:123456789000:policy:test-policy"
E0105 16:16:49.191577   16052 hcl_printer.go:127] 257   }

Note the empty policy = line and name = line

By changing these to omitempty I was able to get the tests to almost pass (after running hack/update-expected.sh):

 type terraformIAMRolePolicy struct {
-       Name           *string            `json:"name"`
+       Name           *string            `json:"name,omitempty"`
        Role           *terraform.Literal `json:"role"`
-       PolicyDocument *terraform.Literal `json:"policy"`
+       PolicyDocument *terraform.Literal `json:"policy,omitempty"`
        PolicyArn      *string            `json:"policy_arn,omitempty"`
 }

It might be cleaner to have a separate terraformIAMRolePolicyAttachment struct rather than reusing the same structure - the idea of this being a separate task might actually make stuff simpler :-)

The tests almost passed because I guess that complex uses CloudFormation, and then I hit:

W0105 16:39:49.118646   17836 executor.go:128] error running task "IAMRolePolicy/master-policyoverride" (29s remaining to succeed): CloudFormation not supported for use with PolicyOverrides.
W0105 16:39:49.118670   17836 executor.go:128] error running task "IAMRolePolicy/node-policyoverride" (29s remaining to succeed): CloudFormation not supported for use with PolicyOverrides.

Two options there - we can split complex into complex and complex_cloudformation and just add it to complex, or we can figure out how to support it for CloudFormation also. I can take a look at the latter...

justinsb · 2020-01-05T16:51:58Z

Oh - we already have the same problem in the load balancer task, so I just copied the hack-around: https://github.com/justinsb/kops/pull/new/tests_for_7837

(In particular e90b40d )

justinsb · 2020-01-05T17:19:46Z

On the second "blocker" - I've changed my mind - I don't think we should define this per IG, because we'd then also need to split the profile name per IG, and we do have the ability to override the profile already, and the interactions would get complicated. So what you've got here is great, and we should just make sure we're happy with the name of the field in the API!

matt0x6F · 2020-02-03T05:46:22Z

Alright, so I've got some updates.

Naming
ExternalPolicies sounds better and is absolutely more accurate. AttachedPolicies is good once the policy has been attached however... in the code I also handle the policy before it's attached and there it's actually called a ManagedPolicy (think of it in terms of like user-managed policy from the AWS end) by AWS.

tests/integration/update_cluster/externalpolicies/in-legacy-v1alpha2.yaml

matt0x6F · 2020-02-04T07:43:44Z

/retest

matt0x6F · 2020-02-05T18:40:03Z

/retest

rifelpet · 2020-02-05T23:13:09Z

tests/integration/update_cluster/externalpolicies/in-v1alpha2.yaml

+    master:
+      - aws:arn:iam:123456789000:policy:test-policy
+    bastion:
+      - aws:arn:iam:123456789000:policy:test-policy


I dont see this policy arn referenced in the terraform output at all.

I'd expect to see something like:

resource "aws_iam_role_policy_attachment" "node-externalpolicy-test-policy" { role = "${aws_iam_role.nodes-complex-example-com.name}" policy_arn = "aws:arn:iam:123456789000:policy:test-policy" }

somewhere in kubernetes.tf below.

ah, I misunderstood how this worked. I'll fix that.

rifelpet · 2020-02-05T23:15:50Z

upup/pkg/fi/cloudup/awstasks/iamrolepolicy.go

+				Role:      e.Role.TerraformLink(),
+				PolicyArn: s(policy),
+			}
+			return t.RenderResource("aws_iam_role_policy_attachment", *e.Name, tf)


This might be related to the resource not showing up in kubernetes.tf, but I think we dont want to return here. We want to render both this aws_iam_role_policy_attachment and the aws_iam_role_policy resource below. I think we may also need to make the resource name (currently *e.Name unique so that multiple aws_iam_role_policy_attachment resources dont conflict. Perhaps the name can include a normalized version of the policy arn?

Let me know if this is working the way it should now. My understanding of how this function was called was wrong.

upup/pkg/fi/cloudup/awstasks/iamrolepolicy.go

tests/integration/update_cluster/externalpolicies/kubernetes.tf

rifelpet · 2020-02-16T02:36:11Z

@mattouille can you squash these commits? I think thats the last blocker to getting this merged (and I remember we have a thread in slack explaining how to do that)

matt0x6F · 2020-02-17T05:53:08Z

@mattouille can you squash these commits? I think thats the last blocker to getting this merged (and I remember we have a thread in slack explaining how to do that)

Done! Sorry for the delay.

rifelpet · 2020-02-17T14:48:40Z

Perfect, thanks!
/approve
/lgtm

k8s-ci-robot · 2020-02-17T14:49:08Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mattouille, rifelpet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [rifelpet]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

This feature was renamed during its [development](kubernetes#7837) and a remnant of that original name was in the docs.

k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Oct 28, 2019

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Oct 28, 2019

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 28, 2019

k8s-ci-robot requested review from granular-ryanbonham and zetaab October 28, 2019 21:38

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Oct 28, 2019

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Nov 15, 2019

justinsb reviewed Jan 5, 2020

View reviewed changes

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 3, 2020

matt0x6F force-pushed the policy-overrides branch from 09dea2e to 90289a4 Compare February 3, 2020 22:09

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 3, 2020

matt0x6F force-pushed the policy-overrides branch from 90289a4 to 28a9c55 Compare February 3, 2020 23:44

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 3, 2020

rifelpet reviewed Feb 3, 2020

View reviewed changes

tests/integration/update_cluster/externalpolicies/in-legacy-v1alpha2.yaml Outdated Show resolved Hide resolved

matt0x6F changed the title ~~Add Policy Overrides (Managed Policies)~~ Add External Policies (AWS managed policy attachments) Feb 3, 2020

rifelpet reviewed Feb 5, 2020

View reviewed changes

rifelpet reviewed Feb 6, 2020

View reviewed changes

upup/pkg/fi/cloudup/awstasks/iamrolepolicy.go Show resolved Hide resolved

tests/integration/update_cluster/externalpolicies/kubernetes.tf Outdated Show resolved Hide resolved

matt0x6F force-pushed the policy-overrides branch from 02708c2 to 157f04c Compare February 17, 2020 05:52

Add External Policies (AWS managed policy attachments)

f025ff0

matt0x6F force-pushed the policy-overrides branch from 157f04c to f025ff0 Compare February 17, 2020 05:55

k8s-ci-robot assigned rifelpet Feb 17, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 17, 2020

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 17, 2020

k8s-ci-robot merged commit b6c849c into kubernetes:master Feb 17, 2020

k8s-ci-robot added this to the v1.18 milestone Feb 17, 2020

This was referenced Feb 24, 2020

Unable to attach external IAM Policy #8621

Closed

Attach an existing managed policy to node.<cluster> IAM role. #1645

Closed

hakman mentioned this pull request Apr 27, 2020

kops update cluster error for externalPolicies #9000

Closed

philnielsen added a commit to philnielsen/kops that referenced this pull request Feb 22, 2023

[docs] Fix External IAM Policy Docs

d978c30

This feature was renamed during its [development](kubernetes#7837) and a remnant of that original name was in the docs.

philnielsen mentioned this pull request Feb 22, 2023

[docs] Fix External IAM Policy Docs #15177

Merged

anthonyhaussman pushed a commit to anthonyhaussman/kops that referenced this pull request Mar 7, 2023

[docs] Fix External IAM Policy Docs

1b6bc8e

This feature was renamed during its [development](kubernetes#7837) and a remnant of that original name was in the docs.

anthonyhaussman pushed a commit to anthonyhaussman/kops that referenced this pull request Mar 14, 2023

[docs] Fix External IAM Policy Docs

c2dea79

This feature was renamed during its [development](kubernetes#7837) and a remnant of that original name was in the docs.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add External Policies (AWS managed policy attachments) #7837

Add External Policies (AWS managed policy attachments) #7837

matt0x6F commented Oct 28, 2019 •

edited

Loading

k8s-ci-robot commented Oct 28, 2019

k8s-ci-robot commented Oct 28, 2019

k8s-ci-robot commented Oct 28, 2019

matt0x6F commented Oct 28, 2019

chrisz100 commented Nov 15, 2019

matt0x6F commented Nov 25, 2019

matt0x6F commented Nov 25, 2019

matt0x6F commented Nov 26, 2019

rifelpet commented Dec 20, 2019

johngmyers commented Dec 23, 2019

matt0x6F commented Dec 26, 2019

rifelpet commented Dec 27, 2019 •

edited

Loading

matt0x6F commented Jan 3, 2020

mikesplain commented Jan 3, 2020 •

edited

Loading

justinsb Jan 5, 2020

justinsb Jan 5, 2020

matt0x6F Feb 3, 2020

justinsb Jan 5, 2020

justinsb Jan 5, 2020

matt0x6F Feb 3, 2020

justinsb commented Jan 5, 2020

justinsb commented Jan 5, 2020 •

edited

Loading

justinsb commented Jan 5, 2020

matt0x6F commented Feb 3, 2020 •

edited

Loading

matt0x6F commented Feb 4, 2020

matt0x6F commented Feb 5, 2020

rifelpet Feb 5, 2020

matt0x6F Feb 6, 2020

rifelpet Feb 5, 2020

matt0x6F Feb 6, 2020

rifelpet commented Feb 16, 2020

matt0x6F commented Feb 17, 2020

rifelpet commented Feb 17, 2020

k8s-ci-robot commented Feb 17, 2020

Add External Policies (AWS managed policy attachments) #7837

Add External Policies (AWS managed policy attachments) #7837

Conversation

matt0x6F commented Oct 28, 2019 • edited Loading

k8s-ci-robot commented Oct 28, 2019

k8s-ci-robot commented Oct 28, 2019

k8s-ci-robot commented Oct 28, 2019

matt0x6F commented Oct 28, 2019

chrisz100 commented Nov 15, 2019

matt0x6F commented Nov 25, 2019

matt0x6F commented Nov 25, 2019

matt0x6F commented Nov 26, 2019

rifelpet commented Dec 20, 2019

johngmyers commented Dec 23, 2019

matt0x6F commented Dec 26, 2019

rifelpet commented Dec 27, 2019 • edited Loading

matt0x6F commented Jan 3, 2020

mikesplain commented Jan 3, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

justinsb commented Jan 5, 2020

justinsb commented Jan 5, 2020 • edited Loading

justinsb commented Jan 5, 2020

matt0x6F commented Feb 3, 2020 • edited Loading

matt0x6F commented Feb 4, 2020

matt0x6F commented Feb 5, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rifelpet commented Feb 16, 2020

matt0x6F commented Feb 17, 2020

rifelpet commented Feb 17, 2020

k8s-ci-robot commented Feb 17, 2020

matt0x6F commented Oct 28, 2019 •

edited

Loading

rifelpet commented Dec 27, 2019 •

edited

Loading

mikesplain commented Jan 3, 2020 •

edited

Loading

justinsb commented Jan 5, 2020 •

edited

Loading

matt0x6F commented Feb 3, 2020 •

edited

Loading