GCE: Create cloud-provider roles and bindings via addons #59686

nicksardo · 2018-02-10T01:39:41Z

What this PR does / why we need it:
This removes the cloud-provider role and role binding from the rbac boostrapper and replaces it with a policy applied via addon mgr. This also creates a new clusterrole allowing the service account to create events for any namespace.

Special notes for your reviewer:
/assign @bowei @timstclair
/cc timstclair

Release note:

GCE: A role and clusterrole will now be provided with GCE/GKE for allowing the cloud-provider to post warning events on all services and watching configmaps in the kube-system namespace.

nicksardo · 2018-02-10T01:42:38Z

/cc @MrHohn

bowei · 2018-02-10T01:54:38Z

/approve no-issue

jhorwit2 · 2018-02-11T03:46:13Z

cluster/gce/addons/loadbalancing/cloud-provider-role.yaml

+  - patch
+  - update
+  - list
+  - watch


Why did the privileges change?

I think there may be a case in the future when we need patch/update on the configmap being used for our cluster id.

Our ingress controller already updates this config map but currently uses the unauthenticated port.

nicksardo · 2018-02-12T17:33:34Z

Outstanding question:
The kube-controller-manager usually starts prior to the addon manager applying these policies. As a result, the configmap watcher in the GCE cloudprovider errors out for tens of seconds. Meanwhile, the service controller will attempt to sync every loadbalancer at start. Since the clusterid hasn't been retrieved, the cloudprovider's EnsureLoadBalancer will error and cause a K8s event.

Need to test this PR more...

/hold

nicksardo · 2018-02-12T22:13:13Z

Actually, the policies don't exist for new clusters, which means services also don't exist.
/unhold

MrHohn

Looks good, some minor comments.

MrHohn · 2018-02-12T23:10:13Z

cluster/gce/addons/loadbalancing/cloud-provider-binding.yaml

+    rbac.authorization.kubernetes.io/autoupdate: "true"
+  labels:
+    addonmanager.kubernetes.io/mode: Reconcile
+    kubernetes.io/cluster-service: "true"


cluster-service label is no longer required for addons, except for those services that need to show up via kubectl cluster-info.

MrHohn · 2018-02-12T23:27:22Z

cluster/gce/gci/configure-helper.sh

 function start-lb-controller {
+  setup-addon-manifests "addons" "loadbalancing"


minor: having both cluster-loadbalancing/ and loadbalancing/ seems confusing. Should we make a cluster-loadbalancing/rbac folder or something similar?

The readme says to avoid naming conflicts. Are subdirs merged correctly?

Sorry, didn't notice this is under cluster/gce. Now it seems good :)

I'm open to better naming. Thankfully, it can easily be changed later when/if we add more manifests.

MrHohn · 2018-02-12T23:30:11Z

plugin/pkg/auth/authorizer/rbac/bootstrappolicy/namespace_policy.go

@@ -85,13 +85,6 @@ func init() {
 			rbac.NewRule("get", "list", "watch").Groups(legacyGroup).Resources("secrets").RuleOrDie(),
 		},
 	})
-	addNamespaceRole(metav1.NamespaceSystem, rbac.Role{
-		// role for the cloud providers to access/create kube-system configmaps
-		ObjectMeta: metav1.ObjectMeta{Name: saRolePrefix + "cloud-provider"},


Just to make sure, is GCE the only consumer of this cloud-provider role?

I checked kubernetes core and didn't see any other users. Naturally, that doesn't stop out-of-tree providers from depending on it. Sooner we can get this in and let people discover the change, the better.

I think this may need to follow the deprecation policy, probably covered by rule #7:

Rule #7: Deprecated behaviors must function for no less than 1 year after their announced deprecation.

Has this removal been announced?

/cc @kubernetes/sig-auth-pr-reviews

nicksardo · 2018-02-14T00:38:17Z

/retest

tallclair

This needs a release note.

tallclair · 2018-02-14T23:16:52Z

plugin/pkg/auth/authorizer/rbac/bootstrappolicy/namespace_policy.go

@@ -85,13 +85,6 @@ func init() {
 			rbac.NewRule("get", "list", "watch").Groups(legacyGroup).Resources("secrets").RuleOrDie(),
 		},
 	})
-	addNamespaceRole(metav1.NamespaceSystem, rbac.Role{
-		// role for the cloud providers to access/create kube-system configmaps
-		ObjectMeta: metav1.ObjectMeta{Name: saRolePrefix + "cloud-provider"},


I think this may need to follow the deprecation policy, probably covered by rule #7:

Rule #7: Deprecated behaviors must function for no less than 1 year after their announced deprecation.

Has this removal been announced?

tallclair · 2018-02-14T23:17:51Z

plugin/pkg/auth/authorizer/rbac/bootstrappolicy/namespace_policy.go

@@ -85,13 +85,6 @@ func init() {
 			rbac.NewRule("get", "list", "watch").Groups(legacyGroup).Resources("secrets").RuleOrDie(),
 		},
 	})
-	addNamespaceRole(metav1.NamespaceSystem, rbac.Role{
-		// role for the cloud providers to access/create kube-system configmaps
-		ObjectMeta: metav1.ObjectMeta{Name: saRolePrefix + "cloud-provider"},


/cc @kubernetes/sig-auth-pr-reviews

nicksardo · 2018-02-14T23:47:48Z

Thanks for the review @tallclair.

How about I create a separate PR which adds a deprecation comment to namespace_policy.go and I set the action required release note there?

nicksardo · 2018-02-15T19:02:31Z

cluster/gce/addons/loadbalancing/cloud-provider-binding.yaml

+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: cloud-provider


@tallclair Is it necessary to create the service account?

Hmm, I think something is already creating it, but I'm not sure what.

liggitt · 2018-02-15T20:29:04Z

cluster/gce/addons/loadbalancing/cloud-provider-binding.yaml

+kind: ClusterRoleBinding
+metadata:
+  annotations:
+    rbac.authorization.kubernetes.io/autoupdate: "true"


remove this annotation, it's not meaningful to the add-on manager

liggitt · 2018-02-15T20:29:07Z

cluster/gce/addons/loadbalancing/cloud-provider-binding.yaml

+kind: RoleBinding
+metadata:
+  annotations:
+    rbac.authorization.kubernetes.io/autoupdate: "true"


remove this annotation, it's not meaningful to the add-on manager

nicksardo · 2018-02-15T21:21:38Z

Removed rbac.authorization.kubernetes.io/autoupdate: "true" from every resource.

tallclair · 2018-02-15T21:47:54Z

Do you want to add a deprecation notice of the built-in role to the release note?

nicksardo · 2018-02-15T22:39:11Z

@tallclair Created #59949 for that purpose. WDYT?

tallclair · 2018-02-16T19:54:09Z

/lgtm

k8s-ci-robot · 2018-02-16T19:54:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bowei, nicksardo, tallclair

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cluster/gce/OWNERS~~ [bowei]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-github-robot · 2018-02-17T00:31:39Z

Automatic merge from submit-queue (batch tested with PRs 59683, 59964, 59841, 59936, 59686). If you want to cherry-pick this change to another branch, please follow the instructions here.

@tallclair

Automatic merge from submit-queue (batch tested with PRs 59052, 59157, 59428, 59949, 60151). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. Start deprecation of role for `cloud-provider` service account in rbac boostrap **What this PR does / why we need it**: See #59686 for reference **Special notes for your reviewer**: /assign @tallclair **Release note**: ```release-note Action Required: The boostrapped RBAC role and rolebinding for the `cloud-provider` service account is now deprecated. If you're currently using this service account, you must create and apply your own RBAC policy for new clusters. ```

@bowei

Automatic merge from submit-queue (batch tested with PRs 65251, 67255, 67224, 67297, 68105). If you want to cherry-pick this change to another branch, please follow the instructions here: https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md. Add namespace for (cluster)role(binding) cloud-provider. **What this PR does / why we need it**: Add namespace for (cluster)role(binding) cloud-provider. Change the addonmanager mode to be from reconcile to EnsureExists. Needs to be cherrypicked together with #59686. **Special notes for your reviewer**: /assign @bowei @tallclair /sig auth **Release note**: ```release-note Role, ClusterRole and their bindings for cloud-provider is put under system namespace. Their addonmanager mode switches to EnsureExists. ``` Manual tested. Cluster can be created succesfully using kube-up.sh with desired (cluster)role(binding)s.

…86-upstream-release-1.9 Cherry pick of #59686 #67224: Add cloud-provider policies to be applied via addon mgr

k8s-ci-robot assigned bowei and timstclair Feb 10, 2018

k8s-ci-robot added the release-note-none Denotes a PR that doesn't merit a release note. label Feb 10, 2018

k8s-ci-robot requested a review from timstclair February 10, 2018 01:39

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 10, 2018

k8s-ci-robot requested a review from MrHohn February 10, 2018 01:42

jhorwit2 reviewed Feb 11, 2018

View reviewed changes

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 12, 2018

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 12, 2018

MrHohn reviewed Feb 12, 2018

View reviewed changes

nicksardo force-pushed the gce-roles branch 2 times, most recently from c04d7f8 to f81a21c Compare February 14, 2018 00:03

tallclair reviewed Feb 14, 2018

View reviewed changes

k8s-ci-robot added sig/auth Categorizes an issue or PR as relevant to SIG Auth. release-note-action-required Denotes a PR that introduces potentially breaking changes that require user action. and removed release-note-none Denotes a PR that doesn't merit a release note. labels Feb 14, 2018

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-action-required Denotes a PR that introduces potentially breaking changes that require user action. labels Feb 14, 2018

nicksardo force-pushed the gce-roles branch from f81a21c to 3a1cc36 Compare February 15, 2018 18:58

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 15, 2018

nicksardo commented Feb 15, 2018

View reviewed changes

liggitt reviewed Feb 15, 2018

View reviewed changes

nicksardo force-pushed the gce-roles branch from 3a1cc36 to b62e868 Compare February 15, 2018 21:21

nicksardo mentioned this pull request Feb 15, 2018

Start deprecation of role for cloud-provider service account in rbac boostrap #59949

Merged

Add cloud-provider policies to be applied via addon mgr

911a082

nicksardo force-pushed the gce-roles branch from b62e868 to 911a082 Compare February 15, 2018 22:49

k8s-ci-robot assigned tallclair Feb 16, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 16, 2018

k8s-github-robot merged commit 3a60b0b into kubernetes:master Feb 17, 2018

This was referenced Aug 1, 2018

Cherry pick of #59686 #67224: Add cloud-provider policies to be applied via addon mgr #66872

Merged

Automated cherry pick of #59686: Add cloud-provider policies to be applied via addon mgr #66939

Closed

grayluck mentioned this pull request Aug 10, 2018

Add namespace for (cluster)role(binding) cloud-provider. #67224

Merged

k8s-ci-robot added a commit that referenced this pull request Sep 18, 2018

Merge pull request #66872 from grayluck/automated-cherry-pick-of-#596…

b65e797

…86-upstream-release-1.9 Cherry pick of #59686 #67224: Add cloud-provider policies to be applied via addon mgr

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GCE: Create cloud-provider roles and bindings via addons #59686

GCE: Create cloud-provider roles and bindings via addons #59686

nicksardo commented Feb 10, 2018 •

edited

Loading

nicksardo commented Feb 10, 2018

bowei commented Feb 10, 2018

jhorwit2 Feb 11, 2018

nicksardo Feb 12, 2018

nicksardo Feb 12, 2018

nicksardo commented Feb 12, 2018

nicksardo commented Feb 12, 2018

MrHohn left a comment

MrHohn Feb 12, 2018

nicksardo Feb 13, 2018

MrHohn Feb 12, 2018

nicksardo Feb 12, 2018

MrHohn Feb 12, 2018

nicksardo Feb 13, 2018

MrHohn Feb 12, 2018

nicksardo Feb 12, 2018 •

edited

Loading

tallclair Feb 14, 2018

tallclair Feb 14, 2018

nicksardo commented Feb 14, 2018

tallclair left a comment

tallclair Feb 14, 2018

tallclair Feb 14, 2018

nicksardo commented Feb 14, 2018 •

edited

Loading

nicksardo Feb 15, 2018

tallclair Feb 15, 2018

liggitt Feb 15, 2018

liggitt Feb 15, 2018

nicksardo commented Feb 15, 2018

tallclair commented Feb 15, 2018

nicksardo commented Feb 15, 2018

tallclair commented Feb 16, 2018

k8s-ci-robot commented Feb 16, 2018

k8s-github-robot commented Feb 17, 2018

		function start-lb-controller {
		setup-addon-manifests "addons" "loadbalancing"

GCE: Create cloud-provider roles and bindings via addons #59686

GCE: Create cloud-provider roles and bindings via addons #59686

Conversation

nicksardo commented Feb 10, 2018 • edited Loading

nicksardo commented Feb 10, 2018

bowei commented Feb 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicksardo commented Feb 12, 2018

nicksardo commented Feb 12, 2018

MrHohn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicksardo Feb 12, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicksardo commented Feb 14, 2018

tallclair left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicksardo commented Feb 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nicksardo commented Feb 15, 2018

tallclair commented Feb 15, 2018

nicksardo commented Feb 15, 2018

tallclair commented Feb 16, 2018

k8s-ci-robot commented Feb 16, 2018

k8s-github-robot commented Feb 17, 2018

nicksardo commented Feb 10, 2018 •

edited

Loading

nicksardo Feb 12, 2018 •

edited

Loading

nicksardo commented Feb 14, 2018 •

edited

Loading