Add GKE Security Features to Deployment Manager config #879

ankushagarwal · 2018-05-25T19:53:51Z

Add option for

Private Clusters
Separate Service Account
masterIpv4CidrBlock
Pod Security Policy
Secure Node Metadata
Authorized Networks for k8s Master
Reduce oauth scopes for worker nodes and replace it with gke-default

This change is

kunmingg · 2018-05-25T19:59:03Z

docs/gke/configs/cluster-kubeflow.yaml

  # kubectl at that point. So we put all resources in a single deployment.
 - name: kubeflow
  type: cluster.jinja
  properties:
    zone: us-east1-d
+    # Set this to v1beta1 to use beta features such as private clusters,
+    apiVersion: v1


v1beta1 or v1?

I want to set a default value of v1. If the end-user wants to leverage beta features, they can set this to v1beta1

Is this the GKE API version? Should we call it gkeApiVersion?

Yes. Updated

jlewi · 2018-05-25T23:48:14Z

docs/gke/configs/cluster.jinja

      nodeConfig:
        machineType: n1-standard-1
+        serviceAccount: {{ properties['serviceAccountName'] }}@{{ env['project'] }}.iam.gserviceaccount.com


Do we want to use the service account as the VM service account? I think a better practice is to move in the direction of creating secrets containing service account keys. This way we can provide finer grained permissions then what we get from node service accounts.

This service account will not be used by pods. This service account is intended to replace the default compute engine service account which has Project Editor Role. We will be using secrets and service account keys to give access to individual pods.

jlewi · 2018-05-26T00:07:05Z

docs/gke/configs/cluster.jinja

@@ -72,59 +83,85 @@ resources:
      monitoringService: monitoring.googleapis.com/kubernetes
      {% else %}
      initialClusterVersion: 1.9.6-gke.1
-      {% endif %}      
+      {% endif %}
+      podSecurityPolicyConfig:


Should we make all the security parameters fields in a dictionary "securityFeatures" just to make the properties a little bit more human readable?

jlewi · 2018-05-26T00:48:39Z

Looks like the test failed because you need the IAM API; you just need to add

- name: iam-api
  action: 'gcp-types/servicemanagement-v1:servicemanagement.services.enable'
  properties: 
    consumerId: {{ 'project:' + env['project'] }}
    serviceName: iam.googleapis.com

and then make sure the patch step depends on that step.

jlewi · 2018-05-26T18:26:42Z

Same error

RROR: (gcloud.deployment-manager.deployments.create) Error in Operation [operation-1527296670560-56d117783ef01-9cc0d413-1104737f]: errors:
- code: RESOURCE_ERROR
 location: /deployments/z79-6e2d4ba-1689-1b9d/resources/kubeflow-service-account
 message: '{"ResourceType":"iam.v1.serviceAccount","ResourceErrorCode":"403","ResourceErrorMessage":{"code":403,"message":"Identity
   and Access Management (IAM) API has not been used in project 593963025935 before
   or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=593963025935
   then retry. If you enabled this API recently, wait a few minutes for the action
   to propagate to our systems and retry.","status":"PERMISSION_DENIED","details":[{"@type":"type.googleapis.com/google.rpc.Help","links":[{"description":
"Google
   developers console API activation","url":"https://console.developers.google.com/apis/api/iam.googleapis.com/overview?project=593963025935"}]}],"statusM
essage":"Forbidden","requestPath":"https://iam.googleapis.com/v1/projects/kubeflow-ci/serviceAccounts","httpMethod":"POST"}}'

/test all

Use gke-default oauth scopes Update to gkeApiVersion Add securityConfig Enable iam api in dm

jlewi · 2018-05-29T19:04:46Z

docs/gke/configs/cluster.jinja

+        - https://www.googleapis.com/auth/logging.write
+        - https://www.googleapis.com/auth/monitoring
+        - https://www.googleapis.com/auth/service.management.readonly
+        - https://www.googleapis.com/auth/servicecontrol


logging and monitoring make sense for the logging and monitoring agents but what about the others?

I assume devstorage is for GCR?

Why do we need service.management, servicecontrol, and trace?

I substituted gke-default [1] with the actual scopes. I will reduce the scopes.

[1] https://cloud.google.com/sdk/gcloud/reference/container/clusters/create

jlewi · 2018-05-29T19:05:22Z

docs/gke/configs/cluster.jinja

@@ -139,7 +144,12 @@ resources:
        machineType: n1-standard-8
        serviceAccount: {{ properties['vmServiceAccountName'] }}@{{ env['project'] }}.iam.gserviceaccount.com
        oauthScopes:
-        - gke-default
+        - https://www.googleapis.com/auth/devstorage.read_only


Can we define a variable and reuse it to avoid code duplication and ensure they stay in sync?

ankushagarwal · 2018-05-29T20:51:39Z

@jlewi This is ready for review

jlewi · 2018-05-30T21:39:23Z

/lgtm
/approve

k8s-ci-robot · 2018-05-30T21:39:25Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jlewi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jlewi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

* Add GKE Security Features to bootstrapper Use gke-default oauth scopes Update to gkeApiVersion Add securityConfig Enable iam api in dm * Add oauth scopes manually. gke-default does not work * Fix oauthscopes * Add PodSecurityPolicy only to v1beta1

* feat: Add event when the reconcile is failed Signed-off-by: Ce Gao <gaoce@caicloud.io> * fix: Use format Signed-off-by: Ce Gao <gaoce@caicloud.io>

…ad (kubeflow#879) * image gcr.io/kubeflow-images-public/notebook-controller:vmaster-g0cb184ad * Image built from kubeflow/kubeflow@0cb184ad

k8s-ci-robot requested review from jlewi and kunmingg May 25, 2018 19:53

k8s-ci-robot added the size/L label May 25, 2018

ankushagarwal changed the title ~~Add GKE Security Features to bootstrapper~~ Add GKE Security Features to Deployment Manager config May 25, 2018

kunmingg reviewed May 25, 2018

View reviewed changes

jlewi reviewed May 25, 2018

View reviewed changes

jlewi reviewed May 26, 2018

View reviewed changes

Ankush Agarwal added 2 commits May 29, 2018 11:29

Add GKE Security Features to bootstrapper

ef927cb

Use gke-default oauth scopes Update to gkeApiVersion Add securityConfig Enable iam api in dm

Add oauth scopes manually. gke-default does not work

74bf7d7

jlewi reviewed May 29, 2018

View reviewed changes

Ankush Agarwal added 2 commits May 29, 2018 12:17

Fix oauthscopes

bab53ac

Add PodSecurityPolicy only to v1beta1

2433c7f

k8s-ci-robot assigned jlewi May 30, 2018

k8s-ci-robot added the lgtm label May 30, 2018

k8s-ci-robot added the approved label May 30, 2018

k8s-ci-robot merged commit f25dd9e into kubeflow:master May 30, 2018

ankushagarwal deleted the securestuff branch June 1, 2018 22:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GKE Security Features to Deployment Manager config #879

Add GKE Security Features to Deployment Manager config #879

ankushagarwal commented May 25, 2018 •

edited

kunmingg May 25, 2018

ankushagarwal May 25, 2018

jlewi May 25, 2018

ankushagarwal May 25, 2018

jlewi May 25, 2018

ankushagarwal May 25, 2018

jlewi May 26, 2018

ankushagarwal May 26, 2018

jlewi commented May 26, 2018

jlewi commented May 26, 2018

jlewi May 29, 2018

ankushagarwal May 29, 2018

jlewi May 29, 2018

ankushagarwal commented May 29, 2018

jlewi commented May 30, 2018

k8s-ci-robot commented May 30, 2018

Add GKE Security Features to Deployment Manager config #879

Add GKE Security Features to Deployment Manager config #879

Conversation

ankushagarwal commented May 25, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jlewi commented May 26, 2018

jlewi commented May 26, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ankushagarwal commented May 29, 2018

jlewi commented May 30, 2018

k8s-ci-robot commented May 30, 2018

ankushagarwal commented May 25, 2018 •

edited