Start deploying the bootstrapper via deployment manager. #823

jlewi · 2018-05-17T20:42:30Z

This config creates the K8s resources needed to run the bootstrapper
Enable the ResourceManager API; this is used to get IAM policies
Add IAM roles to the cloudservices account. This is needed so that
the deployment manager has sufficient RBAC permissions to do what it needs
to.
Delete initialNodeCount and just make the default node pool a 1 CPU node pool.
The bootstrapper isn't running successfully; it looks like its trying
to create a pytorch component but its using an older version of the registry
which doesn't include the pytorch operator.

Related to: #802 Use deployment manager to run bootstrapper

Related to #802 On GCP trigger bootstrapper with deployment manager
Related to #757 Install Kubeflow via deployment manager
Related to #833 Internal errors
/assign @kunmingg

This change is

jlewi · 2018-05-18T14:07:42Z

Test failed because of an IAM permission issue

ERROR: (gcloud.deployment-manager.deployments.create) Error in Operation [operation-1526618835540-56c73a5665820-80e4de3c-fba39e36]: errors:
- code: RESOURCE_ERROR
 location: /deployments/z23-cb334c0-1566-349b/resources/patch-iam-policy
 message: '{"ResourceType":"gcp-types/cloudresourcemanager-v1:cloudresourcemanager.projects.setIamPolicy","ResourceErrorCode":"403","ResourceErrorMessage"
:{"code":403,"message":"The
   caller does not have permission","status":"PERMISSION_DENIED","statusMessage":"Forbidden","requestPath":"https://cloudresourcemanager.googleapis.com/v1
/projects/kubeflow-ci:setIamPolicy","httpMethod":"POST"}}'

The cloudservices account in the test project needs to have IAM set policy permissions.

I added Project IAM Admin Role to the cloudservices account in project: kubeflow-ci

ankushagarwal · 2018-05-18T18:56:11Z

docs/gke/configs/cluster-kubeflow.yaml

+  # 2. Create two separate deployments and launch the boot strapper
+  # after the cluster is created.
+  # 
+  # Two separate deployments doesn't make much sense; we could just use 


Having initialNodeCount and cpu-pool-initialNodeCount can be confusing. Can initialNodeCount property be removed from here and set to 1 by default in cluster.jinja?

ankushagarwal · 2018-05-18T18:57:07Z

docs/gke/configs/cluster.jinja

-{% set K8S_ENDPOINTS = {'': 'api/v1', '-v1beta1-extensions': 'apis/extensions/v1beta1'} %}
+{% set RBAC_TYPE_NAME = TYPE_NAME + '-rbac-v1' %}
+
+{% set K8S_ENDPOINTS = {'': 'api/v1', '-v1beta1-extensions': 'apis/extensions/v1beta1', '-rbac-v1': 'apis/rbac.authorization.k8s.io/v1'}  %}
 {% set CPU_POOL = 'cpu-pool-' + properties['pool-version'] %}


Use {{ CPU_POOL }} on line 68 instead of duplicating this

ankushagarwal · 2018-05-18T19:57:12Z

docs/gke/configs/cluster.jinja

+      # Wait for the type provider to be created.
+      - {{ TYPE_NAME }}
+
+{#  StatefulSet needs a service. #}


I don't think this is correct.
I was able to create a statefulset without a service

kubectl apply -f- <<EOF apiVersion: apps/v1 kind: StatefulSet metadata: name: website spec: serviceName: "nginx" replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: k8s.gcr.io/nginx-slim:0.8 ports: - containerPort: 80 name: web EOF

That's what I thought. Thanks for trying it out.

jlewi · 2018-05-18T23:32:40Z

/test all

ankushagarwal · 2018-05-18T23:33:52Z

/lgtm
/approve

/hold

Feel free to run /hold cancel when ready to submit

jlewi · 2018-05-19T21:34:46Z

Test failed with an internal error trying to tear it down.

Your active configuration is: [default]
+ gcloud deployment-manager --project=kubeflow-ci --quiet deployments delete z23-25b96cd-1584-c7cc
Waiting for delete [operation-1526763570795-56c95584b11f9-22f0bb38-c3ac6396]...
..............failed.
ERROR: (gcloud.deployment-manager.deployments.delete) Delete operation operation-1526763570795-56c95584b11f9-22f0bb38-c3ac6396 failed.
Error in Operation [operation-1526763570795-56c95584b11f9-22f0bb38-c3ac6396]: errors:
- code: INTERNAL_ERROR
 message: "Code: '-3751873619725894346'"

jlewi · 2018-05-19T21:35:22Z

/test all

jlewi · 2018-05-19T22:24:05Z

@kunmingg @ankushagarwal Can you PTAL? I had to fix a bug causing the tests to fail. So I also updated it to actually run the bootstrapper.

ankushagarwal · 2018-05-21T18:37:17Z

/lgtm
/approve

ankushagarwal · 2018-05-21T19:29:00Z

/hold cancel

jlewi · 2018-05-21T20:53:19Z

/test all

ankushagarwal · 2018-05-21T23:37:05Z

/lgtm
/approve

jlewi · 2018-05-21T23:42:50Z

/hold

* This config creates the K8s resources needed to run the bootstrapper * Enable the ResourceManager API; this is used to get IAM policies * Add IAM roles to the cloudservices account. This is needed so that the deployment manager has sufficient RBAC permissions to do what it needs to. * Delete initialNodeCount and just make the default node pool a 1 CPU node pool. * The bootstrapper isn't running successfully; it looks like its trying to create a pytorch component but its using an older version of the registry which doesn't include the pytorch operator. * Set delete policy on K8s resources to ABANDON otherwise we get internal errors. * We can use actions to enable APIs and then we won't try to delete the API when the deployment is deleted which causes errors. fix kubeflow#833

jlewi · 2018-05-22T00:53:41Z

/test all
Most recent test failure was because we ran out of ip address quota.

jlewi · 2018-05-22T01:17:22Z

Another internal error deleting the deployment

+ gcloud deployment-manager --project=kubeflow-ci --quiet deployments delete z23-c4971e3-1605-9f0c
Waiting for delete [operation-1526951147049-56cc104b5902b-342b1ac7-1f9f4298]...
...............failed.
ERROR: (gcloud.deployment-manager.deployments.delete) Delete operation operation-1526951147049-56cc104b5902b-342b1ac7-1f9f4298 failed.
Error in Operation [operation-1526951147049-56cc104b5902b-342b1ac7-1f9f4298]: errors:
- code: INTERNAL_ERROR
 message: "Code: '-7647148574420173012'"

jlewi · 2018-05-22T01:17:28Z

/test all

jlewi · 2018-05-22T02:11:32Z

@ankushagarwal Could you take another look I had to update the tests again and I switched to using actions to enable the APIs so we wouldn't need a separate deployment.

ankushagarwal · 2018-05-22T17:15:37Z

/lgtm
/approve

k8s-ci-robot · 2018-05-22T17:15:42Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ankushagarwal

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ankushagarwal]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jlewi · 2018-05-22T22:54:03Z

/hold cancel

jlewi · 2018-05-23T12:18:37Z

/test all

* This config creates the K8s resources needed to run the bootstrapper * Enable the ResourceManager API; this is used to get IAM policies * Add IAM roles to the cloudservices account. This is needed so that the deployment manager has sufficient RBAC permissions to do what it needs to. * Delete initialNodeCount and just make the default node pool a 1 CPU node pool. * The bootstrapper isn't running successfully; it looks like its trying to create a pytorch component but its using an older version of the registry which doesn't include the pytorch operator. * Set delete policy on K8s resources to ABANDON otherwise we get internal errors. * We can use actions to enable APIs and then we won't try to delete the API when the deployment is deleted which causes errors. fix kubeflow#833

…ubeflow#823) * image gcr.io/kubeflow-images-public/jupyter-web-app:vmaster-gc80316d3 * Image built from kubeflow/kubeflow@c80316d3

k8s-ci-robot assigned kunmingg May 17, 2018

k8s-ci-robot requested review from kunmingg and wbuchwalter May 17, 2018 20:42

k8s-ci-robot added size/M size/L and removed size/M labels May 17, 2018

jlewi force-pushed the gke_bootstrapper branch from cb334c0 to aa0a7ab Compare May 18, 2018 14:11

ankushagarwal reviewed May 18, 2018

View reviewed changes

k8s-ci-robot assigned ankushagarwal May 18, 2018

k8s-ci-robot added do-not-merge/hold lgtm approved and removed lgtm labels May 18, 2018

jlewi force-pushed the gke_bootstrapper branch from 6cd0699 to 25b96cd Compare May 19, 2018 20:47

jlewi mentioned this pull request May 19, 2018

bootstrapper error trying to create the PyTorch Operator #832

Closed

jlewi mentioned this pull request May 19, 2018

GCP deployment manager test handle internal errors #833

Closed

jlewi mentioned this pull request May 21, 2018

E2E test verifies Kubeflow installed via Deployment Manager #836

Closed

k8s-ci-robot added the lgtm label May 21, 2018

k8s-ci-robot removed the do-not-merge/hold label May 21, 2018

k8s-ci-robot removed the lgtm label May 21, 2018

k8s-ci-robot added the lgtm label May 21, 2018

k8s-ci-robot added the do-not-merge/hold label May 21, 2018

jlewi force-pushed the gke_bootstrapper branch from b96c2fe to c4971e3 Compare May 22, 2018 00:27

k8s-ci-robot removed the lgtm label May 22, 2018

k8s-ci-robot added the lgtm label May 22, 2018

k8s-ci-robot removed the do-not-merge/hold label May 22, 2018

k8s-ci-robot merged commit 4cffc20 into kubeflow:master May 23, 2018

snyk-bot mentioned this pull request Aug 28, 2022

[Snyk] Upgrade @kubernetes/client-node from 0.12.2 to 0.17.0 violethaze74/kubeflow#4

Open

snyk-bot mentioned this pull request Sep 14, 2022

[Snyk] Upgrade @kubernetes/client-node from 0.12.2 to 0.17.0 k8s-mcrr/kubeflow-fork#6

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Start deploying the bootstrapper via deployment manager. #823

Start deploying the bootstrapper via deployment manager. #823

jlewi commented May 17, 2018 •

edited

jlewi commented May 18, 2018 •

edited

ankushagarwal May 18, 2018

jlewi May 18, 2018

ankushagarwal May 18, 2018 •

edited

ankushagarwal May 18, 2018

jlewi May 18, 2018

kunmingg May 18, 2018

jlewi commented May 18, 2018

ankushagarwal commented May 18, 2018

jlewi commented May 19, 2018

jlewi commented May 19, 2018

jlewi commented May 19, 2018

ankushagarwal commented May 21, 2018

ankushagarwal commented May 21, 2018

jlewi commented May 21, 2018

ankushagarwal commented May 21, 2018

jlewi commented May 21, 2018

jlewi commented May 22, 2018

jlewi commented May 22, 2018

jlewi commented May 22, 2018

jlewi commented May 22, 2018

ankushagarwal commented May 22, 2018

k8s-ci-robot commented May 22, 2018

jlewi commented May 22, 2018

jlewi commented May 23, 2018

Start deploying the bootstrapper via deployment manager. #823

Start deploying the bootstrapper via deployment manager. #823

Conversation

jlewi commented May 17, 2018 • edited

jlewi commented May 18, 2018 • edited

ankushagarwal May 18, 2018

Choose a reason for hiding this comment

jlewi May 18, 2018

Choose a reason for hiding this comment

ankushagarwal May 18, 2018 • edited

Choose a reason for hiding this comment

ankushagarwal May 18, 2018

Choose a reason for hiding this comment

jlewi May 18, 2018

Choose a reason for hiding this comment

kunmingg May 18, 2018

Choose a reason for hiding this comment

jlewi commented May 18, 2018

ankushagarwal commented May 18, 2018

jlewi commented May 19, 2018

jlewi commented May 19, 2018

jlewi commented May 19, 2018

ankushagarwal commented May 21, 2018

ankushagarwal commented May 21, 2018

jlewi commented May 21, 2018

ankushagarwal commented May 21, 2018

jlewi commented May 21, 2018

jlewi commented May 22, 2018

jlewi commented May 22, 2018

jlewi commented May 22, 2018

jlewi commented May 22, 2018

ankushagarwal commented May 22, 2018

k8s-ci-robot commented May 22, 2018

jlewi commented May 22, 2018

jlewi commented May 23, 2018

jlewi commented May 17, 2018 •

edited

jlewi commented May 18, 2018 •

edited

ankushagarwal May 18, 2018 •

edited