Golang profile controller #2533

lluunn · 2019-02-22T17:11:52Z

Changed the spec a little bit to

apiVersion: kubeflow.org/v1alpha1
kind: Profile
metadata:
  name: john
  namespace: kubeflow
spec:
  namespace: john
  owner:
    kind: User
    apiGroup: rbac.authorization.k8s.io
    name: john@foo.com

I haven't added resource quota field. I am not sure if it makes sense for user to specify their own quota? Or should controller determine the quota somehow?
removed nested spec.template.spec.

The controller:

creates the namespace
creates the role and rolebinding for the user in that namespace

This change is

lluunn · 2019-02-23T00:15:55Z

/assign @kkasravi @jlewi
Ready for review.

lluunn · 2019-02-26T19:50:38Z

/assign @kunmingg

kunmingg

I feel profile scope shouldn't be Namespaced as it owns namespace as child which is not Namespaced.

kunmingg · 2019-02-26T22:07:06Z

components/profile-controller/config/crds/kubeflow_v1alpha1_profile.yaml

+  names:
+    kind: Profile
+    plural: profiles
+  scope: Namespaced


Should we make profile name scope global and have namespace name == profile name?
That simplify usage if we have profile <-> namespace 1 - 1 mapping.

done. I make profile a cluster scope resource, and namespace's names = profile's name

It looks like its still Namespace scoped.

Sorry only updated ksonnet package.
yaml updated now

kunmingg · 2019-02-27T03:02:20Z

components/profile-controller/pkg/controller/profile/profile_controller.go

+		},
+	}
+	if err := controllerutil.SetControllerReference(instance, ns, r.scheme); err != nil {
+		return reconcile.Result{}, err


What if the namespace is already exist and owned by another profile?

this is fixed now that profile is cluster resource

kkasravi · 2019-02-27T19:33:55Z

components/profile-controller/Makefile

+	#go test ./pkg/... ./cmd/... -coverprofile cover.out
+
+# Build manager binary
+manager: generate fmt vet


I tried make manager and got

$ ☞ make manager go generate ./pkg/... ./cmd/... stat ../../vendor/k8s.io/code-generator/cmd/deepcopy-gen/main.go: no such file or directory pkg/apis/apis.go:18: running "go": exit status 1 make: *** [generate] Error 1

I also tried make generate and make docker-build with similar errors.

The problem is deepcopy assumes a vendor directory exists which isn't the case for go modules.
This is due to the fact that kubebuilder doesn't generate a Makefile that works with go modules.

You may want to add the following targets to the Makefile

$(GOPATH)/bin/deepcopy-gen: GO111MODULE=off $(GO) get k8s.io/code-generator/cmd/deepcopy-gen deepcopy: $(GOPATH)/bin/deepcopy-gen $(GOPATH)/bin/deepcopy-gen -i github.com/kubeflow/kubeflow/components/profile-controller/pkg/apis/... -O zz_generated.deepcopy

will it work if I just checkin vendor/sigs.k8s.io/controller-tools/cmd/controller-gen/main.go?

kkasravi · 2019-02-27T19:34:01Z

components/profile-controller/Makefile

+
+# Build manager binary
+manager: generate fmt vet
+	go build -o bin/manager github.com/kubeflow/kubeflow/components/profile-controller/cmd/manager


I would change bin/manager to bin/profiler-controller

kkasravi · 2019-02-27T19:35:13Z

components/profile-controller/Makefile

+	kustomize build config/default | kubectl apply -f -
+
+# Generate manifests e.g. CRD, RBAC etc.
+manifests:


this won't work since there is no vendor directory. I'll update with the correct command a bit later

kkasravi · 2019-02-27T19:37:53Z

components/profile-controller/config/default/kustomization.yaml

@@ -0,0 +1,49 @@
+# Adds namespace to all resources.


you may want to add these kustomization files to the new repo for kustomization manifests (kubeflow/manifests)

ack, will do later

kkasravi · 2019-02-27T19:42:51Z

components/profile-controller/pkg/apis/kubeflow/v1alpha1/doc.go

+// +k8s:deepcopy-gen=package,register
+// +k8s:conversion-gen=github.com/kubeflow/kubeflow/components/profile-controller/pkg/apis/kubeflow
+// +k8s:defaulter-gen=TypeMeta
+// +groupName=kubeflow.kubeflow.org


group name should probably be profiles.kubeflow.org

I think it should be just kubeflow.org? updated

kkasravi · 2019-02-27T19:45:04Z

components/profile-controller/pkg/apis/kubeflow/v1alpha1/profile_types.go

+
+// ProfileStatus defines the observed state of Profile
+type ProfileStatus struct {
+	// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster


we usually have something in here like

Conditions []ProfileCondition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type" protobuf:"bytes,6,rep,name=conditions"`

I will update status in a followup PR

lluunn · 2019-02-28T23:57:26Z

/hold
wait for test to be merged #2404

lluunn · 2019-03-01T01:23:33Z

/hold cancel

lluunn · 2019-03-01T02:48:05Z

e2e test passed, ptal
@jlewi @kunmingg

jlewi · 2019-03-01T16:29:11Z

/lgtm
/approve
/hold for @kunmingg

k8s-ci-robot · 2019-03-01T16:29:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jlewi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jlewi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kunmingg · 2019-03-01T21:55:50Z

/hold seems not working when followed by strings...

kunmingg · 2019-03-01T22:01:55Z

@jlewi @lluunn
Seems we still have problem when the namespace is already exist and owned by another user (admin).
Create an issue to track: #2601

* init * create api * wip * fix * fix * fix * fix test * address review * fix test * make it cluster scope * fix test * fix test * fix

@prodonjs

…forms' (#2547) * revert to prior version * fixes kfctl (golang) rename 'ks' directory and app to ksonnet and ksonnet.go respectively * fixes 'kfctl - Fetch registry automatically' * fixes 'refactor gcp, minikube, docker-for-desktop, ack to be kfctl plugins' * change DefaultDevRepo to point to just the repo not repo/kubeflow * fixes 'refactor gcp, minikube, docker-for-desktop, ack to be kfctl plugins' * plugins for all existing KfApp instances {ksonnet, minikube, foo}. * update golang version * fixes 'docker-for-desktop' * delete meta-controller-cluster-role-binding * update README.md * remove unused var DEBUG * remove bootstrap/cmd/plugins/foo.go * fixes kfctl (golang) rename 'ks' directory and app to ksonnet and ksonnet.go respectively * fixes 'add the gcp platform to kfctl (golang)' * fix merge conflict * cli option --project is an init flag * fix to write out project to app.yaml * fix for nil project check * include fix for #2367 * added updateDM, createSecrets to Apply * fixes to macros * merge changes from upstream master * remove old files, update README.md * fix for init app-name where app-name is not a path * fix for gcp apply * code to create a Gcp Secret, doesn't do correct auth right now * snapshot * update on downloadK8sManifests, gcpInitProject, added TODO's * additional work on gcpInitProject * gcpInitProject now works but requires GOOGLE_APPLICATION_CREDENTIALS env var * initial pass on updateDeployment * update README.md to described static vs dynamic platforms * updates to Gcp.updateDM, added GOOGLE_APPLICATION_CREDENTIALS to the Makefile tests * minor changes on init, generate usage * snapshot * additional global resources identified * do not delete scalingpolicies * change default platform to ksonnet * default --version to master until 0.5.0 * remove verbose from spark-operator apply * address comments from Jeremy * address comments from Jeremy * fixes #2537, refer to no platform as 'none' * add general test for kfctl in Makefile that also tests the plugin capability * remove kftcl config since we now have bootstrap config, fix for DefaultPlatform="none" * change test of platform none to create directory none, updated README * remove reference to docker-for-desktop * update README.md, remove --debug flag (not used) * fixes 'kfctl (golang) Simplify the child/parent relationships in platforms' * fix backendconfig name (#2542) * fix backendconfig name * fix test * pin to the latest verified tf-operator commit in order to enable changes in tf-operator HEAD whcih likely break kubeflow checkout tests * fix (#2506) * Added unit tests for GetUpdatedPolicy at gcpUtils.go - bootstrap/app (#2516) * Added tests for 'GetUpdatedPolicy' - bootstrap/app * Fix typo - gcpUtils_test.go * Added a windows equivalent script for the makefile for central-dashboard (#2518) * update katib component (#2553) * Refactor for Express + Webpack build stack. (#2572) * Refactor for Express + Webpack build stack. - Add webpack to process front-end code in public/ - Add TS and Express for server backend - Update Dockerfile to run the server as Node. - Fix issue with external links. * Updates based on feedback * address review comments * move tests under test directory * Add ESLint to enforce Google JS style in public/ (#2579) * Add ESLint to enforce Google JS style in public/ * Switch to 4 space tabs * Updating python package imports related to tf-operator (#2581) * updated tf-operator imports * fixed a typo * fixed a typo * Profiles e2e test (#2404) * profiles e2e test * fix * review comment * fix * fix * fix * fix * add retry * fix * Update the jupyter-web-app Image (#2589) Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Golang profile controller (#2533) * init * create api * wip * fix * fix * fix * fix test * address review * fix test * make it cluster scope * fix test * fix test * fix * turnoff jupyter images authentication (#2591) * Updated Jupyter image paths for using in notebook controller (#2552) * updated jupyter image paths for using in notebook conroller * removed 1.4.1 image from the list of options * updated jupyter image tags to point to the latest verified ones * updated jupyter images to allow any origin (#2606) * Removing prune for Subresource field in CRD (#2598) * Adding backOffLimit field to metric collector spec (#2597) * remove global resources by label rather than listing all and looking for the label * mv client to coordinator * Fixed Dashboard v0.0.1 w/ Polymer3 + Webpack (#2576) * Multiple issues tackled: - Fixed webpack for windows (components/ is not a valid path in windows) - Spacing and semicolons matched to old-codebase - Added cross-platform support to npm run targets - Added missing dep (babel/polyfill) - Dashboard build version is now corrected and hover will reveal kubeflow vs Dashboard versions - Pug fixed to match syntax for empty-value attributes - Updated webpack for readability * Added babel-polyfill to runtime deps rather than dev deps * [Windows-only] Color coded all make output, Tag check added, failures now have messages, also dirty git output now has a warning during build * Updated new dashboard reference :v20190227-v0.4.0-rc.1-176-g738de035 * Fixed Multiple changes: - Server port fixed to 8082 which is what is needed for GKE deployment - Refactored some code * Final working state for new image * TS Changes * Build updated to match latest :v20190301-v0.4.0-rc.1-183-g07eecd2f * Migrated dev dependency to correct location * Uncommented linter, fixed PR comments from @prodonjs * updated jupyter images based on postsubmit results of 97b94c0 (#2611) * Adding additional printer columns to display status (#2609) * Adding additional printer columns to display status * Fixing tests * Semi-working kfctl go binary (#2587) * Add flag --skip-init-gcp-project * fix gcpInitProject * update error msg * add deployment creation logic * fix bugs on generateDMData * fix bugs * add default config to ipName and hostName * fix bugs on app.yaml * add Fingerprint to update deployment * make network and gcfs as DM optional * add todos * add iam policy handlers * finish patch iam policy * adding poll op to dm api calls * add gke api version * temp * change mysqlPd/nfsPd names * remove * temp * add config types * add basic auth flag * move config to gcp * link use basic auth flag to generate * use basic auth to determine config path * change to gabrielwen repo for now * change config on the fly * change repo path * create new Genereate method * add required packages * implement NewGenerate * clean up default config and add on-the-fly configs * move config handling to ksonnet.go * add ks param set * add application as component * fix typo * add simple TargetConfiguration factory * add fixed backof * temp * fix from merge * make namespace in gke * bind user as default-admin in gke * finish createSecrets * write oauth secret * uncomment ks apply * add ks show * add ks apply logic to k8sutils * uncomment deployment * build rest.Config from remote * impersonate as admin for rbac * use remote config as REST client config * update default.yaml path * complete apply logic * add retry * add ignore * merge * remove temp comment * use backoff in gcp * remove progress as it is not updating * use backoff * use backoff * use default zone * fix zone config * add doc * add doc * update Makefile * fix flag doc * fix * doc * doc * add doc * sort packages and component names * shell out to gcloud and kubectl for now * remove some configs * fix * remove GkeApiVersion * add todo * add notebook-controller back * change names * updates * add field InitRequired * remove KsInitParams * migrate * Add status to notebook (#2558) * wip * wip * update test to check status condition * fix * Add prodonjs to OWNERS (#2613) * Changed sidebar tab from JupyterHub to Notebooks (#2619) * fix (#2620) * pf gcb (#2603) * Setting the base_url flag of Jupyter images (#2627) * set base_url of jupyter * change the env base_url to NB_PREFIX to sync with #2620 * fix (#2631) * resolve merge conflicts from upstream/master, fix error on master when --platform=none or not specified * store non exist parameter differently (#2625) * fix (#2632) * update pipeline system images to 0.1.12 release (#2637) * updated jupyter image paths to point to the latest ones (#2638) * add retry around ks init (#2643) * add retry around ks init * update comments * merge errors * Getting started card completed (#2651) * Getting Started card content added, and un-hidden. Windows support added in webpack to run dev builds * Getting Started card fully implemented from mocks * Spaced out cards a little more * Got eslint working! And fixed eslint issues in webpack

@prodonjs

…forms' (kubeflow#2547) * revert to prior version * fixes kfctl (golang) rename 'ks' directory and app to ksonnet and ksonnet.go respectively * fixes 'kfctl - Fetch registry automatically' * fixes 'refactor gcp, minikube, docker-for-desktop, ack to be kfctl plugins' * change DefaultDevRepo to point to just the repo not repo/kubeflow * fixes 'refactor gcp, minikube, docker-for-desktop, ack to be kfctl plugins' * plugins for all existing KfApp instances {ksonnet, minikube, foo}. * update golang version * fixes 'docker-for-desktop' * delete meta-controller-cluster-role-binding * update README.md * remove unused var DEBUG * remove bootstrap/cmd/plugins/foo.go * fixes kfctl (golang) rename 'ks' directory and app to ksonnet and ksonnet.go respectively * fixes 'add the gcp platform to kfctl (golang)' * fix merge conflict * cli option --project is an init flag * fix to write out project to app.yaml * fix for nil project check * include fix for kubeflow#2367 * added updateDM, createSecrets to Apply * fixes to macros * merge changes from upstream master * remove old files, update README.md * fix for init app-name where app-name is not a path * fix for gcp apply * code to create a Gcp Secret, doesn't do correct auth right now * snapshot * update on downloadK8sManifests, gcpInitProject, added TODO's * additional work on gcpInitProject * gcpInitProject now works but requires GOOGLE_APPLICATION_CREDENTIALS env var * initial pass on updateDeployment * update README.md to described static vs dynamic platforms * updates to Gcp.updateDM, added GOOGLE_APPLICATION_CREDENTIALS to the Makefile tests * minor changes on init, generate usage * snapshot * additional global resources identified * do not delete scalingpolicies * change default platform to ksonnet * default --version to master until 0.5.0 * remove verbose from spark-operator apply * address comments from Jeremy * address comments from Jeremy * fixes kubeflow#2537, refer to no platform as 'none' * add general test for kfctl in Makefile that also tests the plugin capability * remove kftcl config since we now have bootstrap config, fix for DefaultPlatform="none" * change test of platform none to create directory none, updated README * remove reference to docker-for-desktop * update README.md, remove --debug flag (not used) * fixes 'kfctl (golang) Simplify the child/parent relationships in platforms' * fix backendconfig name (kubeflow#2542) * fix backendconfig name * fix test * pin to the latest verified tf-operator commit in order to enable changes in tf-operator HEAD whcih likely break kubeflow checkout tests * fix (kubeflow#2506) * Added unit tests for GetUpdatedPolicy at gcpUtils.go - bootstrap/app (kubeflow#2516) * Added tests for 'GetUpdatedPolicy' - bootstrap/app * Fix typo - gcpUtils_test.go * Added a windows equivalent script for the makefile for central-dashboard (kubeflow#2518) * update katib component (kubeflow#2553) * Refactor for Express + Webpack build stack. (kubeflow#2572) * Refactor for Express + Webpack build stack. - Add webpack to process front-end code in public/ - Add TS and Express for server backend - Update Dockerfile to run the server as Node. - Fix issue with external links. * Updates based on feedback * address review comments * move tests under test directory * Add ESLint to enforce Google JS style in public/ (kubeflow#2579) * Add ESLint to enforce Google JS style in public/ * Switch to 4 space tabs * Updating python package imports related to tf-operator (kubeflow#2581) * updated tf-operator imports * fixed a typo * fixed a typo * Profiles e2e test (kubeflow#2404) * profiles e2e test * fix * review comment * fix * fix * fix * fix * add retry * fix * Update the jupyter-web-app Image (kubeflow#2589) Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Golang profile controller (kubeflow#2533) * init * create api * wip * fix * fix * fix * fix test * address review * fix test * make it cluster scope * fix test * fix test * fix * turnoff jupyter images authentication (kubeflow#2591) * Updated Jupyter image paths for using in notebook controller (kubeflow#2552) * updated jupyter image paths for using in notebook conroller * removed 1.4.1 image from the list of options * updated jupyter image tags to point to the latest verified ones * updated jupyter images to allow any origin (kubeflow#2606) * Removing prune for Subresource field in CRD (kubeflow#2598) * Adding backOffLimit field to metric collector spec (kubeflow#2597) * remove global resources by label rather than listing all and looking for the label * mv client to coordinator * Fixed Dashboard v0.0.1 w/ Polymer3 + Webpack (kubeflow#2576) * Multiple issues tackled: - Fixed webpack for windows (components/ is not a valid path in windows) - Spacing and semicolons matched to old-codebase - Added cross-platform support to npm run targets - Added missing dep (babel/polyfill) - Dashboard build version is now corrected and hover will reveal kubeflow vs Dashboard versions - Pug fixed to match syntax for empty-value attributes - Updated webpack for readability * Added babel-polyfill to runtime deps rather than dev deps * [Windows-only] Color coded all make output, Tag check added, failures now have messages, also dirty git output now has a warning during build * Updated new dashboard reference :v20190227-v0.4.0-rc.1-176-g738de035 * Fixed Multiple changes: - Server port fixed to 8082 which is what is needed for GKE deployment - Refactored some code * Final working state for new image * TS Changes * Build updated to match latest :v20190301-v0.4.0-rc.1-183-g07eecd2f * Migrated dev dependency to correct location * Uncommented linter, fixed PR comments from @prodonjs * updated jupyter images based on postsubmit results of kubeflow@97b94c0 (kubeflow#2611) * Adding additional printer columns to display status (kubeflow#2609) * Adding additional printer columns to display status * Fixing tests * Semi-working kfctl go binary (kubeflow#2587) * Add flag --skip-init-gcp-project * fix gcpInitProject * update error msg * add deployment creation logic * fix bugs on generateDMData * fix bugs * add default config to ipName and hostName * fix bugs on app.yaml * add Fingerprint to update deployment * make network and gcfs as DM optional * add todos * add iam policy handlers * finish patch iam policy * adding poll op to dm api calls * add gke api version * temp * change mysqlPd/nfsPd names * remove * temp * add config types * add basic auth flag * move config to gcp * link use basic auth flag to generate * use basic auth to determine config path * change to gabrielwen repo for now * change config on the fly * change repo path * create new Genereate method * add required packages * implement NewGenerate * clean up default config and add on-the-fly configs * move config handling to ksonnet.go * add ks param set * add application as component * fix typo * add simple TargetConfiguration factory * add fixed backof * temp * fix from merge * make namespace in gke * bind user as default-admin in gke * finish createSecrets * write oauth secret * uncomment ks apply * add ks show * add ks apply logic to k8sutils * uncomment deployment * build rest.Config from remote * impersonate as admin for rbac * use remote config as REST client config * update default.yaml path * complete apply logic * add retry * add ignore * merge * remove temp comment * use backoff in gcp * remove progress as it is not updating * use backoff * use backoff * use default zone * fix zone config * add doc * add doc * update Makefile * fix flag doc * fix * doc * doc * add doc * sort packages and component names * shell out to gcloud and kubectl for now * remove some configs * fix * remove GkeApiVersion * add todo * add notebook-controller back * change names * updates * add field InitRequired * remove KsInitParams * migrate * Add status to notebook (kubeflow#2558) * wip * wip * update test to check status condition * fix * Add prodonjs to OWNERS (kubeflow#2613) * Changed sidebar tab from JupyterHub to Notebooks (kubeflow#2619) * fix (kubeflow#2620) * pf gcb (kubeflow#2603) * Setting the base_url flag of Jupyter images (kubeflow#2627) * set base_url of jupyter * change the env base_url to NB_PREFIX to sync with kubeflow#2620 * fix (kubeflow#2631) * resolve merge conflicts from upstream/master, fix error on master when --platform=none or not specified * store non exist parameter differently (kubeflow#2625) * fix (kubeflow#2632) * update pipeline system images to 0.1.12 release (kubeflow#2637) * updated jupyter image paths to point to the latest ones (kubeflow#2638) * add retry around ks init (kubeflow#2643) * add retry around ks init * update comments * merge errors * Getting started card completed (kubeflow#2651) * Getting Started card content added, and un-hidden. Windows support added in webpack to run dev builds * Getting Started card fully implemented from mocks * Spaced out cards a little more * Got eslint working! And fixed eslint issues in webpack

* init * create api * wip * fix * fix * fix * fix test * address review * fix test * make it cluster scope * fix test * fix test * fix

@prodonjs

…forms' (kubeflow#2547) * revert to prior version * fixes kfctl (golang) rename 'ks' directory and app to ksonnet and ksonnet.go respectively * fixes 'kfctl - Fetch registry automatically' * fixes 'refactor gcp, minikube, docker-for-desktop, ack to be kfctl plugins' * change DefaultDevRepo to point to just the repo not repo/kubeflow * fixes 'refactor gcp, minikube, docker-for-desktop, ack to be kfctl plugins' * plugins for all existing KfApp instances {ksonnet, minikube, foo}. * update golang version * fixes 'docker-for-desktop' * delete meta-controller-cluster-role-binding * update README.md * remove unused var DEBUG * remove bootstrap/cmd/plugins/foo.go * fixes kfctl (golang) rename 'ks' directory and app to ksonnet and ksonnet.go respectively * fixes 'add the gcp platform to kfctl (golang)' * fix merge conflict * cli option --project is an init flag * fix to write out project to app.yaml * fix for nil project check * include fix for kubeflow#2367 * added updateDM, createSecrets to Apply * fixes to macros * merge changes from upstream master * remove old files, update README.md * fix for init app-name where app-name is not a path * fix for gcp apply * code to create a Gcp Secret, doesn't do correct auth right now * snapshot * update on downloadK8sManifests, gcpInitProject, added TODO's * additional work on gcpInitProject * gcpInitProject now works but requires GOOGLE_APPLICATION_CREDENTIALS env var * initial pass on updateDeployment * update README.md to described static vs dynamic platforms * updates to Gcp.updateDM, added GOOGLE_APPLICATION_CREDENTIALS to the Makefile tests * minor changes on init, generate usage * snapshot * additional global resources identified * do not delete scalingpolicies * change default platform to ksonnet * default --version to master until 0.5.0 * remove verbose from spark-operator apply * address comments from Jeremy * address comments from Jeremy * fixes kubeflow#2537, refer to no platform as 'none' * add general test for kfctl in Makefile that also tests the plugin capability * remove kftcl config since we now have bootstrap config, fix for DefaultPlatform="none" * change test of platform none to create directory none, updated README * remove reference to docker-for-desktop * update README.md, remove --debug flag (not used) * fixes 'kfctl (golang) Simplify the child/parent relationships in platforms' * fix backendconfig name (kubeflow#2542) * fix backendconfig name * fix test * pin to the latest verified tf-operator commit in order to enable changes in tf-operator HEAD whcih likely break kubeflow checkout tests * fix (kubeflow#2506) * Added unit tests for GetUpdatedPolicy at gcpUtils.go - bootstrap/app (kubeflow#2516) * Added tests for 'GetUpdatedPolicy' - bootstrap/app * Fix typo - gcpUtils_test.go * Added a windows equivalent script for the makefile for central-dashboard (kubeflow#2518) * update katib component (kubeflow#2553) * Refactor for Express + Webpack build stack. (kubeflow#2572) * Refactor for Express + Webpack build stack. - Add webpack to process front-end code in public/ - Add TS and Express for server backend - Update Dockerfile to run the server as Node. - Fix issue with external links. * Updates based on feedback * address review comments * move tests under test directory * Add ESLint to enforce Google JS style in public/ (kubeflow#2579) * Add ESLint to enforce Google JS style in public/ * Switch to 4 space tabs * Updating python package imports related to tf-operator (kubeflow#2581) * updated tf-operator imports * fixed a typo * fixed a typo * Profiles e2e test (kubeflow#2404) * profiles e2e test * fix * review comment * fix * fix * fix * fix * add retry * fix * Update the jupyter-web-app Image (kubeflow#2589) Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Golang profile controller (kubeflow#2533) * init * create api * wip * fix * fix * fix * fix test * address review * fix test * make it cluster scope * fix test * fix test * fix * turnoff jupyter images authentication (kubeflow#2591) * Updated Jupyter image paths for using in notebook controller (kubeflow#2552) * updated jupyter image paths for using in notebook conroller * removed 1.4.1 image from the list of options * updated jupyter image tags to point to the latest verified ones * updated jupyter images to allow any origin (kubeflow#2606) * Removing prune for Subresource field in CRD (kubeflow#2598) * Adding backOffLimit field to metric collector spec (kubeflow#2597) * remove global resources by label rather than listing all and looking for the label * mv client to coordinator * Fixed Dashboard v0.0.1 w/ Polymer3 + Webpack (kubeflow#2576) * Multiple issues tackled: - Fixed webpack for windows (components/ is not a valid path in windows) - Spacing and semicolons matched to old-codebase - Added cross-platform support to npm run targets - Added missing dep (babel/polyfill) - Dashboard build version is now corrected and hover will reveal kubeflow vs Dashboard versions - Pug fixed to match syntax for empty-value attributes - Updated webpack for readability * Added babel-polyfill to runtime deps rather than dev deps * [Windows-only] Color coded all make output, Tag check added, failures now have messages, also dirty git output now has a warning during build * Updated new dashboard reference :v20190227-v0.4.0-rc.1-176-g738de035 * Fixed Multiple changes: - Server port fixed to 8082 which is what is needed for GKE deployment - Refactored some code * Final working state for new image * TS Changes * Build updated to match latest :v20190301-v0.4.0-rc.1-183-g07eecd2f * Migrated dev dependency to correct location * Uncommented linter, fixed PR comments from @prodonjs * updated jupyter images based on postsubmit results of kubeflow@e896cd3 (kubeflow#2611) * Adding additional printer columns to display status (kubeflow#2609) * Adding additional printer columns to display status * Fixing tests * Semi-working kfctl go binary (kubeflow#2587) * Add flag --skip-init-gcp-project * fix gcpInitProject * update error msg * add deployment creation logic * fix bugs on generateDMData * fix bugs * add default config to ipName and hostName * fix bugs on app.yaml * add Fingerprint to update deployment * make network and gcfs as DM optional * add todos * add iam policy handlers * finish patch iam policy * adding poll op to dm api calls * add gke api version * temp * change mysqlPd/nfsPd names * remove * temp * add config types * add basic auth flag * move config to gcp * link use basic auth flag to generate * use basic auth to determine config path * change to gabrielwen repo for now * change config on the fly * change repo path * create new Genereate method * add required packages * implement NewGenerate * clean up default config and add on-the-fly configs * move config handling to ksonnet.go * add ks param set * add application as component * fix typo * add simple TargetConfiguration factory * add fixed backof * temp * fix from merge * make namespace in gke * bind user as default-admin in gke * finish createSecrets * write oauth secret * uncomment ks apply * add ks show * add ks apply logic to k8sutils * uncomment deployment * build rest.Config from remote * impersonate as admin for rbac * use remote config as REST client config * update default.yaml path * complete apply logic * add retry * add ignore * merge * remove temp comment * use backoff in gcp * remove progress as it is not updating * use backoff * use backoff * use default zone * fix zone config * add doc * add doc * update Makefile * fix flag doc * fix * doc * doc * add doc * sort packages and component names * shell out to gcloud and kubectl for now * remove some configs * fix * remove GkeApiVersion * add todo * add notebook-controller back * change names * updates * add field InitRequired * remove KsInitParams * migrate * Add status to notebook (kubeflow#2558) * wip * wip * update test to check status condition * fix * Add prodonjs to OWNERS (kubeflow#2613) * Changed sidebar tab from JupyterHub to Notebooks (kubeflow#2619) * fix (kubeflow#2620) * pf gcb (kubeflow#2603) * Setting the base_url flag of Jupyter images (kubeflow#2627) * set base_url of jupyter * change the env base_url to NB_PREFIX to sync with kubeflow#2620 * fix (kubeflow#2631) * resolve merge conflicts from upstream/master, fix error on master when --platform=none or not specified * store non exist parameter differently (kubeflow#2625) * fix (kubeflow#2632) * update pipeline system images to 0.1.12 release (kubeflow#2637) * updated jupyter image paths to point to the latest ones (kubeflow#2638) * add retry around ks init (kubeflow#2643) * add retry around ks init * update comments * merge errors * Getting started card completed (kubeflow#2651) * Getting Started card content added, and un-hidden. Windows support added in webpack to run dev builds * Getting Started card fully implemented from mocks * Spaced out cards a little more * Got eslint working! And fixed eslint issues in webpack

* init * create api * wip * fix * fix * fix * fix test * address review * fix test * make it cluster scope * fix test * fix test * fix

@prodonjs

…forms' (kubeflow#2547) * revert to prior version * fixes kfctl (golang) rename 'ks' directory and app to ksonnet and ksonnet.go respectively * fixes 'kfctl - Fetch registry automatically' * fixes 'refactor gcp, minikube, docker-for-desktop, ack to be kfctl plugins' * change DefaultDevRepo to point to just the repo not repo/kubeflow * fixes 'refactor gcp, minikube, docker-for-desktop, ack to be kfctl plugins' * plugins for all existing KfApp instances {ksonnet, minikube, foo}. * update golang version * fixes 'docker-for-desktop' * delete meta-controller-cluster-role-binding * update README.md * remove unused var DEBUG * remove bootstrap/cmd/plugins/foo.go * fixes kfctl (golang) rename 'ks' directory and app to ksonnet and ksonnet.go respectively * fixes 'add the gcp platform to kfctl (golang)' * fix merge conflict * cli option --project is an init flag * fix to write out project to app.yaml * fix for nil project check * include fix for kubeflow#2367 * added updateDM, createSecrets to Apply * fixes to macros * merge changes from upstream master * remove old files, update README.md * fix for init app-name where app-name is not a path * fix for gcp apply * code to create a Gcp Secret, doesn't do correct auth right now * snapshot * update on downloadK8sManifests, gcpInitProject, added TODO's * additional work on gcpInitProject * gcpInitProject now works but requires GOOGLE_APPLICATION_CREDENTIALS env var * initial pass on updateDeployment * update README.md to described static vs dynamic platforms * updates to Gcp.updateDM, added GOOGLE_APPLICATION_CREDENTIALS to the Makefile tests * minor changes on init, generate usage * snapshot * additional global resources identified * do not delete scalingpolicies * change default platform to ksonnet * default --version to master until 0.5.0 * remove verbose from spark-operator apply * address comments from Jeremy * address comments from Jeremy * fixes kubeflow#2537, refer to no platform as 'none' * add general test for kfctl in Makefile that also tests the plugin capability * remove kftcl config since we now have bootstrap config, fix for DefaultPlatform="none" * change test of platform none to create directory none, updated README * remove reference to docker-for-desktop * update README.md, remove --debug flag (not used) * fixes 'kfctl (golang) Simplify the child/parent relationships in platforms' * fix backendconfig name (kubeflow#2542) * fix backendconfig name * fix test * pin to the latest verified tf-operator commit in order to enable changes in tf-operator HEAD whcih likely break kubeflow checkout tests * fix (kubeflow#2506) * Added unit tests for GetUpdatedPolicy at gcpUtils.go - bootstrap/app (kubeflow#2516) * Added tests for 'GetUpdatedPolicy' - bootstrap/app * Fix typo - gcpUtils_test.go * Added a windows equivalent script for the makefile for central-dashboard (kubeflow#2518) * update katib component (kubeflow#2553) * Refactor for Express + Webpack build stack. (kubeflow#2572) * Refactor for Express + Webpack build stack. - Add webpack to process front-end code in public/ - Add TS and Express for server backend - Update Dockerfile to run the server as Node. - Fix issue with external links. * Updates based on feedback * address review comments * move tests under test directory * Add ESLint to enforce Google JS style in public/ (kubeflow#2579) * Add ESLint to enforce Google JS style in public/ * Switch to 4 space tabs * Updating python package imports related to tf-operator (kubeflow#2581) * updated tf-operator imports * fixed a typo * fixed a typo * Profiles e2e test (kubeflow#2404) * profiles e2e test * fix * review comment * fix * fix * fix * fix * add retry * fix * Update the jupyter-web-app Image (kubeflow#2589) Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Golang profile controller (kubeflow#2533) * init * create api * wip * fix * fix * fix * fix test * address review * fix test * make it cluster scope * fix test * fix test * fix * turnoff jupyter images authentication (kubeflow#2591) * Updated Jupyter image paths for using in notebook controller (kubeflow#2552) * updated jupyter image paths for using in notebook conroller * removed 1.4.1 image from the list of options * updated jupyter image tags to point to the latest verified ones * updated jupyter images to allow any origin (kubeflow#2606) * Removing prune for Subresource field in CRD (kubeflow#2598) * Adding backOffLimit field to metric collector spec (kubeflow#2597) * remove global resources by label rather than listing all and looking for the label * mv client to coordinator * Fixed Dashboard v0.0.1 w/ Polymer3 + Webpack (kubeflow#2576) * Multiple issues tackled: - Fixed webpack for windows (components/ is not a valid path in windows) - Spacing and semicolons matched to old-codebase - Added cross-platform support to npm run targets - Added missing dep (babel/polyfill) - Dashboard build version is now corrected and hover will reveal kubeflow vs Dashboard versions - Pug fixed to match syntax for empty-value attributes - Updated webpack for readability * Added babel-polyfill to runtime deps rather than dev deps * [Windows-only] Color coded all make output, Tag check added, failures now have messages, also dirty git output now has a warning during build * Updated new dashboard reference :v20190227-v0.4.0-rc.1-176-g738de035 * Fixed Multiple changes: - Server port fixed to 8082 which is what is needed for GKE deployment - Refactored some code * Final working state for new image * TS Changes * Build updated to match latest :v20190301-v0.4.0-rc.1-183-g07eecd2f * Migrated dev dependency to correct location * Uncommented linter, fixed PR comments from @prodonjs * updated jupyter images based on postsubmit results of kubeflow@598dd06 (kubeflow#2611) * Adding additional printer columns to display status (kubeflow#2609) * Adding additional printer columns to display status * Fixing tests * Semi-working kfctl go binary (kubeflow#2587) * Add flag --skip-init-gcp-project * fix gcpInitProject * update error msg * add deployment creation logic * fix bugs on generateDMData * fix bugs * add default config to ipName and hostName * fix bugs on app.yaml * add Fingerprint to update deployment * make network and gcfs as DM optional * add todos * add iam policy handlers * finish patch iam policy * adding poll op to dm api calls * add gke api version * temp * change mysqlPd/nfsPd names * remove * temp * add config types * add basic auth flag * move config to gcp * link use basic auth flag to generate * use basic auth to determine config path * change to gabrielwen repo for now * change config on the fly * change repo path * create new Genereate method * add required packages * implement NewGenerate * clean up default config and add on-the-fly configs * move config handling to ksonnet.go * add ks param set * add application as component * fix typo * add simple TargetConfiguration factory * add fixed backof * temp * fix from merge * make namespace in gke * bind user as default-admin in gke * finish createSecrets * write oauth secret * uncomment ks apply * add ks show * add ks apply logic to k8sutils * uncomment deployment * build rest.Config from remote * impersonate as admin for rbac * use remote config as REST client config * update default.yaml path * complete apply logic * add retry * add ignore * merge * remove temp comment * use backoff in gcp * remove progress as it is not updating * use backoff * use backoff * use default zone * fix zone config * add doc * add doc * update Makefile * fix flag doc * fix * doc * doc * add doc * sort packages and component names * shell out to gcloud and kubectl for now * remove some configs * fix * remove GkeApiVersion * add todo * add notebook-controller back * change names * updates * add field InitRequired * remove KsInitParams * migrate * Add status to notebook (kubeflow#2558) * wip * wip * update test to check status condition * fix * Add prodonjs to OWNERS (kubeflow#2613) * Changed sidebar tab from JupyterHub to Notebooks (kubeflow#2619) * fix (kubeflow#2620) * pf gcb (kubeflow#2603) * Setting the base_url flag of Jupyter images (kubeflow#2627) * set base_url of jupyter * change the env base_url to NB_PREFIX to sync with kubeflow#2620 * fix (kubeflow#2631) * resolve merge conflicts from upstream/master, fix error on master when --platform=none or not specified * store non exist parameter differently (kubeflow#2625) * fix (kubeflow#2632) * update pipeline system images to 0.1.12 release (kubeflow#2637) * updated jupyter image paths to point to the latest ones (kubeflow#2638) * add retry around ks init (kubeflow#2643) * add retry around ks init * update comments * merge errors * Getting started card completed (kubeflow#2651) * Getting Started card content added, and un-hidden. Windows support added in webpack to run dev builds * Getting Started card fully implemented from mocks * Spaced out cards a little more * Got eslint working! And fixed eslint issues in webpack

lluunn added 3 commits February 19, 2019 16:34

init

e45a6c2

create api

d363b12

wip

e3d48c5

k8s-ci-robot added the do-not-merge/work-in-progress label Feb 22, 2019

googlebot added the cla: yes label Feb 22, 2019

k8s-ci-robot requested review from ellistarn and pdmack February 22, 2019 17:11

k8s-ci-robot added the size/XXL label Feb 22, 2019

lluunn added 3 commits February 22, 2019 13:36

fix

c64db59

fix

3d6f665

fix

e6406d9

lluunn changed the title ~~WIP profile controller~~ Golang profile controller Feb 23, 2019

k8s-ci-robot removed the do-not-merge/work-in-progress label Feb 23, 2019

k8s-ci-robot assigned jlewi and kkasravi Feb 23, 2019

lluunn added 2 commits February 22, 2019 16:37

Merge branch 'master' into pfcr

c6dc79b

fix test

c99152a

k8s-ci-robot assigned kunmingg Feb 26, 2019

kunmingg reviewed Feb 27, 2019

View reviewed changes

kkasravi suggested changes Feb 27, 2019

View reviewed changes

lluunn added 5 commits February 27, 2019 12:03

address review

a1facd6

fix test

61fa470

Merge branch 'master' into pfcr

bfc8bb2

make it cluster scope

7492845

Merge branch 'master' into pfcr

d91b4f8

k8s-ci-robot added the do-not-merge/hold label Feb 28, 2019

Merge branch 'master' into pfcr

d5dda9d

fix test

d6bc2d7

k8s-ci-robot removed the do-not-merge/hold label Mar 1, 2019

lluunn added 2 commits February 28, 2019 17:23

fix test

1a80265

fix

b929f25

k8s-ci-robot added the lgtm label Mar 1, 2019

k8s-ci-robot added the approved label Mar 1, 2019

k8s-ci-robot merged commit 22592d5 into kubeflow:master Mar 1, 2019

prodonjs pushed a commit to prodonjs/kubeflow that referenced this pull request Mar 4, 2019

Golang profile controller (kubeflow#2533)

bbff6d9

* init * create api * wip * fix * fix * fix * fix test * address review * fix test * make it cluster scope * fix test * fix test * fix

kkasravi pushed a commit to kkasravi/kubeflow that referenced this pull request Mar 8, 2019

Golang profile controller (kubeflow#2533)

75346d5

* init * create api * wip * fix * fix * fix * fix test * address review * fix test * make it cluster scope * fix test * fix test * fix

kkasravi pushed a commit to kkasravi/kubeflow that referenced this pull request Mar 8, 2019

Golang profile controller (kubeflow#2533)

ada6e9a

* init * create api * wip * fix * fix * fix * fix test * address review * fix test * make it cluster scope * fix test * fix test * fix

saffaalvi pushed a commit to StatCan/kubeflow that referenced this pull request Feb 11, 2021

Golang profile controller (kubeflow#2533)

e31e6cf

* init * create api * wip * fix * fix * fix * fix test * address review * fix test * make it cluster scope * fix test * fix test * fix

saffaalvi pushed a commit to StatCan/kubeflow that referenced this pull request Feb 12, 2021

Golang profile controller (kubeflow#2533)

8b6742b

* init * create api * wip * fix * fix * fix * fix test * address review * fix test * make it cluster scope * fix test * fix test * fix

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Golang profile controller #2533

Golang profile controller #2533

lluunn commented Feb 22, 2019 •

edited

lluunn commented Feb 23, 2019

lluunn commented Feb 26, 2019

kunmingg left a comment

kunmingg Feb 26, 2019

lluunn Feb 27, 2019

jlewi Feb 28, 2019

lluunn Feb 28, 2019

kunmingg Feb 27, 2019

lluunn Feb 27, 2019

kkasravi Feb 27, 2019

lluunn Feb 27, 2019

kkasravi Feb 27, 2019

kkasravi Feb 27, 2019

kkasravi Feb 27, 2019

lluunn Feb 27, 2019

kkasravi Feb 27, 2019

lluunn Feb 27, 2019

kkasravi Feb 27, 2019

lluunn Feb 27, 2019

lluunn commented Feb 28, 2019

lluunn commented Mar 1, 2019

lluunn commented Mar 1, 2019

jlewi commented Mar 1, 2019

k8s-ci-robot commented Mar 1, 2019

kunmingg commented Mar 1, 2019

kunmingg commented Mar 1, 2019

Golang profile controller #2533

Golang profile controller #2533

Conversation

lluunn commented Feb 22, 2019 • edited

lluunn commented Feb 23, 2019

lluunn commented Feb 26, 2019

kunmingg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lluunn commented Feb 28, 2019

lluunn commented Mar 1, 2019

lluunn commented Mar 1, 2019

jlewi commented Mar 1, 2019

k8s-ci-robot commented Mar 1, 2019

kunmingg commented Mar 1, 2019

kunmingg commented Mar 1, 2019

lluunn commented Feb 22, 2019 •

edited