Use ksonnet to configure/deploy Kubeflow #36

jlewi · 2017-12-14T00:30:35Z

Use ksonnet to customize/deploy Kubeflow.

Its intended to inform the discussion in Tooling to manage configuration and deployment #23
The current implementation supports JupyterHub, TfJob operator, TfServing
The README.ksonnet.md provides instructions on how to use ksonnet.
Provide a prototype for running the TfCnn benchmark.
The directory organization is based on what ksonnet expects. Specifically the directory within the repo gets checked out into the repo. So we use the directory kubeflow so that pkgs will be installed in ".../vendor/kubeflow/".
The core component has an option to specify a list of persistent disks to be exported and mounted via NFS.

googlebot · 2017-12-14T00:30:38Z

So there's good news and bad news.

👍 The good news is that everyone that needs to sign a CLA (the pull request submitter and all commit authors) have done so. Everything is all good there.

😕 The bad news is that it appears that one or more commits were authored by someone other than the pull request submitter. We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that here in the pull request.

Note to project maintainer: This is a terminal state, meaning the cla/google commit status will not change from this State. It's up to you to confirm consent of the commit author(s) and merge this pull request when appropriate.

jlewi · 2017-12-18T00:37:15Z

This is ready for review.

/cc @foxish @vishh

paulbaumgart · 2017-12-18T22:53:45Z

README.ksonnet.md

+
+## Build ks
+
+ksonnet doesn't support adding non default registries yet ((ksonnet/ksonnet/issues/38)[https://github.com/ksonnet/ksonnet/issues/38]) so we need to modify and [build from source](https://ksonnet.io/docs/build-install).


GitHub isn't rendering this link correctly. I believe it should be: [ksonnet/ksonnet/issues/38](https://github.com/ksonnet/ksonnet/issues/38)

paulbaumgart · 2017-12-18T23:04:33Z

README.ksonnet.md

+go get github.com/ksonnet/ksonnet
+```
+
+Checkout PR 228 which adds support for registry add


This has been merged: ksonnet/ksonnet@29e3cd4

jlewi · 2017-12-19T00:44:52Z

Thanks @paulbaumgart PTAL

jlewi · 2017-12-19T01:00:01Z

@foxish @vishh Do you want to review this before I merge it in?

paulbaumgart · 2017-12-19T01:02:08Z

@jlewi The README.ksonnet.md updates look good. Might be worth taking another pass at https://github.com/jlewi/kubeflow/blob/ksonnet_crd/kubeflow/README.md (perhaps just linking to the README.ksonnet.md section?)

The doc-gen link in kubeflow/README.md also seems to be broken.

jlewi · 2017-12-19T06:03:35Z

@paulbaumgart Thanks. I also auto-generated READMEs for each component.

foxish

The abstraction seems ok although quite a bit complicated - it is a bit concerning how much friction here is due to changes we're waiting on in ksonnet.

foxish · 2017-12-19T14:35:47Z

README.ksonnet.md

+
+You need a version of ksonnet newer than the [0.7.0 release](https://github.com/ksonnet/ksonnet/releases).
+
+As of this writing, Heptio hasn't released any newer prebuilt binaries so you will need to [build from source](https://ksonnet.io/docs/build-install).


Will there be a new release soon? This seems like a bit of friction until then.

Hopefully, heptio might push a release this week but no guarantees.

foxish · 2017-12-19T14:37:24Z

README.ksonnet.md

+
+Create the Kubeflow core component. The core component includes 
+  * JupyterHub
+  * TensorFlow job controller


Rename core to training?

Naming something core indicates that it's a dependency for other subsequent steps, which is not strictly true here I think.

JupyterHub isn't training specific and the CRD is needed for subsequent steps. The idea here is that we have certain core components that you probably only deploy once. Whereas for tf-serving and tf-job you probably have multiple ones.

I'm open to other names but training doesn't seem right.

foxish · 2017-12-19T14:39:26Z

README.ksonnet.md

+Parameters can be set using `ks param` e.g. to set the Docker image used
+
+```
+ks param set ${JOB_NAME} image ${IMAGE}


Can we instead recommend the declarative method here? I think people would want to version their config as a reproducible artifact, and specifying it this way doesn't allow doing that.

It is versioned. ks param set modifies params.jsonnet which is checked into code control.

foxish · 2017-12-19T14:40:24Z

README.ksonnet.md

+ks env add cloud
+ks param set --cloud=gke
+```
+   * The cloud parameter triggers a set of curated cloud configs.


Describe what parameters this sets?

Right now nothing, but it should control things like the CRD config for different clusters.

foxish · 2017-12-19T14:43:41Z

hack/convert_manifest_to_jsonnet.py

@@ -0,0 +1,29 @@
+"""A simple utility to convert a manifest to corresponding jsonnet."""


Parts of this script seem brittle, what's the purpose of it?

It was a hack hence its location in "hacks" for converting our existing yaml manifests to jsonnet.
In a subsequent PR when we clean up the old stuff and move to ksonnet we can delete it.

foxish · 2017-12-19T14:45:58Z

kubeflow/core/README.md

+
+### io.ksonnet.pkg.kubeflow-core
+
+Kubeflow core components


Same comment as above about naming parts of it core

foxish · 2017-12-19T14:48:39Z

kubeflow/core/nfs.libsonnet

+      "kind": "ServiceAccount", 
+      "metadata": {
+        "labels": {
+          "app": name + "nfs-provisioner"


Should the nfs-provisioner be its own component? Not sure if it is here already. I see it as - if it launches a new (optional) "thing" in the user's cluster, it should be its own component.

Ideally and long term yes. I think the ideal way to do this would be to have distinct packages for each thing; e.g a package for JupyterHub, a package for Spark, a package for the CRD etc...

We could then have one or more packages with one or more prototypes defining useful combinations of this components. Its not immediately clear what the organizing principle for those higher level packages should be.

In the short term though, its more convenient for users if they have fewer packages to install. So that was the motivation for putting them all in the same package.

Using more packages should be reasonable when ksonnet adds better support for managing package dependencies ; e.g. so that core can depend on some other packages and installing core automatically pulls in the other packages.

foxish · 2017-12-19T14:50:51Z

kubeflow/core/parts.yaml

+   "keywords": [
+      "kubeflow",
+      "tensorflow",
+      "database"


jlewi · 2017-12-19T19:47:00Z

@foxish this should be ready for another look.

foxish · 2017-12-20T00:34:10Z

Please squash before merging. Also - can we have an issue tracking the fixes that need to be done afterwards (like removing the hack/ directory, separating out NFS provisioner, etc)

…ace.

* Jupyter is hitting problems in non-default namespaces because it relies on default service account.

…ving problems with NFS.

…ponents.

…dor directory.

* Remove instructions to PRs that have been merged. * Assume the PR adding ksonnet registry to google/kubeflow has been merged.

* Auto generate README files for each component.

jlewi · 2017-12-20T00:59:27Z

I had to force the merge because the CLA check wouldn't run.

* initial seldon manifests * Namespaced and clusterwide deployment * Remove labels added by commonLabels * initial seldon operator kustomize update * add application and recreate * update from review * change name to seldon-core-operator * update service selector

Signed-off-by: Ce Gao <gaoce@caicloud.io>

[pull] master from kubeflow:master

jlewi mentioned this pull request Dec 15, 2017

Enable PVCs for per-user persistent storage #22

Closed

jlewi changed the title ~~[Wip] Example of using ksonnet to configure/deploy Kubeflow~~ Use ksonnet to configure/deploy Kubeflow Dec 18, 2017

paulbaumgart reviewed Dec 18, 2017

View reviewed changes

foxish reviewed Dec 19, 2017

View reviewed changes

foxish previously approved these changes Dec 20, 2017

View reviewed changes

This was referenced Dec 20, 2017

Clean up repo after switching to ksonnet #41

Closed

Split up core ksonnet package into separate packages #42

Closed

jlewi added 14 commits December 19, 2017 16:54

Initialize a ksonnet Kubeflow app.

c174474

Create a basic deployment component.

de61b89

Create jsonnet configs for the resources needed to deploy the crd.

3225a20

Figure out to append volume options to the spawner.

c69e3b0

Add all the components needed for jupyterhub.

ac20e66

Add a variable for namespace.

055f3d4

Refactor the tf job components into a library parameterized by namesp…

d54b6bc

…ace.

juptyer config now works.

e0b4069

* Add Kubeflow component to configure things like namespace and cloud.

fb023a7

* Jupyter is hitting problems in non-default namespaces because it relies on default service account.

Latest.

5113d00

Resolve the permissions issues with JuptyerHub launching pods; now ha…

04a124a

…ving problems with NFS.

Comment out the use of NFS volumes since we haven't added the NFS com…

c195537

…ponents.

Move the files into the ksonnet directory.

2c8b6c4

Revert changes unrelated to ksonnet.

70ba3e0

jlewi added 19 commits December 19, 2017 16:54

Update the readme.

2634bf7

Update the README.

6e66a84

Rename the directory so that it gets checked out correctly in the ven…

973683f

…dor directory.

Move kubeflow so that it gets checked out correctly in vendor.

65dbd1b

Update imports

d62aeed

Move script into hack directory.

616bf5e

update comment.

5ef0b59

Fix imports.

6f85aaa

Start a tf-job template.

03d073f

Add a TfJob prototype

e6ed8ce

Create a prototype for running the TF CNN benchmark

a24f826

Remove test app from source control.

6b813b1

Some cleanup.

8b9ec63

Update the README since changes to ksonnet have been merged.

1f0003a

Update the README.

06b6429

* Remove instructions to PRs that have been merged. * Assume the PR adding ksonnet registry to google/kubeflow has been merged.

Fix a bug.

8345e25

Fix README.

c88cb47

* Fix bugs in parts.yaml

d523d37

* Auto generate README files for each component.

Fix keywords.

feb54f2

jlewi dismissed foxish’s stale review via feb54f2 December 20, 2017 00:54

jlewi force-pushed the ksonnet_crd branch from 256563b to feb54f2 Compare December 20, 2017 00:54

jlewi merged commit fe59730 into kubeflow:master Dec 20, 2017

jlewi mentioned this pull request Dec 20, 2017

Doc gen for kubeflow ksonnet registry #39

Closed

kimwnasptd pushed a commit to arrikto/kubeflow that referenced this pull request Mar 5, 2019

Add gabrielwen to google-team.members (kubeflow#36)

4a9f981

yanniszark pushed a commit to arrikto/kubeflow that referenced this pull request Feb 15, 2021

docs: Update katib (kubeflow#36)

9bfd388

Signed-off-by: Ce Gao <gaoce@caicloud.io>

VaishnaviHire pushed a commit to VaishnaviHire/kubeflow that referenced this pull request Dec 6, 2022

Merge pull request kubeflow#36 from kubeflow/master

a12faf1

[pull] master from kubeflow:master

Apetree100122 mentioned this pull request Dec 28, 2023

[Snyk] Security upgrade prometheus-query from 3.3.2 to 3.3.3 Apetree100122/kubeflow#4

Open

Apetree100122 mentioned this pull request Jan 7, 2024

[Snyk] Security upgrade prometheus-query from 3.3.2 to 3.3.3 Apetree100122/kubeflow#11

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use ksonnet to configure/deploy Kubeflow #36

Use ksonnet to configure/deploy Kubeflow #36

jlewi commented Dec 14, 2017 •

edited

Loading

googlebot commented Dec 14, 2017

jlewi commented Dec 18, 2017

paulbaumgart Dec 18, 2017

paulbaumgart Dec 18, 2017

jlewi commented Dec 19, 2017 •

edited

Loading

jlewi commented Dec 19, 2017

paulbaumgart commented Dec 19, 2017

jlewi commented Dec 19, 2017

foxish left a comment

foxish Dec 19, 2017

jlewi Dec 19, 2017

foxish Dec 19, 2017

foxish Dec 19, 2017 •

edited

Loading

jlewi Dec 19, 2017

foxish Dec 19, 2017

jlewi Dec 19, 2017

foxish Dec 19, 2017

jlewi Dec 19, 2017

foxish Dec 19, 2017

jlewi Dec 19, 2017

foxish Dec 19, 2017

foxish Dec 19, 2017 •

edited

Loading

jlewi Dec 19, 2017

foxish Dec 19, 2017

jlewi commented Dec 19, 2017

foxish commented Dec 20, 2017

jlewi commented Dec 20, 2017


		## Build ks

		ksonnet doesn't support adding non default registries yet ((ksonnet/ksonnet/issues/38)[https://github.com/ksonnet/ksonnet/issues/38]) so we need to modify and [build from source](https://ksonnet.io/docs/build-install).


		You need a version of ksonnet newer than the [0.7.0 release](https://github.com/ksonnet/ksonnet/releases).

		As of this writing, Heptio hasn't released any newer prebuilt binaries so you will need to [build from source](https://ksonnet.io/docs/build-install).

		@@ -0,0 +1,29 @@
		"""A simple utility to convert a manifest to corresponding jsonnet."""

Use ksonnet to configure/deploy Kubeflow #36

Use ksonnet to configure/deploy Kubeflow #36

Conversation

jlewi commented Dec 14, 2017 • edited Loading

googlebot commented Dec 14, 2017

jlewi commented Dec 18, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jlewi commented Dec 19, 2017 • edited Loading

jlewi commented Dec 19, 2017

paulbaumgart commented Dec 19, 2017

jlewi commented Dec 19, 2017

foxish left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

foxish Dec 19, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

foxish Dec 19, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jlewi commented Dec 19, 2017

foxish commented Dec 20, 2017

jlewi commented Dec 20, 2017

jlewi commented Dec 14, 2017 •

edited

Loading

jlewi commented Dec 19, 2017 •

edited

Loading

foxish Dec 19, 2017 •

edited

Loading

foxish Dec 19, 2017 •

edited

Loading