Ambassador refactor (#1207)

Jupyterhub refactor

Central Dashboard Refactor

TFJob Refactor

Spartakus

WIP

wip

update docs

bootstrap/cmd/bootstrap/app/server.go

@@ -517,7 +517,7 @@ func Run(opt *options.ServerOption) error {
 
 	// Component customization
 	for _, component := range bootConfig.App.Components {
-		if component.Name == "kubeflow-core" {
+		if component.Name == "jupyterhub" {
 			pvcMount := ""
 			if hasDefault {
 				pvcMount = "/home/jovyan"
@@ -526,7 +526,7 @@ func Run(opt *options.ServerOption) error {
 			err = actions.RunParamSet(map[string]interface{}{
 				actions.OptionApp:   kfApp,
 				actions.OptionName:  component.Name,
-				actions.OptionPath:  "jupyterNotebookPVCMount",
+				actions.OptionPath:  "notebookPVCMount",
 				actions.OptionValue: pvcMount,
 			})
 

bootstrap/config/default.yaml

@@ -8,5 +8,13 @@ app:
     - name: tf-serving
       registry: kubeflow
   components:
-    - name: kubeflow-core
-      prototype: kubeflow-core
+    - name: jupyterhub
+      prototype: jupyterhub
+    - name: ambassador
+      prototype: ambassador
+    - name: centraldashboard
+      prototype: centraldashboard
+    - name: tf-job-operator
+      prototype: tf-job-operator
+    - name: spartakus
+      prototype: spartakus

bootstrap/config/gcp_prototype.yaml

@@ -8,8 +8,16 @@ app:
     - name: tf-serving
       registry: kubeflow
   components:
-    - name: kubeflow-core
-      prototype: kubeflow-core
+    - name: jupyterhub
+      prototype: jupyterhub
+    - name: ambassador
+      prototype: ambassador
+    - name: centraldashboard
+      prototype: centraldashboard
+    - name: tf-job-operator
+      prototype: tf-job-operator
+    - name: spartakus
+      prototype: spartakus
     - name: cloud-endpoints
       prototype: cloud-endpoints
     - name: cert-manager
@@ -32,6 +40,15 @@ app:
       name: hostname
       # TODO: replace with Name of GCP project. This is fully qualified domain name to use with ingress.
       value: kubeflow.endpoints.<Project>.cloud.goog
-    - component: kubeflow-core
+    - component: jupyterhub
       name: jupyterHubAuthenticator
       value: iap
+    - component: jupyterhub
+      name: cloud
+      value: gke
+    - component: ambassador
+      name: cloud
+      value: gke
+    - component: spartakus
+      name: reportUsage
+      value: true

components/jupyterhub/README.md

@@ -69,7 +69,7 @@ Configuration for JupyterHub is shipped separately and contained within the conf
 the core component. To see a list of ksonnet parameters run
 
 ```
-ks prototype describe kubeflow-core
+ks prototype describe jupyterhub
 ```
 
 If the provided parameters don't provide the flexibility you need, you can take advantage of ksonnet to customize the core component and use a config file fully specified by you.
@@ -78,14 +78,14 @@ Configuration includes sections for KubeSpawner and Authenticators. Spawner para
 Jupyter notebooks, and configuration defining how JupyterHub creates and interacts with Kubernetes pods for individual notebooks.
 Authenticator parameters correspond to the authentication mechanism used by JupyterHub.
 
-Additional information about configuration can be found in the 
+Additional information about configuration can be found in the
 [Zero to JupyterHub with Kubernetes guide](https://zero-to-jupyterhub.readthedocs.io/en/latest/)
 and the [JupyterHub documentation](https://jupyterhub.readthedocs.io/en/latest/).
 
 
 ## Usage
 
-If you're using the quick-start, the external IP address of the JupyterHub 
+If you're using the quick-start, the external IP address of the JupyterHub
 instance can be obtained from `kubectl get svc`.
 
 ```commandline
@@ -127,7 +127,7 @@ via the default `PATH`.
 
 After creating the initial Hub and exposing it on a public IP address, you can add GitHub based authentication. First, you'll need to create a [GitHub oauth application](https://github.com/settings/applications/new). The callback URL would be of the form `http://xx.yy.zz.ww/hub/oauth_callback`.
 
-Once the GitHub application is created in the GitHub UI, update the 
+Once the GitHub application is created in the GitHub UI, update the
 `manifest/config.yaml` with the `callback_url`, `client_id` and `client_secret`
 provided by GitHub UI. You should comment out the `DummyAuthenticator` and
 set the JupyterHub `authenticator_class` to `GitHubOAuthenticator`. You will
@@ -150,7 +150,7 @@ kubectl delete pod tf-hub-0
 ```
 
 By deleting the old pod, a new pod will come up with the new configuration
-and be configured to use the GitHub authenticator you specified in the 
+and be configured to use the GitHub authenticator you specified in the
 previous step. You can additionally modify the JupyterHub configuration to add
 whitelists and admin users. For example, to limit the hub to only GitHub users,
 user1 and user2, one might use the following configuration:

docs_dev/ksonnet_packages.md

@@ -5,7 +5,9 @@
 - [Developer Guide For Ksonnet Packages](#developer-guide-for-ksonnet-packages)
   - [Creating New Ksonnet Components](#creating-new-ksonnet-components)
   - [Testing changes to ksonnet components](#testing-changes-to-ksonnet-components)
+      - [ERROR Component parameters for 'jupyterhub' already exists](#error-component-parameters-for-jupyterhub-already-exists)
     - [New Packages](#new-packages)
+      - [Pull Requests](#pull-requests)
 
 <!-- END doctoc generated TOC please keep comment here to allow auto update -->
 
@@ -25,7 +27,7 @@ Here are some instructions for creating new Kubeflow packages.
 			- Optional packages probably don't belong in core
 
 		- Otherwise, if there's an existing package that it belongs with put
-			it in that package 
+			it in that package
 		- If no existing package is a good fit create a new one
 
 1. If you are creating a new package follow these steps
@@ -58,10 +60,10 @@ Here are some instructions for creating new Kubeflow packages.
 1. Create one or more prototype files in the prototypes directory for your package
 
 	- Use [tf-prototype.jsonnet](https://github.com/kubeflow/kubeflow/tree/master/kubeflow/new-package-stub/prototypes/tf-prototype.jsonnet) as a template.
-	
+
 ## Testing changes to ksonnet components
 
-The easiest way to test ksonnet components is to follow the normal instructions for setting up a 
+The easiest way to test ksonnet components is to follow the normal instructions for setting up a
 ksonnet app to deploy kubeflow.
 
 Then replace the directory `vendor/kubeflow` with a symbolic link to `${GIT_KUBEFLOW}/kubeflow`
@@ -73,16 +75,16 @@ ln -sf ${GIT_KUBEFLOW}/kubeflow  ${APP_DIR}/vendor/kubeflow
 this way changes to your .libsonnet files will automaticaly be reflected in your components.
 
 If you make changes to prototypes you need to regenerate the prototype. You can just delete the `.jsonnet`
-file in your app's component directory and then regenerate the component using `ks generate`. 
+file in your app's component directory and then regenerate the component using `ks generate`.
 
 ```
-rm -rf ${APP_DIR}/components/kubeflow-core.jsonnet 
-ks generate kubeflow-core kubeflow-core
-INFO  Writing component at 'components/kubeflow-core'
-ERROR Component parameters for 'kubeflow-core' already exists
+rm -rf ${APP_DIR}/components/jupyterhub.jsonnet
+ks generate jupyterhub jupyterhub
+INFO  Writing component at 'components/jupyterhub'
+ERROR Component parameters for 'jupyterhub' already exists
 ```
 
-#### ERROR Component parameters for 'kubeflow-core' already exists
+#### ERROR Component parameters for 'jupyterhub' already exists
 If you use the same component name you will preserve the values of any parameters you have set.
 ksonnet will print a warning but it works.
 

kubeflow/core/README.md

@@ -3,70 +3,9 @@
 **Table of Contents**  *generated with [DocToc](https://github.com/thlorenz/doctoc)*
 
 - [core](#core)
-  - [Quickstart](#quickstart)
-  - [Using the library](#using-the-library)
-    - [io.ksonnet.pkg.kubeflow-core](#ioksonnetpkgkubeflow-core)
-      - [Example](#example)
-      - [Parameters](#parameters)
 
 <!-- END doctoc generated TOC please keep comment here to allow auto update -->
 
 # core
 
-> Core components of Kubeflow.
-
-
-* [Quickstart](#quickstart)
-* [Using Prototypes](#using-prototypes)
-  * [io.ksonnet.pkg.kubeflow-core](#io.ksonnet.pkg.kubeflow-core)
-
-## Quickstart
-
-*The following commands use the `io.ksonnet.pkg.kubeflow` prototype to generate Kubernetes YAML for core, and then deploys it to your Kubernetes cluster.*
-
-First, create a cluster and install the ksonnet CLI (see root-level [README.md](rootReadme)).
-
-If you haven't yet created a [ksonnet application](linkToSomewhere), do so using `ks init <app-name>`.
-
-Finally, in the ksonnet application directory, run the following:
-
-```shell
-# Expand prototype as a Jsonnet file, place in a file in the
-# `components/` directory. (YAML and JSON are also available.)
-$ ks prototype use io.ksonnet.pkg.kubeflow-core \
-  --name core \
-  --namespace default \
-  --disks 
-
-# Apply to server.
-$ ks apply -f core.jsonnet
-```
-
-## Using the library
-
-The library files for core define a set of relevant *parts* (_e.g._, deployments, services, secrets, and so on) that can be combined to configure core for a wide variety of scenarios. For example, a database like Redis may need a secret to hold the user password, or it may have no password if it's acting as a cache.
-
-This library provides a set of pre-fabricated "flavors" (or "distributions") of core, each of which is configured for a different use case. These are captured as ksonnet *prototypes*, which allow users to interactively customize these distributions for their specific needs.
-
-These prototypes, as well as how to use them, are enumerated below.
-
-### io.ksonnet.pkg.kubeflow-core
-
-Kubeflow core components
-#### Example
-
-```shell
-# Expand prototype as a Jsonnet file, place in a file in the
-# `components/` directory. (YAML and JSON are also available.)
-$ ks prototype use io.ksonnet.pkg.kubeflow-core core \
-  --name YOUR_NAME_HERE
-```
-
-#### Parameters
-
-The available options to pass prototype are:
-
-* `--name=<name>`: Name to give to each of the components [string]
-
-
-[rootReadme]: https://github.com/ksonnet/mixins
+This ksonnet package contains kubeflow core prototypes such as jupyterhub, ambassador, spartakus, etc. You can install this using `ks pkg install kubeflow/core`. `ks prototype list` should list the available prototypes. `ks prototype describe <name>` should describe the prototype.