First iteration of the new Katib UI (#1427)

* Create a folder for the new-ui

We will create a `new-ui` folder under the `pkg` dir to add the new UI.
This will ensure that we won't break any existing functionality.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Initial code for the frontend

This PR introduces the new UI. We hope that this will be the last big PR
in this repo and all of the subsequent ones will be smaller bit-sized
PRs.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* backend: Expose the entire status of an experiment

We want the table in the main UI page to show more information for each
experiment. This information lives in the status of each Experiment CR,
so we expand the API to also return the entire status for each
Experiment.

In the future we will probably need to just send the entire CR to the
frontend and not parse it at all in the backend.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* backend: Return the KFP run uid

We want to return the Pipeline UID for a Trial, if such exists.

When combined with Kale, a Trial initiates a KFP run. In this case,
there is an annotation with the KFP run ID, which we can use to navigate
the user to the KFP UI for the specific run.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* backend: sereve an Angular SPA

To serve an SPA the backend must return the index.html for any non-API
route. The index.html must be sent for any request to the app's page.
Then, once the javascript loads, the app will show to the user the
correct view.

In this commit we also completely remove any caching of the index.html,
for the browser to always request the latest version. This eliminates
the need to hard reload the page to view changes to the frontend code.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Dockerfile for the new Katib web app

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Extend the dockerignore for the new UI

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Update the README with build commands

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* review: use port 8080 instead of 80 in backend

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* review: use lowercase fields when fetchin exps

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* review: Add seconds to the x-axis of Trial info

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* review: Unify the npm run build commands

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* review: Move TypeMeta values to a common place

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* review: Remove section for max_old_space_size in README

Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* review: add katib prefix to docs link

Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* review: Correct link for new UI in README

Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* review: Remove unused 'format' npm script

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* review: Ensure format checks work with Travis

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* review: Remove unused space

Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* review: Use create_experiment route

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* review: fix travis govet test

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* review: Rename the Bayesian settings

Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* review: Rename the ParametersSpec

Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* review: Remove setting TypeMeta and ObjectMeta

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* review: Update README for build:watch

Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* review: Fix a typo

Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* review: Remove unused css

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* review: Use types from k8s.models file

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Add kfp-run column if UID is present in trials

With Kale a Trial can launch a distinct KF Pipeline. The UID of this
pipeline will be also set as an annotation to the Trial owning the
Pipeline.

In this case the UI should have one extra column for this UID.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Properly expose the NAS fields

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* review: Move MetricCollector enums to global enums file

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* review: Remove unused volume enum

Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

* review: Don't send empty settings

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* review: Add parameters for TPE

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

.dockerignore

@@ -5,3 +5,5 @@ examples
 manifests
 pkg/ui/*/frontend/node_modules
 pkg/ui/*/frontend/build
+pkg/new-ui/*/frontend/node_modules
+pkg/new-ui/*/frontend/build

cmd/new-ui/v1beta1/Dockerfile

@@ -0,0 +1,55 @@
+# --- Clone the kubeflow/kubeflow code ---
+FROM ubuntu AS fetch-kubeflow-kubeflow
+
+RUN apt-get update && apt-get install git -y
+
+WORKDIR /kf
+RUN git clone https://github.com/kubeflow/kubeflow.git && \
+    cd kubeflow && \
+    git checkout a349284
+
+# --- Build the frontend kubeflow library ---
+FROM node:12 AS frontend-kubeflow-lib
+
+WORKDIR /src
+
+ARG LIB=/kf/kubeflow/components/crud-web-apps/common/frontend/kubeflow-common-lib
+COPY --from=fetch-kubeflow-kubeflow $LIB/package*.json ./
+RUN npm ci
+
+COPY --from=fetch-kubeflow-kubeflow $LIB/ ./
+RUN npm run build
+
+# --- Build the frontend ---
+FROM node:12 AS frontend
+
+WORKDIR /src
+COPY ./pkg/new-ui/v1beta1/frontend/package*.json ./
+RUN npm ci
+
+COPY ./pkg/new-ui/v1beta1/frontend/ .
+COPY --from=frontend-kubeflow-lib /src/dist/kubeflow/ ./node_modules/kubeflow/
+
+RUN npm run build:prod
+
+# --- Build the backend ---
+FROM golang:alpine AS go-build
+
+# The GOPATH in the image is /go.
+ADD . /go/src/github.com/kubeflow/katib
+WORKDIR /go/src/github.com/kubeflow/katib/cmd/new-ui
+RUN if [ "$(uname -m)" = "ppc64le" ] || [ "$(uname -m)" = "aarch64" ]; then \
+    apk --update add gcc musl-dev  && \
+    go build -o katib-ui ./v1beta1; \
+    else \
+    go build -o katib-ui ./v1beta1; \
+    fi
+
+# --- compose the web app ---
+FROM alpine:3.7
+WORKDIR /app
+COPY --from=go-build /go/src/github.com/kubeflow/katib/cmd/new-ui/katib-ui /app/
+COPY --from=frontend /src/dist/static /app/build/static/
+
+USER 1000
+ENTRYPOINT ["./katib-ui"]

cmd/new-ui/v1beta1/main.go

@@ -0,0 +1,58 @@
+package main
+
+import (
+	"flag"
+	"fmt"
+	"log"
+	"net/http"
+
+	_ "k8s.io/client-go/plugin/pkg/client/auth/gcp"
+
+	common_v1beta1 "github.com/kubeflow/katib/pkg/common/v1beta1"
+	ui "github.com/kubeflow/katib/pkg/new-ui/v1beta1"
+)
+
+var (
+	port, host, buildDir, dbManagerAddr *string
+)
+
+func init() {
+	port = flag.String("port", "8080", "The port to listen to for incoming HTTP connections")
+	host = flag.String("host", "0.0.0.0", "The host to listen to for incoming HTTP connections")
+	buildDir = flag.String("build-dir", "/app/build", "The dir of frontend")
+	dbManagerAddr = flag.String("db-manager-address", common_v1beta1.GetDBManagerAddr(), "The address of Katib DB manager")
+}
+
+func main() {
+	flag.Parse()
+	kuh := ui.NewKatibUIHandler(*dbManagerAddr)
+
+	log.Printf("Serving the frontend dir %s", *buildDir)
+	frontend := http.FileServer(http.Dir(*buildDir))
+	http.HandleFunc("/katib/", kuh.ServeIndex(*buildDir))
+	http.Handle("/katib/static/", http.StripPrefix("/katib/", frontend))
+
+	http.HandleFunc("/katib/fetch_experiments/", kuh.FetchAllExperiments)
+
+	http.HandleFunc("/katib/create_experiment/", kuh.CreateExperiment)
+
+	http.HandleFunc("/katib/delete_experiment/", kuh.DeleteExperiment)
+
+	http.HandleFunc("/katib/fetch_experiment/", kuh.FetchExperiment)
+	http.HandleFunc("/katib/fetch_suggestion/", kuh.FetchSuggestion)
+
+	http.HandleFunc("/katib/fetch_hp_job_info/", kuh.FetchHPJobInfo)
+	http.HandleFunc("/katib/fetch_hp_job_trial_info/", kuh.FetchHPJobTrialInfo)
+	http.HandleFunc("/katib/fetch_nas_job_info/", kuh.FetchNASJobInfo)
+
+	http.HandleFunc("/katib/fetch_trial_templates/", kuh.FetchTrialTemplates)
+	http.HandleFunc("/katib/add_template/", kuh.AddTemplate)
+	http.HandleFunc("/katib/edit_template/", kuh.EditTemplate)
+	http.HandleFunc("/katib/delete_template/", kuh.DeleteTemplate)
+	http.HandleFunc("/katib/fetch_namespaces", kuh.FetchNamespaces)
+
+	log.Printf("Serving at %s:%s", *host, *port)
+	if err := http.ListenAndServe(fmt.Sprintf("%s:%s", *host, *port), nil); err != nil {
+		panic(err)
+	}
+}

pkg/new-ui/v1beta1/README.md

@@ -0,0 +1,126 @@
+# Katib User Interface
+
+This is the source code for the Katib UI. Current version of Katib UI is v1beta1. On the official Kubeflow website [here](https://www.kubeflow.org/docs/components/katib/experiment/#running-the-experiment-from-the-katib-ui) you can find information how to use Katib UI.
+We are using [Angular](https://angular.io/) framework to create frontend and Go as a backend.
+
+We are using [Material UI](https://material.angular.io/) to design frontend. Try to use Material UI components to implement new Katib UI features.
+
+## Folder structure
+
+1. You can find `Dockerfile` and `main.go` - file to serve the UI under `cmd/new-ui/v1beta1`
+
+1. You can find Go backend under `pkg/new-ui/v1beta1`
+
+1. You can find Angular frontend under `pkg/new-ui/v1beta1/frontend`
+
+## Requirements
+
+To make changes to the UI you need to install:
+
+- Tools, defined [here](https://github.com/kubeflow/katib/blob/master/docs/developer-guide.md#requirements).
+
+- `node` (v12 or later) and `npm` (v6.13 or later). Recommended to install `node` and `npm` using [`nvm`](https://github.com/nvm-sh/nvm). After installing `nvm`, you can run `nvm install 12.18.1` to install `node` version 12.18.1 and run `nvm use 12.18.1` to use that version.
+
+## Development
+
+While development you have different ways to run Katib UI.
+1. Build and serve only the frontend. The dev server will also be proxying requests to the backend
+2. Build the frontend and serve it via the backend locally
+
+### Serve only the frontend
+
+You can run a webpack dev server that only exposes the frontend files, which can be useful for testing only the UI of the app. There's also a `proxy.conf.json` file which configures the dev server to send the backend requests to port `8000`.
+
+In order to build the UI locally, and expose it with a webpack dev server you will need to:
+1. Create a module from the [common library](https://github.com/kubeflow/kubeflow/tree/master/components/crud-web-apps/common/frontend/kubeflow-common-lib)
+2. Install the node modules of the app and also link the common-library module
+
+
+
+You can build the common library with:
+```bash
+cd /tmp && git clone https://github.com/kubeflow/kubeflow.git \
+  && cd kubeflow \
+  && git checkout a349284 \
+  && cd components/crud-web-apps/common/frontend/kubeflow-common-lib
+
+# build the common library module
+npm i
+npm run build
+
+# link the module to your npm packages
+# depending on where you npm stores the global packages you
+# might need to use sudo
+npm link dist/kubeflow
+```
+
+And then build and run the UI locally, on `localhost:4200`, with:
+```bash
+# If you've already cloned the repo then skip this step and just
+# navigate to the pkg/new-ui/v1beta1/frontend dir
+cd /tmp && git clone https://github.com/kubeflow/katib.git \
+  && cd katib/pkg/new-ui/v1beta1/frontend
+
+npm i
+npm link kubeflow
+npm run start
+```
+
+### Serve the UI  from the backend
+
+This is the recommended way to test the web app e2e. In order to build the UI and serve it via the backend, locally, you will need to:
+
+1. Build the UI locally. You have to follow the steps from the previous section, but instead of running `npm run start` you need to run `npm run build:prod`. It builds the frontend artifacts under `frontend/dist/static` folder.
+
+    Moreover, you are able to run `npm run build:watch` instead of `npm run build:prod`. In that case, it starts a process which is watching the source code changes and building the frontend artifacts under `frontend/dist/static` folder.
+
+    Learn more about Angular scripts in the [official guide](https://angular.io/cli/build).
+1. Run `kubectl port-forward svc/katib-db-manager 6789 -n kubeflow` to expose `katib-db-manager` service for external access. You can use [different ways](https://kubernetes.io/docs/tasks/access-application-cluster/) to get external address for Kubernetes service. After exposing service, you should be able to receive information by running `wget <external-ip>:<service-port>`. In case of port-forwarding above, you have to run `wget localhost:6789`.
+
+1. Go to `cmd/new-ui/v1beta1`.
+
+1. Run `main.go` file with appropriate flags, where:
+
+   - `--build-dir` - directory with the frontend artifacts.
+   - `--port` - port to access Katib UI.
+   - `--db-manager-address` - Katib DB manager external IP and port address.
+
+   For example, if you use port-forwarding to expose `katib-db-manager`, run this command:
+
+   ```
+   go run main.go --build-dir=../../../pkg/new-ui/v1beta1/frontend/dist --port=8080 --db-manager-address=localhost:6789
+   ```
+
+After that, you can access the UI using this URL: `http://localhost:8080/katib/`.
+
+## Production
+
+To run Katib UI in Production, after all changes in frontend and backend, you need to create an image for the UI. Under `/katib` directory run this: `docker build . -f cmd/new-ui/v1beta1/Dockerfile -t <name of your image>` to build the image.
+
+After that, you can modify UI [deployment](https://github.com/kubeflow/katib/blob/master/manifests/v1beta1/ui/deployment.yaml#L24) with your new image. Then, follow [these steps](https://www.kubeflow.org/docs/components/katib/hyperparameter/#accessing-the-katib-ui) to access Katib UI.
+
+## Code style
+
+To make frontend code consistent and easy to review we use [Prettier](https://prettier.io/). You can find Prettier config [here](https://github.com/kubeflow/katib/tree/master/pkg/new-ui/v1beta1/frontend/.prettierrc.yaml).
+Check [here](https://prettier.io/docs/en/install.html), how to install Prettier CLI to check and format your code.
+
+### IDE integration
+
+For VSCode you can install plugin: "Prettier - Code formatter" and it picks Prettier config automatically.
+
+You can edit [settings.json](https://code.visualstudio.com/docs/getstarted/settings#_settings-file-locations) file for VSCode to autoformat on save.
+
+```json
+  "settings": {
+    "editor.formatOnSave": true
+  }
+```
+
+For others IDE see [this](https://prettier.io/docs/en/editors.html).
+
+### Check and format code
+
+Before submitting PR check and format your code. To check your code run `npm run format:check` under `/frontend` folder. To format your code run `npm run format:write` under `/frontend` folder.
+If all files formatted you can submit the PR.
+
+If you don't want to format some code, [here](https://prettier.io/docs/en/ignore.html) is an instruction how to disable Prettier.