Update v0.2.0 branch (#365)

* Add dev doc links to CONTRIBUTING.md (#312) Resolves a task from #304 Signed-off-by: Rafael Vasquez <rafael.vasquez@ryerson.com> * Fixes the references to latest kfctl release (#311) Resolves a task from #304 Signed-off-by: Rafael Vasquez <rafael.vasquez@ryerson.com> * Do not verify links in project dependencies (#320) Resolves #319 Signed-off-by: Christian Kadner <ckadner@us.ibm.com> * Add navigation and description table of docs (#314) * Create readme for docs * Add description table to readme * Add links to table * update relative links * Change documentation to document Signed-off-by: Rafael Vasquez <rafael.vasquez@ryerson.com> * Add script to update the docs table (#317) Signed-off-by: Rafael Vasquez <rafael.vasquez@ryerson.com> * Modified the way sed version is set (#315) Instead of checking the operating system or shell emulator, test which version of `sed` is actually installed in the local environment. Resolves #301 Signed-off-by: Krishna Kumar <krishnakumar@ibm.com> * Bump waitress from 2.0.0 to 2.1.1 in /api/server (#321) Bumps [waitress](https://github.com/Pylons/waitress) from 2.0.0 to 2.1.1. - [Release notes](https://github.com/Pylons/waitress/releases) - [Changelog](https://github.com/Pylons/waitress/blob/master/CHANGES.txt) - [Commits](Pylons/waitress@v2.0.0...v2.1.1) --- updated-dependencies: - dependency-name: waitress dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump jupyter-server from 1.13.4 to 1.15.4 in /api/server (#324) Bumps [jupyter-server](https://github.com/jupyter/jupyter_server) from 1.13.4 to 1.15.4. - [Release notes](https://github.com/jupyter/jupyter_server/releases) - [Changelog](https://github.com/jupyter-server/jupyter_server/blob/main/CHANGELOG.md) - [Commits](jupyter-server/jupyter_server@v1.13.4...v1.15.4) --- updated-dependencies: - dependency-name: jupyter-server dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump minimist from 1.2.5 to 1.2.6 in /dashboard/origin-mlx (#326) Bumps [minimist](https://github.com/substack/minimist) from 1.2.5 to 1.2.6. - [Release notes](https://github.com/substack/minimist/releases) - [Commits](https://github.com/substack/minimist/compare/1.2.5...1.2.6) --- updated-dependencies: - dependency-name: minimist dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump notebook from 6.4.8 to 6.4.10 in /api/server (#327) Bumps [notebook](http://jupyter.org) from 6.4.8 to 6.4.10. --- updated-dependencies: - dependency-name: notebook dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update API developer docs (#325) * Add codegen workflow diagram * Describe codegen workflow * Describe API server package modules Closes #325 Signed-off-by: Pavan Pss <psspavan96@gmail.com> Co-authored-by: Christian Kadner <ckadner@us.ibm.com> * Update UI developer docs (#323) Closes #323 Signed-off-by: RRM123 <rithikmamidi@gmail.com> * Correct description for make update_doc_table (#329) Signed-off-by: ezinneanne <onwukaezinneanne@gmail.com> * Update Kubernetes high version in deployment docs (#318) (#332) * Add Troubleshooting section * Limit K8s version for KIND cluster using `--image` flag Resolves #318 Signed-off-by: Kiran-Patel <kiran.patel@ibm.com> Signed-off-by: Kiran Patel 104844270+kiranp2396@users.noreply.github.com Signed-off-by: Christian Kadner <ckadner@us.ibm.com> Co-authored-by: Kiran-Patel <kiran.patel@ibm.com> Co-authored-by: Christian Kadner <ckadner@us.ibm.com> * fix errors in mlx-ui startup (#338) * Run mlx-ui as non-root user (#339) Closes #337 Signed-off-by: Christian Kadner <ckadner@us.ibm.com> * Update MLX setup instructions for KF 1.5 (#346) * Update MLX setup instructions for KF 1.5 Signed-off-by: Christian Kadner <ckadner@us.ibm.com> * fix selected assets show up in different menus (#342) Signed-off-by: Jiaxuan-Yang <jiaxuan.yang@ibm.com> * Consolidate readme (#356) Signed-off-by: Rafael Vasquez <rafael.vasquez@ryerson.com> Signed-off-by: Rafael Vasquez <rafael.vasquez@ryerson.com> Co-authored-by: Rafael Vasquez <rafael.vasquez@ryerson.com> * Document MLX Models Workshop (#352) Signed-off-by: Christian Kadner <ckadner@us.ibm.com> * Add GitHub action to verify doc links (#357) Signed-off-by: Rafael Vasquez <raf.vasquez@ibm.com> Signed-off-by: Rafael Vasquez <rafael.vasquez@ryerson.com> Signed-off-by: Christian Kadner <ckadner@us.ibm.com> Signed-off-by: Krishna Kumar <krishnakumar@ibm.com> Signed-off-by: RRM123 <rithikmamidi@gmail.com> Signed-off-by: ezinneanne <onwukaezinneanne@gmail.com> Signed-off-by: Jiaxuan-Yang <jiaxuan.yang@ibm.com> Signed-off-by: Rafael Vasquez <raf.vasquez@ibm.com> Co-authored-by: Rafael Vasquez <rafael.vasquez@ryerson.ca> Co-authored-by: Krishna Kumar Ramachandran <krishnakumar@gatech.edu> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pavan Pss <psspavan96@gmail.com> Co-authored-by: RRM123 <40575227+rrm123@users.noreply.github.com> Co-authored-by: Ezinne Anne Emilia <onwukaezinneanne@gmail.com> Co-authored-by: Kiran Patel <104844270+kiranp2396@users.noreply.github.com> Co-authored-by: Kiran-Patel <kiran.patel@ibm.com> Co-authored-by: jbusche <101845348+jbusche@users.noreply.github.com> Co-authored-by: Joanna <Jiaxuan.Yang@ibm.com> Co-authored-by: Rafael Vasquez <rafael.vasquez@ryerson.com>
machine-learning-exchange · Oct 28, 2022 · d304ccc · d304ccc
1 parent e9f95ee
commit d304ccc
Show file tree

Hide file tree

Showing 36 changed files with 6,633 additions and 3,392 deletions.
diff --git a/.github/workflows/verify-links.yml b/.github/workflows/verify-links.yml
@@ -0,0 +1,38 @@
+# Copyright 2021 The MLX Contributors
+#
+# SPDX-License-Identifier: Apache-2.0
+
+name: Verify Links
+
+on:
+  push:
+    branches: 
+    - main
+
+    tags:
+    - v*
+
+  pull_request:
+
+jobs:
+  check-links:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: [3.7]
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install dependencies
+        run: pip install requests
+
+      - name: Link check
+        run: |
+          ./tools/python/verify_doc_links.py
+
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -32,6 +32,14 @@ maintainers of each component affected.
 
 For a list of the maintainers, see the [MAINTAINERS.md](MAINTAINERS.md) page.
 
+## Getting Started 
+
+ **MLX Dashboard Web UI**
+- Information regarding configuration, deployment, and development setup can be found [here](https://github.com/machine-learning-exchange/mlx/blob/main/dashboard/origin-mlx/README.md).
+
+**MLX API**
+- Quickstart and development setup documentation can be found [here](https://github.com/machine-learning-exchange/mlx/tree/main/api).
+
 ## Legal
 
 Each source file must include a license header for the Apache
@@ -72,14 +80,6 @@ git commit -s
 ## Communication
 Please feel free to connect with us on our [Slack channel](https://lfaifoundation.slack.com/archives/C0264LKNH63).
 
-## Setup
-**FIXME** Please add any special setup instructions for your project to help the developer
-become productive quickly.
-
-## Testing
-**FIXME** Please provide information that helps the developer test any changes they make
-before submitting.
-
 ## Coding style guidelines
 **FIXME** Optional, but recommended: please share any specific style guidelines you might
 have for your project.

diff --git a/Makefile b/Makefile
@@ -26,6 +26,11 @@ check_doc_links: ## Check markdown files for invalid links
 	@python3 tools/python/verify_doc_links.py
 	@echo "$@: OK"
 
+.PHONY: update_doc_table
+update_doc_table: ## Regenerate the /docs/README.md file
+	@python3 tools/python/update_doc_table.py
+	@echo "$@: OK"
+
 .PHONY: check_license
 check_license: ## Make sure source files have license header
 	@git grep -L "SPDX-License-Identifier: Apache-2.0" -- *.py *.yml *.yaml *.sh *.html *.js *.css *.ts *.tsx ':!*.bundle.js' | \

diff --git a/README.md b/README.md
@@ -13,11 +13,6 @@ Allows upload, registration, execution, and deployment of:
  - Datasets
  - Notebooks
 
-For more details about the project please follow this [announcement blog post](https://lfaidata.foundation/blog/2021/09/28/machine-learning-exchange-mlx/). 
-
-
-<img src="docs/images/mlx.png" height="90%" width="90%">
-
 Additionally it provides:
 
  - Automated sample pipeline code generation to execute registered models, datasets and notebooks
@@ -28,6 +23,12 @@ Additionally it provides:
  - Serving engine by [KFServing](https://github.com/kubeflow/kfserving)
  - Model Metadata schemas
 
+For more details about the project check out this [announcement blog post](https://lfaidata.foundation/blog/2021/09/28/machine-learning-exchange-mlx/). 
+
+
+<img src="docs/images/mlx.png" height="90%" width="90%">
+
+
 ## 1. Deployment
 <img src="docs/images/mlx-architecture-4.png" height="40%" width="40%">
 
@@ -48,30 +49,28 @@ For a full deployment, we use [Kubeflow Kfctl](https://github.com/kubeflow/kfctl
 * #### [MLX on an existing Kubeflow Cluster](./docs/install-mlx-on-kubeflow.md)
 
 
-## 2. Access the MLX UI
+## 2. Access the MLX UI and import Assets to the Catalog
 
 By default, the MLX UI is available at http://<cluster_node_ip>:30380/mlx/
 
-If you deployed on a **Kubernetes** cluster, run the following and look for the External-IP column to find the public IP of a node.
+If you deployed on a **Kubernetes** cluster or using **OpenShift**, run the following and look for the External-IP column to find the public IP of a node.
 
 ```bash
 kubectl get node -o wide
 ```
 
-If you deployed using **OpenShift**, you can use IstioIngresGateway Route. You can find it in the OpenShift Console or using the CLI.
+For information on how to import data and AI assets using MLX's catalog importer, use this [guide](/docs/import-assets.md).
 
-```bash
-oc get route -n istio-system
-```
+## 3. Use MLX
 
-## 3. Import Data and AI Assets in MLX Catalog
+For information on how to use MLX and create assets check out this [guide](/docs/usage-steps.md).
 
-[Import data and AI assets using MLX's catalog importer](/docs/import-assets.md)
+## 4. How to Contribute
+
+For information about adding new features, bug fixing, communication
+or UI and API setup, refer to this [document](CONTRIBUTING.md).
 
-## 4. Use MLX
 
-[MLX Usage Instructions](/docs/usage-steps.md)
-
 ## 5. Troubleshooting
 
 [MLX Troubleshooting Instructions](/docs/troubleshooting.md)

diff --git a/api/README.md b/api/README.md
@@ -1,29 +1,59 @@
-# Machine Learning Exchange API - Python Client and Python-Flask Server
+# MLX API - Python Client and Python-Flask Server
 
-An extension to the Kubeflow Pipeline API for Components and Models
+The MLX API is an extension to the Kubeflow Pipeline API with additional API
+endpoints for Dataset, Models, Notebooks and Pipeline Components.
+We use [OpenAPI v2 (fka Swagger)](https://swagger.io/specification/v2/) for the
+[API specification](swagger/swagger.yaml).
 
----
 
-# Quickstart    
+# Deploy to Kubernetes
 
-## Deploy to Kubernetes
+If you already have a Kubeflow Pipelines deployment on a Kubernetes cluster, you
+can use the following steps to deploy the MLX API on top of it. However, for a full
+deployment of MLX we recommend following on of these [guides](../README.md#1-deployment) 
 
-    kubectl apply -f ./server/mlx-api.yml
+1) Run kubectl command to apply the manifest
 
-## Find API Server Host and Port
+    `kubectl apply -f ./server/mlx-api.yml`
 
-    export API_HOST=$(kubectl get nodes -o jsonpath='{.items[].status.addresses[?(@.type=="ExternalIP")].address}')
-    export API_PORT=$(kubectl get service mlx-api -n kubeflow -o jsonpath='{.spec.ports[0].nodePort}')
+2) Find API Server Host and Port
 
-## Open the Swagger UI in a Web Browser
+    `export API_HOST=$(kubectl get nodes -o jsonpath='{.items[].status.addresses[?(@.type=="ExternalIP")].address}')`
+    `export API_PORT=$(kubectl get service mlx-api -n kubeflow -o jsonpath='{.spec.ports[0].nodePort}')`
 
-    open "http://${API_HOST}:${API_PORT}/apis/v1alpha1/ui/"
+3) Open the Swagger UI in a Web Browser
+
+    `open "http://${API_HOST}:${API_PORT}/apis/v1alpha1/ui/" `
 
+
 ---
 
-# Development Setup
+# API Development
+
+## Code Generation Overview
+
+![API Codegeneration Workflow](codegen_workflow.png)
+
+- Changes/additions to the API are done in the API [spec](swagger/swagger.yaml)
+- The [`generate_code.sh`](generate_code.sh) script validates the API [spec](swagger/swagger.yaml)
+  and generates [`client`](client) and [`server`](server) Python packages.
+  The [examples](examples) package is hand-written based on the usage examples
+  inside the `client` package 
+- Instead of a static HTML documentation for the API endpoints, a Swagger UI web
+  interface will be generated on the fly when the Python server is started
+- The API endpoints and the JSON data model is documented under [`client/docs`](client/docs)
+  in Markdown format which works well when browsing through the GitHub repo
+- After the code generation, we need to copy any new API method stubs (if any new)
+  from the [`server/swagger_server/controllers`](server/swagger_server/controllers) folder
+  to the [`server/swagger_server/controllers_impl`](server/swagger_server/controllers_impl)
+  folder and implement the actual business logic
+- If existing API method signatures got updated, we need to update the existing
+  `controller_impl` methods respectively
+
 
-## Swagger Codegen 2.4
+## Development Setup
+
+### Swagger Codegen 2.4
 
 To generate our API we are using [`swagger-codegen`](https://github.com/swagger-api/swagger-codegen/tree/v2.4.8#prerequisites)
 version [`2.4`](https://repo1.maven.org/maven2/io/swagger/swagger-codegen-cli/2.4.8/swagger-codegen-cli-2.4.8.jar)
@@ -48,20 +78,29 @@ will automatically download the _"correct"_ version of the `swagger-codegen-cli.
     # brew install swagger-codegen@2
     # brew link --force swagger-codegen@2
 
-## Create a Python Virtual Environment for Development
+### Create a Python Virtual Environment for Development
 
     python3 -m venv .venv
     source .venv/bin/activate
 
-## Install the Python Package Dependencies
+### Install the Python Package Dependencies
 
     # cd <mlx_root_dir>
     # cd api
 
     pip install -r ./requirements.txt
 
+
+
 ## (Re-)Generate Swagger Client and Server Code
 
+If there are changes to the [API spec](swagger/swagger.yaml) then we need to re-
+generate the API client and server code. Do not run the script `codegen.sh`
+on its own, since Swagger generates a lot of unwanted and some breaking changes.
+Instead use the [generate_code.sh](generate_code.sh) script which runs the Swagger
+codegen and tries to undo some of the code and documentation changes that are not
+desired. 
+
     ./generate_code.sh
 
 ## Build the Docker Image
@@ -86,7 +125,8 @@ or:
     kubectl delete -f ./server/mlx-api.yml
     kubectl apply -f ./server/mlx-api.yml
 
-## Testing API Code Changes with Docker Compose
+
+## Testing API Code Changes Locally with Docker Compose
 
 You can test most code changes without a Kubernetes cluster. A K8s cluster is only
 required to `run` the generated sample pipeline code. Running the Quickstart with
@@ -189,3 +229,7 @@ Docker Compose stack with the `-v` option (`docker compose --project-name no_api
         -d @catalog_upload.json \
         http://localhost:8080/apis/v1alpha1/catalog
 
+# Troubleshooting
+
+Report any issues with at https://github.com/machine-learning-exchange/mlx/issues
+
diff --git a/api/codegen.sh b/api/codegen.sh
@@ -44,18 +44,13 @@ swagger-codegen generate -i swagger/swagger.yaml -l python-flask -o server 2>&1
 # set interactive mode to enable defining a gsed alias
 shopt -s expand_aliases
 
-# we use sed to make in-file text replacements, but sed works differently on macOS and Linux
-if [[ "$OSTYPE" == "linux-gnu"* ]]; then  # Linux
-    alias gsed="sed -i"
-elif [[ "$OSTYPE" == "darwin"* ]]; then  # macOS
-    alias gsed="sed -i ''"
-elif [[ "$OSTYPE" == "cygwin" ]]; then  # POSIX compatible emulation for Windows
-    alias gsed="sed -i"
-elif [[ "$OSTYPE" == "msys"* ]]; then  # Git Bash (Windows)
-    alias gsed="sed -i"
+# we use sed to make in-file text replacements, but sed works differently depending on the version
+if ! sed -i '1s/^/test/' $(mktemp) 2> /dev/null; then
+    # macOS (BSD) version of sed
+    alias gsed="sed -i ''" 
 else
-    echo "FAILED. OS not compatible with script '${BASH_SOURCE[0]}'"
-    exit 1
+    # POSIX compliant version of sed 
+    alias gsed="sed -i"
 fi
 export gsed
 

diff --git a/api/codegen_workflow.png b/api/codegen_workflow.png
diff --git a/api/server/README.md b/api/server/README.md
@@ -1,14 +1,29 @@
-# Swagger generated server
+# MLX API Server
 
 ## Overview
-This server was generated by the [swagger-codegen](https://github.com/swagger-api/swagger-codegen)
+
+The MLX API server was generated by the [swagger-codegen](https://github.com/swagger-api/swagger-codegen)
 project. By using the [OpenAPI-Spec](https://github.com/swagger-api/swagger-core/wiki) from a remote
 server, you can easily generate a server stub. This is an example of building a Swagger-enabled
 Flask server.
 
 This example uses the [Connexion](https://github.com/zalando/connexion) library on top of Flask.
 
+## MLX API Server Modules
+
+| Python package/script  | Purpose |
+| ------------- | ------------- |
+| `code_templates`  | Code templates are used to generate sample Kubeflow pipeline DSL scripts for each asset type  |
+| `controllers`  | Controllers are the code stubs (“interfaces”) for the API endpoints generated by Swagger  |
+| `controllers_impl` | Controller implementation of the generated API code stubs (“interfaces”), originally copied from respective controller and then filled in with the actual business logic. API calls get forwarded from the `controllers` to the `controllers_impl` by injection of some code magic in `util.py` (see `codegen.sh`) |
+| `data_access` | Contains 2 clients - MySQL (relational database) and Minio (object storage) for data access |
+| `gateways` | Connects the API server to various services on the Kubernetes cluster like KFServing, Kubeflow Pipelines, and the Kubernetes API |
+| `models` | Swagger generated API classes used to transfer data between API client and API server |
+| `encoder.py` | Implementation of JSON encoder (generated by Swagger) |
+| `util.py` | Collection of helper functions, most notably to “glue” the API `controllers` code stubs to the `controllers_impl` API implementations |
+
 ## Requirements
+
 Python 3.6.1+
 
 ## Usage
@@ -33,6 +48,7 @@ http://localhost:8080/apis/v1alpha1/swagger.json
 ```
 
 To launch the integration tests, use tox:
+
 ```
 sudo pip install tox
 tox

diff --git a/api/server/requirements.txt b/api/server/requirements.txt
@@ -14,10 +14,6 @@ ansiwrap==0.8.4
     # via papermill
 anyio==3.5.0
     # via jupyter-server
-appnope==0.1.2
-    # via
-    #   ipykernel
-    #   ipython
 argon2-cffi==21.3.0
     # via
     #   jupyter-server
@@ -188,10 +184,7 @@ ipython==8.0.1
     #   ipykernel
     #   jupyterlab
 ipython-genutils==0.2.0
-    # via
-    #   jupyter-server
-    #   nbformat
-    #   notebook
+    # via notebook
 isodate==0.6.1
     # via openapi-schema-validator
 isort==5.10.1
@@ -244,7 +237,7 @@ jupyter-lsp==1.5.1
     # via jupyterlab-lsp
 jupyter-resource-usage==0.6.1
     # via elyra-server
-jupyter-server==1.13.4
+jupyter-server==1.15.4
     # via
     #   elyra-server
     #   jupyter-lsp
@@ -327,7 +320,7 @@ nbdime==3.1.1
     # via
     #   elyra-server
     #   jupyterlab-git
-nbformat==5.1.3
+nbformat==5.2.0
     # via
     #   elyra-server
     #   jupyter-server
@@ -345,7 +338,7 @@ nest-asyncio==1.5.4
     #   notebook
 networkx==2.6.3
     # via elyra-server
-notebook==6.4.8
+notebook==6.4.10
     # via nbclassic
 numpy==1.22.1
     # via
@@ -612,7 +605,7 @@ urllib3==1.26.8
     #   kubernetes
     #   minio
     #   requests
-waitress==2.0.0
+waitress==2.1.1
     # via -r requirements.in
 watchdog==2.1.6
     # via elyra-server