From 55978a0a2a83892cf1f09cc83d45ab5859300a3e Mon Sep 17 00:00:00 2001 From: Omer Spillinger Date: Thu, 28 Nov 2019 12:08:20 -0800 Subject: [PATCH 1/3] Update docs --- README.md | 59 ++++++++++++++----- docs/{cluster => cluster-management}/aws.md | 0 docs/{cluster => cluster-management}/cli.md | 0 .../{cluster => cluster-management}/config.md | 0 .../install.md | 4 +- .../security.md | 0 .../uninstall.md | 0 .../{cluster => cluster-management}/update.md | 0 docs/{ => contributing}/development.md | 34 +++++------ .../python-packages.md | 0 .../system-packages.md | 4 +- docs/deployments/compute.md | 2 +- docs/deployments/onnx.md | 2 +- docs/deployments/predictor.md | 2 +- docs/deployments/request-handlers.md | 2 +- docs/deployments/tensorflow.md | 2 +- docs/{packaging => packaging-models}/onnx.md | 4 +- .../tensorflow.md | 4 +- docs/summary.md | 26 ++++---- examples/README.md | 20 +++---- 20 files changed, 96 insertions(+), 69 deletions(-) rename docs/{cluster => cluster-management}/aws.md (100%) rename docs/{cluster => cluster-management}/cli.md (100%) rename docs/{cluster => cluster-management}/config.md (100%) rename docs/{cluster => cluster-management}/install.md (93%) rename docs/{cluster => cluster-management}/security.md (100%) rename docs/{cluster => cluster-management}/uninstall.md (100%) rename docs/{cluster => cluster-management}/update.md (100%) rename docs/{ => contributing}/development.md (88%) rename docs/{dependencies => dependency-management}/python-packages.md (100%) rename docs/{dependencies => dependency-management}/system-packages.md (92%) rename docs/{packaging => packaging-models}/onnx.md (96%) rename docs/{packaging => packaging-models}/tensorflow.md (97%) diff --git a/README.md b/README.md index 8f26b1bbd9..5155d4a069 100644 --- a/README.md +++ b/README.md @@ -15,25 +15,42 @@ Cortex is an open source platform for deploying machine learning models—traine ## Key features -- **Autoscaling:** Cortex automatically scales APIs to handle production workloads. +* **Autoscaling:** Cortex automatically scales APIs to handle production workloads. -- **Multi framework:** Cortex supports TensorFlow, PyTorch, scikit-learn, XGBoost, and more. +* **Multi framework:** Cortex supports TensorFlow, PyTorch, scikit-learn, XGBoost, and more. -- **CPU / GPU support:** Cortex can run inference on CPU or GPU infrastructure. +* **CPU / GPU support:** Cortex can run inference on CPU or GPU infrastructure. -- **Spot instances:** Cortex supports EC2 spot instances. +* **Spot instances:** Cortex supports EC2 spot instances. -- **Rolling updates:** Cortex updates deployed APIs without any downtime. +* **Rolling updates:** Cortex updates deployed APIs without any downtime. -- **Log streaming:** Cortex streams logs from deployed models to your CLI. +* **Log streaming:** Cortex streams logs from deployed models to your CLI. -- **Prediction monitoring:** Cortex monitors network metrics and tracks predictions. +* **Prediction monitoring:** Cortex monitors network metrics and tracks predictions. -- **Minimal configuration:** Deployments are defined in a single `cortex.yaml` file. +* **Minimal configuration:** Deployments are defined in a single `cortex.yaml` file.
-## Usage +## Spinning up a Cortex cluster + +Cortex is designed to be self-hosted on any AWS account. You can spin up a Cortex cluster with a single command: + +```bash +$ cortex cluster up + +aws region: us-west-2 +aws instance type: p2.xlarge +min instances: 0 +max instances: 10 +spot instances: yes + +￮ spinning up your cluster ... +your cortex cluster is ready! +``` + +## Deploying a model ### Implement your predictor @@ -98,17 +115,27 @@ negative 4
-## How it works +## What is Cortex an alternative to? + +Cortex is an open source alternative to serving models with SageMaker or building your own model deployment platform on top of AWS services like Elastic Kubernetes Service (EKS), Elastic Container Service (ECS), Lambda, Fargate, and Elastic Compute Cloud (EC2) or open source projects like Docker, Kubernetes, and TensorFlow Serving. + +
+ +## How does Cortex work? The CLI sends configuration and code to the cluster every time you run `cortex deploy`. Each model is loaded into a Docker container, along with any Python packages and request handling code. The model is exposed as a web service using Elastic Load Balancing (ELB), TensorFlow Serving, and ONNX Runtime. The containers are orchestrated on Elastic Kubernetes Service (EKS) while logs and metrics are streamed to CloudWatch.
-## Examples +## Examples of Cortex deployments -- [Sentiment analysis](https://github.com/cortexlabs/cortex/tree/0.11/examples/tensorflow/sentiment-analyzer) in TensorFlow with BERT -- [Image classification](https://github.com/cortexlabs/cortex/tree/0.11/examples/tensorflow/image-classifier) in TensorFlow with Inception -- [Text generation](https://github.com/cortexlabs/cortex/tree/0.11/examples/pytorch/text-generator) in PyTorch with DistilGPT2 -- [Reading comprehension](https://github.com/cortexlabs/cortex/tree/0.11/examples/pytorch/reading-comprehender) in PyTorch with ELMo-BiDAF -- [Iris classification](https://github.com/cortexlabs/cortex/tree/0.11/examples/sklearn/iris-classifier) in scikit-learn +* [Sentiment analysis](https://github.com/cortexlabs/cortex/tree/0.11/examples/tensorflow/sentiment-analyzer): deploy a BERT model for sentiment analysis. + +* [Image classification](https://github.com/cortexlabs/cortex/tree/0.11/examples/tensorflow/image-classifier): deploy an Inception model to classify images. + +* [Search completion](https://github.com/cortexlabs/cortex/tree/0.11/examples/tensorflow/search-completer): deploy Facebook's RoBERTa model to complete search terms. + +* [Text generation](https://github.com/cortexlabs/cortex/tree/0.11/examples/pytorch/text-generator): deploy Hugging Face's DistilGPT2 model to generate text. + +* [Iris classification](https://github.com/cortexlabs/cortex/tree/0.11/examples/sklearn/iris-classifier): deploy a scikit-learn model to classify iris flowers. diff --git a/docs/cluster/aws.md b/docs/cluster-management/aws.md similarity index 100% rename from docs/cluster/aws.md rename to docs/cluster-management/aws.md diff --git a/docs/cluster/cli.md b/docs/cluster-management/cli.md similarity index 100% rename from docs/cluster/cli.md rename to docs/cluster-management/cli.md diff --git a/docs/cluster/config.md b/docs/cluster-management/config.md similarity index 100% rename from docs/cluster/config.md rename to docs/cluster-management/config.md diff --git a/docs/cluster/install.md b/docs/cluster-management/install.md similarity index 93% rename from docs/cluster/install.md rename to docs/cluster-management/install.md index 4a08ad2646..5b6efb455d 100644 --- a/docs/cluster/install.md +++ b/docs/cluster-management/install.md @@ -42,8 +42,8 @@ cortex get classifier # classify a sample curl -X POST -H "Content-Type: application/json" \ - -d '{ "sepal_length": 5.2, "sepal_width": 3.6, "petal_length": 1.4, "petal_width": 0.3 }' \ - + -d '{ "sepal_length": 5.2, "sepal_width": 3.6, "petal_length": 1.4, "petal_width": 0.3 }' \ + ``` ## Cleanup diff --git a/docs/cluster/security.md b/docs/cluster-management/security.md similarity index 100% rename from docs/cluster/security.md rename to docs/cluster-management/security.md diff --git a/docs/cluster/uninstall.md b/docs/cluster-management/uninstall.md similarity index 100% rename from docs/cluster/uninstall.md rename to docs/cluster-management/uninstall.md diff --git a/docs/cluster/update.md b/docs/cluster-management/update.md similarity index 100% rename from docs/cluster/update.md rename to docs/cluster-management/update.md diff --git a/docs/development.md b/docs/contributing/development.md similarity index 88% rename from docs/development.md rename to docs/contributing/development.md index 68492d33f5..d672c52ed6 100644 --- a/docs/development.md +++ b/docs/contributing/development.md @@ -1,13 +1,13 @@ -# Development Environment +# Development ## Prerequisites 1. Go (>=1.12.9) -1. Docker -1. eksctl -1. kubectl +2. Docker +3. eksctl +4. kubectl -## Cortex Dev Environment +## Cortex dev environment Clone the project: @@ -109,7 +109,7 @@ make cli # The binary will be placed in path/to/cortex/bin/cortex path/to/cortex/bin/cortex configure ``` -### Cortex Cluster +### Cortex cluster Start Cortex: @@ -123,35 +123,35 @@ Tear down the Cortex cluster: make cortex-down ``` -### Deployment an Example +### Deploy an example ```bash cd examples/iris-classifier path/to/cortex/bin/cortex deploy ``` -## Off-cluster Operator +## Off-cluster operator If you're making changes in the operator and want faster iterations, you can run an off-cluster operator. 1. `make operator-stop` to stop the in-cluster operator -1. `make devstart` to run the off-cluster operator (which rebuilds the CLI and restarts the Operator when files change) -1. `path/to/cortex/bin/cortex configure` (on a separate terminal) to configure your cortex CLI to use the off-cluster operator. When prompted for operator URL, use `http://localhost:8888` +2. `make devstart` to run the off-cluster operator (which rebuilds the CLI and restarts the Operator when files change) +3. `path/to/cortex/bin/cortex configure` (on a separate terminal) to configure your cortex CLI to use the off-cluster operator. When prompted for operator URL, use `http://localhost:8888` Note: `make cortex-up-dev` will start Cortex without installing the operator. If you want to switch back to the in-cluster operator: 1. `` to stop your off-cluster operator -1. `make operator-start` to install the operator in your cluster -1. `path/to/cortex/bin/cortex configure` to configure your cortex CLI to use the in-cluster operator. When prompted for operator URL, use the URL shown when running `make cortex-info` +2. `make operator-start` to install the operator in your cluster +3. `path/to/cortex/bin/cortex configure` to configure your cortex CLI to use the in-cluster operator. When prompted for operator URL, use the URL shown when running `make cortex-info` -## Dev Workflow +## Dev workflow 1. `make cortex-up-dev` -1. `make devstart` -1. Make changes -1. `make registry-dev` -1. Test your changes with projects in `examples` or your own +2. `make devstart` +3. Make changes +4. `make registry-dev` +5. Test your changes with projects in `examples` or your own See `Makefile` for additional dev commands diff --git a/docs/dependencies/python-packages.md b/docs/dependency-management/python-packages.md similarity index 100% rename from docs/dependencies/python-packages.md rename to docs/dependency-management/python-packages.md diff --git a/docs/dependencies/system-packages.md b/docs/dependency-management/system-packages.md similarity index 92% rename from docs/dependencies/system-packages.md rename to docs/dependency-management/system-packages.md index fe7423b60e..b1df28ea2d 100644 --- a/docs/dependencies/system-packages.md +++ b/docs/dependency-management/system-packages.md @@ -1,8 +1,8 @@ # System packages -Cortex uses Docker images to deploy your models. These images can be replaced with custom images that you can augment with your system packages and libraries. You will need to push your custom images to a container registry that your cluster has access to (e.g. [Docker Hub](https://hub.docker.com/) or [AWS ECR](https://aws.amazon.com/ecr/)). +Cortex uses Docker images to deploy your models. These images can be replaced with custom images that you can augment with your system packages and libraries. You will need to push your custom images to a container registry that your cluster has access to (e.g. [Docker Hub](https://hub.docker.com) or [AWS ECR](https://aws.amazon.com/ecr)). -See the `image paths` section in [cluster configuration](../cluster/config.md) for a complete list of customizable images. +See the `image paths` section in [cluster configuration](../cluster-management/config.md) for a complete list of customizable images. ## Create a custom image diff --git a/docs/deployments/compute.md b/docs/deployments/compute.md index 296810dd29..5cb8de5fe7 100644 --- a/docs/deployments/compute.md +++ b/docs/deployments/compute.md @@ -27,6 +27,6 @@ One unit of memory is one byte. Memory can be expressed as an integer or by usin ## GPU 1. Make sure your AWS account is subscribed to the [EKS-optimized AMI with GPU Support](https://aws.amazon.com/marketplace/pp/B07GRHFXGM). -2. You may need to [file an AWS support ticket](https://console.aws.amazon.com/support/cases#/create?issueType=service-limit-increase&limitType=ec2-instances) to incease the limit for your desired instance type. +2. You may need to [file an AWS support ticket](https://console.aws.amazon.com/support/cases#/create?issueType=service-limit-increase&limitType=ec2-instances) to increase the limit for your desired instance type. 3. Set instance type to an AWS GPU instance (e.g. p2.xlarge) when installing Cortex. 4. Note that one unit of GPU corresponds to one virtual GPU on AWS. Fractional requests are not allowed. diff --git a/docs/deployments/onnx.md b/docs/deployments/onnx.md index 0dac8e3c6e..d6705a7ac6 100644 --- a/docs/deployments/onnx.md +++ b/docs/deployments/onnx.md @@ -26,7 +26,7 @@ Deploy ONNX models as web services. mem: # memory request per replica (default: Null) ``` -See [packaging ONNX models](../packaging/onnx.md) for information about exporting ONNX models. +See [packaging ONNX models](../packaging-models/onnx.md) for information about exporting ONNX models. ## Example diff --git a/docs/deployments/predictor.md b/docs/deployments/predictor.md index 34f78f11f6..6801d17cda 100644 --- a/docs/deployments/predictor.md +++ b/docs/deployments/predictor.md @@ -154,4 +154,4 @@ torchvision==0.4.2 xgboost==0.90 ``` -Learn how to install additional packages [here](../dependencies/python-packages.md). +Learn how to install additional packages [here](../dependency-management/python-packages.md). diff --git a/docs/deployments/request-handlers.md b/docs/deployments/request-handlers.md index 55931a2853..26c36352ac 100644 --- a/docs/deployments/request-handlers.md +++ b/docs/deployments/request-handlers.md @@ -84,4 +84,4 @@ tensorflow-hub==0.7.0 # TensorFlow runtime only tensorflow==2.0.0 # TensorFlow runtime only ``` -Learn how to install additional packages [here](../dependencies/python-packages.md). +Learn how to install additional packages [here](../dependency-management/python-packages.md). diff --git a/docs/deployments/tensorflow.md b/docs/deployments/tensorflow.md index 723d3e243a..9e42f84a64 100644 --- a/docs/deployments/tensorflow.md +++ b/docs/deployments/tensorflow.md @@ -27,7 +27,7 @@ Deploy TensorFlow models as web services. mem: # memory request per replica (default: Null) ``` -See [packaging TensorFlow models](../packaging/tensorflow.md) for how to export a TensorFlow model. +See [packaging TensorFlow models](../packaging-models/tensorflow.md) for how to export a TensorFlow model. ## Example diff --git a/docs/packaging/onnx.md b/docs/packaging-models/onnx.md similarity index 96% rename from docs/packaging/onnx.md rename to docs/packaging-models/onnx.md index 08ff7a6da5..fbaec4b439 100644 --- a/docs/packaging/onnx.md +++ b/docs/packaging-models/onnx.md @@ -1,8 +1,8 @@ -# Packaging ONNX models +# ONNX Export your trained model to the ONNX model format. Here is an example of an sklearn model being exported to ONNX: -```Python +```python from sklearn.linear_model import LogisticRegression from onnxmltools import convert_sklearn from onnxconverter_common.data_types import FloatTensorType diff --git a/docs/packaging/tensorflow.md b/docs/packaging-models/tensorflow.md similarity index 97% rename from docs/packaging/tensorflow.md rename to docs/packaging-models/tensorflow.md index 93fe0b8e35..ef48e978fb 100644 --- a/docs/packaging/tensorflow.md +++ b/docs/packaging-models/tensorflow.md @@ -1,9 +1,9 @@ -# Packaging TensorFlow models +# TensorFlow Export your trained model and upload the export directory, or a checkpoint directory containing the export directory (which is usually the case if you used `estimator.train_and_evaluate`). An example is shown below (here is the [complete example](https://github.com/cortexlabs/cortex/blob/master/examples/tensorflow/sentiment-analyzer)): -```Python +```python import tensorflow as tf ... diff --git a/docs/summary.md b/docs/summary.md index 2b49e57936..ab24ac7ce8 100644 --- a/docs/summary.md +++ b/docs/summary.md @@ -1,7 +1,7 @@ -# Summary +# Table of contents * [Deploy machine learning models in production](../README.md) -* [Install](cluster/install.md) +* [Install](cluster-management/install.md) * [Tutorial](../examples/sklearn/iris-classifier/README.md) * [GitHub](https://github.com/cortexlabs/cortex) * [Examples](https://github.com/cortexlabs/cortex/tree/master/examples) @@ -19,28 +19,28 @@ * [Autoscaling](deployments/autoscaling.md) * [Prediction monitoring](deployments/prediction-monitoring.md) * [Compute](deployments/compute.md) -* [CLI commands](cluster/cli.md) * [API statuses](deployments/statuses.md) * [Python client](deployments/python-client.md) ## Packaging models -* [TensorFlow](packaging/tensorflow.md) -* [ONNX](packaging/onnx.md) +* [TensorFlow](packaging-models/tensorflow.md) +* [ONNX](packaging-models/onnx.md) ## Dependency management -* [Python packages](dependencies/python-packages.md) -* [System packages](dependencies/system-packages.md) +* [Python packages](dependency-management/python-packages.md) +* [System packages](dependency-management/system-packages.md) ## Cluster management -* [Cluster configuration](cluster/config.md) -* [AWS credentials](cluster/aws.md) -* [Security](cluster/security.md) -* [Update](cluster/update.md) -* [Uninstall](cluster/uninstall.md) +* [CLI commands](cluster-management/cli.md) +* [Cluster configuration](cluster-management/config.md) +* [AWS credentials](cluster-management/aws.md) +* [Security](cluster-management/security.md) +* [Update](cluster-management/update.md) +* [Uninstall](cluster-management/uninstall.md) ## Contributing -* [Development](development.md) +* [Development](contributing/development.md) diff --git a/examples/README.md b/examples/README.md index 6a246f25be..6d528a21ec 100644 --- a/examples/README.md +++ b/examples/README.md @@ -4,29 +4,29 @@ - [Iris classification](tensorflow/iris-classifier): deploy a model to classify iris flowers. -- [Text generation with GPT-2](tensorflow/text-generator): deploy OpenAI's GPT-2 to generate text. +- [Text generation](tensorflow/text-generator): deploy OpenAI's GPT-2 to generate text. -- [Image classification with Inception V3](tensorflow/image-classifier): deploy an Inception V3 model to classify images. +- [Sentiment analysis](tensorflow/sentiment-analyzer): deploy a BERT model for sentiment analysis. -- [Sentiment analysis with BERT](tensorflow/sentiment-analyzer): deploy a BERT model for sentiment analysis. +- [Image classification](tensorflow/image-classifier): deploy an Inception model to classify images. ## PyTorch - [Iris classification](pytorch/iris-classifier): deploy a model to classify iris flowers. -- [Search completion](pytorch/search-completer): deploy a model to complete search terms. +- [Search completion](pytorch/search-completer): deploy a Facebook's RoBERTa model to complete search terms. -- [Text generation with DistilGPT2](pytorch/text-generator): deploy Hugging Face's DistilGPT2 model to generate text. +- [Text generation](pytorch/text-generator): deploy Hugging Face's DistilGPT2 model to generate text. -- [Answer generation with DialoGPT](pytorch/answer-generator): deploy Microsoft's DialoGPT model to answer questions. +- [Answer generation](pytorch/answer-generator): deploy Microsoft's DialoGPT model to answer questions. -- [Text summarization with BERT](pytorch/text-summarizer): deploy a BERT model to summarize text. +- [Text summarization](pytorch/text-summarizer): deploy a BERT model to summarize text. -- [Reading comprehension with AllenNLP](pytorch/reading-comprehender): deploy an AllenNLP model for reading comprehension. +- [Reading comprehension](pytorch/reading-comprehender): deploy an AllenNLP model for reading comprehension. -- [Language identification with fastText](pytorch/language-identifier): deploy a fastText model to identify languages. +- [Language identification](pytorch/language-identifier): deploy a fastText model to identify languages. -- [Image Classification with AlexNet](pytorch/image-classifier): deploy an AlexNet model from TorchVision to classify images. +- [Image Classification](pytorch/image-classifier): deploy an AlexNet model from TorchVision to classify images. ## XGBoost From 480a9c2332fb23c9946fa5867cc37d57c62904f4 Mon Sep 17 00:00:00 2001 From: Omer Spillinger Date: Thu, 28 Nov 2019 12:12:33 -0800 Subject: [PATCH 2/3] Update README.md --- README.md | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/README.md b/README.md index 5155d4a069..20ff5a1b06 100644 --- a/README.md +++ b/README.md @@ -16,19 +16,12 @@ Cortex is an open source platform for deploying machine learning models—traine ## Key features * **Autoscaling:** Cortex automatically scales APIs to handle production workloads. - * **Multi framework:** Cortex supports TensorFlow, PyTorch, scikit-learn, XGBoost, and more. - * **CPU / GPU support:** Cortex can run inference on CPU or GPU infrastructure. - * **Spot instances:** Cortex supports EC2 spot instances. - * **Rolling updates:** Cortex updates deployed APIs without any downtime. - * **Log streaming:** Cortex streams logs from deployed models to your CLI. - * **Prediction monitoring:** Cortex monitors network metrics and tracks predictions. - * **Minimal configuration:** Deployments are defined in a single `cortex.yaml` file.
@@ -131,11 +124,7 @@ The CLI sends configuration and code to the cluster every time you run `cortex d * [Sentiment analysis](https://github.com/cortexlabs/cortex/tree/0.11/examples/tensorflow/sentiment-analyzer): deploy a BERT model for sentiment analysis. - * [Image classification](https://github.com/cortexlabs/cortex/tree/0.11/examples/tensorflow/image-classifier): deploy an Inception model to classify images. - * [Search completion](https://github.com/cortexlabs/cortex/tree/0.11/examples/tensorflow/search-completer): deploy Facebook's RoBERTa model to complete search terms. - * [Text generation](https://github.com/cortexlabs/cortex/tree/0.11/examples/pytorch/text-generator): deploy Hugging Face's DistilGPT2 model to generate text. - * [Iris classification](https://github.com/cortexlabs/cortex/tree/0.11/examples/sklearn/iris-classifier): deploy a scikit-learn model to classify iris flowers. From 496cbb122af03746a26a07656d2a1c8178517866 Mon Sep 17 00:00:00 2001 From: Omer Spillinger Date: Thu, 28 Nov 2019 12:25:16 -0800 Subject: [PATCH 3/3] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 20ff5a1b06..80d6e36c5e 100644 --- a/README.md +++ b/README.md @@ -40,7 +40,7 @@ max instances: 10 spot instances: yes ￮ spinning up your cluster ... -your cortex cluster is ready! +your cluster is ready! ``` ## Deploying a model