From cfa8b749e5762643bed9534f2de369292114a808 Mon Sep 17 00:00:00 2001 From: Jeannie Finks <74554921+jeanniefinks@users.noreply.github.com> Date: Mon, 25 Apr 2022 13:35:10 -0400 Subject: [PATCH 1/8] Update README.md various grammatical edits --- examples/aws-sagemaker/README.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/examples/aws-sagemaker/README.md b/examples/aws-sagemaker/README.md index 315c480391..05b98ae392 100644 --- a/examples/aws-sagemaker/README.md +++ b/examples/aws-sagemaker/README.md @@ -17,7 +17,7 @@ limitations under the License. # Deploy DeepSparse with Amazon SageMaker [Amazon SageMaker](https://docs.aws.amazon.com/sagemaker/index.html) -offers easy to use infrastructure for deploying deep learning models at scale. +offers easy-to-use infrastructure for deploying deep learning models at scale. This directory provides a guided example for deploying a [DeepSparse](https://github.com/neuralmagic/deepsparse) inference server on SageMaker. Using both of these tools, deployments benefit from sparse-CPU acceleration from @@ -25,8 +25,8 @@ DeepSparse and automatic scaling from SageMaker. ## Contents -In addition to the step-by-step instructions in this guide, this directory contains -additional files to aide in the deployment. +In addition to the step-by-step instructions in this guide, the directory contains +additional files to aid in the deployment. ### Dockerfile The included `Dockerfile` builds an image on top of the standard `python:3.8` image @@ -36,11 +36,11 @@ with `deepsparse` installed and creates an executable command `serve` that runs `invocations/` endpoint. For general customization of the server, changes should not need to be made -to the dockerfile, but to the `config.yaml` file that the dockerfile reads from +to the dockerfile, but to the `config.yaml` file that the Dockerfile reads from instead. ### config.yaml -`config.yaml` used to configure the DeepSparse serve running in the Dockerfile. +`config.yaml` is used to configure the DeepSparse server running in the Dockerfile. It is important that the config contains the line `integration: sagemaker` so endpoints may be provisioned correctly to match SageMaker specifications. @@ -56,7 +56,7 @@ More information on the DeepSparse server and its configuration can be found ## Deploying to SageMaker -The following steps are required to provision and deploy DeepSparse to sagemaker +The following steps are required to provision and deploy DeepSparse to SageMaker for inference: * Build the DeepSparse-SageMaker `Dockerfile` into a local docker image * Create an [Amazon ECR](https://aws.amazon.com/ecr/) repository to host the image @@ -82,7 +82,7 @@ docker build -t deepsparse-sagemaker-example . ``` ### Create an ECR Repository -The following code snippet can be used in python to create an ECR repository. +The following code snippet can be used in Python to create an ECR repository. The `region_name` can be swapped to a preferred region. The repository will be named `deepsparse-sagemaker`. If the repository is already created, this step may be skipped. @@ -204,7 +204,7 @@ endpoint_res = sm_boto3.create_endpoint( ) ``` -After creating the endpoint, it's status can be checked by running the following. +After creating the endpoint, its status can be checked by running the following. Initially, the `EndpointStatus` will be `Creating`. Checking after the image is successfully launched, it will be `InService`. If there are any errors, it will become `Failed`. @@ -215,9 +215,9 @@ pprint(sm_boto3.describe_endpoint(EndpointName=endpoint_name)) ``` -## Making a reqest to the Endpoint +## Making a request to the Endpoint After the endpoint is in service, requests can be made to it through the -`invoke_endpoint` api. Inputs will be passed as a json payload. +`invoke_endpoint` api. Inputs will be passed as a JSON payload. ```python import json @@ -255,7 +255,7 @@ sm_boto3.delete_model(ModelName=model_name) ## Next Steps These steps create an invokable SageMaker inference endpoint powered with the DeepSparse -engine. The `EndpointConfig` settings may be adjusted to set instance scaling rules based +Engine. The `EndpointConfig` settings may be adjusted to set instance scaling rules based on deployment needs. More information on deploying custom models with SageMaker can be found From adf2f203c30a1e4a96746c2ed89b996541ad17e8 Mon Sep 17 00:00:00 2001 From: Jeannie Finks <74554921+jeanniefinks@users.noreply.github.com> Date: Mon, 25 Apr 2022 13:41:01 -0400 Subject: [PATCH 2/8] Update README.md additional edits for section headline consistency --- examples/aws-sagemaker/README.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/examples/aws-sagemaker/README.md b/examples/aws-sagemaker/README.md index 05b98ae392..9f2890a4f1 100644 --- a/examples/aws-sagemaker/README.md +++ b/examples/aws-sagemaker/README.md @@ -14,7 +14,7 @@ See the License for the specific language governing permissions and limitations under the License. --> -# Deploy DeepSparse with Amazon SageMaker +# Deploying DeepSparse with Amazon SageMaker [Amazon SageMaker](https://docs.aws.amazon.com/sagemaker/index.html) offers easy-to-use infrastructure for deploying deep learning models at scale. @@ -73,7 +73,7 @@ credentials, tools, and libraries are also required: * [Docker and the `docker` cli](https://docs.docker.com/get-docker/) * The `boto3` python AWS sdk (`pip install boto3`) -### Build the DeepSparse-SageMaker image locally +### Building the DeepSparse-SageMaker Image Locally The `Dockerfile` can be build from this directory from a bash shell using the following command. The image will be tagged locally as `deepsparse-sagemaker-example`. @@ -81,7 +81,7 @@ The image will be tagged locally as `deepsparse-sagemaker-example`. docker build -t deepsparse-sagemaker-example . ``` -### Create an ECR Repository +### Creating an ECR Repository The following code snippet can be used in Python to create an ECR repository. The `region_name` can be swapped to a preferred region. The repository will be named `deepsparse-sagemaker`. If the repository is already created, this step may be skipped. @@ -93,7 +93,7 @@ ecr = boto3.client("ecr", region_name='us-east-1') create_repository_res = ecr.create_repository(repositoryName="deepsparse-sagemaker") ``` -### Push local image to ECR Repository +### Pushing the Local Image to the ECR Repository Once the image is built and the ECR repository is created, the image can be pushed using the following bash commands. @@ -129,7 +129,7 @@ c18b71656bcf: Waiting latest: digest: sha256:XXX size: 3884 ``` -### Create SageMaker Model +### Creating a SageMaker Model A SageMaker `Model` can now be created referencing the pushed image. The example model will be named `question-answering-example`. As mentioned in the requirements, `ROLE_ARN` should be a string arn of an AWS @@ -159,7 +159,7 @@ More information about options for configuring SageMaker `Model` instances can be found [here](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateModel.html). -### Build SageMaker EndpointConfig +### Building a SageMaker EndpointConfig The `EndpointConfig` is used to set the instance type to provision, how many, scaling rules, and other deployment settings. The following code snippet defines an endpoint with a single machine using an `ml.c5.large` CPU. @@ -193,7 +193,7 @@ endpoint_config = { endpoint_config_res = sm_boto3.create_endpoint_config(**endpoint_config) ``` -### Launch SageMaker Endpoint +### Launching a SageMaker Endpoint Once the `EndpointConfig` is defined, the endpoint can be easily launched using the `create_endpoint` command: @@ -215,7 +215,7 @@ pprint(sm_boto3.describe_endpoint(EndpointName=endpoint_name)) ``` -## Making a request to the Endpoint +## Making a Request to the Endpoint After the endpoint is in service, requests can be made to it through the `invoke_endpoint` api. Inputs will be passed as a JSON payload. From 87584b7590af8543b29c3818084c6da8fe631d69 Mon Sep 17 00:00:00 2001 From: Jeannie Finks <74554921+jeanniefinks@users.noreply.github.com> Date: Wed, 27 Apr 2022 09:49:59 -0400 Subject: [PATCH 3/8] Update examples/aws-sagemaker/README.md Co-authored-by: Rahul Tuli --- examples/aws-sagemaker/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/aws-sagemaker/README.md b/examples/aws-sagemaker/README.md index 9f2890a4f1..9eb8f66a06 100644 --- a/examples/aws-sagemaker/README.md +++ b/examples/aws-sagemaker/README.md @@ -17,7 +17,7 @@ limitations under the License. # Deploying DeepSparse with Amazon SageMaker [Amazon SageMaker](https://docs.aws.amazon.com/sagemaker/index.html) -offers easy-to-use infrastructure for deploying deep learning models at scale. +offers an easy-to-use infrastructure for deploying deep learning models at scale. This directory provides a guided example for deploying a [DeepSparse](https://github.com/neuralmagic/deepsparse) inference server on SageMaker. Using both of these tools, deployments benefit from sparse-CPU acceleration from From b4de8d26ab112c95b07a697b2a82403aaa088b05 Mon Sep 17 00:00:00 2001 From: Jeannie Finks <74554921+jeanniefinks@users.noreply.github.com> Date: Wed, 27 Apr 2022 09:50:22 -0400 Subject: [PATCH 4/8] Update examples/aws-sagemaker/README.md Co-authored-by: Rahul Tuli --- examples/aws-sagemaker/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/aws-sagemaker/README.md b/examples/aws-sagemaker/README.md index 9eb8f66a06..099801cd91 100644 --- a/examples/aws-sagemaker/README.md +++ b/examples/aws-sagemaker/README.md @@ -41,7 +41,7 @@ instead. ### config.yaml `config.yaml` is used to configure the DeepSparse server running in the Dockerfile. -It is important that the config contains the line `integration: sagemaker` so +The config must contain the line `integration: sagemaker` so endpoints may be provisioned correctly to match SageMaker specifications. Notice that the `model_path` and `task` are set to run a sparse-quantized From 6d39187bb3d2d995b4436bc021dc1167c952fb43 Mon Sep 17 00:00:00 2001 From: Jeannie Finks <74554921+jeanniefinks@users.noreply.github.com> Date: Wed, 27 Apr 2022 09:50:51 -0400 Subject: [PATCH 5/8] Update examples/aws-sagemaker/README.md Co-authored-by: Rahul Tuli --- examples/aws-sagemaker/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/aws-sagemaker/README.md b/examples/aws-sagemaker/README.md index 099801cd91..58d809a026 100644 --- a/examples/aws-sagemaker/README.md +++ b/examples/aws-sagemaker/README.md @@ -20,7 +20,7 @@ limitations under the License. offers an easy-to-use infrastructure for deploying deep learning models at scale. This directory provides a guided example for deploying a [DeepSparse](https://github.com/neuralmagic/deepsparse) inference server on SageMaker. -Using both of these tools, deployments benefit from sparse-CPU acceleration from +Deployments benefit from both sparse-CPU acceleration with DeepSparse and automatic scaling from SageMaker. From de8445d07a4c10c2f6c875c383e4622242978f0e Mon Sep 17 00:00:00 2001 From: Jeannie Finks <74554921+jeanniefinks@users.noreply.github.com> Date: Wed, 27 Apr 2022 09:51:07 -0400 Subject: [PATCH 6/8] Update examples/aws-sagemaker/README.md Co-authored-by: Rahul Tuli --- examples/aws-sagemaker/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/aws-sagemaker/README.md b/examples/aws-sagemaker/README.md index 58d809a026..7d86464b01 100644 --- a/examples/aws-sagemaker/README.md +++ b/examples/aws-sagemaker/README.md @@ -254,7 +254,7 @@ sm_boto3.delete_model(ModelName=model_name) ``` ## Next Steps -These steps create an invokable SageMaker inference endpoint powered with the DeepSparse +These steps create an invokable SageMaker inference endpoint powered by the DeepSparse Engine. The `EndpointConfig` settings may be adjusted to set instance scaling rules based on deployment needs. From 21e5c6a2a4027fb709f3669145f76900098f7996 Mon Sep 17 00:00:00 2001 From: Jeannie Finks <74554921+jeanniefinks@users.noreply.github.com> Date: Wed, 27 Apr 2022 09:52:54 -0400 Subject: [PATCH 7/8] Update README.md --- examples/aws-sagemaker/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/aws-sagemaker/README.md b/examples/aws-sagemaker/README.md index 7d86464b01..460537ec0d 100644 --- a/examples/aws-sagemaker/README.md +++ b/examples/aws-sagemaker/README.md @@ -31,7 +31,7 @@ additional files to aid in the deployment. ### Dockerfile The included `Dockerfile` builds an image on top of the standard `python:3.8` image with `deepsparse` installed and creates an executable command `serve` that runs -`deepsparse.server` on port 8080. SageMaker will execute this image by running +`deepsparse.server` on port 8080. SageMaker will execute this image by running `docker run serve` and expects the image to serve inference requests at the `invocations/` endpoint. From f2899c6a6e46e8e3dbd06d9dad37494f89c542b9 Mon Sep 17 00:00:00 2001 From: Jeannie Finks <74554921+jeanniefinks@users.noreply.github.com> Date: Wed, 27 Apr 2022 10:08:29 -0400 Subject: [PATCH 8/8] Update README.md --- examples/aws-sagemaker/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/aws-sagemaker/README.md b/examples/aws-sagemaker/README.md index 460537ec0d..9f7a4f7590 100644 --- a/examples/aws-sagemaker/README.md +++ b/examples/aws-sagemaker/README.md @@ -36,7 +36,7 @@ with `deepsparse` installed and creates an executable command `serve` that runs `invocations/` endpoint. For general customization of the server, changes should not need to be made -to the dockerfile, but to the `config.yaml` file that the Dockerfile reads from +to the Dockerfile, but to the `config.yaml` file that the Dockerfile reads from instead. ### config.yaml