The AWS Envision Engineering team has partnered with Boston Dynamics to create a demo for deploying customer code and machine learning models to Spot® robots. The objective of this demo is to demonstrate how AWS can bring AI/ML applications to the edge, and help Spot identify objects of interest via the Spot CAM+, a combination of a 360-degree and pan-tilt-zoom camera. Using SageMaker to train object detection/image classification models (with built-in SageMaker algorithms, or popular ML frameworks such as TensorFlow and PyTorch), the models can then be optimized and packaged within a Greengrass v2 component, and deployed to the edge to run on AWS IoT Greengrass. Greengrass will run on Spot Compute AI, a hardened Linux-based computer that mounts to the back of the robot, and includes a Nvidia P5000 GPU.
The demo is comprised of two major parts: 1) building/starting a Docker image for Spot that includes Greengrass, CUDA, and Nvidia drivers; and 2) a Python-based CDK application that deploys all the resources necessary for Greengrass v2 components and associated repos/build pipelines, and SageMaker Edge Manager.
sudo usermod -aG docker $USER
sudo rebootEnsure you have AWS CLI installed, a IAM user with an access key, and a named profile configured:
- Installing, updating, and uninstalling the AWS CLI
- Configuration basics
- Named profiles
- Minimum IAM policy required for deploying solution
cd spot_demo_cdk
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install -r requirements.txt
chmod +x package_lambdas_py3.sh
chmod +x deploy.shThis will download the 'ssd_512_mobilenet1.0_voc' pre-trained object detection from GluonCV Model Zoo, and create a model archive used for the demonstration.
python3 download_export_model.pyNOTE: This will be stored in AWS Secrets Manager, do not hard-code these variables in your application!
export AWS_DEFAULT_PROFILE=<YOUR_AWS_NAMED_CLI_PROFILE>
export AWS_DEFAULT_REGION=<YOUR_AWS_REGION>
export SPOT_USER=<YOUR_SPOT_USERNAME>
export SPOT_PASSWORD=<YOUR_SPOT_PASSWORD>
export ROBOT_IP=<YOUR_ROBOT_IP_ADDRESS>
export THING_NAME=<YOUR_SPOT_ROBOT_NAME>
export TARGET=<cpu | spotcore_ai>
export SPOT_SDK_VERSION=<MAJOR.MINOR.PATCH>./deploy.shssh-copy-id -p 20022 spot@$ROBOT_IPcd spot_container/
./publish-to-ecr-ssh.sh $AWS_DEFAULT_PROFILE spot $ROBOT_IP greengrass-v2-spot-robot-demo $TARGET $SPOT_SDK_VERSION
cd ..- Go to https://developer.nvidia.com/nvidia-tensorrt-download, and login (join if you don't have an account)
- Click TensorRT 7 under Available Versions
- Check I Agree and click TensorRT 7.2.1
- Download TensorRT 7.2.1 for Ubuntu 18.04 and CUDA 10.2 TAR package to the spot_container directory
- You should now have TensorRT-7.2.1.6.Ubuntu-18.04.x86_64-gnu.cuda-10.2.cudnn8.0.tar.gz in ggv2/container/gpu, which is required for building the container
cd spot_container/
./publish-to-ecr-ssh.sh $AWS_DEFAULT_PROFILE spot $ROBOT_IP greengrass-v2-spot-robot-demo $TARGET $SPOT_SDK_VERSION
cd ..NOTE: running this the first time will automatically provision resources necessary to run Greengrass v2
NOTE: edit this script to include restart policies to manage your Docker container after shutdown/reboot
./run-ggv2-ssh.sh $AWS_DEFAULT_PROFILE spot $ROBOT_IP 20022 greengrass-v2-spot-robot-demo $TARGET $SPOT_SDK_VERSION $THING_NAMEaws secretsmanager create-secret --name spot_secrets --profile $AWS_DEFAULT_PROFILE
aws secretsmanager describe-secret --secret-id spot_secrets --profile $AWS_DEFAULT_PROFILE
aws secretsmanager put-secret-value --profile $AWS_DEFAULT_PROFILE --secret-id spot_secrets --secret-string "{\"spot_user\":\"$SPOT_USER\",\"spot_password\":\"$SPOT_PASSWORD\"}"NOTE: make sure to note the secret ARN. You'll need this later when configuring Greengrass.
aws iam attach-role-policy --profile $AWS_DEFAULT_PROFILE --role-name GreengrassV2TokenExchangeRole --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess
aws iam attach-role-policy --profile $AWS_DEFAULT_PROFILE --role-name GreengrassV2TokenExchangeRole --policy-arn arn:aws:iam::aws:policy/SecretsManagerReadWriteTo create a new Greengrass deployment with the custom components for Spot, run the following:
python3 create_greengrass_deployment.pyAfter logging onto Spot Tablet, navigate to Settings >> Actions.
Then tap Create New Action.
Under the prompt Choose a template action, select SageMaker Server, and tap Create.
Verify the message The service [sagemaker-server] is currently registered and has no service faults. Then select your image source - for example, Spot CAM Image Service Ptz. Finally, select a specific processing model (or tap Choose Later to select when performing the action). When completed, tap Save
Verify you have a new action named SageMaker Server.
On the main Spot Tablet screen, tap the red action button in the bottom right corner.
Then, choose SageMaker Server as your action from the provided list.
Next, aim your camera at the area you'd like to capture and run inference on. Tap Next.
Then, select the model you'd like to use for inference - for example, gluoncv-model. Tap Next.
Finally, review the image capture and inference results. Tap Create to add metadata attributes and save as mission data.
See CONTRIBUTING for more information.
This repository is licensed under the MIT-0 License. See the LICENSE file.
This repository contains modified source code licensed under the Boston Dynamics Inc. ("Boston Dynamics") Software Development Kit License (20191101-BDSDK-SL).