Sagemaker Idle

We want to be able to shutdown sagemaker notebooks automatically, to avoid paying for time we aren't using!

Lifecycle configuration script

• Runs each time notebook starts
• Script details:
  • Runs as root
    • Notebooks run as ec2-user, so to get that user in shell: sudo -u ec2-user
  • Limit of 16384 characters
  • Limit of 5 minute runtime
  • In scripts, $PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
• Can setup connections to other AWS resources as well
• CloudWatch Logs for issues
  • Log group: /aws/sagemaker/NotebookInstances
  • Log stream: [notebook-instance-name]/[LifecycleConfigHook]

Tons more details for iterating through Conda environments.

Some example scripts from AWS can be found here.

Recommended Setup

1. Turn off notebook
2. Create new lifecycle configuration with the below script
3. Click "Edit" notebook in AWS console, associate lifecycle with notebook
4. Restart notebook and check CloudWatch logs to make sure it worked

#!/bin/bash

set -e

# OVERVIEW
# This script stops a SageMaker notebook once it's idle for more than 1 hour (default time)
# You can change the idle time for stop using the environment variable below.
# If you want the notebook the stop only if no browsers are open, remove the --ignore-connections flag
#
# Note that this script will fail if either condition is not met
#   1. Ensure the Notebook Instance has internet connectivity to fetch the example config
#   2. Ensure the Notebook Instance execution role permissions to SageMaker:StopNotebookInstance to stop the notebook 
#       and SageMaker:DescribeNotebookInstance to describe the notebook.
#

# PARAMETERS
IDLE_TIME=10800

echo "Fetching the autostop script"
wget https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-notebook-instance-lifecycle-config-samples/master/scripts/auto-stop-idle/autostop.py

echo "Starting the SageMaker autostop script in cron"

(crontab -l 2>/dev/null; echo "5 * * * * /usr/bin/python $PWD/autostop.py --time $IDLE_TIME --ignore-connections") | crontab -

Sagemaker Knockout

Seems a promising way to manage and use CPU/GPU load.

Note that if you use the Sagemaker Knockout starting script, it will fail in the AWS console!

Uncaught DOMException: Failed to execute 'btoa' on 'Window': The string to be encoded contains characters outside of the Latin1 range.

You have to remove the 🥊 character.

I haven't been able to make this work yet, pending this pidfile issue.

Using AWS Shell to manage

If you want to update your lifecycle configuration for existing notebook:

pip3 install -r requirements.txt
aws-shell  # will take a bit to create autocomplete index...

Create an IAM user with SagemakerFullAccess policy. This lets it control sagemaker along with a few other adjacent things, like particular S3 buckets and keys.

I usually add these to ~/.aws/credentials file in the format:

[myprofile]
output = json
region = us-east-1
aws_access_key_id = xxxxxx
aws_secret_access_key = xxxxxxx

Then in the shell:

aws-shell
aws> .profile myprofile  # activate your profile!
aws> sagemaker update-notebook-instance --lifecycle-config-name my-config --notebook-instance-name mynotebook-name
aws> .exit

If you prefer, you can also do this with boto3.