Skip to content


Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

Sagemaker Idle

We want to be able to shutdown sagemaker notebooks automatically, to avoid paying for time we aren't using!

Lifecycle configuration script

  • Runs each time notebook starts
  • Script details:
    • Runs as root
      • Notebooks run as ec2-user, so to get that user in shell: sudo -u ec2-user
    • Limit of 16384 characters
    • Limit of 5 minute runtime
    • In scripts, $PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
  • Can setup connections to other AWS resources as well
  • CloudWatch Logs for issues
    • Log group: /aws/sagemaker/NotebookInstances
    • Log stream: [notebook-instance-name]/[LifecycleConfigHook]

Tons more details for iterating through Conda environments.

Some example scripts from AWS can be found here.

Recommended Setup

  1. Turn off notebook
  2. Create new lifecycle configuration with the below script
  3. Click "Edit" notebook in AWS console, associate lifecycle with notebook
  4. Restart notebook and check CloudWatch logs to make sure it worked

set -e

# This script stops a SageMaker notebook once it's idle for more than 1 hour (default time)
# You can change the idle time for stop using the environment variable below.
# If you want the notebook the stop only if no browsers are open, remove the --ignore-connections flag
# Note that this script will fail if either condition is not met
#   1. Ensure the Notebook Instance has internet connectivity to fetch the example config
#   2. Ensure the Notebook Instance execution role permissions to SageMaker:StopNotebookInstance to stop the notebook 
#       and SageMaker:DescribeNotebookInstance to describe the notebook.


echo "Fetching the autostop script"

echo "Starting the SageMaker autostop script in cron"

(crontab -l 2>/dev/null; echo "5 * * * * /usr/bin/python $PWD/ --time $IDLE_TIME --ignore-connections") | crontab -

Sagemaker Knockout

Seems a promising way to manage and use CPU/GPU load.

Note that if you use the Sagemaker Knockout starting script, it will fail in the AWS console!

Uncaught DOMException: Failed to execute 'btoa' on 'Window': The string to be encoded contains characters outside of the Latin1 range.

You have to remove the 🥊 character.

I haven't been able to make this work yet, pending this pidfile issue.

Using AWS Shell to manage

If you want to update your lifecycle configuration for existing notebook:

pip3 install -r requirements.txt
aws-shell  # will take a bit to create autocomplete index...

Create an IAM user with SagemakerFullAccess policy. This lets it control sagemaker along with a few other adjacent things, like particular S3 buckets and keys.

I usually add these to ~/.aws/credentials file in the format:

output = json
region = us-east-1
aws_access_key_id = xxxxxx
aws_secret_access_key = xxxxxxx

Then in the shell:

aws> .profile myprofile  # activate your profile!
aws> sagemaker update-notebook-instance --lifecycle-config-name my-config --notebook-instance-name mynotebook-name
aws> .exit

If you prefer, you can also do this with boto3.


Tools / notes for working with Sagemaker notebooks






No releases published


No packages published