diff --git a/README.md b/README.md index 20449e0..876216f 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,9 @@ This repository contains examples of Docker images that are valid custom images - [r-image](examples/r-image) - This example contains the `ir` kernel and a selection of R packages, along with the AWS Python SDK (boto3) and the SageMaker Python SDK which can be used from R using `reticulate` - [rapids-image](examples/rapids-image) - This example uses the offical rapids.ai image from Dockerhub. Use with a GPU instance on Studio - [scala-image](examples/scala-image) - This example adds a Scala kernel based on [Almond Scala Kernel](https://almond.sh/). -- [tf2.3-image](examples/tf23-image) - This examples uses the official TensorFlow 2.3 image from DockerHub and demonstrates bundling custom files along with the image. +- [tf2.3-image](examples/tf23-image) - This example uses the official TensorFlow 2.3 image from DockerHub and demonstrates bundling custom files along with the image. +- [conda-efs-image](examples/conda-efs-image) - This example creates a custom image in Amazon SageMaker Studio which will load a `custom` conda environment from a users [Elastic File System](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-tasks-manage-storage.html) (EFS) home folder. + #### One-time setup All examples have a one-time setup to create an ECR repository diff --git a/examples/conda-efs-image/Dockerfile b/examples/conda-efs-image/Dockerfile new file mode 100644 index 0000000..4abb0d5 --- /dev/null +++ b/examples/conda-efs-image/Dockerfile @@ -0,0 +1,52 @@ +FROM public.ecr.aws/ubuntu/ubuntu:20.04 + +ARG NB_USER="sagemaker-user" +ARG NB_UID="1000" +ARG NB_GID="100" +ARG NB_ENV="custom" + +###################### +# OVERVIEW +# 1. Creates the `sagemaker-user` user with UID/GID 1000/100. +# 2. Ensures this user can `sudo` by default. +# 5. Make the default shell `bash`. This enhances the experience inside a Jupyter terminal as otherwise Jupyter defaults to `sh` +###################### + +# Setup the "sagemaker-user" user with root privileges. +RUN \ + apt-get update && \ + apt-get install -y sudo wget nano && \ + useradd -m -s /bin/bash -N -u $NB_UID $NB_USER && \ + chmod g+w /etc/passwd && \ + echo "${NB_USER} ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers && \ + # Prevent apt-get cache from being persisted to this layer. + rm -rf /var/lib/apt/lists/* + +# installing miniconda +RUN wget -q https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh +RUN bash Miniconda3-latest-Linux-x86_64.sh -b -p /miniconda +ENV PATH=$PATH:/miniconda/condabin:/miniconda/bin + +# install libraries for the base environment +RUN conda install ipykernel + +# update kernelspec to load our custom environment strored on EFS +RUN rm -rf /miniconda/share/jupyter/kernels/python3 +# write new kernel which initialises uses conda run for custom env +# see: https://github.com/ipython/ipykernel/issues/416 +COPY custom_kernel_spec/ /miniconda/share/jupyter/kernels/python3 + +# Make the default shell bash (vs "sh") for a better Jupyter terminal UX +ENV SHELL=/bin/bash \ + NB_USER=$NB_USER \ + NB_ENV=$NB_ENV \ + NB_UID=$NB_UID \ + NB_GID=$NB_GID \ + HOME=/home/$NB_USER + +# Set the conda envs path to map to EFS (without of requiring .condarc) +ENV CONDA_AUTO_ACTIVATE_BASE=false \ + CONDA_ENVS_PATH=/home/$NB_USER/.conda/envs + +WORKDIR $HOME +USER $NB_UID \ No newline at end of file diff --git a/examples/conda-efs-image/README.md b/examples/conda-efs-image/README.md new file mode 100644 index 0000000..227b9c0 --- /dev/null +++ b/examples/conda-efs-image/README.md @@ -0,0 +1,95 @@ +## Conda EFS Image + +### Overview + +This example creates a custom image in Amazon SageMaker Studio which will load a `custom` conda environment from a users [Elastic File System](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-tasks-manage-storage.html) (EFS) home folder. + +The Dockerfile is built from `ubuntu:20.04` and installs the latest [Minconda3](https://docs.conda.io/en/latest/miniconda.html) distribution. + +### Building the image + +Build the Docker image and push to Amazon ECR. +``` +# Modify these as required. The Docker registry endpoint can be tuned based on your current region from https://docs.aws.amazon.com/general/latest/gr/ecr.html#ecr-docker-endpoints +REGION= +ACCOUNT_ID= + +# Build the image +IMAGE_NAME=custom-conda-efs +aws --region ${REGION} ecr get-login-password | docker login --username AWS --password-stdin ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/smstudio-custom +docker build . -t ${IMAGE_NAME} -t ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/smstudio-custom:${IMAGE_NAME} --build-arg NB_ENV=custom +``` + +NOTE: The `NB_ENV` maps to the conda environment which will need to be created on EFS (see below) + +``` +docker push ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/smstudio-custom:${IMAGE_NAME} +``` + +### Using it with SageMaker Studio + +Create a SageMaker Image with the image in ECR. + +``` +# Role in your account to be used for the SageMaker Image +ROLE_ARN= + +aws --region ${REGION} sagemaker create-image \ + --image-name ${IMAGE_NAME} \ + --display-name "Conda EFS env" \ + --role-arn ${ROLE_ARN} + +aws --region ${REGION} sagemaker create-image-version \ + --image-name ${IMAGE_NAME} \ + --base-image "${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/smstudio-custom:${IMAGE_NAME}" + +# Verify the image-version is created successfully. Do NOT proceed if image-version is in CREATE_FAILED state or in any other state apart from CREATED. +aws --region ${REGION} sagemaker describe-image-version --image-name ${IMAGE_NAME} +``` + +Create an AppImageConfig for this image. + +``` +aws --region ${REGION} sagemaker create-app-image-config --cli-input-json file://app-image-config-input.json +``` + +Create a Domain, providing the SageMaker Image and AppImageConfig in the Domain creation. Replace the placeholders for VPC ID, Subnet IDs, and Execution Role in `create-domain-input.json`. + +``` +aws --region ${REGION} sagemaker create-domain --cli-input-json file://create-domain-input.json +``` + +If you have an existing Domain, you can use the `update-domain` command. + +``` +DOMAIN_ID= +aws --region ${REGION} sagemaker update-domain --domain-id ${DOMAIN_ID} --cli-input-json file://update-domain-input.json +``` + +### Create your custom conda environment on EFS + +Use the [Amazon SageMaker Studio Launcher](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-launcher.html) to open a **System terminal** and run the following commands to setup your custom environment: + +```bash +mkdir -p ~/.conda/envs +conda create -p ~/.conda/envs/custom +conda activate custom # or ". activate ~/.conda/envs/custom" +conda install ipykernel +``` + +### Verifying your installation + +Use the [Amazon SageMaker Studio Launcher](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-launcher.html) to launch a **Notebook** for your *Conda EFS env* SageMaker Image and run the following command in the first cell to verify you are now using a `custom` conda environment: + +``` +!conda info +``` + +### Installing additional libraries + +Use the [Amazon SageMaker Studio Launcher](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-launcher.html) to launch an **Image terminal** and run the following commands to install new libraries in your custom environment such as `numpy`: + +```bash +. activate /home/$NB_USER/.conda/envs/$NB_ENV +conda install numpy +``` \ No newline at end of file diff --git a/examples/conda-efs-image/app-image-config-input.json b/examples/conda-efs-image/app-image-config-input.json new file mode 100644 index 0000000..5a08191 --- /dev/null +++ b/examples/conda-efs-image/app-image-config-input.json @@ -0,0 +1,16 @@ +{ + "AppImageConfigName": "custom-conda-efs-env-config", + "KernelGatewayImageConfig": { + "KernelSpecs": [ + { + "Name": "python3", + "DisplayName": "Python 3 (custom EFS env)" + } + ], + "FileSystemConfig": { + "MountPath": "/home/sagemaker-user", + "DefaultUid": 1000, + "DefaultGid": 100 + } + } +} diff --git a/examples/conda-efs-image/create-domain-input.json b/examples/conda-efs-image/create-domain-input.json new file mode 100644 index 0000000..2498c1d --- /dev/null +++ b/examples/conda-efs-image/create-domain-input.json @@ -0,0 +1,19 @@ +{ + "DomainName": "domain-with-conda-efs-image", + "VpcId": "", + "SubnetIds": [ + "" + ], + "DefaultUserSettings": { + "ExecutionRole": "", + "KernelGatewayAppSettings": { + "CustomImages": [ + { + "ImageName": "custom-conda-efs", + "AppImageConfigName": "custom-conda-efs-env-config" + } + ] + } + }, + "AuthMode": "IAM" +} \ No newline at end of file diff --git a/examples/conda-efs-image/custom_kernel_spec/kernel.json b/examples/conda-efs-image/custom_kernel_spec/kernel.json new file mode 100644 index 0000000..40ed268 --- /dev/null +++ b/examples/conda-efs-image/custom_kernel_spec/kernel.json @@ -0,0 +1,9 @@ +{ + "argv": [ + "bash", + "-c", + "conda run -n $NB_ENV python -m ipykernel_launcher -f '{connection_file}'" + ], + "display_name": "Conda EFS env", + "language": "python" +} \ No newline at end of file diff --git a/examples/conda-efs-image/custom_kernel_spec/logo-32x32.png b/examples/conda-efs-image/custom_kernel_spec/logo-32x32.png new file mode 100644 index 0000000..be81330 Binary files /dev/null and b/examples/conda-efs-image/custom_kernel_spec/logo-32x32.png differ diff --git a/examples/conda-efs-image/custom_kernel_spec/logo-64x64.png b/examples/conda-efs-image/custom_kernel_spec/logo-64x64.png new file mode 100644 index 0000000..eebbff6 Binary files /dev/null and b/examples/conda-efs-image/custom_kernel_spec/logo-64x64.png differ diff --git a/examples/conda-efs-image/update-domain-input.json b/examples/conda-efs-image/update-domain-input.json new file mode 100644 index 0000000..f780309 --- /dev/null +++ b/examples/conda-efs-image/update-domain-input.json @@ -0,0 +1,12 @@ +{ + "DefaultUserSettings": { + "KernelGatewayAppSettings": { + "CustomImages": [ + { + "ImageName": "custom-conda-efs", + "AppImageConfigName": "custom-conda-efs-env-config" + } + ] + } + } +} \ No newline at end of file