SAMPLE10-MLOps-SageMaker-GitHub-Codepipeline-CodeBuild-CodeDeploy.md

SAMPLE-10: Implementing MLOps Pipeline using GitHub, AWS CodePipeline, AWS CodeBuild, AWS CodeDeploy, and AWS Sagemaker (Endpoint)

This sample shows:

• how to create MLOps Pipeline
• how to use GitHub Hooks (Getting Source Code from Github to CodePipeline)
• how to create Build CodePipeline (Source, Build), CodeBuild (modelbuild_buildspec.yml), Deploy CodePipeline (Source, Build, DeployStaging, DeployProd), CodeBuild (modeldeploy_buildspec.yml)
• how to save the model and artifacts on S3
• how to create and test models using Notebooks

Notes:

• Original Source code was pulled, updated, and adapted:
  • https://github.com/aws-samples/aws-mlops-pipelines-terraform
• "Modelbuild_Pipeline" and "Modeldeploy_Pipeline" are uploaded before. After applying terraform, Webhook in the CodePipeline pulls it from GitHub to inject it into AWS.
• If you run it the first time, please open to request to AWS for the instance: "ml.m5.xlarge" ("ResourceLimitExceeded").
  • https://repost.aws/knowledge-center/sagemaker-resource-limit-exceeded-error

Code: https://github.com/omerbsezer/Fast-Terraform/tree/main/samples/mlops-sagemaker-github-codepipeline-codebuild-codedeploy

Architecture:

• Model Build Pipeline Source Code, modelbuild_pipeline: https://github.com/omerbsezer/modelbuild_pipeline

• Model Deploy Pipeline Source Code, modeldeploy_pipeline: https://github.com/omerbsezer/modeldeploy_pipeline

• AWS Code Pipeline for Model Build (CI):

  • AWS Code Pipeline (modelbuild_ci_pipeline.tf)
    • Stage: Source (Hook from GitHub: modelbuild_hooks.tf)
    • Stage: Build (modelbuild_codebuild.tf: artifacts, environment, source (modelbuild_buildspec.yml: run build pipeline))
  • Sagemaker Data Pipeline (modelbuild_pipeline project: pipeline.py):
    • Preprocessing Step (modelbuild_pipeline project: pipeline.py => preprocess.py)
    • Model Training Step (modelbuild_pipeline project: pipeline.py => XGB BuiltIn Container)
    • Evaluation Step (modelbuild_pipeline project: pipeline.py => evaluate.py, ConditionStep to evaluate model quality)
    • Registering Model Step (modelbuild_pipeline project: pipeline.py => RegisterModel)

• AWS Code Pipeline for Model Deploy (CD):

  • AWS Code Pipeline (modeldeploy_cd_pipeline.tf)
    • Stage: Source (Hook from GitHub: modeldeploy_hooks.tf)
    • Stage: Build (modeldeploy_codebuild.tf: artifacts, environment, source (modeldeploy_buildspec.yml: run deploy pipeline => modeldeploy_pipeline project: build.py, cloud formation to create endpoint))
    • Stage: DeployStaging:
      • Action: Deploy (Cloudformation, DeployResourcesStaging: modeldeploy_pipeline project => endpoint-config-template.yml)
      • Action: Build (Test Staging: modeldeploy_testbuild.tf => modeldeploy_pipeline project: test/test_buildspec.yml)
      • Action: Approval (Manual Approval by User)
    • Stage: DeployProd:
      • Action: Deploy (Cloudformation, DeployResourcesProd: modeldeploy_pipeline project => endpoint-config-template.yml)

• Notebooks (for testing) (region: us-east-1)

  • End2end.ipynb
    • https://github.com/omerbsezer/Fast-Terraform/blob/main/samples/mlops-sagemaker-github-codepipeline-codebuild-codedeploy/Notebooks/SageMaker_Customer_Churn_XGB_end2end.ipynb
  • Pipeline.ipynb (Sagemaker Data Pipeline)
    • https://github.com/omerbsezer/Fast-Terraform/blob/main/samples/mlops-sagemaker-github-codepipeline-codebuild-codedeploy/Notebooks/SageMaker_Customer_Churn_XGB_Pipeline.ipynb

  

Prerequisite

• You should have a look following lab:
  • LAB-00: Terraform Install, AWS Configuration with Terraform

Steps

• Before running Terraform, upload "Modelbuild_Pipeline" and "Modeldeploy_Pipeline" in your GitHub account.
• Run:

cd terraform
terraform init
terraform validate
terraform plan
terraform apply

• After run:

  

• AWS CodePipeline:

  

• ModelBuild:

  

• ModelBuild Log:

  

• AWS S3:

  

• ModelBuild was done successfully:

  

• CloudWatch to see the training accuracy:

  

• CloudWatch, Log Groups, Train Error:

  

• ModelDeploy:

  

  

• CloudFormation, stack: deploy-staging

  

• SageMaker Dashboard, staging endpoint in service:

  

• SageMaker, Model:

  

• S3, Model, possible to download:

  

• Try Staging Endpoint with notebook (end2end.ipynb, last cell, enter the endpointname):

import pandas as pd
import numpy as np
import sagemaker
import boto3
from sagemaker import get_execution_role

test_data=pd.read_csv('test.csv',header=None)
testdata1=test_data.iloc[0:1,1:]

runtime = boto3.client("sagemaker-runtime")
Endpoint_name='aws-ml-11052023-staging-0306' #<your endpoint name> # update to your own endpoint name

prediction = runtime.invoke_endpoint(
    EndpointName=Endpoint_name,
    Body=testdata1.to_csv(header=False, index=False).encode("utf-8"),
    ContentType="text/csv",
    Accept= "text/csv",
)

print(prediction["Body"].read())

• Endpoint returned the result:

  

• Approve the Product on the CodePipeline:

  

• SageMaker Dashboard, 2 Endpoints are in-service:

  

• SageMaker, Prod Endpoint:

  

• CloudFormation:

  

• Test Prod Endpoint, returns results:

  

• Delete Endpoints manually, if the endpoints are in-service, you have to pay their cost:

  

• Delete stacks manually in Cloudformation.

• Download artifacts on S3:

aws s3 sync s3://artifact-ml-11052023 C:\Users\oesezer\Desktop\aws-artifacts

• Downloaded to the local PC:

  

• Destroy with "terraform destroy":

  

• Check whether all created artifacts are deleted on CodePipeline, S3, CloudFormation, SageMaker, and CloudWatch (LogGroups) or not. If still some of the artifacts are in the AWS, please delete them all.

References

• https://github.com/aws-samples/aws-mlops-pipelines-terraform