Train, deploy and monitor a XGBoost regression model with the Amazon SageMaker and alert using AWS Lambda and Amazon SNS
This repository contains a sample to train, deploy and monitor a XGBoost regression model in Amazon SageMaker and alert using AWS Lambda and Amazon SNS. SageMaker's Model Monitor will be used to monitor data quality drift using the Data Quality Monitor and regression metrics like MAE, MSE, RMSE and R2 using the Model Quality Monitor.
Amazon SageMaker is a fully managed machine learning service. With SageMaker, you have the option of using the built-in algorithms or you can bring your own algorithms and frameworks. One of the features offered by Amazon SageMaker is Model Monitor which continuously monitors the quality of Amazon SageMaker machine learning models in production. With Model Monitor, you can set alerts that notify you when there are deviations from the specified baseline for the specified monitoring type.
With the Data Quality Monitor, you can detect data quality drift by tracking the difference between data that was used to train the models versus the data that is being presented to the model to score.
With the Model Quality Monitor, you can monitor model characteristics (such as MAE, MSE, RMSE, R2, precision, accuracy, recall and more) of your ML models in real time. SageMaker Model monitor reports how well a ML model is predicting outcomes by comparing model prediction to ground truth data.
This example contains a Jupyter Notebook that demonstrates how to use the SageMaker's built-in XGBoost algorithm to train a regression model on the California Housing dataset, deploy it on a SageMaker inference endpoint and monitor using SageMaker Model Monitor. Of the various monitoring types supported by the Model Monitor, we will demonstrate Data Quality Monitor and Model Quality Monitor in this notebook. When you complete running through all the steps in this notebook, you will notice both Data Quality and Model Quality violations assuming you have not changed the data or the processing logic. Finally, this notebook will provide an overview of the process involved in setting up alerting using AWS Lambda and Amazon SNS to send out e-mail or text based alerts. A code sample for the Lambda function is also included.
This repository contains
-
A Jupyter Notebook to get started.
-
An AWS Lambda function in Python3 that contains the sample code for processing the violations files generated by the SageMaker Model Monitor.
See CONTRIBUTING for more information.
This library is licensed under the MIT-0 License. See the LICENSE file.