Repository for the MLops Meetup on the 24th November 2021 Model Monitoring: The Million Dollar Problem by Loka team.
In order, demos shown in the video are the following:
Explains how you can train an ML model, generate a baseline (for comparison) and creating a quality monitoring job using SageMaker. This Jupyter notebook is intended to be executed in a SageMaker studio notebook instance.
Code serves an ML model locally using flask and how you can add observability to your model predictions using whylogs Python module. Video further explains how you can generate comparisons between data profiles using whylabs paid platform.
A data profile is a set of summary statistics, so what is actually sent to platform are statistics of your data rather than your data itself.
Go to whylogs notebook to check the demo.
Note: this demo can't be replicated without having a whylabs API key and account.
This code demonstrates how to use evidently.ai open source package to generate different reports to detect changes between two distributions (testing and training) used to evaluate data drifting, target drifting among others.
Go to evidently notebook to check the demo.
