Skip to content

Latest commit

 

History

History
 
 

05-monitoring

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
config
config
 
 
dashboards
dashboards
 
 
data
data
 
 
images
images
 
 
models
models
 
 
README.md
README.md
 
 
baseline_model_nyc_taxi_data.ipynb
baseline_model_nyc_taxi_data.ipynb
 
 
debugging_nyc_taxi_data.ipynb
debugging_nyc_taxi_data.ipynb
 
 
docker-compose.yml
docker-compose.yml
 
 
dummy_metrics_calculation.py
dummy_metrics_calculation.py
 
 
evidently_metrics_calculation.py
evidently_metrics_calculation.py
 
 
meta.json
meta.json
 
 
requirements.txt
requirements.txt
 
 

README.md

5. Model Monitoring

5.1 Intro to ML monitoring

5.2 Environment setup

5.3 Prepare reference and model

5.4 Evidently metrics calculation

5.5 Dummy monitoring

5.6 Data quality monitoring

5.7 Save Grafana Dashboard

5.8 Debugging with test suites and reports

5.8 Homework

More information here

Notes

Did you take notes? Add them here:

Monitoring example

Prerequisites

You need following tools installed:

  • docker
  • docker-compose (included to Docker Desktop for Mac and Docker Desktop for Windows )

Preparation

Note: all actions expected to be executed in repo folder.

  • Create virtual environment and activate it (eg. python -m venv venv && source ./venv/bin/activate or conda create -n venv python=3.11 && conda activate venv)
  • Install required packages pip install -r requirements.txt
  • Run baseline_model_nyc_taxi_data.ipynb for downloading datasets, training model and creating reference dataset

Monitoring Example

Starting services

To start all required services, execute:

docker-compose up

It will start following services:

  • db - PostgreSQL, for storing metrics data
  • adminer - database management tool
  • grafana - Visual dashboarding tool

Sending data

To calculate evidently metrics with prefect and send them to database, execute:

python evidently_metrics_calculation.py

This script will simulate batch monitoring. Every 10 seconds it will collect data for a daily batch, calculate metrics and insert them into database. This metrics will be available in Grafana in preconfigured dashboard.

Accsess dashboard

  • In your browser go to a localhost:3000 The default username and password are admin

  • Then navigate to General/Home menu and click on Home.

  • In the folder General you will see New Dashboard. Click on it to access preconfigured dashboard.

Ad-hoc debugging

Run debugging_nyc_taxi_data.ipynb to see how you can perform a debugging with help of Evidently TestSuites and Reports

Stopping services

To stop all services, execute:

docker-compose down