Note
The following example uses timestamps<Timestamp>. These are optional but have an impact on the way data is chunked and results are plotted. You can read more about them in the data requirements<data_requirements_columns_timestamp>.
Advanced configuration
- Set up custom chunking [
tutorial <chunking>] [API reference] - Set up custom thresholds [
tutorial <thresholds>] [API reference]
For simplicity the guide is based on a synthetic dataset where the monitored model predicts which type of credit card product new customers should be assigned to. You can learn more about this dataset<dataset-synthetic-multiclass>.
In order to monitor a model, NannyML needs to learn about it from a reference dataset. Then it can monitor the data that is subject to actual analysis, provided as the analysis dataset. You can read more about this in our section on data periods<data-drift-periods>
The analysis_targets dataframe contains the target results of the analysis period. This is kept separate in the synthetic data because it is not used during performance estimation.<performance-estimation>. But it is required to calculate performance, so the first thing we need to in this case is set up the right data in the right dataframes. The analysis target values are joined on the analysis frame by the identifier column.
Next a ~nannyml.performance_calculation.calculator.PerformanceCalculator is created using a list of metrics to calculate (or just one metric), the data columns required for these metrics, and an optional chunking<chunking> specification.
The list of metrics specifies which performance metrics of the monitored model will be calculated. The following metrics are currently supported:
roc_auc- one-vs-the-rest, macro-averagedf1- macro-averagedprecision- macro-averagedrecall- macro-averagedspecificity- macro-averagedaccuracy
For more information on metrics, check the ~nannyml.performance_calculation.metrics module.
The new ~nannyml.performance_calculation.calculator.PerformanceCalculator is fitted using the ~nannyml.performance_calculation.calculator.PerformanceCalculator.fit method on the reference data.
The fitted ~nannyml.performance_calculation.calculator.PerformanceCalculator can then be used to calculate realized performance metrics on all data which has target values available with the ~nannyml.performance_calculation.calculator.PerformanceCalculator.calculate method. NannyML can output a dataframe that contains all the results of the analysis data.
There results from the reference data are also available.
Apart from chunking and chunk and period-related columns, the results data have the a set of columns for each calculated metric.
targets_missing_rate- The fraction of missing target data.value- the realized metric value for a specific chunk.sampling_error- the estimate of theSampling Error.upper_thresholdandlower_threshold- crossing these thresholds will raise an alert on significant performance change. The thresholds are calculated based on the actual performance of the monitored model on chunks in thereferencepartition. The thresholds are 3 standard deviations away from the mean performance calculated on chunks. They are calculated duringfitphase.alert- flag indicating potentially significant performance change.Trueif estimated performance crosses upper or lower threshold.
The results can be plotted for visual inspection:
After reviewing the performance calculation results, we should be able to clearly see how the model is performing against the targets, according to whatever metrics we wish to track.
If we decide further investigation is needed, the Data Drift<data-drift> functionality can help us to see what feature changes may be contributing to any performance changes.
It is also wise to check whether the model's performance is satisfactory according to business requirements. This is an ad-hoc investigation that is not covered by NannyML.