diff --git a/md-docs/imgs/monitoring/drift-explainability/concept-fi.svg b/md-docs/imgs/monitoring/drift-explainability/concept-fi.svg new file mode 100644 index 0000000..4e46236 --- /dev/null +++ b/md-docs/imgs/monitoring/drift-explainability/concept-fi.svg @@ -0,0 +1 @@ +
0.300.300.250.250.200.200.150.150.100.100.050.050.000.00Discriminative PowerX_0X_0X_1X_1X_2X_2X_3X_3X_4X_4X_5X_5X_6X_6X_7X_7X_8X_8X_9X_9X_10X_10yyFeature \ No newline at end of file diff --git a/md-docs/imgs/monitoring/drift-explainability/fi.svg b/md-docs/imgs/monitoring/drift-explainability/fi.svg new file mode 100644 index 0000000..777e0e9 --- /dev/null +++ b/md-docs/imgs/monitoring/drift-explainability/fi.svg @@ -0,0 +1,53 @@ +
Reference
Production
0.500.500.400.400.300.300.200.200.100.100.000.00Feature ImportanceX_0X_0X_1X_1X_2X_2X_3X_3X_4X_4X_5X_5X_6X_6X_7X_7X_8X_8X_9X_9X_10X_10Feature \ No newline at end of file diff --git a/md-docs/imgs/monitoring/drift-explainability/score.svg b/md-docs/imgs/monitoring/drift-explainability/score.svg new file mode 100644 index 0000000..ed98365 --- /dev/null +++ b/md-docs/imgs/monitoring/drift-explainability/score.svg @@ -0,0 +1,53 @@ +
Test statistic
Threshold
17.0017.0016.0016.0015.0015.0014.0014.0013.0013.0012.0012.0011.0011.0010.0010.009.009.008.008.007.007.006.006.005.005.004.004.003.003.002.002.001.001.000.000.00-1.00-1.00-2.00-2.00Drift Score07 Jul07 Jul08 Jul08 Jul09 Jul09 Jul10 Jul10 Jul11 Jul11 Jul12 Jul12 Jul13 Jul13 Jul14 Jul14 Jul15 Jul15 Jul16 Jul16 Jul17 Jul17 Jul18 Jul18 Jul19 Jul19 Jul20 Jul20 Jul21 Jul21 Jul22 Jul22 Jul23 Jul23 Jul24 Jul24 Jul25 Jul25 Jul26 Jul26 Jul27 Jul27 JulTimeWarning OnDrift On - LowDrift On - MediumDrift On - High \ No newline at end of file diff --git a/md-docs/imgs/monitoring/overview.svg b/md-docs/imgs/monitoring/overview.svg new file mode 100644 index 0000000..668eb81 --- /dev/null +++ b/md-docs/imgs/monitoring/overview.svg @@ -0,0 +1 @@ +InputPredictionTrue valueInputConceptErrorMetricsMetricsPredictionMetrics \ No newline at end of file diff --git a/md-docs/imgs/monitoring/states.svg b/md-docs/imgs/monitoring/states.svg new file mode 100644 index 0000000..764f8c6 --- /dev/null +++ b/md-docs/imgs/monitoring/states.svg @@ -0,0 +1 @@ +.InitialstateOKWARNINGDRIFTDrift ONWarningONDrift OFFWarning OFFSet newreferenceDrift ONSetnewreferenceDrift OFF \ No newline at end of file diff --git a/md-docs/stylesheets/extra.css b/md-docs/stylesheets/extra.css index a78dfad..49b3f25 100644 --- a/md-docs/stylesheets/extra.css +++ b/md-docs/stylesheets/extra.css @@ -30,4 +30,12 @@ background-color: rgb(43, 155, 70); -webkit-mask-image: var(--md-admonition-icon--code-block); mask-image: var(--md-admonition-icon--code-block); -} \ No newline at end of file +} + +.nice-list ul{ + list-style-type: circle; +} + +.mermaid { + text-align: center; + } \ No newline at end of file diff --git a/md-docs/user_guide/data.md b/md-docs/user_guide/data.md index a113c94..10f85cf 100644 --- a/md-docs/user_guide/data.md +++ b/md-docs/user_guide/data.md @@ -19,7 +19,7 @@ Available categories are: The [Data Schema] created for the [Task] contains a list of Column objects, each of which has a _Role_. Naturally, there is a relationship between the Column's Role and the Data Category. In fact, each Data Category comprises a set of Column objects with certain Roles. -So that, when you upload samples belonging to a Data Category, they must contains all the Columns objects declared on the Data Schema to be considered valid. +When you upload samples belonging to a Data Category, they must contain all the Columns objects declared on the Data Schema to be considered valid. The following table shows these relationships: @@ -130,7 +130,7 @@ For RAG Tasks, reference data can be used to indicate the type of data expected You can set reference data as follow: ``` py - job_id = job_id = client.set_model_reference( + job_id = client.set_model_reference( model_id=model_id, from_timestamp=from_timestamp, to_timestamp=to_timestamp, diff --git a/md-docs/user_guide/detection_event_rules.md b/md-docs/user_guide/detection_event_rules.md deleted file mode 100644 index 93a2de8..0000000 --- a/md-docs/user_guide/detection_event_rules.md +++ /dev/null @@ -1,52 +0,0 @@ -# Detection Event Rules - -This section provides an overview of how you can setup automation rules after a detection event occurs in order to receive notifications or to start retraining. - -When a detection event occurs, the platform evaluates your set detection event rules. -If a rule matches the event, the specified actions will be triggered. -These rules are specific to a task and require the following parameters for configuration: - -- `name`: A descriptive label for your rule, helping you understand its purpose quickly. -- `task_id`: The unique identifier of the task to which the rule is applicable. -- `severity`: Indicates the severity level of the event - it can be `HIGH`, `MEDIUM`, or `LOW`. -- `detection_event_type`: Currently, only `DRIFT` events are available for detection. -- `monitoring_target`: Specifies what is being monitored, which can be `MODEL`, `INPUT`, or `CONCEPT`. If the value is `MODEL`, you need to provide a corresponding `model_name`. -- `actions`: A sequential list of actions to be executed when the rule is triggered. - -## Supported Actions -Two types of actions are currently supported: notification and retrain. - - -### Notifications -- `SlackNotificationAction`: sends a notification to a Slack channel via webhook. -- `DiscordNotificationAction`: sends a notification to a Discord channel via webhook. -- `EmailNotificationAction`: sends an email to the provided email address. -- `TeamsNotificationAction`: sends a notification to Microsoft Teams via webhook. -- `MqttNotificationAction`: sends a notification to an MQTT broker. - -### Retrain Actions - -Retrain actions let you retrain your model, therefore, they are only available when the rule monitoring target is a model. -The retrain action does not need any parameter because it is automatically inferred from the `model_name` attribute of the rule. -Of course, it is mandatory that the model has a retrain trigger associated in order to add a retrain action to the rule. - -!!! example - The following code snippet demonstrates how to create a rule that matches high severity drift events for a specific model. When triggered, it sends a notification to the `ml3-platform-notifications` channel on your Slack workspace using the provided webhook URL and then start the retraining of the model. - - ```py - rule_id = client.create_detection_event_rule( - name='Retrain model with notification', - task_id='my-task-id, - model_name='my-model', - severity=DetectionEventSeverity.HIGH, - detection_event_type=DetectionEventType.DRIFT, - monitoring_target=MonitoringTarget.MODEL, - actions=[ - SlackNotificationAction( - webhook='https://hooks.slack.com/services/...', - channel='ml3-platform-notifications' - ), - RetrainAction() - ], - ) - ``` \ No newline at end of file diff --git a/md-docs/user_guide/index.md b/md-docs/user_guide/index.md index 1f55726..0c3d7a5 100644 --- a/md-docs/user_guide/index.md +++ b/md-docs/user_guide/index.md @@ -53,7 +53,7 @@ A **Task** is specified by several attributes, the most important are: - `type`: regression, classification, object detection ... - `data structure`: tabular data, image data, ... -- `optional target`: if the target is not always available. This happen when input samples are labeled and the most part of production data do not have a label +- `optional target`: if the target is not always available. This happens when input samples are labeled and the most part of production data do not have a label - `data schema`: specifies the inputs and the target of the task, see [Data Schema](data_schema.md) section for more details - `cost info`: information about the economic costs of the error on the target @@ -110,7 +110,7 @@ Now that you have clear the basic concepts we invite you to explore the other ML Discover how to setup automation rules to increase your reactivity. - [:octicons-arrow-right-24: More info](detection_event_rules.md) + [:octicons-arrow-right-24: More info](monitoring/detection_event_rules.md) - :material-lock:{ .lg .middle } **Roles and access** diff --git a/md-docs/user_guide/model.md b/md-docs/user_guide/model.md index ace4641..ba492e2 100644 --- a/md-docs/user_guide/model.md +++ b/md-docs/user_guide/model.md @@ -1 +1,20 @@ -# Model \ No newline at end of file +# Model + + + + +[//]: # () +[//]: # () +[//]: # (What is additional probabilistic output?) + +[//]: # () +[//]: # (What is metric?) + +[//]: # () +[//]: # (What is suggestion type?) + +[//]: # () +[//]: # (What is retraining cost?) + +[//]: # () +[//]: # (What is retraining trigger?) \ No newline at end of file diff --git a/md-docs/user_guide/modules/index.md b/md-docs/user_guide/modules/index.md index 879ac58..375976c 100644 --- a/md-docs/user_guide/modules/index.md +++ b/md-docs/user_guide/modules/index.md @@ -13,7 +13,7 @@ Modules can be always active or on-demand: Monitoring module and Drift Explainab Data drift detection over data. - [:octicons-arrow-right-24: More info](user_guide/company.md) + [:octicons-arrow-right-24: More info](../monitoring/index.md) - :material-compare:{ .lg .middle } **Drift Explainability** @@ -21,7 +21,7 @@ Modules can be always active or on-demand: Monitoring module and Drift Explainab Understand the nature of detected drift. - [:octicons-arrow-right-24: More info](user_guide/modules/index.md) + [:octicons-arrow-right-24: More info](../monitoring/drift_explainability.md) - :material-speedometer:{ .lg .middle } **Retraining** diff --git a/md-docs/user_guide/monitoring/detection_event.md b/md-docs/user_guide/monitoring/detection_event.md new file mode 100644 index 0000000..27d2ff6 --- /dev/null +++ b/md-docs/user_guide/monitoring/detection_event.md @@ -0,0 +1,49 @@ +# Detection Event + +A Detection Event is raised by the ML cube Platform when a significant change is detected in one of the entities being monitored. + +An event is characterized by the following attributes: + +- `Event Type`: the type of the event. It's possible values are: +
+ +
+- `Severity`: the severity of the event. It's provided only for drift events and it can be `Low`, `Medium`, or `High`. +- `Monitoring Target`: the [Monitoring Target](index.md#monitoring-metrics) being monitored. +- `Monitoring Metric`: the [Monitoring Metric](index.md#monitoring-metrics) being monitored. +- `Model Name`: the name of the model that raised the event. It's present only if the event is related to a model. +- `Model Version`: the version of the model that raised the event. It's present only if the event is related to a model. +- `Insert datetime`: the time when the event was raised. +- `Sample timestamp`: the timestamp of the sample that triggered the event. +- `Sample customer ID`: the id of the customer that triggered the event. +- `User feedback`: the feedback provided by the user on whether the event was expected or not. + +## Retrieve Detection Events + +You can access the detection events generated by the Platform in two ways: + +- **SDK**: it can be used to retrieve all detection events for a specific task programmatically. +- **WebApp**: navigate to the **`Detection `** section located in the task page's sidebar. Here, all detection events are displayed in a table, + with multiple filtering options available for useful event management. Additionally, the latest detection events identified are shown in the Task homepage, + in the section named "Latest Detection Events". + +## User Feedback + +When a detection event is raised, you can provide feedback on whether the event was expected or not. This feedback is then used +to tune the monitoring algorithms and improve their performance. The feedback can be provided through the WebApp, in the +**`Detection `** section of the task page, or through the SDK. + + +## Detection Event Rules + +To automate actions upon the reception of a detection event, you can set up detection event rules. +You can learn more about how to configure them in the [Detection Event Rules] section. + + +[Monitoring]: index.md +[Detection Event Rules]: detection_event_rules.md \ No newline at end of file diff --git a/md-docs/user_guide/monitoring/detection_event_rules.md b/md-docs/user_guide/monitoring/detection_event_rules.md new file mode 100644 index 0000000..69c4036 --- /dev/null +++ b/md-docs/user_guide/monitoring/detection_event_rules.md @@ -0,0 +1,66 @@ +# Detection Event Rules + +This section outlines how to configure automation to receive notifications or start retraining after a [Detection Event] occurs. + +When a detection event is produced, the ML cube Platform reviews all the detection event rules you have set +and triggers those matching the event. + +Rules are specific to a task and are characterized by the following attributes: + +- `Name`: a descriptive label of the rule. +- `Detection Event Type`: the type of event that triggers the rule. +- `Severity`: the severity of the event that triggers the rule. It is only applicable to drift events. If not specified, the rule will be triggered by drift events of any severity. +- `Monitoring Target`: the [Monitoring Target](index.md#monitoring-targets) whose event should trigger the rule. +- `Monitoring Metric`: the [Monitoring Metric](index.md#monitoring-metrics) whose event should trigger the rule. +- `Model name`: the name of the model to which the rule applies. This is only required when the monitoring target is related to a model + (such as `ERROR` or `PREDICTION`). +- `Actions`: A list of actions to be executed sequentially when the rule is triggered. + +## Detection Event Actions +Three types of actions are currently supported: notification, plot configuration and retrain. + +### Notifications + +These actions send notifications to external services when a detection event is triggered. The following notification actions are available: + +- `Slack Notification`: sends a notification to a Slack channel via webhook. +- `Discord Notification`: sends a notification to a Discord channel via webhook. +- `Email Notification`: sends an email to the provided email address. +- `Teams Notification`: sends a notification to Microsoft Teams via webhook. +- `Mqtt Notification`: sends a notification to an MQTT broker. + +### Plot Configuration + +This action consists in creating two plot configurations when a detection event is triggered: the first one includes +data preceding the event, while the second one includes data following the event. + +### Retrain + +Retrain Action enables the automatic retraining of your model. Therefore, it is only available when the target of the rule is related to a model. +The retrain action does not need any parameter because it is automatically inferred from the `Model Name` attribute of the rule. +Of course, the model must already have a retrain trigger associated before setting up this action. + +!!! example + The following code snippet demonstrates how to create a rule that matches high severity drift events on the error of a model. + When triggered, it first sends a notification to the `ml3-platform-notifications` channel on your Slack workspace, using the + provided webhook URL, and then starts the retraining of the model. + + ```py + rule_id = client.create_detection_event_rule( + name='Retrain model with notification', + task_id='my-task-id', + model_name='my-model', + severity=DetectionEventSeverity.HIGH, + detection_event_type=DetectionEventType.DRIFT_ON, + monitoring_target=MonitoringTarget.ERROR, + actions=[ + SlackNotificationAction( + webhook='https://hooks.slack.com/services/...', + channel='ml3-platform-notifications' + ), + RetrainAction() + ], + ) + ``` + +[Detection Event]: detection_event.md \ No newline at end of file diff --git a/md-docs/user_guide/monitoring/drift_explainability.md b/md-docs/user_guide/monitoring/drift_explainability.md new file mode 100644 index 0000000..dd0575e --- /dev/null +++ b/md-docs/user_guide/monitoring/drift_explainability.md @@ -0,0 +1,64 @@ +# Drift Explainability + +[Monitoring] is a crucial aspect of the machine learning lifecycle, as it enables tracking the model's performance and its data over time, +ensuring the model continues to function as expected. However, monitoring only is not enough when it comes to the adaptation phase. + +In order to make the right decisions, you need to understand what were the main factors that led to the drift in the first place, so that +the correct actions can be taken to mitigate it. + +The ML cube Platform supports this process by offering what we refer to as **Drift Explainability Reports**, +automatically generated upon the detection of a drift and containing several elements that should help you diagnose the root causes +of the change occurred. + +You can access the reports in the WebApp, by navigating to the `Drift Explainability` tab in the sidebar of the Task page. + +## Structure + +A Drift Explainability Report consists in comparing the reference data and the portion of production data where the drift was identified, hence +those belonging to the new data distribution. Notice that these reports are generated after a sufficient amount of samples has been collected +after the drift, in order to ensure statistical reliability of the results. +If the data distribution moves back to the reference before enough samples are collected, the report might not be generated. + +Each report is composed of several entities, each providing a different perspective on the data and the drift occurred. +Most of them are specific to a certain `Data Structure`, so they might not be available for all tasks. + +These entities can take the form of tables, plots, or textual explanations. +Observed and analyzed together, they should provide a comprehensive understanding of the drift and its underlying causes. +These are the entities currently available: + +- `Feature Importance`: it's a barplot that illustrates how the significance of each feature differs between the reference + and the production datasets. Variations in a feature's values might suggest that its contribution to the model's predictions + has changed over time. This entity is available only for tasks with tabular data. + +
+ ![Feature Importance](../../imgs/monitoring/drift-explainability/fi.svg) +
Example of a feature importance plot.
+
+ +- `Variable discriminative power`: it's also a bar plot displays the influence of each feature, as well as the target, + in differentiating between the reference and the production datasets. + The values represent how strongly a given feature helps to distinguish the datasets, with higher values representing stronger + separating power. This entity is available only for tasks with tabular data. + +
+ ![Variable discriminative power](../../imgs/monitoring/drift-explainability/concept-fi.svg) +
Example of a variable discriminative power plot.
+
+ +- `Drift Score`: it's a line plot that shows the evolution of the drift score over time. The drift score is a + measure of the statistical distance between a sliding window of the production data and the reference data. It also shows the threshold, + which is the value that the drift score must exceed to raise a drift alarm, and all the [Detection Events] that were triggered in + the time frame of the report. This plot helps in understanding how the drift evolved over time and the moments in which the difference + between the two datasets was higher. Notice that some postprocessing is applied on the events to account for the functioning of the drift detection algorithms. + Specifically, + we shift back the drift on events by a certain offset, aiming to point at the precise time when the drift actually started. As a result, + drift on events might be shown before the threshold is exceeded. This explainability entity is available for all tasks. + + +
+ ![Drift score](../../imgs/monitoring/drift-explainability/score.svg) +
Example of a drift score plot with detection events of increasing severity displayed.
+
+ +[Monitoring]: index.md +[Detection Events]: detection_event.md \ No newline at end of file diff --git a/md-docs/user_guide/monitoring/index.md b/md-docs/user_guide/monitoring/index.md new file mode 100644 index 0000000..1cfd14d --- /dev/null +++ b/md-docs/user_guide/monitoring/index.md @@ -0,0 +1,159 @@ +# Monitoring + +The monitoring module is a key feature of the ML cube Platform. +It enables continuous tracking of AI models performance over time, helping to identify potential issues. +It also implements the monitoring of production data to preemptively detect distribution changes, ensuring +that the model continues to perform as expected and aligns with business requirements. + +## Why do you need Monitoring? + +Machine Learning algorithms are based on the assumption that the distribution of the data used for training is the same as the one from which +production data are drawn from. This assumption never holds in practice, as the real world is characterized by dynamic and ever-changing conditions. +These distributional changes, if not addressed properly, can cause a drop in the model's performance, leading to bad estimations or predictions, which +in turn can have a negative impact on the business. + +Monitoring, also known as __Drift Detection__ in the literature, refers the process of continuously tracking the performance of a model +and the distribution of the data it is operating on to identify significant changes. + +## How does the ML cube Platform perform Monitoring? + +The ML cube platform performs monitoring by employing statistical techniques to compare a certain reference (for instance, data used for training or the +performance of a model on the test set) to incoming production data. + +These statistical techniques, also known as _monitoring algorithms_, are tailored to the type of data +being observed; for instance, univariate data requires different monitoring techniques than multivariate data. +However, you don't need to worry about +the specifics of these algorithms, as the ML cube Platform takes care of selecting the most appropriate ones for your task. + +If a significant difference between reference and production data is detected, an alarm is raised, signaling that the monitored entity +is drifting away from the expected behavior and that corrective actions should be taken. + +In practical terms, you can use the SDK to specify the time period where the reference of a given model should be placed. +As a consequence, all algorithms associated with the specified model (not just those monitoring the performance, but also those operating +on the data used by the model) will +be initialized on the specified reference. Of course, you should provide to the +Platform the data you want to use as a reference before setting the reference itself. This can be done through the SDK as well. + +After setting the reference, you can send production data to the platform, still using the SDK. This data will be analyzed by the monitoring algorithms +and, if a significant difference is detected, an alarm will be raised, in the form of a [Detection Event]. +You can explore more about detection events and how you can set up automatic actions upon their reception in the [Detection Event] +and the [Detection Event Rule] sections respectively. + +### Targets and Metrics + +After explaining why monitoring is so important in modern AI systems and detailing how it is performed in the ML cube Platform, +we can introduce the concepts of Monitoring Targets and Monitoring Metrics. They both represent quantities that the ML cube Platform monitors, +but they differ in their nature. + +Targets and Metrics are defined by the ML cube platform based on the [Task] attributes, such as the Task type and the data structure, and their monitoring +is automatically enabled upon the task creation. The idea underlying defining many entities to monitor, rather than monitoring +only the model error, is to provide a comprehensive view of the model's +performance and the data distribution, easing the identification of the root causes of a drift and thus facilitating the corrective actions. + +
+ ![Monitoring Overview](../../imgs/monitoring/overview.svg) +
Monitoring Targets and Metrics overview
+
+ +#### Monitoring Targets + +A Monitoring Target is a relevant entity involved in a [Task]. They represent the main quantities monitored by the platform, those whose +variation can have a significant impact on the AI task success. + +The ML cube platform supports the following monitoring targets: + +- `INPUT`: the input distribution, $P(X)$. +- `CONCEPT`: the joint distribution of input and target, $P(X, Y)$. +- `PREDICTION`: the prediction of the model, $P(\hat{Y})$. +- `INPUT_PREDICTION`: the joint distribution of input and prediction, $P(X, \hat{Y})$. +- `ERROR`: the error of the model, whose computation depends on the task type. +- `USER_INPUT`: the input provided by the user, usually in the form of a query. This target is only available in tasks of type RAG. +- `USER_INPUT_RETRIEVED_CONTEXT`: the similarity between the user input and the context retrieved by the RAG system. This target is only available in tasks of type RAG. +- `USER_INPUT_MODEL_OUTPUT`: the similarity between the user input and the response of the Large Language Model. This target is only available in tasks of type RAG. +- `MODEL_OUTPUT_RETRIEVED_CONTEXT`: the similarity between the response of the Large Language Model and the context retrieved by the RAG system. This target is only available in tasks of type RAG. + +As mentioned, some targets are available only for specific task types. The following table shows all the available monitoring targets in relation with the task type. +Notice that while some targets were specifically designed for a certain task type, others are more general and can be used in different contexts. +Nonetheless, the platform might not support yet all possible combinations. The table will be updated as new targets are added to the product. + +| **Monitoring Target** | **REGRESSION** | **CLASSIFICATION_BINARY** | **CLASSIFICATION_MULTICLASS** | **CLASSIFICATION_MULTILABEL** | **OBJECT_DETECTION** | **RAG** | +|--------------------------------|:------------------:|:-------------------------:|:-----------------------------:|:-----------------------------:|:--------------------:|:------------------:| +| INPUT | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | +| CONCEPT | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | | +| PREDICTION | :white_check_mark: | :white_check_mark: | :white_check_mark: | | | | +| INPUT_PREDICTION | :white_check_mark: | :white_check_mark: | :white_check_mark: | | | | +| ERROR | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: | | | +| USER_INPUT | | | | | | :white_check_mark: | +| USER_INPUT_RETRIEVED_CONTEXT | | | | | | :white_check_mark: | +| USER_INPUT_MODEL_OUTPUT | | | | | | :white_check_mark: | +| MODEL_OUTPUT_RETRIEVED_CONTEXT | | | | | | :white_check_mark: | + +#### Monitoring Metrics + +A Monitoring Metric is a generic quantity that can be computed on a Monitoring Target. They enable the monitoring of specific +aspects of an entity, which might help in identifying the root cause of a drift, as well as defining the corrective actions to be taken. + +The following table displays the monitoring metrics supported, along with their monitoring target and the conditions +under which they are actually monitored. The possible values that each metric can assume are also provided. +Notice that also this table is subject to changes, as new metrics will be added. + +| **Monitoring Metric** | Description | **Monitoring Target** | **Conditions** | **Possible values** | +|:---------------------:|-----------------------------------------------------------------------|:------------------------------------------------:|:--------------------------------------:|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| TEXT_TOXICITY | The toxicity of the text | INPUT, USER_INPUT, PREDICTION | When the data structure is text | Either _neutral_ or _toxic_. | +| TEXT_EMOTION | The emotion of the text | INPUT, USER_INPUT | When the data structure is text | If the Task text language is Italian, one between these: _anger_, _joy_, _sadness_, _fear_.

Otherwise, _one_ between these: _admiration_, _amusement_, _anger_, _annoyance_, _approval_, _caring_, _confusion_, _curiosity_, _desire_, _disappointment_, _disapproval_, _disgust_, _embarrassment_, _excitement_, _fear_, _gratitude_, _grief_, _joy_, _love_, _nervousness_, _optimism_, _pride_, _realization_, _relief_, _remorse_, _sadness_, _surprise_, _neutral_. | +| TEXT_SENTIMENT | The sentiment of the text | INPUT, USER_INPUT | When the data structure is text | If the Task text language is Italian, one between these: _POSITIVE_, _NEGATIVE_. Otherwise, one between these: _negative_, _neutral_, _positive_ | +| TEXT_LENGTH | The length of the text | INPUT, USER_INPUT, RETRIEVED_CONTEXT, PREDICTION | When the data structure is text | An integer value. | +| MODEL_PERPLEXITY | A measure of the uncertainty of an LLM when predicting the next words | PREDICTION | When the task type is RAG | A floating point value. | +| IMAGE_BRIGHTNESS | The brightness of the image | INPUT | When the data structure is image | A floating point value. | +| IMAGE_CONTRAST | The contrast of the image | INPUT | When the data structure is image | A floating point value. | +| BBOXES_AREA | The average area of the predicted bounding boxes | PREDICTION | When the task type is Object Detection | A floating point value. | +| BBOXES_QUANTITY | The average number of predicted bounding boxes per image | PREDICTION | When the task type is Object Detection | An integer value. | + + +### Monitoring Status + +All the entities being monitored are associated with a status, which can be one of the following: + +- `OK`: the entity is behaving as expected. +- `WARNING`: the entity has shown signs of drifts, but it is still within the acceptable range. +- `DRIFT`: the entity has experienced a significant change and corrective actions should be taken. + +The following diagram illustrates the possible transitions between the statuses. +Each transition is triggered by a [Detection Event] and the status of the entity is updated accordingly. + +```mermaid +stateDiagram-v2 + + [*] --> OK : Initial State + + OK --> WARNING : Warning On + WARNING --> OK : Set new reference + WARNING --> OK : Warning Off + + + WARNING --> DRIFT : Drift On + DRIFT --> WARNING : Drift Off + + DRIFT --> OK : Set new reference + DRIFT --> OK : Drift Off +``` + + +Notice that a Drift OFF event can either bring the entity back to the `OK` status or to the `WARNING` status, +depending on the velocity of the change and the monitoring algorithm's sensitivity. The same applies +to the Drift ON events, which can both happen when the entity is in the `WARNING` status or in the `OK` status. + +The only transition which is not due to a detection event is the one caused by the specification of a new reference. In this case, the status of the entity is reset to `OK` +for every entity as all the monitoring algorithms are reinitialized on the new reference. + +You can check the status of the monitored entities in two ways: + +- **WebApp**: The homepage of the task displays the status of both monitoring targets and metrics. +- **SDK**: there are a couple of methods to retrieve the status of the monitored entities programmatically. You can either get the status of a specific entity + or retrieve the status of all the entities associated with a task. + + +[Task]: ../task.md +[Detection Event Rule]: detection_event_rules.md +[Detection Event]: detection_event.md +[get_task]: ../../api/python/client.md#get_task \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 065d848..2d9cae5 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -129,10 +129,16 @@ nav: - user_guide/data_schema.md - user_guide/data.md - user_guide/rbac.md + + - Monitoring: + - user_guide/monitoring/index.md + - user_guide/monitoring/detection_event.md + - user_guide/monitoring/detection_event_rules.md + - user_guide/monitoring/drift_explainability.md + - Modules: - user_guide/modules/index.md - - user_guide/modules/monitoring.md - - user_guide/detection_event_rules.md +# - user_guide/modules/monitoring.md - user_guide/modules/retraining.md - user_guide/modules/topic_modeling.md - user_guide/modules/rag_evaluation.md diff --git a/requirements.txt b/requirements.txt index 263d14a..191ba2c 100644 --- a/requirements.txt +++ b/requirements.txt @@ -14,8 +14,6 @@ anyio==4.4.0 # via # httpx # jupyter-server -appnope==0.1.4 - # via ipykernel argon2-cffi==23.1.0 # via jupyter-server argon2-cffi-bindings==21.2.0 @@ -51,7 +49,12 @@ charset-normalizer==3.3.2 click==8.1.7 # via mkdocs colorama==0.4.6 - # via mkdocs-material + # via + # click + # ipython + # mkdocs + # mkdocs-material + # tqdm comm==0.2.2 # via # ipykernel @@ -337,8 +340,6 @@ pathspec==0.12.1 # via # mkdocs # mkdocs-macros-plugin -pexpect==4.9.0 - # via ipython pillow==10.3.0 # via # ml3-platform-docs (pyproject.toml) @@ -360,10 +361,6 @@ psutil==5.9.8 # via # accelerate # ipykernel -ptyprocess==0.7.0 - # via - # pexpect - # terminado pure-eval==0.2.2 # via stack-data pyarrow==16.1.0 @@ -399,6 +396,13 @@ python-json-logger==2.0.7 # via jupyter-events pytz==2024.1 # via pandas +pywin32==308 + # via jupyter-core +pywinpty==2.0.14 + # via + # jupyter-server + # jupyter-server-terminals + # terminado pyyaml==6.0.1 # via # accelerate