Update to the monitoring docs (#1116)

evidentlyai · May 15, 2024 · 195e72a · 195e72a
1 parent b846259
commit 195e72a
Show file tree

Hide file tree

Showing 23 changed files with 1,233 additions and 646 deletions.
diff --git a/docs/book/.gitbook/assets/cloud/alerts.png b/docs/book/.gitbook/assets/cloud/alerts.png
diff --git a/docs/book/.gitbook/assets/monitoring/distribution_panels.png b/docs/book/.gitbook/assets/monitoring/distribution_panels.png
diff --git a/docs/book/.gitbook/assets/monitoring/field_path.png b/docs/book/.gitbook/assets/monitoring/field_path.png
diff --git a/docs/book/.gitbook/assets/monitoring/metric_panels.png b/docs/book/.gitbook/assets/monitoring/metric_panels.png
diff --git a/docs/book/.gitbook/assets/monitoring/panel_dist_group-min.png b/docs/book/.gitbook/assets/monitoring/panel_dist_group-min.png
diff --git a/docs/book/.gitbook/assets/monitoring/panel_dist_group_2-min.png b/docs/book/.gitbook/assets/monitoring/panel_dist_group_2-min.png
diff --git a/docs/book/.gitbook/assets/monitoring/panel_dist_overlay-min.png b/docs/book/.gitbook/assets/monitoring/panel_dist_overlay-min.png
diff --git a/docs/book/.gitbook/assets/monitoring/panel_dist_relative-min.png b/docs/book/.gitbook/assets/monitoring/panel_dist_relative-min.png
diff --git a/docs/book/.gitbook/assets/monitoring/panel_dist_stacked-min.png b/docs/book/.gitbook/assets/monitoring/panel_dist_stacked-min.png
diff --git a/docs/book/.gitbook/assets/monitoring/panel_dist_stacked_2-min.png b/docs/book/.gitbook/assets/monitoring/panel_dist_stacked_2-min.png
diff --git a/docs/book/.gitbook/assets/monitoring/test_panels.png b/docs/book/.gitbook/assets/monitoring/test_panels.png
diff --git a/docs/book/SUMMARY.md b/docs/book/SUMMARY.md
@@ -43,11 +43,14 @@
   * [Run Evidently on Spark](tests-and-reports/spark.md)
 * [Monitoring](monitoring/README.md)
   * [Monitoring overview](monitoring/monitoring_overview.md)
-  * [Add workspace and project](monitoring/workspace_project.md)
+  * [Set up your Workspace](monitoring/workspace.md)
+  * [Create a Project](monitoring/add_project.md)
   * [Log snapshots](monitoring/snapshots.md)
-  * [Add dashboard panels](monitoring/design_dashboard.md)
-  * [Monitoring UI](monitoring/monitoring_ui.md)
-  * [Collector service (Alpha)](monitoring/collector_service.md) 
+  * [Get a Dashboard](monitoring/add_dashboard_tabs.md)
+  * [Panel types](monitoring/design_dashboard.md)
+  * [Add monitoring Panels](monitoring/design_dashboard_api.md)
+  * [Send alerts](monitoring/alerting.md)
+  * [Collector service](monitoring/collector_service.md) 
 * [Customization](customization/README.md)
   * [Data drift parameters](customization/options-for-statistical-tests.md)
   * [Embeddings drift parameters](customization/embeddings-drift-parameters.md)

diff --git a/docs/book/monitoring/add_dashboard_tabs.md b/docs/book/monitoring/add_dashboard_tabs.md
@@ -0,0 +1,50 @@
+---
+description: How to get a pre-built monitoring dashboard using templates.
+---   
+
+# What is a Dashboard? 
+
+Each Project has a monitoring Dashboard to visualize metrics and test results over time. A Dashboard can have multiple monitoring Panels, such as counters, line or bar plots, etc.
+
+![](../.gitbook/assets/main/evidently_ml_monitoring_main.png)
+
+
+{% hint style="info" %}
+**Data source**. To populate the Dashboard, you must send the relevant data inside the snapshots. The Panels will be empty otherwise. Read more about [sending snapshots](snapshots.md).  
+{% endhint %}
+
+You choose how exactly to organize your Dashboard and which values to plot. By default, the Dashboard for a new Project is empty. 
+
+For both Evidently Cloud and open-source, you can define the composition of monitoring Panels via API. This is great for version control.
+
+In Evidently Cloud, you can also:
+* Get pre-built Dashboards for Data Quality, Data Drift, etc.
+* Add and modify Panels directly in the user interface.
+* Add multiple Tabs on the Dashboard to logically group the Panels.
+
+# Pre-built dashboards
+{% hint style="success" %}
+Dashboard templates is a Pro feature available in the Evidently Cloud. 
+{% endhint %}
+
+Starting with template Dashboard Tabs is convenient: you get a set of monitoring Panels out of the box without adding them individually.
+
+To use a template:
+* Enter the “Edit” mode clicking on the top right corner of the Dashboard. 
+* Click on the “Add tab” button.
+* Choose a template Tab in the dropdown.
+
+Optionally, give a custom name to the Tab.
+
+You can choose between the following options:
+
+| Tab Template | Description | Data source |
+|---|---|---|
+| Columns | Shows column values (e.g., mean, quantiles) over time for categorical and numerical columns. | Capture the `DataQualityPreset()` or `ColumnSummaryMetric()` for individual columns. |
+| Data Quality | Shows data quality metrics (e.g., missing values, duplicates) over time for the complete dataset and results of Data Quality Tests. | For the Metric Panels, capture the `DataQualityPreset()` or `DatasetSummaryMetric()`. For the Test Panel, include any individual Tests from Data Quality or Data Integrity groups.|
+| Data Drift | Shows the share of drifting features over time, and the results of Column Drift Tests. | For the Metric Panel, capture the `DataDriftPreset()` or `DataDriftTestPreset()`. For the Test Panel, include individual `TestColumnDrift()` or `DataDriftTestPreset()`. |
+
+# What’s next?
+
+* See available individual [monitoring Panels types](design_dashboard.md).
+* How to add [custom monitoring Panels and Tabs to your dashboard](design_dashboard_api.md).
diff --git a/docs/book/monitoring/add_project.md b/docs/book/monitoring/add_project.md
@@ -0,0 +1,158 @@
+---
+description: How to create a Project for your monitoring use case.
+---   
+
+# What is a Project?
+
+A Project helps gather all Reports and Test Suites associated with the same use case. Each Project has a dedicated monitoring dashboard and snapshot storage.
+
+{% hint style="info" %}
+**Should you have one Project for one ML model?** You will often create one project per ML model or dataset, but this is not a strict rule. For example, you can log the performance of a champion and challenger models to the same Project. Or, store data on related models (such as demand forecasting models by country) in one Project and use tags to organize them. You can also set up your monitoring for any data pipeline or dataset.
+{% endhint %}
+
+Once you create a Project, you can connect to it from your Python environment to send the data or edit the dashboards. In Evidently Cloud, you can work both via API and a graphic user interface.
+
+# Create a Project
+
+## Add a new Project
+
+{% tabs %}
+
+{% tab title="API" %} 
+
+To create a Project inside a workspace `ws`, assign a name and description, and save the changes:
+
+```
+project = ws.create_project("My project name")
+project.description = "My project description"
+project.save()
+```
+
+{% endtab %}
+
+{% tab title="UI" %} 
+
+Click on the “plus” sign on the home page and type your Project name and description.
+
+![](../.gitbook/assets/cloud/add_project_wide-min.png)
+
+After creating a Project, you can click to open a Dashboard. Since there's no data yet, it will be empty. 
+
+{% endtab %}
+
+{% endtabs %}
+
+**Project ID**. Once you run `create_project`, you will see the Project ID. You can later use it to reference the Project. You can also copy the Project ID directly from the UI: it appears above the monitoring dashboard.
+
+## Add a Team Project
+
+{% hint style="success" %}
+Team management is a Pro feature available in the Evidently Cloud.
+{% endhint %}
+
+You can associate a Project with a particular Team, such as a "Marketing team" for related ML models. A Project inside the Team will be visible to all Team members. 
+
+You must create a Team before adding a Project. Navigate to the “Teams” section in the left menu, and add a new one. You can add other users to this Team at any point after creating it.
+
+{% tabs %}
+
+{% tab title="API" %} 
+
+After creating the team, copy the `team_ID` from the team page. To add a Project to a Team, reference the team_id when creating the Project:
+
+```
+project = ws.create_project("Add your project name", team_id="TEAM ID")
+project.description = "Add your project description"
+project.save()
+```
+
+{% endtab %}
+
+{% tab title="UI" %} 
+
+Click on the “plus” sign on the home page and type your Project name and description. Select the team name from the dropdown menu to add a Project to a team.
+
+![](../.gitbook/assets/cloud/add_project_wide-min.png)
+
+{% endtab %}
+
+{% endtabs %}
+
+{% hint style="info" %}
+**What's next?** Head to the next section to see how to [send data to a Project](snapshots.md).
+{% endhint %}
+
+## Connect to a Project
+
+To connect to an existing Project from your Python environment (for example, if you first created the Project in the UI and now want to send data to it), use the `get_project` method.
+
+```python
+project = ws.get_project("PROJECT_ID")
+```
+
+## Save changes
+
+After you make any changes to a Project via API (such as editing description or adding new monitoring panels), you must use the `save()` command:
+
+```python
+project.save()
+```
+
+## Browse Projects
+
+You can see all available Projects on the monitoring homepage, or request a list programmatically. To get a list of all Projects in a workspace `ws`, use:
+
+```python
+ws.list_projects()
+```
+
+To find a specific Project by its name, use the `search_project` method: 
+
+```python
+ws.search_project("project_name")
+```
+
+# [DANGER] Delete Project 
+
+{% hint style="danger" %}
+**You are deleting the data in a Project**. If you delete a Project, you will delete all the associated snapshots stored in a Project.
+{% endhint %}
+
+{% tabs %}
+
+{% tab title="API" %} 
+
+To delete the Project and all the data inside it:
+
+```
+# ws.delete_project("PROJECT ID")
+```
+
+{% endtab %}
+
+{% tab title="UI" %} 
+
+Go to the "home page", and hover over a Project name. Click on the bin sign and confirm that you want to delte the Project.
+
+{% endtab %}
+
+{% endtabs %}
+
+## Project parameters
+
+Each Project has the following parameters.
+
+| Parameter | Description |
+|---|---|
+| `name: str` | Project name. |
+| `id: UUID4 = Field(default_factory=uuid.uuid4)` | Unique identifier of the Project. Assigned automatically. |
+| `description: Optional[str] = None` | Optional description. Visible when you browse Projects. |
+| `dashboard: DashboardConfig` | Configuration of the Project dashboard. It describes the monitoring Panels that appear on the dashboard.<br><br>**Note**: Explore the [Dashboard Design](design_dashboard_api.md) section for details. There is no need to explicitly pass `DashboardConfig` as a parameter if you use the `.dashboard.add_panel` method to add Panels. |
+| `date_from: Optional[datetime.datetime] = None` | Start DateTime of the monitoring dashboard. By default, Evidently shows data for all available periods based on the snapshot timestamps. <br><br>You can set a specific date or a relative DateTime. For example, to refer to the last 30 days:<br>`from datetime import datetime, timedelta`<br>`datetime.now() + timedelta(-30)`<br>When you view the dashboard, the data will be visible from this start date. You can switch to other dates in the interface. |
+| `date_to: Optional[datetime.datetime] = None` | End datetime of the monitoring dashboard. <br>Works the same as above. |
+
+# What’s next?
+
+Once you create or connect to a Project, you can:
+* [Send snapshots](snapshots.md) using the `add_report` or `add_test_suite` methods. 
+* Configure the monitoring dashboard in the [user interface](add_dashboard_tabs.md) or via the [Python API](design_dashboard_api.md).
diff --git a/docs/book/monitoring/alerting.md b/docs/book/monitoring/alerting.md
@@ -0,0 +1,36 @@
+---
+description: How to send alerts. 
+---   
+
+{% hint style="success" %} 
+Built-in alerting is a Pro feature available in the Evidently Cloud. 
+{% endhint %}
+
+To enable alerts, open the Project and navigate to the "Alerts" section in the left menu. To enable alerts, you must set:
+* A notification channel.
+* An alert condition. 
+
+# Notification channels
+
+You can choose between the following options:
+* **Email**. Add email addresses to send alerts to. 
+* **Slack**. Add a Slack webhook. 
+* **Discord**. Add a Discord webhook.
+
+# Alert conditions
+
+## Failed tests
+
+If you use Test Suites, you can tie alerting to the failed Tests in a Test Suite. Toggle this option on the Alerts page. Evidently will set an alert to the defined channel if any of the Tests fail.
+
+{% hint style="info" %} 
+**How to avoid alert fatigue?** When you create a Test Suite, you can [mark certain conditions as Warnings](../tests-and-reports/custom-test-suite.md) using the `is_critical` parameters. This helps distinguish between critical failures that trigger alerts (set `is_critical` as `True`; default) and non-critical ones for which no alerts will be generated (set `is_critical` as `False`).
+{% endhint %}
+
+## Custom conditions 
+
+You can also set alerts on individual Metric values for both Reports and Test Suites. For example, you can generate Alerts when the share of drifting features is above a certain threshold. 
+
+Click on the plus sign below the “Add new Metric alert” and follow the prompts to set an alert condition. 
+
+![](../.gitbook/assets/cloud/alerts.png)
diff --git a/docs/book/monitoring/collector_service.md b/docs/book/monitoring/collector_service.md
@@ -1,5 +1,5 @@
 ---
-description: Deployment scenario for near real-time monitoring.
+description: Send data in near real-time.
 ---   
 
 # Overview
@@ -44,11 +44,11 @@ You can configure the following parameters:
 | `report_config`   | `ReportConfig`    | Configures the contents of the snapshot: Report or TestSuite computed for each batch of data. |
 | `reference_path`  | Optional[str]     | Local path to a *.parquet* file with the reference dataset.                                   |
 | `cache_reference` | bool              | Defines whether to cache reference data or re-read it each time.                              |
-| `api_url`         | str               | URL where the Evidently UI Service runs and snapshots will be saved to.                       |
+| `api_url`         | str               | URL where the Evidently UI Service runs and snapshots will be saved to. For Evidently Cloud, use `api_url="https://app.evidently.cloud"`|
 | `api_secret`      | Optional[str]     | Evidently UI Service secrets.                                                                 |
 | `project_id`      | str               | ID of the project to save snapshots to.                                                       |
 
-You can create a `ReportConfig` object from Report or TestSuite objects. You must run them first so that all Metrics and Tests are collected (including when you use Presets or Test/Metric generators).
+You can create a `ReportConfig` object from `Report` or `TestSuite` objects. You must run them first so that all `Metrics` and `Tests` are collected (including when you use Presets or Test/Metric generators).
 
 ```python
 report = Report(...) 
@@ -66,7 +66,7 @@ report_config = ReportConfig.from_test_suite(test_suite)
 
 Currently, there are two options available: 
 * `IntervalTrigger`: triggers the snapshot calculation each interval seconds 
-* `RowsCountTrigger`: triggers the snapshot calculation every time the configured amount of rows have been sent to the collector service
+* `RowsCountTrigger`: triggers the snapshot calculation every time the configured number of rows has been sent to the collector service
 
 **Note**: we are also working on `CronTrigger` and other triggers. Would you like to see additional scenarios? Please open a GitHub issue with your suggestions.
 
@@ -102,7 +102,7 @@ First, run the collector service:
 evidently collector
 ```
 
-Then, use the `CollectorClient` to add new collector config:
+Then, use the `CollectorClient` to add a new collector config:
 
 ```python
 config = CollectorConfig(
@@ -159,12 +159,8 @@ This is how it looks in the Terminal.
 
 Sending data:
 
-![](../.gitbook/assets/monitoring/collector_send-min.png)
-
-The data is received by the collector service:
-
 ![](../.gitbook/assets/monitoring/collector_data-min.png)
 
+The data is received by the collector service:
 
-
-
+![](../.gitbook/assets/monitoring/collector_send-min.png)