diff --git a/site/faq/faq-testing.qmd b/site/faq/faq-testing.qmd index dcfc73b817..07d7e14536 100644 --- a/site/faq/faq-testing.qmd +++ b/site/faq/faq-testing.qmd @@ -115,7 +115,7 @@ Check out our introductory series — [**{{< var vm.product >}} for validation** [^12]: [Manage record stakeholder types](/guide/configuration/manage-record-stakeholder-types.qmd) -[^13]: [Assess compliance](/guide/validation/assess-compliance.qmd#link-validator-evidence) +[^13]: [Assess compliance](/guide/validation/assess-compliance.qmd#map-evidence) [^14]: [Can we configure, customize, or add our own tests?](#can-we-configure-customize-or-add-our-own-tests) diff --git a/site/get-started/validator/quickstart-validator.qmd b/site/get-started/validator/quickstart-validator.qmd index 460d16c157..db62f00678 100644 --- a/site/get-started/validator/quickstart-validator.qmd +++ b/site/get-started/validator/quickstart-validator.qmd @@ -118,7 +118,7 @@ After you successfully run the notebook, return to the {{< var validmind.platfor - Your validation report is broken down into sections, defined by your report templates.[^3] - The report includes risk assessment summaries at the overall and per-section level, overviews of your compliance assessments provided within your validation report.[^4] - - Each sub-section allows you to attach test results as evidence,[^5] as well as log artifacts based on your evaluation of datasets or records.[^6] + - Each sub-section (or guideline) allows you to attach test results as evidence,[^5] as well as log artifacts based on your evaluation of datasets or records.[^6] :::: {.flex .flex-wrap .justify-around} @@ -138,13 +138,13 @@ After you successfully run the notebook, return to the {{< var validmind.platfor #### Add evidence to your report -Expand any section of the validation report to attach your test results and drafts of test result summaries generated by the {{< var validmind.developer >}} as evidence by clicking on **{{< fa link >}} Link Evidence to Report**. +Expand any section of the validation report to attach your test results and drafts of test result summaries generated by the {{< var validmind.developer >}} as evidence by clicking **Evidence** on any guideline to expand the evidence panel. For example: -1. Locate the 2.2.1. Data Quality section and click on **Class Imbalance Assessment**. +1. Within your validation report, click on **Class Imbalance Assessment**. -2. Under the Class Imbalance Assessment sub-section, locate Validator Evidence then click **{{< fa link >}} Link Evidence to Report**. +2. Under the Class Imbalance Assessment guideline click **{{< fa link >}} Link Evidence**, then select **Validator Evidence** as the [evidence type]{.smallcaps}. 3. Select the Class Imbalance test results we logged: **ValidMind Data Validation Class Imbalance** @@ -173,9 +173,9 @@ Now that you've grasped the basics of using {{< var vm.product >}} for validatio [^3]: [Working with document templates](/guide/templates/working-with-document-templates.qmd) -[^4]: [Assess compliance](/guide/validation/assess-compliance.qmd#provide-compliance-assessments) +[^4]: [Assess compliance](/guide/validation/assess-compliance.qmd#assign-compliance-assessments) -[^5]: [Assess compliance](/guide/validation/assess-compliance.qmd#link-validator-evidence) +[^5]: [Assess compliance](/guide/validation/assess-compliance.qmd#map-evidence) [^6]: [Add and manage artifacts](/guide/validation/add-manage-artifacts.qmd#add-artifacts-on-validation-reports) diff --git a/site/guide/documentation/content-editing-toolbar.png b/site/guide/documentation/content-editing-toolbar.png index 846109487e..68cf696412 100644 Binary files a/site/guide/documentation/content-editing-toolbar.png and b/site/guide/documentation/content-editing-toolbar.png differ diff --git a/site/guide/documentation/work-with-content-blocks.qmd b/site/guide/documentation/work-with-content-blocks.qmd index 6776962b17..5d5e23d7b4 100644 --- a/site/guide/documentation/work-with-content-blocks.qmd +++ b/site/guide/documentation/work-with-content-blocks.qmd @@ -45,9 +45,10 @@ The content editing toolbar is a rich text editor that enables you to: - Format your text, including adding hyperlinks and code blocks - Reference related record and artifact field values with variables[^2] - Insert LaTex formulas[^3] +- Reference other sections of the document[^4] - Attach images via upload or URL -You can also use the toolbar to suggest changes, save named versions of content, and leave comments.[^4] +You can also use the toolbar to suggest changes, save named versions of content, and leave comments.[^5] ::: {.callout title="Stuck on where to start?"} Use {{< var vm.product >}} to assist you with generating content via AI!^[[Generate content drafts with AI](#generate-content)] @@ -58,9 +59,9 @@ Use {{< var vm.product >}} to assist you with generating content via AI!^[[Gener ## Prerequisites - [x] {{< var link.login >}} -- [x] There are records registered in the inventory.[^5] -- [x] Documents exist and are completed or are in progress for your record.[^6] -- [x] You are a [{{< fa code >}} Developer]{.bubble} or [{{< fa circle-check >}} Validator]{.bubble}, or assigned another role with sufficient permissions to perform the tasks in this guide.[^7] +- [x] There are records registered in the inventory.[^6] +- [x] Documents exist and are completed or are in progress for your record.[^7] +- [x] You are a [{{< fa code >}} Developer]{.bubble} or [{{< fa circle-check >}} Validator]{.bubble}, or assigned another role with sufficient permissions to perform the tasks in this guide.[^8] ::: @@ -79,26 +80,26 @@ Use {{< var vm.product >}} to assist you with generating content via AI!^[[Gener #### [from library]{.smallcaps} Text Block - : Inserts a text block from a template in your block library:[^8] + : Inserts a text block from a template in your block library:[^9] a. Select the reusable blocks you want to add. b. Click **Insert # Text Block(s) to Document**. - Test-Driven[^9] + Test-Driven[^10] : Adds a new section with logged test results. - Metric Over Time[^10] + Metric Over Time[^11] : Adds a new section with logged metric over time results. ::: -7. After adding the block to your document, click on the text to make changes or add comments.[^11] +7. After adding the block to your document, click on the text to make changes or add comments.[^12] ### Reference field values While editing a simple text block within documents, you can reference values in the form of variables from: ::: {.callout} -- To reference field values in the form of variables, your organization must not have tracked changes enabled by default.[^12] +- To reference field values in the form of variables, your organization must not have tracked changes enabled by default.[^13] - Note that while you are able to select fields with empty values within available record or artifact type fields, no value will be displayed in the content block until the field is populated. ::: @@ -118,8 +119,8 @@ While editing a simple text block within documents, you can insert math equation While editing a simple text block within documents, you can have {{< var vm.product >}} assist you with generating content drafts. ::: {.callout title="How can generate content drafts with AI?"} -- To use the generate content drafts with AI feature, your organization must not have tracked changes enabled by default.[^13] -- Generating content drafts works best after you've logged tests with the {{< var validmind.developer >}},[^14] as existing test descriptions and results provide more context for the {{< var vm.product >}} AI Content Builder to draw upon. +- To use the generate content drafts with AI feature, your organization must not have tracked changes enabled by default.[^14] +- Generating content drafts works best after you've logged tests with the {{< var validmind.developer >}},[^15] as existing test descriptions and results provide more context for the {{< var vm.product >}} AI Content Builder to draw upon. ::: @@ -128,7 +129,7 @@ To generate content drafts: {{< include content_blocks/_generate-with-ai.qmd >}} -When generating content drafts with AI, accepted versions and edits are retained in your {{< fa wifi >}} Activity[^15] just like other updates to your documents. +When generating content drafts with AI, accepted versions and edits are retained in your {{< fa wifi >}} Activity[^16] just like other updates to your documents.
@@ -176,7 +177,7 @@ While editing a simple text block within documents, you can directly reference o ::: -Hyperlinks will also take you to the referenced section in PDF document exports.[^16] +Hyperlinks will also take you to the referenced section in PDF document exports.[^17] ## Remove content blocks @@ -186,11 +187,11 @@ Test-driven or metric over time blocks can be re-added later on but **text block 1. In the left sidebar, click **{{< fa cubes >}} Inventory**. -2. Select a record or find your record by applying a filter or searching for it.[^17] +2. Select a record or find your record by applying a filter or searching for it.[^18] -3. In the left sidebar that appears for your record, click **{{< fa file >}} Documents** and select the **Latest** tab.[^18] +3. In the left sidebar that appears for your record, click **{{< fa file >}} Documents** and select the **Latest** tab.[^19] -4. Click on the document file you want to remove a block from.[^19] +4. Click on the document file you want to remove a block from.[^20] 5. Click on a section header to expand that section and remove content. @@ -209,34 +210,36 @@ Test-driven or metric over time blocks can be re-added later on but **text block [^3]: [Insert mathematical formulas](#insert-mathematical-formulas) -[^4]: [Collaborate with others](collaborate-with-others.qmd) +[^4]: [Reference document sections](#reference-document-sections) -[^5]: [Register records in the inventory](/guide/inventory/register-records-in-inventory.qmd) +[^5]: [Collaborate with others](collaborate-with-others.qmd) -[^6]: [Working with documents](/guide/templates/working-with-documents.qmd) +[^6]: [Register records in the inventory](/guide/inventory/register-records-in-inventory.qmd) -[^7]: [Manage permissions](/guide/configuration/manage-permissions.qmd) +[^7]: [Working with documents](/guide/templates/working-with-documents.qmd) -[^8]: [Manage text block library](/guide/templates/manage-text-block-library.qmd) +[^8]: [Manage permissions](/guide/configuration/manage-permissions.qmd) -[^9]: [Work with test results](/guide/documentation/work-with-test-results.qmd) +[^9]: [Manage text block library](/guide/templates/manage-text-block-library.qmd) -[^10]: [Work with metrics over time](/guide/monitoring/work-with-metrics-over-time.qmd) +[^10]: [Work with test results](/guide/documentation/work-with-test-results.qmd) -[^11]: [Collaborate with others](/guide/documentation/collaborate-with-others.qmd) +[^11]: [Work with metrics over time](/guide/monitoring/work-with-metrics-over-time.qmd) -[^12]: [Managing your organization](/guide/configuration/managing-your-organization.qmd#manage-document-defaults) +[^12]: [Collaborate with others](/guide/documentation/collaborate-with-others.qmd) [^13]: [Managing your organization](/guide/configuration/managing-your-organization.qmd#manage-document-defaults) -[^14]: [Run tests and test suites](/developer/how-to/testing-overview.qmd) +[^14]: [Managing your organization](/guide/configuration/managing-your-organization.qmd#manage-document-defaults) -[^15]: [View model activity](/guide/inventory/view-record-activity.qmd) +[^15]: [Run tests and test suites](/developer/how-to/testing-overview.qmd) -[^16]: [Export documents](/guide/reporting/export-documents.qmd) +[^16]: [View model activity](/guide/inventory/view-record-activity.qmd) -[^17]: [Working with the model inventory](/guide/inventory/working-with-the-inventory.qmd#search-filter-and-sort-models) +[^17]: [Export documents](/guide/reporting/export-documents.qmd) -[^18]: [Work with document versions](/guide/documentation/work-with-document-versions.qmd) +[^18]: [Working with the model inventory](/guide/inventory/working-with-the-inventory.qmd#search-filter-and-sort-models) -[^19]: [Working with documents](/guide/templates/working-with-documents.qmd) \ No newline at end of file +[^19]: [Work with document versions](/guide/documentation/work-with-document-versions.qmd) + +[^20]: [Working with documents](/guide/templates/working-with-documents.qmd) \ No newline at end of file diff --git a/site/guide/validation/_add-artifacts-on-documentation.qmd b/site/guide/validation/_add-artifacts-on-documentation.qmd index 6018aa579d..ecd4c4fe44 100644 --- a/site/guide/validation/_add-artifacts-on-documentation.qmd +++ b/site/guide/validation/_add-artifacts-on-documentation.qmd @@ -23,32 +23,26 @@ SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial --> 1. On the modal that opens, provide information for your artifact. -1. When you are done, click **Save** to submit the artifact. +1. When you are done, click **Add {Artifact Type}** where `{Artifact Type}` is the artifact type you want to add to submit the artifact. Once created, you can then add supporting documentation to your artifact.^[[Manage attachments on artifacts](/guide/validation/add-manage-artifacts.qmd#manage-attachments)] :::: :::: {.content-hidden unless-format="revealjs"} -1. In the left sidebar, click **{{< fa cubes >}} Inventory**. - -1. Select a record or [find your record by applying a filter or searching for it](/guide/inventory/working-with-the-inventory.qmd#search-filter-and-sort-records){target="_blank"}. - -1. In the left sidebar that appears for your record, click **Development** under {{< fa file >}} Documents. +a. In the left sidebar that appears for your record, click **Development** under {{< fa file >}} Documents. - - -1. On your record's documentation page, click **{{< fa plus >}} Add Artifact**. +b. On your record's documentation page, click **{{< fa plus >}} Add Artifact**. -1. Select the [artifact type](/guide/validation/manage-artifact-types.qmd){target="_blank"} you want to add: +c. Select the [artifact type](/guide/validation/manage-artifact-types.qmd){target="_blank"} you want to add: - Validation Issue - Policy Exception - Limitation -1. On the modal that opens, provide information for your artifact. +d. On the modal that opens, provide information for your artifact. -1. When you are done, click **Save** to submit the artifact. +e. When you are done, click **Add {Artifact Type}** where `{Artifact Type}` is the artifact type you want to add to submit the artifact. Once created, you can then [add supporting documentation to your artifact](/guide/validation/add-manage-artifacts.qmd#manage-attachments){target="_blank"}. diff --git a/site/guide/validation/_assess-evidence-for-compliance.qmd b/site/guide/validation/_assess-evidence-for-compliance.qmd index c45fdef072..c6c25d79a2 100644 --- a/site/guide/validation/_assess-evidence-for-compliance.qmd +++ b/site/guide/validation/_assess-evidence-for-compliance.qmd @@ -9,7 +9,7 @@ SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial --> ![Assess Evidence panel](/guide/validation/assess-evidence-panel.png){fig-alt="Assess Evidence panel showing option to identify potential risks and compliance gaps based on linked evidence." .screenshot} :::: -Assess Evidence analyzes the linked evidence and generates a structured compliance assessment, identifying potential risks and compliance gaps. +Assess Evidence uses AI to analyze the linked evidence and generate a structured compliance assessment, identifying potential risks and compliance gaps. 1. In the left sidebar, click **{{< fa cubes >}} Inventory**. @@ -19,21 +19,39 @@ Assess Evidence analyzes the linked evidence and generates a structured complian 4. Click on a Validation type file.^[[Preparing validation reports](/guide/validation/preparing-validation-reports.qmd#validation-overview)] -5. Navigate to a section that has linked evidence. +5. You can assess evidence for compliance at three levels: -6. Expand the **Evidence** panel. +::: {.panel-tabset} -7. Click **{{< fa wand-magic-sparkles >}} Assess Evidence**. +#### For the entire report -8. The AI analyzes the linked evidence and generates an **Evidence Assessment** containing: +Click **{{< fa wand-magic-sparkles >}} Assess Evidence** in the right sidebar to expand the evidence assessment panel for the entire report. - - **Guideline Assessment** — A compliance conclusion indicating whether the guideline requirements are fully met, partially met, or not met, with an explanation of the evidence quality. +#### For an entire section - - **Validation Observations** — Specific findings about gaps or issues in the evidence, with each observation covering a single concern and suggesting actions for developers. +Sections include subsections with guidelines, and a risk assessment summary at the top of the page. + +a. Click on a section of the validation report to expand that section. +b. Click **{{< fa wand-magic-sparkles >}} Assess Evidence** at the top of the page to open the evidence assessment panel for that section. + +#### For individual guidelines + +Guidelines are organized into subsections under a parent section. + +a. Click on a section of the validation report to expand that section. +b. In any subsection, click **{{< fa wand-magic-sparkles >}} Assess Evidence** to open the evidence assessment panel for that guideline. + +::: + +6. Click **Assess Evidence** to run an analysis on the linked evidence and generate an Evidence Assessment containing: + - **Guideline Assessment** — A compliance conclusion indicating whether the guideline requirements are fully met, partially met, or not met, with an explanation of the evidence quality. + - **Validation Observations** — Specific findings about gaps or issues in the evidence, with each observation covering a single concern and suggesting actions for developers. - **Evidence Review** — A detailed analysis of what the evidence demonstrates, including references to specific test outputs, documentation, and any limitations. - The panel displays how many assessments are available to review. + If this is your first time running Assess Evidence, review the information presented to you explaining the assessment process, then click **Assess Evidence** to run the assessment.^[To hide the additional information on assessment modal for future runs, select the **Don't show this again** checkbox.] + +When evidence assessment is complete, the [#]{.bubble} of assessment items to review is displayed. Review the assessment items and approve or reject the suggestions as needed. ### Review and approve evidence assessments @@ -43,35 +61,37 @@ After running Assess Evidence, you can review and approve assessments at three l #### For the entire report -a. Open the validation report and look at the right sidebar. +If you have run evidence assessment for the entire report, you can review and approve suggestions at the entire report level. -b. The **Assess Evidence** panel shows how many assessments need review across the entire report. +The **{{< fa wand-magic-sparkles >}} Assess Evidence** panel in the right sidebar for your validation report shows how many items need review across the entire report — click to expand the panel if it is collapsed: -c. Use **Approve All** to accept all generated assessments, or **Reject All** to dismiss all assessments. + - Select **[{{< fa check >}} Approve All]{.green}** to link all suggested assessments across all guidelines, or **[{{< fa xmark >}} Reject All]{.red}** to dismiss all suggestions. + - Click into any section of the report with approved assessments to confirm that any approved assessments appear in the Evidence Assessment panel for guidelines as expected. + - To re-run the assessment, click **{{< fa sync >}} Reassess Evidence**. #### For an entire section -a. Navigate to a specific section in the validation report. +Sections include subsections with guidelines, and a risk assessment summary at the top of the page. If you have run mapping for an entire section, you can review and approve suggestions at the section level: -b. In the section header, click **{{< fa wand-magic-sparkles >}} Assess Evidence** to open the assessment panel. +a. Click on a section of the validation report to expand that section. -c. Use **Approve All** to accept all generated assessments for that section, or **Reject All** to dismiss them. +b. Click **{{< fa wand-magic-sparkles >}} Assess Evidence** at the top of the page to open the evidence assessment panel for that section: -d. To regenerate assessments with updated evidence, click **Re-assess Evidence**. + - Select **[{{< fa check >}} Approve All]{.green}** to link all suggested assessments for guidelines in that section, or **[{{< fa xmark >}} Reject All]{.red}** to dismiss all section suggestions. + - Confirm that any approved assessments appear in the Evidence Assessment panels for the guidelines in that section as expected. + - To re-run the assessment, click **{{< fa sync >}} Reassess Evidence**. #### For individual guidelines -a. Navigate to a specific section in the validation report. - -b. Expand the **Evidence Assessment** panel for a guideline. Assessments pending review show a [Review]{.bubble} status. +Guidelines are organized into subsections under a parent section. If you have run mapping for an individual guideline, you can review and approve suggestions at the guideline level: -c. Review the generated assessment content. +a. Click on a section of the validation report to expand that section. -d. Click **Approve** to accept the assessment, or **Reject** to dismiss it. +b. In any subsection, click **Evidence Assessment** to expand the evidence assessment panel for that guideline: -e. After approving, you can edit the assessment content as needed — changes are auto-saved. - -f. To regenerate an assessment, click **{{< fa wand-magic-sparkles >}} Reassess Evidence** to run the AI analysis again with any updated evidence. + - Click **[{{< fa check >}} Approve]{.green}** to approve the assessment. + - Click **[{{< fa xmark >}} Reject]{.red}** to dismiss the assessment. + - To re-run the assessment, click **{{< fa wand-magic-sparkles >}} Reassess Evidence**. ::: @@ -79,57 +99,89 @@ f. To regenerate an assessment, click **{{< fa wand-magic-sparkles >}} Reassess :::: {.content-hidden unless-format="revealjs"} -Assess Evidence analyzes the linked evidence and generates a structured compliance assessment, identifying potential risks and compliance gaps. +Assess Evidence uses AI to analyze the linked evidence and generate a structured compliance assessment, identifying potential risks and compliance gaps. -1. Navigate to a section that has linked evidence. +1. In the left sidebar, click **{{< fa cubes >}} Inventory**. -2. Expand the **Evidence** panel. +2. Select a record or find your record by [applying a filter or searching for it](/guide/inventory/working-with-the-inventory.qmd#search-filter-and-sort-records){target="_blank"}. -3. Click **{{< fa wand-magic-sparkles >}} Assess Evidence**. +3. In the left sidebar that appears for your record, click **Validation** under **{{< fa file >}} Documents**. -4. The AI analyzes the linked evidence and generates an **Evidence Assessment** containing: +4. You can assess evidence for compliance at three levels: - - **Guideline Assessment** — A compliance conclusion indicating whether the guideline requirements are fully met, partially met, or not met, with an explanation of the evidence quality. +::: {.panel-tabset} - - **Validation Observations** — Specific findings about gaps or issues in the evidence, with each observation covering a single concern and suggesting actions for developers. +#### For the entire report + +Click **{{< fa wand-magic-sparkles >}} Assess Evidence** in the right sidebar to expand the evidence assessment panel for the entire report. + +#### For an entire section + +Sections include subsections with guidelines, and a risk assessment summary at the top of the page. + +a. Click on a section of the validation report to expand that section. +b. Click **{{< fa wand-magic-sparkles >}} Assess Evidence** at the top of the page to open the evidence assessment panel for that section. +#### For individual guidelines + +Guidelines are organized into subsections under a parent section. + +a. Click on a section of the validation report to expand that section. +b. In any subsection, click **{{< fa wand-magic-sparkles >}} Assess Evidence** to open the evidence assessment panel for that guideline. + +::: + +6. Click **Assess Evidence** to run an analysis on the linked evidence and generate an Evidence Assessment containing: + + - **Guideline Assessment** — A compliance conclusion indicating whether the guideline requirements are fully met, partially met, or not met, with an explanation of the evidence quality. + - **Validation Observations** — Specific findings about gaps or issues in the evidence, with each observation covering a single concern and suggesting actions for developers. - **Evidence Review** — A detailed analysis of what the evidence demonstrates, including references to specific test outputs, documentation, and any limitations. - The panel displays how many assessments are available to review. + If this is your first time running Assess Evidence, review the information presented to you explaining the assessment process, then click **Assess Evidence** to run the assessment. + +When evidence assessment is complete, the [#]{.bubble} of assessment items to review is displayed. Review the assessment items and approve or reject the suggestions as needed. + +#### Review and approve evidence assessments After running Assess Evidence, you can review and approve assessments at three levels: +::: {.panel-tabset} + #### For the entire report -a. Open the validation report and look at the right sidebar. +If you have run evidence assessment for the entire report, you can review and approve suggestions at the entire report level. -b. The **Assess Evidence** panel shows how many assessments need review across the entire report. +The **{{< fa wand-magic-sparkles >}} Assess Evidence** panel in the right sidebar for your validation report shows how many items need review across the entire report — click to expand the panel if it is collapsed: -c. Use **Approve All** to accept all generated assessments, or **Reject All** to dismiss all assessments. + - Select **[{{< fa check >}} Approve All]{.green}** to link all suggested assessments across all guidelines, or **[{{< fa xmark >}} Reject All]{.red}** to dismiss all suggestions. + - Click into any section of the report with approved assessments to confirm that any approved assessments appear in the Evidence Assessment panel for guidelines as expected. + - To re-run the assessment, click **{{< fa sync >}} Reassess Evidence**. #### For an entire section -a. Navigate to a specific section in the validation report. +Sections include subsections with guidelines, and a risk assessment summary at the top of the page. If you have run mapping for an entire section, you can review and approve suggestions at the section level: -b. In the section header, click **{{< fa wand-magic-sparkles >}} Assess Evidence** to open the assessment panel. +a. Click on a section of the validation report to expand that section. -c. Use **Approve All** to accept all generated assessments for that section, or **Reject All** to dismiss them. +b. Click **{{< fa wand-magic-sparkles >}} Assess Evidence** at the top of the page to open the evidence assessment panel for that section: -d. To regenerate assessments with updated evidence, click **Re-assess Evidence**. + - Select **[{{< fa check >}} Approve All]{.green}** to link all suggested assessments for guidelines in that section, or **[{{< fa xmark >}} Reject All]{.red}** to dismiss all section suggestions. + - Confirm that any approved assessments appear in the Evidence Assessment panels for the guidelines in that section as expected. + - To re-run the assessment, click **{{< fa sync >}} Reassess Evidence**. #### For individual guidelines -a. Navigate to a specific section in the validation report. +Guidelines are organized into subsections under a parent section. If you have run mapping for an individual guideline, you can review and approve suggestions at the guideline level: -b. Expand the **Evidence Assessment** panel for a guideline. Assessments pending review show a [Review]{.bubble} status. +a. Click on a section of the validation report to expand that section. -c. Review the generated assessment content. +b. In any subsection, click **Evidence Assessment** to expand the evidence assessment panel for that guideline: -d. Click **Approve** to accept the assessment, or **Reject** to dismiss it. + - Click **[{{< fa check >}} Approve]{.green}** to approve the assessment. + - Click **[{{< fa xmark >}} Reject]{.red}** to dismiss the assessment. + - To re-run the assessment, click **{{< fa wand-magic-sparkles >}} Reassess Evidence**. -e. After approving, you can edit the assessment content as needed — changes are auto-saved. - -f. To regenerate an assessment, click **{{< fa wand-magic-sparkles >}} Reassess Evidence** to run the AI analysis again with any updated evidence. +::: :::: diff --git a/site/guide/validation/_provide-compliance-assessments.qmd b/site/guide/validation/_assign-compliance-assessments.qmd similarity index 52% rename from site/guide/validation/_provide-compliance-assessments.qmd rename to site/guide/validation/_assign-compliance-assessments.qmd index ad5456259d..f4a5c61660 100644 --- a/site/guide/validation/_provide-compliance-assessments.qmd +++ b/site/guide/validation/_assign-compliance-assessments.qmd @@ -2,6 +2,8 @@ Refer to the LICENSE file in the root of this repository for details. SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial --> +After you have mapped and assessed available evidence, and generated or attached relevant artifacts, assign compliance assessments to each subsection of your validation report: + :::: {.content-visible unless-format="revealjs"} ::: {.callout title="Assessment options available for selection depend on configuration for individual validation report templates."} Your organization can customize these options to match its own assessment framework, or choose to remove them from templates entirely.^[[Configure assessment options](/guide/validation/configure-assessment-options.qmd)] @@ -15,32 +17,29 @@ Your organization can customize these options to match its own assessment framew 1. Click on a Validation type file.^[[Preparing validation reports](/guide/validation/preparing-validation-reports.qmd#validation-overview)] - Expand any subsection of the validation report you would like to work with. - -1. In any section of the documentation where the **[assessment]{.smallcaps}** dropdown menu is available, select one of the available assessment options. + Expand any subsection of the validation report you have attached evidence and artifacts for. - Confirm that the compliance summary shown for each subsection provides an accurate overview for current qualitative and quantitative risk assessments. +1. Under Guideline, click **Select a value for this guideline** to choose one of the available Assessment options. -1. Click under **Risk Assessment Notes** to add your risk assessment notes for that section using the content editing toolbar.^[[Work with content blocks](/guide/documentation/work-with-content-blocks.qmd#content-editing-toolbar)] + At the top of the page for that subsection, confirm that the compliance summary shown provides an accurate overview for current qualitative and quantitative risk assessments for that subsection. :::: :::: {.content-hidden unless-format="revealjs"} -1. In the same Class Imbalance Assessment sub-section above the Risk Assessment Notes, locate the Guideline with the **[assessment]{.smallcaps}** drop-down. +1. In the left sidebar, click **{{< fa cubes >}} Inventory**. -1. Select one of the available assessment options. +2. Select a record or find your record [by applying a filter or searching for it](/guide/inventory/working-with-the-inventory.qmd#search-filter-and-sort-records){target="_blank"}. -1. Confirm that the compliance summary shown for each subsection under 2. Validation provides an accurate overview for current qualitative and quantitative risk assessments: +3. In the left sidebar that appears for your record, click **Validation** under **{{< fa file >}} Documents**. - - ![Risk assessment compliance summary](/guide/validation/assess-compliance-summary.png){fig-alt="A screenshot of the risk assessment compliance summary" .screenshot group="compliance"} + Expand any subsection of the validation report you have attached evidence and artifacts for. -:::: {.pa3 .mt3 .f5 .embed} -**Assessment options available for selection depend on configuration for individual validation report templates.** +4. Under Guideline, click **Select a value for this guideline** to choose one of the available Assessment options. -Your organization can customize these options to match its own assessment framework, or choose to remove them from templates entirely. + At the top of the page for that subsection, confirm that the compliance summary shown provides an accurate overview for current qualitative and quantitative risk assessments for that subsection: -::: + + ![Risk assessment compliance summary](/guide/validation/assess-compliance-summary.png){fig-alt="A screenshot of the risk assessment compliance summary" .screenshot group="compliance"} :::: diff --git a/site/guide/validation/_autogenerate-validation-issues.qmd b/site/guide/validation/_autogenerate-validation-issues.qmd index 013d19ecf3..805bd861a0 100644 --- a/site/guide/validation/_autogenerate-validation-issues.qmd +++ b/site/guide/validation/_autogenerate-validation-issues.qmd @@ -2,39 +2,111 @@ Refer to the LICENSE file in the root of this repository for details. SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial --> -After you complete your risk assessment notes for a validation guideline, you can use AI to automatically suggest validation issues based on the gaps identified in your assessment. + + +::::: {.content-visible unless-format="revealjs"} + +::: {.callout-important title="AI suggestions require review!"} +Always review AI-generated validation issues before approving them. The suggestions are based on patterns in your risk assessment notes and may need adjustment to accurately reflect your findings. -::: {.callout title="This feature requires AI features to be enabled for your organization."} -Contact your administrator if you don't see the **Suggest Validation Issues** option. ::: -1. In the left sidebar, click **{{< fa cubes >}} Inventory**. +Use AI to automatically suggest validation issues based on the gaps identified in your assessment:^[This feature requires AI capabilities to be enabled for your organization — contact an administrator if you cannot access the **{{< fa wand-magic-sparkles >}} Suggest Validation Issues** interface.] -1. Select a record or find your record by applying a filter or searching for it.^[[Working with the inventory](/guide/inventory/working-with-the-inventory.qmd#search-filter-and-sort-records)] +a. In the Artifacts section under your evidence, click **{{< fa wand-magic-sparkles >}} Suggest Validation Issues**. -1. In the left sidebar, click **Validation** to open the validation report. +b. Click **Suggest Validation Issues** to have {{< var vm.product >}} analyze your risk assessment notes along with the guideline context and identify any gaps that warrant validation issues. -1. Expand any subsection of the validation report and complete your **Risk Assessment Notes** for a guideline. +c. Review the suggested validation issues. Each suggestion includes: -1. Click **{{< fa wand-magic-sparkles >}} Suggest Validation Issues** next to the guideline you assessed. + - A proposed title and description outlining the issue + - A preliminary severity level based on the identified gap + - A proposed remediation plan - The AI analyzes your risk assessment notes along with the guideline context to identify any gaps that warrant validation issues. + Click **See more {{< fa chevron-down >}}** on each suggested issue to expand for more details. -1. Review the suggested validation issues: +d. Approve, reject, or regenerate the suggested validation issues: - Each suggestion includes: + ::: {.panel-tabset} - - A proposed title describing the issue - - A severity level based on the identified gap - - Recommended remediation actions + #### Approve/reject individual issues -1. For each suggested issue, choose one of the following actions: + Under each individual issue: - Click **{{< fa check >}} Approve** to create the validation issue and link it to your report. - - Click **{{< fa xmark >}} Dismiss** to reject the suggestion if it doesn't apply. + - Click **{{< fa xmark >}} Reject** to reject the suggestion if it doesn't apply. -1. After approving a validation issue, you can further customize it by clicking on the issue to edit fields such as owner, due date, and status.^[[Add and manage artifacts](/guide/validation/add-manage-artifacts.qmd)] + #### Approve/reject all issues -::: {.callout-important title="AI suggestions require review"} -Always review AI-generated validation issues before approving them. The suggestions are based on patterns in your risk assessment notes and may need adjustment to accurately reflect your findings. + Click **{{< fa wand-magic-sparkles >}} Suggest Validation Issues [#]{.bubble}** and select: + + - **{{< fa check >}} Approve All** to approve all suggested issues at once. + - **{{< fa xmark >}} Reject All** to reject all suggested issues at once. + + #### Regenerate new suggestions + + Click **{{< fa wand-magic-sparkles >}} Suggest Validation Issues [#]{.bubble}** and select **{{< fa sync >}} Regenerate Issues**. + + ::: + +:::: {.content-visible when-format="html" when-meta="includes.compliance"} +e. After approving a validation issue, you can further customize it by clicking on the issue to edit fields such as owner, due date, and status.^[[Add and manage artifacts](/guide/validation/add-manage-artifacts.qmd)] + +:::: + +:::: {.content-visible when-format="html" unless-meta="includes.compliance"} +e. After approving a validation issue, you can further customize it by clicking on the issue to edit fields such as owner, due date, and status. + +:::: + + +::::: + + + +::::: {.content-hidden unless-format="revealjs"} + +:::: {.mb2 .mr4 .pa3 .f5 .embed} +- **Always review AI-generated validation issues before approving them.** The suggestions are based on patterns in your risk assessment notes and may need adjustment to accurately reflect your findings. +- This feature requires AI capabilities to be enabled for your organization. Contact an administrator if you cannot access the **{{< fa wand-magic-sparkles >}} Suggest Validation Issues** interface. ::: + +You can of course log artifacts manually on your validation report, but you can also use AI to automatically suggest validation issues based on the gaps identified in your assessment: + +a. In the left sidebar that appears for your record, click **Validation** under {{< fa file >}} Documents. + +b. In the Artifacts section under your evidence, click **{{< fa wand-magic-sparkles >}} Suggest Validation Issues**. + +c. Click **Suggest Validation Issues** to have {{< var vm.product >}} analyze your risk assessment notes along with the guideline context and identify any gaps that warrant validation issues. + +d. Review the suggested validation issues. Each suggestion includes: + + - A proposed title and description outlining the issue + - A preliminary severity level based on the identified gap + - A proposed remediation plan + + Click **See more {{< fa chevron-down >}}** on each suggested issue to expand for more details. + +e. Approve, reject, or regenerate the suggested validation issues: + + #### Approve/reject individual issues + + Under each individual issue: + + - Click **{{< fa check >}} Approve** to create the validation issue and link it to your report. + - Click **{{< fa xmark >}} Reject** to reject the suggestion if it doesn't apply. + + #### Approve/reject all issues + + Click **{{< fa wand-magic-sparkles >}} Suggest Validation Issues [#]{.bubble}** and select: + + - **{{< fa check >}} Approve All** to approve all suggested issues at once. + - **{{< fa xmark >}} Reject All** to reject all suggested issues at once. + + #### Regenerate new suggestions + + Click **{{< fa wand-magic-sparkles >}} Suggest Validation Issues [#]{.bubble}** and select **{{< fa sync >}} Regenerate Issues**. + +f. After approving a validation issue, you can further customize it by clicking on the issue to edit fields such as owner, due date, and status. + +::::: diff --git a/site/guide/validation/_link-artifacts-to-reports.qmd b/site/guide/validation/_link-artifacts-to-reports.qmd index db4b14c661..bdbc706c92 100644 --- a/site/guide/validation/_link-artifacts-to-reports.qmd +++ b/site/guide/validation/_link-artifacts-to-reports.qmd @@ -3,52 +3,52 @@ Refer to the LICENSE file in the root of this repository for details. SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial --> :::: {.content-visible unless-format="revealjs"} -1. In the left sidebar, click **{{< fa cubes >}} Inventory**. +a. Under your evidence, click **Artifacts** to expand the artifacts panel. -1. Select a record or find your record by applying a filter or searching for it.^[[Working with the inventory](/guide/inventory/working-with-the-inventory.qmd#search-filter-and-sort-records)] - -1. In the left sidebar that appears for your record, click **{{< fa file >}} Documents** and select the **Latest** tab.^[[Work with document versions](/guide/documentation/work-with-document-versions.qmd)] - -1. Click on a Validation type file.^[[Preparing validation reports](/guide/validation/preparing-validation-reports.qmd#validation-overview)] - -1. Expand any subsection of the validation report you would like to work with. - -1. In any section of the report where the button is available, click **{{< fa link >}} Link Artifact to Report** and select the artifact type^[[Manage artifact types](/guide/validation/manage-artifact-types.qmd)] you want to add: +b. Click **{{< fa link >}} Link Artifact** and select the artifact type^[[Manage artifact types](/guide/validation/manage-artifact-types.qmd)] you want to add: - Validation Issue - Policy Exception - Limitation -1. On the **Link Artifact to Report** modal that opens, select from the list of available artifacts, or click **{{< fa plus >}} Update Linked Artifacts**.^[[Add and manage artifacts](/guide/validation/add-manage-artifacts.qmd#add-artifacts-on-validation-reports)] +c. On the Link Artifact to Report modal that opens: -1. Click **Update Linked Artifacts**. + - Select from the list of available artifacts. + - Click **{{< fa plus >}} Add {Artifact Type}** where `{Artifact Type}` is the artifact type you want to add to create a new artifact,[^create-artifact] then select the new artifact from the list. - Confirm that newly linked artifact shown under Artifacts is accurate. +d. Click **Update Linked Artifacts**. + + Confirm that the newly linked artifacts shown under Artifacts is accurate. :::: + + +[^create-artifact]: + + 1. Click **{{< fa plus >}} Add {Artifact Type}** where `{Artifact Type}` is the artifact type you want to add to create a new artifact. + 2. Enter in the details for your artifact. + 3. Click **Add {Artifact Type}** to submit the artifact. :::: {.content-hidden unless-format="revealjs"} -1. From the **{{< fa cubes >}} Inventory** in the ValidMind Platform, go to the model you connected to earlier. +1. From the **{{< fa cubes >}} Inventory** in the {{< var validmind.platform >}}, go to the model you connected to earlier. 1. In the left sidebar that appears for your model, click **Validation** under {{< fa file >}} Documents. -1. Locate the Data Preparation section and click on **2.2.2. Model Performance** to expand that section. - -1. Under the Model Performance Metrics section, locate Artifacts then click **Link Artifact to Report**. +1. Click **2.2.2. Model Performance** to expand that section. -1. Select **Validation Issue** as the [type of artifact](/guide/validation/manage-artifact-types.qmd){target="_blank"}. +1. Under the Model Performance Metrics guideline, click to expand the **Artifacts** panel. -1. Click **{{< fa plus >}} Add Validation Issue** to add a validation issue type artifact. +1. Click **{{< fa link >}} Link Artifact** and select **Validation Issue** as the [type of artifact](/guide/validation/manage-artifact-types.qmd){target="_blank"}. -1. Enter in the details for your validation issue, for example: +1. Click **{{< fa plus >}} Add Validation Issue** and enter in the details for your validation issue, for example: - **[title]{.smallcaps}** — Champion Logistic Regression Model Fails Minimum Accuracy Threshold - **[risk area]{.smallcaps}** — Model Performance - **[documentation section]{.smallcaps}** — 3.2. Model Evaluation - **[description]{.smallcaps}** — The logistic regression champion model was subjected to a Minimum Accuracy test to determine whether its predictive accuracy meets the predefined performance threshold of 0.7. The model achieved an accuracy score of 0.6136, which falls below the required minimum. As a result, the test produced a Fail outcome. -1. Click **Save**. +1. Click **Add Validation Issue** to submit the validation issue. 1. Select the validation issue you just added to link to your validation report. diff --git a/site/guide/validation/_link-validator-evidence.qmd b/site/guide/validation/_link-validator-evidence.qmd deleted file mode 100644 index 8563d37877..0000000000 --- a/site/guide/validation/_link-validator-evidence.qmd +++ /dev/null @@ -1,68 +0,0 @@ - - -:::: {.content-visible unless-format="revealjs"} -::: {.callout} -To link validator evidence to a report, you must first log tests as a validator with the {{< var validmind.developer >}}.^[[{{< var validmind.developer >}}](/developer/validmind-library.qmd)] -::: - -1. In the left sidebar, click **{{< fa cubes >}} Inventory**. - -1. Select a record or find your record by applying a filter or searching for it.^[[Working with the inventory](/guide/inventory/working-with-the-inventory.qmd#search-filter-and-sort-records)] - -1. In the left sidebar that appears for your record, click **{{< fa file >}} Documents** and select the **Latest** tab.^[[Work with document versions](/guide/documentation/work-with-document-versions.qmd)] - -1. Click on a Validation type file.^[[Preparing validation reports](/guide/validation/preparing-validation-reports.qmd#validation-overview)] - - You can now expand any subsection of the validation report you would like to work with. - -1. In any section of the report where the button is available, click **{{< fa link >}} Link Evidence to Report**. - -1. On the **Link Validator Evidence to Validation Document** page that opens, select the evidence that is related to your assessment. - - If you are not sure if something is relevant, you can expand the section by clicking **{{< fa chevron-down >}}** for more details. - -1. Click **Update Linked Evidence**. - - - Confirm that the newly linked-to evidence shown under Developer Evidence is accurate. - - Evidence that needs your attention gets flagged with [{{< fa triangle-exclamation >}} Requires Attention]{.bubble .yellow-bg}. These sections get flagged automatically whenever a test result is above or below a certain threshold. - -:::: - - -:::: {.content-hidden unless-format="revealjs"} -With some test results logged, let's head to the model we connected to at the beginning of this notebook and insert our test results into the validation report as evidence. - -While the example below focuses on a specific test result, you can follow the same general procedure for your other results: - -::: {.panel-tabset} - -### 1. Link data quality test results -a. From the **{{< fa cubes >}} Inventory** in the {{< var validmind.platform >}}, go to the model you connected to earlier. - -a. In the left sidebar that appears for your model, click **Validation** under {{< fa file >}} Documents. - -a. Locate the Data Preparation section and click on **2.2.1. Data Quality** to expand that section. - -a. Under the Class Imbalance Assessment section, locate Validator Evidence then click **Link Evidence to Report**. - -a. Select the Class Imbalance test results we logged: **ValidMind Data Validation Class Imbalance** - - ![The ClassImbalance tests selected](/notebooks/tutorials/validation/selecting-class-imbalance-results.png){fig-alt="Screenshot showing the ClassImbalance tests selected" .screenshot} - -a. Click **Update Linked Evidence** to add the test results to the validation report. - - Confirm that the results for the Class Imbalance test you inserted has been correctly inserted into section **2.2.1. Data Quality** of the report. - -### 2. Review Class Imbalance test results - -- Once linked as evidence to section **2.2.1. Data Quality** note that the ValidMind Data Validation Class Imbalance test results are flagged as **Requires Attention** — as they include comparative results from our initial raw dataset. -- Click **See evidence details** to review the LLM-generated description that summarizes the test results, that confirm that our final preprocessed dataset actually passes our test: - - ![ClassImbalance test generated description in the text editor](/notebooks/tutorials/validation/class-imbalance-results-detail.png){fig-alt="Screenshot showing the ClassImbalance test generated description in the text editor" .screenshot} - -::: - - -:::: \ No newline at end of file diff --git a/site/guide/validation/_map-evidence-to-guidelines.qmd b/site/guide/validation/_map-evidence-to-guidelines.qmd index 2c0f7c9b3a..659b50b5d1 100644 --- a/site/guide/validation/_map-evidence-to-guidelines.qmd +++ b/site/guide/validation/_map-evidence-to-guidelines.qmd @@ -19,63 +19,102 @@ Map Evidence uses AI to suggest relevant evidence for each validation guideline, 4. Click on a Validation type file.^[[Preparing validation reports](/guide/validation/preparing-validation-reports.qmd#validation-overview)] -5. Navigate to a section and expand the **Evidence** panel. +5. You can map evidence at three levels: -6. Click **{{< fa wand-magic-sparkles >}} Map Evidence**. +::: {.panel-tabset} + +#### For the entire report + +Click **{{< fa wand-magic-sparkles >}} Map Evidence** in the right sidebar to expand the mapping panel for the entire report. + +#### For an entire section + +Sections include subsections with guidelines, and a risk assessment summary at the top of the page. + +a. Click on a section of the validation report to expand that section. +b. Click **{{< fa wand-magic-sparkles >}} Map Evidence** at the top of the page to open the mapping panel for that section. + +#### For individual guidelines + +Guidelines are organized into subsections under a parent section. + +a. Click on a section of the validation report to expand that section. +b. In any subsection, click **{{< fa wand-magic-sparkles >}} Map Evidence** to open the mapping panel for that guideline. + +::: -7. Configure the mapping options: - - Toggle **Developer Evidence** to include evidence logged via the {{< var validmind.developer >}}. - - Toggle **Validator Evidence** to include evidence uploaded or created by validators. - - Adjust the **Relevance Threshold** slider — lower values return more results while higher values show only the most relevant matches. +6. Configure the mapping options: -8. Click **Map Evidence** to run the AI mapping. + - Toggle the **[evidence type]{.smallcaps}** toggle to include evidence from the selected types: + - **Developer Evidence** — Include evidence logged on the Development document associated with the record.^[[Manage document types](/guide/templates/manage-document-types.qmd)] + - **Validator Evidence** — Include evidence logged on the current validation report. + - Adjust the **[relevance threshold]{.smallcaps}** slider — lower values return more results while higher values show only the most relevant matches. - The panel displays how many evidence items are available to review for each guideline in the section. +7. Click **Map Evidence** to start the mapping. + + If this is your first time running Map Evidence, review the information presented to you explaining the mapping process, then click **Map Evidence** to run the mapping.^[To hide the additional information on mapping modal for future runs, select the **Don't show this again** checkbox.] + +When mapping is complete, the [#]{.bubble} of evidence items to review is displayed. Review the evidence items and approve or reject the suggestions as needed. ### Review and approve mapped evidence -After running Map Evidence, you can review and approve suggestions at three levels: +After running Map Evidence, you can review and approve suggestions at four levels: ::: {.panel-tabset} #### For the entire report -a. Open the validation report and look at the right sidebar. +If you have run mapping for the entire report, you can review and approve suggestions at the entire report level. -b. The **Map Evidence** panel shows how many items need review across the entire report. +The **{{< fa wand-magic-sparkles >}} Map Evidence** panel in the right sidebar for your validation report shows how many items need review across the entire report — click to expand the panel if it is collapsed: -c. Use **Approve All** to link all suggested evidence across all guidelines, or **Reject All** to dismiss all suggestions. - -d. To re-run mapping with different settings, click **Remap Evidence**. This lets you adjust the relevance threshold or change which evidence types to include, then generate new suggestions. + - Select **[{{< fa check >}} Approve All]{.green}** to link all suggested evidence across all guidelines, or **[{{< fa xmark >}} Reject All]{.red}** to dismiss all suggestions. + - Click into any section of the report with approved mapped evidence to confirm that any approved evidence appears in the Evidence panel for guidelines as expected. + - To re-run mapping with different settings, click **{{< fa sync >}}Remap Evidence**. + - (Optional) Adjust the relevance threshold or change which evidence types to include, then generate new suggestions. #### For an entire section -a. Navigate to a specific section in the validation report. +Sections include subsections with guidelines, and a risk assessment summary at the top of the page. If you have run mapping for an entire section, you can review and approve suggestions at the section level: -b. In the section header, click **{{< fa wand-magic-sparkles >}} Map Evidence** to open the mapping panel. +a. Click on a section of the validation report to expand that section. -c. Use **Approve All** to link all suggested evidence for guidelines in that section, or **Reject All** to dismiss all section suggestions. +b. Click **{{< fa wand-magic-sparkles >}} Map Evidence** at the top of the page to open the mapping panel for that section: -d. To re-run mapping with different settings, click **Remap Evidence**. + - Select **[{{< fa check >}} Approve All]{.green}** to link all suggested evidence for guidelines in that section, or **[{{< fa xmark >}} Reject All]{.red}** to dismiss all section suggestions. + - Confirm that any approved evidence appears in the Evidence panels for the guidelines in that section as expected. + - To re-run mapping with different settings, click **{{< fa sync >}} Remap Evidence**. + - (Optional) Adjust the relevance threshold or change which evidence types to include, then generate new suggestions. #### For individual guidelines -a. Navigate to a specific section in the validation report. +Guidelines are organized into subsections under a parent section. If you have run mapping for an individual guideline, you can review and approve suggestions at the guideline level: + +a. Click on a section of the validation report to expand that section. + +b. In any subsection, click **{{< fa wand-magic-sparkles >}} Map Evidence** to open the mapping panel for that guideline: + + - Select **[{{< fa check >}} Approve All]{.green}** to link all suggested evidence for that guideline, or **[{{< fa xmark >}} Reject All]{.red}** to dismiss all guideline suggestions. + - Confirm that any approved evidence appears in the Evidence panel for that guideline as expected. + - To re-run mapping with different settings, click **{{< fa sync >}} Remap Evidence**. + - (Optional) Adjust the relevance threshold or change which evidence types to include, then generate new suggestions. + +#### For individual suggestions -b. Expand the **Evidence** panel for a guideline. +If you have run mapping at any level, you can review and approve individual suggestions at the individual suggestion level: -c. Click **{{< fa wand-magic-sparkles >}} Map Evidence** to open the mapping panel for that guideline. +a. Click on a section of the validation report to expand that section. -d. Review individual evidence suggestions: +b. In any subsection, click **Evidence** to expand the evidence panel for that guideline. + +c. Review individual evidence suggestions on the **Developer Evidence** or **Validator Evidence** tabs: - Each item shows the evidence block name and a relevance score. - Click **See Relevance Analysis** to view why the evidence was suggested. - - Click **Approve** to link an individual item to the guideline. - - Click **Reject** to dismiss an individual suggestion. - -e. Or use **Approve All** / **Reject All** to handle all suggestions for that guideline at once. + - Click **[{{< fa check >}} Approve]{.green}** to link an individual item to the guideline. + - Click **[{{< fa xmark >}} Reject]{.red}** to dismiss an individual suggestion. -Approved evidence appears in the Evidence panel for that guideline, organized by evidence type (Developer Evidence or Validator Evidence). +d. Confirm that any approved evidence appears in the Evidence panel for that guideline as expected. ::: @@ -86,59 +125,110 @@ Approved evidence appears in the Evidence panel for that guideline, organized by :::: {.content-hidden unless-format="revealjs"} Map Evidence uses AI to suggest relevant evidence for each validation guideline, helping you find and link supporting documentation from both developers and validators. -1. Navigate to a section and expand the **Evidence** panel. - -2. Click **{{< fa wand-magic-sparkles >}} Map Evidence**. +1. In the left sidebar, click **{{< fa cubes >}} Inventory**. -3. Configure the mapping options: - - Toggle **Developer Evidence** to include evidence logged via the {{< var validmind.developer >}}. - - Toggle **Validator Evidence** to include evidence uploaded or created by validators. - - Adjust the **Relevance Threshold** slider — lower values return more results while higher values show only the most relevant matches. +2. Select a record or find your record by [applying a filter or searching for it](/guide/inventory/working-with-the-inventory.qmd#search-filter-and-sort-records){target="_blank"}. -4. Click **Map Evidence** to run the AI mapping. +3. In the left sidebar that appears for your record, click **Validation** under **{{< fa file >}} Documents**. - The panel displays how many evidence items are available to review for each guideline in the section. +4. You can map evidence at three levels: -After running Map Evidence, you can review and approve suggestions at three levels: +::: {.panel-tabset} #### For the entire report -a. Open the validation report and look at the right sidebar. +Click **{{< fa wand-magic-sparkles >}} Map Evidence** in the right sidebar to expand the mapping panel for the entire report. -b. The **Map Evidence** panel shows how many items need review across the entire report. +#### For an entire section -c. Use **Approve All** to link all suggested evidence across all guidelines, or **Reject All** to dismiss all suggestions. +Sections include subsections with guidelines, and a risk assessment summary at the top of the page. -d. To re-run mapping with different settings, click **Remap Evidence**. This lets you adjust the relevance threshold or change which evidence types to include, then generate new suggestions. +a. Click on a section of the validation report to expand that section. +b. Click **{{< fa wand-magic-sparkles >}} Map Evidence** at the top of the page to open the mapping panel for that section. + +#### For individual guidelines + +Guidelines are organized into subsections under a parent section. + +a. Click on a section of the validation report to expand that section. +b. In any subsection, click **{{< fa wand-magic-sparkles >}} Map Evidence** to open the mapping panel for that guideline. + +::: + +6. Configure the mapping options: + + - Toggle the **[evidence type]{.smallcaps}** toggle to include evidence from the selected types: + - **Developer Evidence** — Include evidence logged on the [Development document associated with the record](/guide/templates/manage-document-types.qmd){target="_blank"}. + - **Validator Evidence** — Include evidence logged on the current validation report. + - Adjust the **[relevance threshold]{.smallcaps}** slider — lower values return more results while higher values show only the most relevant matches. + +7. Click **Map Evidence** to start the mapping. + + If this is your first time running Map Evidence, review the information presented to you explaining the mapping process, then click **Map Evidence** to run the mapping. + +When mapping is complete, the [#]{.bubble} of evidence items to review is displayed. Review the evidence items and approve or reject the suggestions as needed. + +#### Review and approve mapped evidence + +After running Map Evidence, you can review and approve suggestions at four levels: + +::: {.panel-tabset} + +#### For the entire report + +If you have run mapping for the entire report, you can review and approve suggestions at the entire report level. + +The **{{< fa wand-magic-sparkles >}} Map Evidence** panel in the right sidebar for your validation report shows how many items need review across the entire report — click to expand the panel if it is collapsed: + + - Select **[{{< fa check >}} Approve All]{.green}** to link all suggested evidence across all guidelines, or **[{{< fa xmark >}} Reject All]{.red}** to dismiss all suggestions. + - Click into any section of the report with approved mapped evidence to confirm that any approved evidence appears in the Evidence panel for guidelines as expected. + - To re-run mapping with different settings, click **{{< fa sync >}}Remap Evidence**. + - (Optional) Adjust the relevance threshold or change which evidence types to include, then generate new suggestions. #### For an entire section -a. Navigate to a specific section in the validation report. +Sections include subsections with guidelines, and a risk assessment summary at the top of the page. If you have run mapping for an entire section, you can review and approve suggestions at the section level: -b. In the section header, click **{{< fa wand-magic-sparkles >}} Map Evidence** to open the mapping panel. +a. Click on a section of the validation report to expand that section. -c. Use **Approve All** to link all suggested evidence for guidelines in that section, or **Reject All** to dismiss all section suggestions. +b. Click **{{< fa wand-magic-sparkles >}} Map Evidence** at the top of the page to open the mapping panel for that section: -d. To re-run mapping with different settings, click **Remap Evidence**. + - Select **[{{< fa check >}} Approve All]{.green}** to link all suggested evidence for guidelines in that section, or **[{{< fa xmark >}} Reject All]{.red}** to dismiss all section suggestions. + - Confirm that any approved evidence appears in the Evidence panels for the guidelines in that section as expected. + - To re-run mapping with different settings, click **{{< fa sync >}} Remap Evidence**. + - (Optional) Adjust the relevance threshold or change which evidence types to include, then generate new suggestions. #### For individual guidelines -a. Navigate to a specific section in the validation report. +Guidelines are organized into subsections under a parent section. If you have run mapping for an individual guideline, you can review and approve suggestions at the guideline level: -b. Expand the **Evidence** panel for a guideline. +a. Click on a section of the validation report to expand that section. -c. Click **{{< fa wand-magic-sparkles >}} Map Evidence** to open the mapping panel for that guideline. +b. In any subsection, click **{{< fa wand-magic-sparkles >}} Map Evidence** to open the mapping panel for that guideline: -d. Review individual evidence suggestions: + - Select **[{{< fa check >}} Approve All]{.green}** to link all suggested evidence for that guideline, or **[{{< fa xmark >}} Reject All]{.red}** to dismiss all guideline suggestions. + - Confirm that any approved evidence appears in the Evidence panel for that guideline as expected. + - To re-run mapping with different settings, click **{{< fa sync >}} Remap Evidence**. + - (Optional) Adjust the relevance threshold or change which evidence types to include, then generate new suggestions. + +#### For individual suggestions + +If you have run mapping at any level, you can review and approve individual suggestions at the individual suggestion level: + +a. Click on a section of the validation report to expand that section. + +b. In any subsection, click **Evidence** to expand the evidence panel for that guideline. + +c. Review individual evidence suggestions on the **Developer Evidence** or **Validator Evidence** tabs: - Each item shows the evidence block name and a relevance score. - Click **See Relevance Analysis** to view why the evidence was suggested. - - Click **Approve** to link an individual item to the guideline. - - Click **Reject** to dismiss an individual suggestion. + - Click **[{{< fa check >}} Approve]{.green}** to link an individual item to the guideline. + - Click **[{{< fa xmark >}} Reject]{.red}** to dismiss an individual suggestion. -e. Or use **Approve All** / **Reject All** to handle all suggestions for that guideline at once. +d. Confirm that any approved evidence appears in the Evidence panel for that guideline as expected. -Approved evidence appears in the Evidence panel for that guideline, organized by evidence type (Developer Evidence or Validator Evidence). +::: :::: diff --git a/site/guide/validation/add-manage-artifacts.qmd b/site/guide/validation/add-manage-artifacts.qmd index 0e7a43aac7..13e709f57b 100644 --- a/site/guide/validation/add-manage-artifacts.qmd +++ b/site/guide/validation/add-manage-artifacts.qmd @@ -1,4 +1,4 @@ - --- +--- # Copyright © 2023-2026 ValidMind Inc. All rights reserved. # Refer to the LICENSE file in the root of this repository for details. # SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial @@ -32,7 +32,7 @@ Add artifacts at the record or documentation section level, update your artifact ## Add artifacts on validation reports -When linking artifacts to validation reports,[^5] you can add artifacts as you assess a record's compliance: +While assessing a record's compliance,[^5] you can add artifacts as you complete your validation report: 1. In the left sidebar, click **{{< fa cubes >}} Inventory**. @@ -42,23 +42,27 @@ When linking artifacts to validation reports,[^5] you can add artifacts as you a 4. Click on a Validation type file.[^8] -5. In any section of the report where the button is available, click **{{< fa link >}} Link Artifact to Report** and select the artifact type[^9] you want to add: + Expand any subsection of the validation report you have attached evidence for to autogenerate validation issues or manually link artifacts to your report: + +::: {.panel-tabset} + +### Autogenerate validation issues - - Validation Issue - - Policy Exception - - Limitation +{{< include _autogenerate-validation-issues.qmd >}} -6. On the Link Artifact to Report modal that opens, click **{{< fa plus >}} Add {Artifact Type}**. +### Manually add artifacts + +{{< include _link-artifacts-to-reports.qmd >}} + +::: -7. On the modal that opens, provide information for your artifact. -8. When you are done, click **Add {Artifact Type}** to submit the artifact. ## Update artifacts As artifacts get resolved or require other changes during the validation process, you can update them: -1. Locate the artifact you want to update.[^10] +1. Locate the artifact you want to update.[^9] 2. Click on a field to make changes. @@ -101,16 +105,16 @@ Uploaded files must be less than 50 MB each in size. ## Delete artifacts ::: {.callout-important title="Artifact deletion is permanent."} -- While artifact deletion will be logged under your {{< fa wifi >}} Activity,[^11] deleted artifacts cannot be retrieved. -- Deleting an artifact will also remove it from any validation reports[^12] it was previously attached to. +- While artifact deletion will be logged under your {{< fa wifi >}} Activity,[^10] deleted artifacts cannot be retrieved. +- Deleting an artifact will also remove it from any validation reports[^11] it was previously attached to. ::: If you logged an artifact in error or otherwise no longer need to track that artifact, you can delete it: -1. Locate the artifact you want to delete.[^13] +1. Locate the artifact you want to delete.[^12] -2. On the artifact's detail page, click **{{< fa trash-can>}} Delete {Artifact Type}**[^14] in the right sidebar. +2. On the artifact's detail page, click **{{< fa trash-can>}} Delete {Artifact Type}**[^13] in the right sidebar. 3. Click **Yes, Delete {Artifact Type}** to confirm permanent deletion of that artifact. @@ -125,7 +129,7 @@ If you logged an artifact in error or otherwise no longer need to track that art [^4]: [Manage permissions](/guide/configuration/manage-permissions.qmd) -[^5]: [Link artifacts to reports](/guide/validation/assess-compliance.qmd#link-artifacts-to-reports) +[^5]: [Assess compliance](/guide/validation/assess-compliance.qmd#link-artifacts-to-reports) [^6]: [Working with the inventory](/guide/inventory/working-with-the-inventory.qmd#search-filter-and-sort-records) @@ -133,14 +137,12 @@ If you logged an artifact in error or otherwise no longer need to track that art [^8]: [Preparing validation reports](/guide/validation/preparing-validation-reports.qmd#validation-overview) -[^9]: [Manage artifact types](manage-artifact-types.qmd) - -[^10]: [View and filter artifacts](view-filter-artifacts.qmd) +[^9]: [View and filter artifacts](view-filter-artifacts.qmd) -[^11]: [View record activity](/guide/inventory/view-record-activity.qmd) +[^10]: [View record activity](/guide/inventory/view-record-activity.qmd) -[^12]: [Link artifacts to reports](/guide/validation/assess-compliance.qmd#link-artifacts-to-reports) +[^11]: [Link artifacts to reports](/guide/validation/assess-compliance.qmd#link-artifacts-to-reports) -[^13]: [View and filter artifacts](view-filter-artifacts.qmd) +[^12]: [View and filter artifacts](view-filter-artifacts.qmd) -[^14]: [Manage artifact types](manage-artifact-types.qmd) \ No newline at end of file +[^13]: [Manage artifact types](manage-artifact-types.qmd) \ No newline at end of file diff --git a/site/guide/validation/assess-compliance.qmd b/site/guide/validation/assess-compliance.qmd index e87cb793d1..b06fa534f0 100644 --- a/site/guide/validation/assess-compliance.qmd +++ b/site/guide/validation/assess-compliance.qmd @@ -7,6 +7,8 @@ date: last-modified aliases: - /guide/assess-compliance.html - /guide/model-validation/assess-compliance.html +includes: + compliance: true --- Use the {{< var validmind.platform >}} to assess compliance of your records with guidelines based on analyzing evidence and artifacts. @@ -15,8 +17,7 @@ Compliance assessments provide an unbiased starting point that enables more effi 1. [Link evidence to reports](#link-evidence-to-reports) 2. [Link artifacts to reports](#link-artifacts-to-reports) -3. [Autogenerate validation issues](#autogenerate-validation-issues) -4. [Provide compliance assessments](#provide-compliance-assessments) +3. [Assign compliance assessments](#assign-compliance-assessments) ::: {.callout title="Compliance assessments cannot be performed within static PDF documents."} You can only assess compliance within PDFs converted to editable documents,[^1] validation reports generated by the {{< var validmind.developer >}}, or validation reports created in the {{< var validmind.platform >}}. @@ -41,9 +42,7 @@ Use {{< var vm.product >}}'s AI-assisted tools to automate mapping and assessing ::: -### Link developer evidence - -To link evidence logged by developers^[[{{< var validmind.developer >}}](/developer/validmind-library.qmd)] to your validation report: +Include sections from other documents and logged test results as evidence to validation report, then assess the linked evidence for compliance: 1. In the left sidebar, click **{{< fa cubes >}} Inventory**. @@ -55,33 +54,98 @@ To link evidence logged by developers^[[{{< var validmind.developer >}}](/develo Expand any subsection of the validation report you would like to work with. -5. In any section of the report where the button is available, click **{{< fa link >}} Link Developer Evidence**. +5. Link evidence to reports in two steps: + a. [Map evidence](#map-evidence) + b. [Assess evidence](#assess-evidence) + +### a. Map evidence + + +1. In any section of the report where linking evidence is available, click **Evidence** to expand the evidence panel. + +2. Click **{{< fa link >}} Link Evidence** and select the [evidence type]{.smallcaps}: + + ::: {.panel-tabset} + + #### Developer Evidence + + Select evidence logged on Development, Monitoring, or custom document types attached to the record: + + i. Select **Developer Evidence** as the [evidence type]{.smallcaps}. + ii. In the left sidebar, click on the document you want to include evidence from. + iii. In the right panel, check off the content blocks and/or test results you want to include as evidence:[^10] + - Expand a section by clicking **{{< fa chevron-down >}}** to review any sub-sections. + - If you are not sure if a block is relevant, click on a block to preview the contents. + - To include all blocks in a section, check off the section header. + iv. When you are ready to insert the evidence, click **Update Linked Evidence**. + + #### Validator Evidence + + Select evidence logged on the same Validation report:[^11] -6. On the **Link Developer Evidence to Validation Document** page that opens, select the evidence that is related to your assessment. + i. Select **Validator Evidence** as the [evidence type]{.smallcaps}. + ii. Check off the test results you want to include as evidence. + If you are not sure if a test result is relevant, click on a test result to preview the contents. + iii. When you are ready to insert the evidence, click **Update Linked Evidence**. - - Available evidence is associated with the relevant documentation section. - - If you are not sure if something is relevant, you can expand the section by clicking **{{< fa chevron-down >}}** for more details. + ::: -7. Click **Update Linked Evidence**. +3. Confirm that the newly linked evidence appears in the Evidence panel: - - Confirm that the newly linked-to evidence shown under Developer Evidence is accurate and is connected to the correct documentation section. + ::: {.panel-tabset} + + #### Developer Evidence + + - Verify that the evidence shown under the **Developer Evidence** tab is accurate and is connected to the correct document section from the right document. + - Evidence that needs your attention gets flagged with [{{< fa triangle-exclamation >}} Requires Attention]{.bubble .yellow-bg}. These sections get flagged automatically whenever a test result is above or below a certain threshold. + + #### Validator Evidence + + - Verify that the evidence shown under the **Validator Evidence** tab is accurate and is connected to the correct test result. - Evidence that needs your attention gets flagged with [{{< fa triangle-exclamation >}} Requires Attention]{.bubble .yellow-bg}. These sections get flagged automatically whenever a test result is above or below a certain threshold. -### Link validator evidence + ::: -{{< include _link-validator-evidence.qmd >}} +### b. Assess evidence + + +Once you have linked evidence to a section of your report, assess the linked evidence for compliance:[^12] + +1. In any section of the report where evidence has been linked, click **Evidence Assessment** to expand the evidence assessment panel. + +2. Click **Add your assessment** to use the content editing toolbar[^13] to enter in your assessment notes. + +3. (Optional) When you are finished editing your assessment notes, hover over the content block and click **{{< fa unlock >}} Lock Assessment** to prevent additional changes. ## Link artifacts to reports -{{< include _link-artifacts-to-reports.qmd >}} +After you have mapped and assessed available evidence, link artifacts to reports to provide additional context and support your assessment: -## Autogenerate validation issues +1. In the left sidebar, click **{{< fa cubes >}} Inventory**. + +2. Select a record or find your record by applying a filter or searching for it.[^14] + +3. In the left sidebar that appears for your record, click **{{< fa file >}} Documents** and select the **Latest** tab.[^15] + +4. Click on a Validation type file.[^16] + + Expand any subsection of the validation report you have attached evidence for to autogenerate validation issues or manually link artifacts to your report: + +::: {.panel-tabset} + +### Autogenerate validation issues {{< include _autogenerate-validation-issues.qmd >}} -## Provide compliance assessments +### Manually link artifacts + +{{< include _link-artifacts-to-reports.qmd >}} + +::: + +## Assign compliance assessments -{{< include _provide-compliance-assessments.qmd >}} +{{< include _assign-compliance-assessments.qmd >}} @@ -103,3 +167,18 @@ To link evidence logged by developers^[[{{< var validmind.developer >}}](/develo [^8]: [Work with document versions](/guide/documentation/work-with-document-versions.qmd) [^9]: [Preparing validation reports](/guide/validation/preparing-validation-reports.qmd#validation-overview) + +[^10]: Only sections with populated content blocks and/or test results are available for selection as evidence — empty sections will have their checkboxes disabled and grayed out. + +[^11]: To link validator evidence to a report, you must first log tests on your report with the {{< var validmind.developer >}}: **[{{< var validmind.developer >}}](/developer/validmind-library.qmd)** + +[^12]: [a. Map evidence](#map-evidence) + +[^13]: [Work with content blocks](/guide/documentation/work-with-content-blocks.qmd#content-editing-toolbar) + +[^14]: [Working with the inventory](/guide/inventory/working-with-the-inventory.qmd#search-filter-and-sort-records) + +[^15]: [Work with document versions](/guide/documentation/work-with-document-versions.qmd) + +[^16]: [Preparing validation reports](/guide/validation/preparing-validation-reports.qmd#validation-overview) + diff --git a/site/guide/validation/map-and-assess-evidence.qmd b/site/guide/validation/map-and-assess-evidence.qmd index b6363e4810..9eb1abdd90 100644 --- a/site/guide/validation/map-and-assess-evidence.qmd +++ b/site/guide/validation/map-and-assess-evidence.qmd @@ -21,6 +21,13 @@ Use {{< var vm.product >}}'s AI-assisted tools to streamline your validation wor {{< include _how-does-mapping-assessment-work.qmd >}} +::: {.callout} +## Currently, automatic mapping and assessment retrieves only Developer Evidence from the default Development document on the record, or Validator Evidence logged on the Validation report you are mapping evidence for.[^4] + +To manually add and assess evidence from other document types, refer to [Assess compliance](/guide/validation/assess-compliance.qmd). + +::: + {{< include _map-evidence-to-guidelines.qmd >}} {{< include _assess-evidence-for-compliance.qmd >}} @@ -32,3 +39,5 @@ Use {{< var vm.product >}}'s AI-assisted tools to streamline your validation wor [^2]: [Submit for approval](/guide/documentation/submit-documents.qmd) [^3]: [Manage permissions](/guide/configuration/manage-permissions.qmd) + +[^4]: [Manage document types](/guide/templates/manage-document-types.qmd) diff --git a/site/notebooks.zip b/site/notebooks.zip index eac4468bc8..3706a851af 100644 Binary files a/site/notebooks.zip and b/site/notebooks.zip differ diff --git a/site/notebooks/EXECUTED/validation/2-start_validation_process.ipynb b/site/notebooks/EXECUTED/validation/2-start_validation_process.ipynb index 8d73210605..e729c2ee30 100644 --- a/site/notebooks/EXECUTED/validation/2-start_validation_process.ipynb +++ b/site/notebooks/EXECUTED/validation/2-start_validation_process.ipynb @@ -106,7 +106,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Make sure the ValidMind Library is installed\n", "\n", @@ -128,9 +130,7 @@ " # model=\"...\",\n", " document=\"validation-report\",\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -152,7 +152,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "from validmind.datasets.classification import customer_churn as demo_dataset\n", "\n", @@ -162,9 +164,7 @@ "\n", "raw_df = demo_dataset.load_data()\n", "raw_df.head()" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -195,23 +195,23 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Get the list of available task types\n", "sorted(vm.tests.list_tasks())" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Get the list of available tags\n", "sorted(vm.tests.list_tags())" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -224,12 +224,12 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "vm.tests.list_tests(task=\"classification\", tags=[\"tabular_data\", \"data_quality\"])" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -259,7 +259,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# vm_raw_dataset is now a VMDataset object that you can pass to any ValidMind test\n", "vm_raw_dataset = vm.init_dataset(\n", @@ -267,9 +269,7 @@ " input_id=\"raw_dataset\",\n", " target_column=\"Exited\",\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -302,12 +302,12 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "vm.tests.describe_test(\"validmind.data_validation.DescriptiveStatistics\")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -318,16 +318,16 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "result2 = vm.tests.run_test(\n", " test_id=\"validmind.data_validation.ClassImbalance\",\n", " inputs={\"dataset\": vm_raw_dataset},\n", " params={\"min_percent_threshold\": 30},\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -340,7 +340,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "import pandas as pd\n", "\n", @@ -352,9 +354,7 @@ "\n", "balanced_raw_df = pd.concat([exited_df, not_exited_df])\n", "balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -367,7 +367,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest\n", "vm_balanced_raw_dataset = vm.init_dataset(\n", @@ -375,13 +377,13 @@ " input_id=\"balanced_raw_dataset\",\n", " target_column=\"Exited\",\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Pass the initialized `balanced_raw_dataset` as input into the test run\n", "result = vm.tests.run_test(\n", @@ -389,9 +391,7 @@ " inputs={\"dataset\": vm_balanced_raw_dataset},\n", " params={\"min_percent_threshold\": 30},\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -410,16 +410,16 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "corr_result = vm.tests.run_test(\n", " test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n", " params={\"max_threshold\": 0.3},\n", " inputs={\"dataset\": vm_balanced_raw_dataset},\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -432,16 +432,16 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "print(type(corr_result))\n", "print(\"Result ID: \", corr_result.result_id)\n", "print(\"Params: \", corr_result.params)\n", "print(\"Passed: \", corr_result.passed)\n", "print(\"Tables: \", corr_result.tables)" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -454,25 +454,25 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Extract table from `corr_result.tables`\n", "features_df = corr_result.tables[0].data\n", "features_df" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Extract list of features that failed the test\n", "high_correlation_features = features_df[features_df[\"Pass/Fail\"] == \"Fail\"][\"Columns\"].tolist()\n", "high_correlation_features" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -483,13 +483,13 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "high_correlation_features = [feature.split(\",\")[0].strip(\"()\") for feature in high_correlation_features]\n", "high_correlation_features" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -502,7 +502,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Remove the highly correlated features from the dataset\n", "balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)\n", @@ -513,9 +515,7 @@ " input_id=\"raw_dataset_preprocessed\",\n", " target_column=\"Exited\",\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -526,16 +526,16 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "corr_result = vm.tests.run_test(\n", " test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n", " params={\"max_threshold\": 0.3},\n", " inputs={\"dataset\": vm_raw_dataset_preprocessed},\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -546,15 +546,15 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "corr_result = vm.tests.run_test(\n", " test_id=\"validmind.data_validation.PearsonCorrelationMatrix\",\n", " inputs={\"dataset\": vm_raw_dataset_preprocessed},\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -589,7 +589,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Individual test config with inputs specified\n", "test_config = {\n", @@ -602,9 +604,7 @@ " \"params\": {\"max_threshold\": 0.3}\n", " },\n", "}" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -615,7 +615,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "for t in test_config:\n", " print(t)\n", @@ -635,9 +637,7 @@ " vm.tests.run_test(t, inputs=test_config[t]['inputs']).log()\n", " except Exception as e:\n", " print(f\"Error running test {t}: {str(e)}\")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -666,7 +666,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "result = vm.tests.run_test(\n", " test_id=\"validmind.data_validation.HighPearsonCorrelation:balanced_raw_dataset\",\n", @@ -674,9 +676,7 @@ " inputs={\"dataset\": vm_balanced_raw_dataset},\n", ")\n", "result.log()" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -694,28 +694,23 @@ "\n", "2. In the left sidebar that appears for your model, click **Validation** under Documents.\n", "\n", - "3. Locate the Data Preparation section and click on **2.2.1. Data Quality** to expand that section.\n", + "3. Click on **2.2.1. Data Quality** to expand that section.\n", "\n", - "4. Under the Class Imbalance Assessment section, locate Validator Evidence then click **Link Evidence to Report**:\n", + "4. Under the Class Imbalance Assessment guideline, click **Evidence** to expand the evidence panel.\n", "\n", - " \"Screenshot\n", - "

\n", + "5. Click **Link Evidence**, then select **Validator Evidence**.\n", "\n", - "5. Select the Class Imbalance test results we logged: **ValidMind Data Validation Class Imbalance** \n", + "6. Select the Class Imbalance test results we logged: **ValidMind Data Validation Class Imbalance**\n", "\n", " \"Screenshot\n", "

\n", "\n", - "6. Click **Update Linked Evidence** to add the test results to the validation report.\n", + "7. Click **Update Linked Evidence** to add the test results to the validation report.\n", "\n", - " Confirm that the results for the Class Imbalance test you inserted has been correctly inserted into section **2.2.1. Data Quality** of the report:\n", + "8. Confirm that the results for the Class Imbalance test you inserted has been correctly inserted into section **2.2.1. Data Quality** of the report.\n", "\n", - " \"Screenshot\n", - "

\n", - "\n", - "7. Note that these test results are flagged as **Requires Attention** — as they include comparative results from our initial raw dataset.\n", - "\n", - " Click **See evidence details** to review the LLM-generated description that summarizes the test results, that confirm that our final preprocessed dataset actually passes our test:\n", + " - Note that these test results are flagged as **Requires Attention** — as they include comparative results from our initial raw dataset.\n", + " - Click **See evidence details** to review the LLM-generated description that summarizes the test results, that confirm that our final preprocessed dataset actually passes our test:\n", "\n", " \"Screenshot\n", "

\n", @@ -749,12 +744,12 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "balanced_raw_no_age_df.head()" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -768,15 +763,15 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "balanced_raw_no_age_df = pd.get_dummies(\n", " balanced_raw_no_age_df, columns=[\"Geography\", \"Gender\"], drop_first=True\n", ")\n", "balanced_raw_no_age_df.head()" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -790,7 +785,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", "\n", @@ -800,9 +797,7 @@ "y_train = train_df[\"Exited\"]\n", "X_test = test_df.drop(\"Exited\", axis=1)\n", "y_test = test_df[\"Exited\"]" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -817,7 +812,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "vm_train_ds = vm.init_dataset(\n", " input_id=\"train_dataset_final\",\n", @@ -830,9 +827,7 @@ " dataset=test_df,\n", " target_column=\"Exited\",\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", diff --git a/site/notebooks/EXECUTED/validation/3-developing_potential_challenger.ipynb b/site/notebooks/EXECUTED/validation/3-developing_potential_challenger.ipynb index ca9cb71f68..3c5db2507d 100644 --- a/site/notebooks/EXECUTED/validation/3-developing_potential_challenger.ipynb +++ b/site/notebooks/EXECUTED/validation/3-developing_potential_challenger.ipynb @@ -111,7 +111,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Make sure the ValidMind Library is installed\n", "\n", @@ -133,9 +135,7 @@ " # model=\"...\",\n", " document=\"validation-report\",\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -150,7 +150,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Load the sample dataset\n", "from validmind.datasets.classification import customer_churn as demo_dataset\n", @@ -160,9 +162,7 @@ ")\n", "\n", "raw_df = demo_dataset.load_data()" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -177,7 +177,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "import pandas as pd\n", "\n", @@ -189,9 +191,7 @@ "\n", "balanced_raw_df = pd.concat([exited_df, not_exited_df])\n", "balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -204,7 +204,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest\n", "vm_balanced_raw_dataset = vm.init_dataset(\n", @@ -212,9 +214,7 @@ " input_id=\"balanced_raw_dataset\",\n", " target_column=\"Exited\",\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -225,7 +225,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Run HighPearsonCorrelation test with our balanced dataset as input and return a result object\n", "corr_result = vm.tests.run_test(\n", @@ -233,42 +235,40 @@ " params={\"max_threshold\": 0.3},\n", " inputs={\"dataset\": vm_balanced_raw_dataset},\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# From result object, extract table from `corr_result.tables`\n", "features_df = corr_result.tables[0].data\n", "features_df" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Extract list of features that failed the test\n", "high_correlation_features = features_df[features_df[\"Pass/Fail\"] == \"Fail\"][\"Columns\"].tolist()\n", "high_correlation_features" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Extract feature names from the list of strings\n", "high_correlation_features = [feature.split(\",\")[0].strip(\"()\") for feature in high_correlation_features]\n", "high_correlation_features" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -279,7 +279,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Remove the highly correlated features from the dataset\n", "balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)\n", @@ -290,13 +292,13 @@ " input_id=\"raw_dataset_preprocessed\",\n", " target_column=\"Exited\",\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Re-run the test with the reduced feature set\n", "corr_result = vm.tests.run_test(\n", @@ -304,9 +306,7 @@ " params={\"max_threshold\": 0.3},\n", " inputs={\"dataset\": vm_raw_dataset_preprocessed},\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -321,20 +321,22 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Encode categorical features in the dataset\n", "balanced_raw_no_age_df = pd.get_dummies(\n", " balanced_raw_no_age_df, columns=[\"Geography\", \"Gender\"], drop_first=True\n", ")\n", "balanced_raw_no_age_df.head()" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", "\n", @@ -345,13 +347,13 @@ "y_train = train_df[\"Exited\"]\n", "X_test = test_df.drop(\"Exited\", axis=1)\n", "y_test = test_df[\"Exited\"]" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Initialize the split datasets\n", "vm_train_ds = vm.init_dataset(\n", @@ -365,9 +367,7 @@ " dataset=test_df,\n", " target_column=\"Exited\",\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -382,16 +382,16 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Import the champion model\n", "import pickle as pkl\n", "\n", "with open(\"lr_model_champion.pkl\", \"rb\") as f:\n", " log_reg = pkl.load(f)" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -421,7 +421,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Import the Random Forest Classification model\n", "from sklearn.ensemble import RandomForestClassifier\n", @@ -434,9 +436,7 @@ "\n", "# Train the model\n", "rf_model.fit(X_train, y_train)" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -456,7 +456,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Initialize the champion logistic regression model\n", "vm_log_model = vm.init_model(\n", @@ -469,9 +471,7 @@ " rf_model,\n", " input_id=\"rf_model\",\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -491,7 +491,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Champion — Logistic regression model\n", "vm_train_ds.assign_predictions(model=vm_log_model)\n", @@ -500,9 +502,7 @@ "# Challenger — Random forest classification model\n", "vm_train_ds.assign_predictions(model=vm_rf_model)\n", "vm_test_ds.assign_predictions(model=vm_rf_model)" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -530,13 +530,13 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "\n", "vm.tests.list_tests(tags=[\"model_performance\"], task=\"classification\")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -555,7 +555,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "mpt = [\n", " \"validmind.model_validation.sklearn.ClassifierPerformance:logreg_champion\",\n", @@ -564,9 +566,7 @@ " \"validmind.model_validation.sklearn.MinimumF1Score:logreg_champion\",\n", " \"validmind.model_validation.sklearn.ROCCurve:logreg_champion\"\n", "]" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -584,7 +584,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "for test in mpt:\n", " vm.tests.run_test(\n", @@ -593,9 +595,7 @@ " \"dataset\": vm_test_ds, \"model\" : vm_log_model,\n", " },\n", " ).log()" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -620,29 +620,28 @@ "\n", "2. In the left sidebar that appears for your model, click **Validation** under Documents.\n", "\n", - "3. Locate the Data Preparation section and click on **2.2.2. Model Performance** to expand that section.\n", - "\n", - "4. Under the Model Performance Metrics section, locate Artifacts then click **Link Artifact to Report**:\n", - "\n", - " \"Screenshot\n", - "

\n", + "3. Click on **2.2.2. Model Performance** to expand that section.\n", "\n", - "5. Select **Validation Issue** as the type of artifact.\n", + "4. Under the Model Performance Metrics guideline, click to expand the **Artifacts** panel.\n", "\n", - "6. Click **+ Add Validation Issue** to add a validation issue type artifact.\n", + "5. Click **Link Artifact** and select **Validation Issue** as the type of artifact.\n", "\n", - "76. Enter in the details for your validation issue, for example:\n", + "6. Click **+ Add Validation Issue** and enter in the details for your validation issue, for example:\n", "\n", " - **TITLE** — Champion Logistic Regression Model Fails Minimum Accuracy Threshold\n", " - **RISK AREA** — Model Performance\n", " - **DOCUMENTATION SECTION** — 3.2. Model Evaluation\n", " - **DESCRIPTION** — The logistic regression champion model was subjected to a Minimum Accuracy test to determine whether its predictive accuracy meets the predefined performance threshold of 0.7. The model achieved an accuracy score of 0.6136, which falls below the required minimum. As a result, the test produced a Fail outcome.\n", "\n", - "8. Click **Save**.\n", + "7. Click **Add Validation Issue** to submit the validation issue.\n", + "\n", + "8. Select the validation issue you just added to link to your validation report.\n", + "\n", + "9. Click **Update Linked Artifacts** to insert your validation issue.\n", "\n", - "9. Select the validation issue you just added to link to your validation report and click **Update Linked Artifacts** to insert your validation issue.\n", + "10. Confirm that validation issue you inserted has been correctly inserted into section 2.2.2. Model Performance of the report.\n", "\n", - "10. Click on the validation issue to expand the issue, where you can adjust details such as severity, owner, due date, status, etc. as well as include proposed remediation plans or supporting documentation as attachments." + "11. Click on the validation issue to expand the issue, where you can adjust details such as severity, owner, due date, status, etc. as well as include proposed remediation plans or supporting documentation as attachments." ] }, { @@ -660,7 +659,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "mpt_chall = [\n", " \"validmind.model_validation.sklearn.ClassifierPerformance:champion_vs_challenger\",\n", @@ -669,9 +670,7 @@ " \"validmind.model_validation.sklearn.MinimumF1Score:champion_vs_challenger\",\n", " \"validmind.model_validation.sklearn.ROCCurve:champion_vs_challenger\"\n", "]" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -682,7 +681,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "for test in mpt_chall:\n", " vm.tests.run_test(\n", @@ -691,9 +692,7 @@ " \"dataset\": [vm_test_ds], \"model\" : [vm_log_model,vm_rf_model]\n", " }\n", " ).log()" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -719,12 +718,12 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "vm.tests.list_tests(tags=[\"model_diagnosis\"], task=\"classification\")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -740,7 +739,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "vm.tests.run_test(\n", " test_id=\"validmind.model_validation.sklearn.OverfitDiagnosis:champion_vs_challenger\",\n", @@ -749,9 +750,7 @@ " \"model\" : [vm_log_model,vm_rf_model]\n", " }\n", ").log()" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -764,7 +763,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "vm.tests.run_test(\n", " test_id=\"validmind.model_validation.sklearn.RobustnessDiagnosis:Champion_vs_LogRegression\",\n", @@ -773,9 +774,7 @@ " \"model\" : [vm_log_model,vm_rf_model]\n", " },\n", ").log()" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -792,14 +791,14 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Store the feature importance tests\n", "FI = vm.tests.list_tests(tags=[\"feature_importance\"], task=\"classification\",pretty=False)\n", "FI" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -810,7 +809,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Run and log our feature importance tests for both models for the testing dataset\n", "for test in FI:\n", @@ -820,9 +821,7 @@ " \"dataset\": [vm_test_ds], \"model\" : [vm_log_model,vm_rf_model]\n", " },\n", " ).log()" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", diff --git a/site/notebooks/EXECUTED/validation/inserted-class-imbalance-results.png b/site/notebooks/EXECUTED/validation/inserted-class-imbalance-results.png deleted file mode 100644 index 2efea1a09b..0000000000 Binary files a/site/notebooks/EXECUTED/validation/inserted-class-imbalance-results.png and /dev/null differ diff --git a/site/notebooks/EXECUTED/validation/link-artifact.png b/site/notebooks/EXECUTED/validation/link-artifact.png deleted file mode 100644 index 23edc1c9eb..0000000000 Binary files a/site/notebooks/EXECUTED/validation/link-artifact.png and /dev/null differ diff --git a/site/notebooks/EXECUTED/validation/link-validator-evidence.png b/site/notebooks/EXECUTED/validation/link-validator-evidence.png deleted file mode 100644 index 660a19cfab..0000000000 Binary files a/site/notebooks/EXECUTED/validation/link-validator-evidence.png and /dev/null differ diff --git a/site/notebooks/how_to/tests/custom_tests/implement_custom_tests.ipynb b/site/notebooks/how_to/tests/custom_tests/implement_custom_tests.ipynb index b7169e96b9..cd8af2d278 100644 --- a/site/notebooks/how_to/tests/custom_tests/implement_custom_tests.ipynb +++ b/site/notebooks/how_to/tests/custom_tests/implement_custom_tests.ipynb @@ -92,7 +92,7 @@ "\n", "### Key concepts\n", "\n", - "**Model documentation**: A structured and detailed record pertaining to a model, encompassing key components such as its underlying assumptions, methodologies, data sources, inputs, performance metrics, evaluations, limitations, and intended uses. It serves to ensure transparency, adherence to regulatory requirements, and a clear understanding of potential risks associated with the model’s application.\n", + "**Model documentation**: A structured and detailed record pertaining to a model, encompassing key components such as its underlying assumptions, methodologies, data sources, inputs, performance metrics, evaluations, limitations, and intended uses. It serves to ensure transparency, adherence to regulatory requirements, and a clear understanding of potential risks associated with the model\u2019s application.\n", "\n", "**Documentation template**: Functions as a test suite and lays out the structure of model documentation, segmented into various sections and sub-sections. Documentation templates define the structure of your model documentation, specifying the tests that should be run, and how the results should be displayed.\n", "\n", @@ -807,9 +807,14 @@ " \"My Cool Table\": table,\n", " \"Another Table\": table2,\n", " },\n", - " fig1,\n", - " fig2,\n", - " fig3,\n", + " {\n", + " # Figures support the same dict-of-titles convention as tables.\n", + " # These titles flow into the document media registry as\n", + " # \"Figure N. \" alongside table captions.\n", + " \"Random Line Plot\": fig1,\n", + " \"Random Bar Plot\": fig2,\n", + " \"Random Scatter Plot\": fig3,\n", + " },\n", " )\n", "\n", "\n", @@ -821,7 +826,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Notice how you can return the tables as a dictionary where the key is the title of the table and the value is the table itself. You could also just return the tables by themselves but this way you can give them a title to more easily identify them in the result.\n", + "Notice how you can return the tables as a dictionary where the key is the title of the table and the value is the table itself. The same convention works for **figures** \u2014 wrap them in a dict whose keys are the titles you want shown in the document media registry (e.g. *Figure 7. Random Line Plot*). You could also just return the figures by themselves but this way you can give them a title to more easily identify them in the result.\n", "\n", "![screenshot showing multiple tables and plots](./multiple-tables-plots-custom-metric.png)" ] @@ -996,7 +1001,7 @@ "\n", "## Next steps\n", "\n", - "You can look at the results of this test suite right in the notebook where you ran the code, as you would expect. But there is a better way — use the ValidMind Platform to work with your model documentation.\n", + "You can look at the results of this test suite right in the notebook where you ran the code, as you would expect. But there is a better way \u2014 use the ValidMind Platform to work with your model documentation.\n", "\n", "<a id='toc8_1__'></a>\n", "\n", @@ -1029,7 +1034,7 @@ "\n", "## Upgrade ValidMind\n", "\n", - "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\">After installing ValidMind, you’ll want to periodically make sure you are on the latest version to access any new features and other enhancements.</div>\n", + "<div class=\"alert alert-block alert-info\" style=\"background-color: #B5B5B510; color: black; border: 1px solid #083E44; border-left-width: 5px; box-shadow: 2px 2px 4px rgba(0, 0, 0, 0.2);border-radius: 5px;\">After installing ValidMind, you\u2019ll want to periodically make sure you are on the latest version to access any new features and other enhancements.</div>\n", "\n", "Retrieve the information for the currently installed version of ValidMind:" ] @@ -1072,7 +1077,7 @@ "\n", "***\n", "\n", - "Copyright © 2023-2026 ValidMind Inc. All rights reserved.<br>\n", + "Copyright \u00a9 2023-2026 ValidMind Inc. All rights reserved.<br>\n", "Refer to [LICENSE](https://github.com/validmind/validmind-library/blob/main/LICENSE) for details.<br>\n", "SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial</small>" ] diff --git a/site/notebooks/tutorials/validation/2-start_validation_process.ipynb b/site/notebooks/tutorials/validation/2-start_validation_process.ipynb index 8d73210605..e729c2ee30 100644 --- a/site/notebooks/tutorials/validation/2-start_validation_process.ipynb +++ b/site/notebooks/tutorials/validation/2-start_validation_process.ipynb @@ -106,7 +106,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Make sure the ValidMind Library is installed\n", "\n", @@ -128,9 +130,7 @@ " # model=\"...\",\n", " document=\"validation-report\",\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -152,7 +152,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "from validmind.datasets.classification import customer_churn as demo_dataset\n", "\n", @@ -162,9 +164,7 @@ "\n", "raw_df = demo_dataset.load_data()\n", "raw_df.head()" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -195,23 +195,23 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Get the list of available task types\n", "sorted(vm.tests.list_tasks())" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Get the list of available tags\n", "sorted(vm.tests.list_tags())" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -224,12 +224,12 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "vm.tests.list_tests(task=\"classification\", tags=[\"tabular_data\", \"data_quality\"])" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -259,7 +259,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# vm_raw_dataset is now a VMDataset object that you can pass to any ValidMind test\n", "vm_raw_dataset = vm.init_dataset(\n", @@ -267,9 +269,7 @@ " input_id=\"raw_dataset\",\n", " target_column=\"Exited\",\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -302,12 +302,12 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "vm.tests.describe_test(\"validmind.data_validation.DescriptiveStatistics\")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -318,16 +318,16 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "result2 = vm.tests.run_test(\n", " test_id=\"validmind.data_validation.ClassImbalance\",\n", " inputs={\"dataset\": vm_raw_dataset},\n", " params={\"min_percent_threshold\": 30},\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -340,7 +340,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "import pandas as pd\n", "\n", @@ -352,9 +354,7 @@ "\n", "balanced_raw_df = pd.concat([exited_df, not_exited_df])\n", "balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -367,7 +367,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest\n", "vm_balanced_raw_dataset = vm.init_dataset(\n", @@ -375,13 +377,13 @@ " input_id=\"balanced_raw_dataset\",\n", " target_column=\"Exited\",\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Pass the initialized `balanced_raw_dataset` as input into the test run\n", "result = vm.tests.run_test(\n", @@ -389,9 +391,7 @@ " inputs={\"dataset\": vm_balanced_raw_dataset},\n", " params={\"min_percent_threshold\": 30},\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -410,16 +410,16 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "corr_result = vm.tests.run_test(\n", " test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n", " params={\"max_threshold\": 0.3},\n", " inputs={\"dataset\": vm_balanced_raw_dataset},\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -432,16 +432,16 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "print(type(corr_result))\n", "print(\"Result ID: \", corr_result.result_id)\n", "print(\"Params: \", corr_result.params)\n", "print(\"Passed: \", corr_result.passed)\n", "print(\"Tables: \", corr_result.tables)" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -454,25 +454,25 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Extract table from `corr_result.tables`\n", "features_df = corr_result.tables[0].data\n", "features_df" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Extract list of features that failed the test\n", "high_correlation_features = features_df[features_df[\"Pass/Fail\"] == \"Fail\"][\"Columns\"].tolist()\n", "high_correlation_features" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -483,13 +483,13 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "high_correlation_features = [feature.split(\",\")[0].strip(\"()\") for feature in high_correlation_features]\n", "high_correlation_features" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -502,7 +502,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Remove the highly correlated features from the dataset\n", "balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)\n", @@ -513,9 +515,7 @@ " input_id=\"raw_dataset_preprocessed\",\n", " target_column=\"Exited\",\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -526,16 +526,16 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "corr_result = vm.tests.run_test(\n", " test_id=\"validmind.data_validation.HighPearsonCorrelation\",\n", " params={\"max_threshold\": 0.3},\n", " inputs={\"dataset\": vm_raw_dataset_preprocessed},\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -546,15 +546,15 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "corr_result = vm.tests.run_test(\n", " test_id=\"validmind.data_validation.PearsonCorrelationMatrix\",\n", " inputs={\"dataset\": vm_raw_dataset_preprocessed},\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -589,7 +589,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Individual test config with inputs specified\n", "test_config = {\n", @@ -602,9 +604,7 @@ " \"params\": {\"max_threshold\": 0.3}\n", " },\n", "}" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -615,7 +615,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "for t in test_config:\n", " print(t)\n", @@ -635,9 +637,7 @@ " vm.tests.run_test(t, inputs=test_config[t]['inputs']).log()\n", " except Exception as e:\n", " print(f\"Error running test {t}: {str(e)}\")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -666,7 +666,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "result = vm.tests.run_test(\n", " test_id=\"validmind.data_validation.HighPearsonCorrelation:balanced_raw_dataset\",\n", @@ -674,9 +676,7 @@ " inputs={\"dataset\": vm_balanced_raw_dataset},\n", ")\n", "result.log()" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -694,28 +694,23 @@ "\n", "2. In the left sidebar that appears for your model, click **Validation** under Documents.\n", "\n", - "3. Locate the Data Preparation section and click on **2.2.1. Data Quality** to expand that section.\n", + "3. Click on **2.2.1. Data Quality** to expand that section.\n", "\n", - "4. Under the Class Imbalance Assessment section, locate Validator Evidence then click **Link Evidence to Report**:\n", + "4. Under the Class Imbalance Assessment guideline, click **Evidence** to expand the evidence panel.\n", "\n", - " <img src= \"link-validator-evidence.png\" alt=\"Screenshot showing the validation report with the link validator evidence to report option highlighted\" style=\"border: 2px solid #083E44; border-radius: 8px; border-right-width: 2px; border-bottom-width: 3px;\">\n", - " <br><br>\n", + "5. Click **Link Evidence**, then select **Validator Evidence**.\n", "\n", - "5. Select the Class Imbalance test results we logged: **ValidMind Data Validation Class Imbalance** \n", + "6. Select the Class Imbalance test results we logged: **ValidMind Data Validation Class Imbalance**\n", "\n", " <img src= \"selecting-class-imbalance-results.png\" alt=\"Screenshot showing the ClassImbalance test selected\" style=\"border: 2px solid #083E44; border-radius: 8px; border-right-width: 2px; border-bottom-width: 3px;\">\n", " <br><br>\n", "\n", - "6. Click **Update Linked Evidence** to add the test results to the validation report.\n", + "7. Click **Update Linked Evidence** to add the test results to the validation report.\n", "\n", - " Confirm that the results for the Class Imbalance test you inserted has been correctly inserted into section **2.2.1. Data Quality** of the report:\n", + "8. Confirm that the results for the Class Imbalance test you inserted has been correctly inserted into section **2.2.1. Data Quality** of the report.\n", "\n", - " <img src= \"inserted-class-imbalance-results.png\" alt=\"Screenshot showing the ClassImbalance test inserted into the validation report\" style=\"border: 2px solid #083E44; border-radius: 8px; border-right-width: 2px; border-bottom-width: 3px;\">\n", - " <br><br>\n", - "\n", - "7. Note that these test results are flagged as **Requires Attention** — as they include comparative results from our initial raw dataset.\n", - "\n", - " Click **See evidence details** to review the LLM-generated description that summarizes the test results, that confirm that our final preprocessed dataset actually passes our test:\n", + " - Note that these test results are flagged as **Requires Attention** — as they include comparative results from our initial raw dataset.\n", + " - Click **See evidence details** to review the LLM-generated description that summarizes the test results, that confirm that our final preprocessed dataset actually passes our test:\n", "\n", " <img src= \"class-imbalance-results-detail.png\" alt=\"Screenshot showing the ClassImbalance test generated description in the text editor\" style=\"border: 2px solid #083E44; border-radius: 8px; border-right-width: 2px; border-bottom-width: 3px;\">\n", " <br><br>\n", @@ -749,12 +744,12 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "balanced_raw_no_age_df.head()" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -768,15 +763,15 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "balanced_raw_no_age_df = pd.get_dummies(\n", " balanced_raw_no_age_df, columns=[\"Geography\", \"Gender\"], drop_first=True\n", ")\n", "balanced_raw_no_age_df.head()" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -790,7 +785,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", "\n", @@ -800,9 +797,7 @@ "y_train = train_df[\"Exited\"]\n", "X_test = test_df.drop(\"Exited\", axis=1)\n", "y_test = test_df[\"Exited\"]" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -817,7 +812,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "vm_train_ds = vm.init_dataset(\n", " input_id=\"train_dataset_final\",\n", @@ -830,9 +827,7 @@ " dataset=test_df,\n", " target_column=\"Exited\",\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", diff --git a/site/notebooks/tutorials/validation/3-developing_potential_challenger.ipynb b/site/notebooks/tutorials/validation/3-developing_potential_challenger.ipynb index ca9cb71f68..3c5db2507d 100644 --- a/site/notebooks/tutorials/validation/3-developing_potential_challenger.ipynb +++ b/site/notebooks/tutorials/validation/3-developing_potential_challenger.ipynb @@ -111,7 +111,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Make sure the ValidMind Library is installed\n", "\n", @@ -133,9 +135,7 @@ " # model=\"...\",\n", " document=\"validation-report\",\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -150,7 +150,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Load the sample dataset\n", "from validmind.datasets.classification import customer_churn as demo_dataset\n", @@ -160,9 +162,7 @@ ")\n", "\n", "raw_df = demo_dataset.load_data()" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -177,7 +177,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "import pandas as pd\n", "\n", @@ -189,9 +191,7 @@ "\n", "balanced_raw_df = pd.concat([exited_df, not_exited_df])\n", "balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -204,7 +204,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest\n", "vm_balanced_raw_dataset = vm.init_dataset(\n", @@ -212,9 +214,7 @@ " input_id=\"balanced_raw_dataset\",\n", " target_column=\"Exited\",\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -225,7 +225,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Run HighPearsonCorrelation test with our balanced dataset as input and return a result object\n", "corr_result = vm.tests.run_test(\n", @@ -233,42 +235,40 @@ " params={\"max_threshold\": 0.3},\n", " inputs={\"dataset\": vm_balanced_raw_dataset},\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# From result object, extract table from `corr_result.tables`\n", "features_df = corr_result.tables[0].data\n", "features_df" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Extract list of features that failed the test\n", "high_correlation_features = features_df[features_df[\"Pass/Fail\"] == \"Fail\"][\"Columns\"].tolist()\n", "high_correlation_features" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Extract feature names from the list of strings\n", "high_correlation_features = [feature.split(\",\")[0].strip(\"()\") for feature in high_correlation_features]\n", "high_correlation_features" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -279,7 +279,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Remove the highly correlated features from the dataset\n", "balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)\n", @@ -290,13 +292,13 @@ " input_id=\"raw_dataset_preprocessed\",\n", " target_column=\"Exited\",\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Re-run the test with the reduced feature set\n", "corr_result = vm.tests.run_test(\n", @@ -304,9 +306,7 @@ " params={\"max_threshold\": 0.3},\n", " inputs={\"dataset\": vm_raw_dataset_preprocessed},\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -321,20 +321,22 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Encode categorical features in the dataset\n", "balanced_raw_no_age_df = pd.get_dummies(\n", " balanced_raw_no_age_df, columns=[\"Geography\", \"Gender\"], drop_first=True\n", ")\n", "balanced_raw_no_age_df.head()" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", "\n", @@ -345,13 +347,13 @@ "y_train = train_df[\"Exited\"]\n", "X_test = test_df.drop(\"Exited\", axis=1)\n", "y_test = test_df[\"Exited\"]" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Initialize the split datasets\n", "vm_train_ds = vm.init_dataset(\n", @@ -365,9 +367,7 @@ " dataset=test_df,\n", " target_column=\"Exited\",\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -382,16 +382,16 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Import the champion model\n", "import pickle as pkl\n", "\n", "with open(\"lr_model_champion.pkl\", \"rb\") as f:\n", " log_reg = pkl.load(f)" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -421,7 +421,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Import the Random Forest Classification model\n", "from sklearn.ensemble import RandomForestClassifier\n", @@ -434,9 +436,7 @@ "\n", "# Train the model\n", "rf_model.fit(X_train, y_train)" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -456,7 +456,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Initialize the champion logistic regression model\n", "vm_log_model = vm.init_model(\n", @@ -469,9 +471,7 @@ " rf_model,\n", " input_id=\"rf_model\",\n", ")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -491,7 +491,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Champion — Logistic regression model\n", "vm_train_ds.assign_predictions(model=vm_log_model)\n", @@ -500,9 +502,7 @@ "# Challenger — Random forest classification model\n", "vm_train_ds.assign_predictions(model=vm_rf_model)\n", "vm_test_ds.assign_predictions(model=vm_rf_model)" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -530,13 +530,13 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "\n", "vm.tests.list_tests(tags=[\"model_performance\"], task=\"classification\")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -555,7 +555,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "mpt = [\n", " \"validmind.model_validation.sklearn.ClassifierPerformance:logreg_champion\",\n", @@ -564,9 +566,7 @@ " \"validmind.model_validation.sklearn.MinimumF1Score:logreg_champion\",\n", " \"validmind.model_validation.sklearn.ROCCurve:logreg_champion\"\n", "]" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -584,7 +584,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "for test in mpt:\n", " vm.tests.run_test(\n", @@ -593,9 +595,7 @@ " \"dataset\": vm_test_ds, \"model\" : vm_log_model,\n", " },\n", " ).log()" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -620,29 +620,28 @@ "\n", "2. In the left sidebar that appears for your model, click **Validation** under Documents.\n", "\n", - "3. Locate the Data Preparation section and click on **2.2.2. Model Performance** to expand that section.\n", - "\n", - "4. Under the Model Performance Metrics section, locate Artifacts then click **Link Artifact to Report**:\n", - "\n", - " <img src=\"link-artifact.png\" alt=\"Screenshot showing the validation report with the link artifact option highlighted\" style=\"border: 2px solid #083E44; border-radius: 8px; border-right-width: 2px; border-bottom-width: 3px;\">\n", - " <br><br>\n", + "3. Click on **2.2.2. Model Performance** to expand that section.\n", "\n", - "5. Select **Validation Issue** as the type of artifact.\n", + "4. Under the Model Performance Metrics guideline, click to expand the **Artifacts** panel.\n", "\n", - "6. Click **+ Add Validation Issue** to add a validation issue type artifact.\n", + "5. Click **Link Artifact** and select **Validation Issue** as the type of artifact.\n", "\n", - "76. Enter in the details for your validation issue, for example:\n", + "6. Click **+ Add Validation Issue** and enter in the details for your validation issue, for example:\n", "\n", " - **TITLE** — Champion Logistic Regression Model Fails Minimum Accuracy Threshold\n", " - **RISK AREA** — Model Performance\n", " - **DOCUMENTATION SECTION** — 3.2. Model Evaluation\n", " - **DESCRIPTION** — The logistic regression champion model was subjected to a Minimum Accuracy test to determine whether its predictive accuracy meets the predefined performance threshold of 0.7. The model achieved an accuracy score of 0.6136, which falls below the required minimum. As a result, the test produced a Fail outcome.\n", "\n", - "8. Click **Save**.\n", + "7. Click **Add Validation Issue** to submit the validation issue.\n", + "\n", + "8. Select the validation issue you just added to link to your validation report.\n", + "\n", + "9. Click **Update Linked Artifacts** to insert your validation issue.\n", "\n", - "9. Select the validation issue you just added to link to your validation report and click **Update Linked Artifacts** to insert your validation issue.\n", + "10. Confirm that validation issue you inserted has been correctly inserted into section 2.2.2. Model Performance of the report.\n", "\n", - "10. Click on the validation issue to expand the issue, where you can adjust details such as severity, owner, due date, status, etc. as well as include proposed remediation plans or supporting documentation as attachments." + "11. Click on the validation issue to expand the issue, where you can adjust details such as severity, owner, due date, status, etc. as well as include proposed remediation plans or supporting documentation as attachments." ] }, { @@ -660,7 +659,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "mpt_chall = [\n", " \"validmind.model_validation.sklearn.ClassifierPerformance:champion_vs_challenger\",\n", @@ -669,9 +670,7 @@ " \"validmind.model_validation.sklearn.MinimumF1Score:champion_vs_challenger\",\n", " \"validmind.model_validation.sklearn.ROCCurve:champion_vs_challenger\"\n", "]" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -682,7 +681,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "for test in mpt_chall:\n", " vm.tests.run_test(\n", @@ -691,9 +692,7 @@ " \"dataset\": [vm_test_ds], \"model\" : [vm_log_model,vm_rf_model]\n", " }\n", " ).log()" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -719,12 +718,12 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "vm.tests.list_tests(tags=[\"model_diagnosis\"], task=\"classification\")" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -740,7 +739,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "vm.tests.run_test(\n", " test_id=\"validmind.model_validation.sklearn.OverfitDiagnosis:champion_vs_challenger\",\n", @@ -749,9 +750,7 @@ " \"model\" : [vm_log_model,vm_rf_model]\n", " }\n", ").log()" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -764,7 +763,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "vm.tests.run_test(\n", " test_id=\"validmind.model_validation.sklearn.RobustnessDiagnosis:Champion_vs_LogRegression\",\n", @@ -773,9 +774,7 @@ " \"model\" : [vm_log_model,vm_rf_model]\n", " },\n", ").log()" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -792,14 +791,14 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Store the feature importance tests\n", "FI = vm.tests.list_tests(tags=[\"feature_importance\"], task=\"classification\",pretty=False)\n", "FI" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", @@ -810,7 +809,9 @@ }, { "cell_type": "code", + "execution_count": null, "metadata": {}, + "outputs": [], "source": [ "# Run and log our feature importance tests for both models for the testing dataset\n", "for test in FI:\n", @@ -820,9 +821,7 @@ " \"dataset\": [vm_test_ds], \"model\" : [vm_log_model,vm_rf_model]\n", " },\n", " ).log()" - ], - "execution_count": null, - "outputs": [] + ] }, { "cell_type": "markdown", diff --git a/site/notebooks/tutorials/validation/inserted-class-imbalance-results.png b/site/notebooks/tutorials/validation/inserted-class-imbalance-results.png deleted file mode 100644 index 2efea1a09b..0000000000 Binary files a/site/notebooks/tutorials/validation/inserted-class-imbalance-results.png and /dev/null differ diff --git a/site/notebooks/tutorials/validation/link-artifact.png b/site/notebooks/tutorials/validation/link-artifact.png deleted file mode 100644 index 23edc1c9eb..0000000000 Binary files a/site/notebooks/tutorials/validation/link-artifact.png and /dev/null differ diff --git a/site/notebooks/tutorials/validation/link-validator-evidence.png b/site/notebooks/tutorials/validation/link-validator-evidence.png deleted file mode 100644 index 660a19cfab..0000000000 Binary files a/site/notebooks/tutorials/validation/link-validator-evidence.png and /dev/null differ diff --git a/site/training/developer-fundamentals/finalizing-documentation.qmd b/site/training/developer-fundamentals/finalizing-documentation.qmd index cc4620cb77..ddfc6a166c 100644 --- a/site/training/developer-fundamentals/finalizing-documentation.qmd +++ b/site/training/developer-fundamentals/finalizing-documentation.qmd @@ -199,7 +199,7 @@ You can also use the content editing toolbar to revise the description of test r For example: -1. Within your documentation, locate the Data Preparation section and click on **2.3 Correlations and Interactions** to expand that section. +1. Within your documentation, click on **2.3 Correlations and Interactions** to expand that section. 2. Click on the text to edit the description for our individually inserted `HighPearsonCorrelation:balanced_raw_dataset` test: diff --git a/site/training/developer-fundamentals/implementing-custom-tests.qmd b/site/training/developer-fundamentals/implementing-custom-tests.qmd index 879e3b5381..278a6aa0e9 100644 --- a/site/training/developer-fundamentals/implementing-custom-tests.qmd +++ b/site/training/developer-fundamentals/implementing-custom-tests.qmd @@ -348,7 +348,7 @@ With the custom test results logged, let's head to the model we connected to at 2. In the left sidebar that appears for your model, click **Development** under {{< fa file >}} Documents. -3. Locate the Data Preparation section and click on **3.2 Model Evaluation** to expand that section. +3. Click on **3.2 Model Evaluation** to expand that section. 4. Hover under the Pearson Correlation Matrix content block until a horizontal line with a **{{< fa plus >}}** button appears, indicating that you can insert a new block. @@ -376,7 +376,7 @@ With the custom test results logged, let's head to the model we connected to at 1. Select the name of your model you registered for this course to open up the model details page. 2. In the left sidebar that appears for your model, click **Development** under {{< fa file >}} Documents. -3. Locate the Data Preparation section and click on **3.2 Model Evaluation** to expand that section. +3. Click on **3.2 Model Evaluation** to expand that section. 4. Hover under the Pearson Correlation Matrix content block until a horizontal line with a **{{< fa plus >}}** button appears, indicating that you can insert a new block. 5. Click **{{< fa plus >}}** and then select **Test-Driven Block** under [from library]{.smallcaps}: - Click on **Custom** under [test-driven]{.smallcaps} in the left sidebar. diff --git a/site/training/developer-fundamentals/learning-to-run-tests.qmd b/site/training/developer-fundamentals/learning-to-run-tests.qmd index 62772da446..c720e5c9b5 100644 --- a/site/training/developer-fundamentals/learning-to-run-tests.qmd +++ b/site/training/developer-fundamentals/learning-to-run-tests.qmd @@ -307,7 +307,7 @@ Work with test results 1. Select the name of your model you registered for this course to open up the model details page. 2. In the left sidebar that appears for your model, click **Development** under {{< fa file >}} Documents. -3. Locate the Data Preparation section and click on **2.3 Correlations and Interactions** to expand that section. +3. Click on **2.3 Correlations and Interactions** to expand that section. 4. Hover under the Pearson Correlation Matrix content block until a horizontal line with a **{{< fa plus >}}** button appears, indicating that you can insert a new block. 5. Click **{{< fa plus >}}** and then select **Test-Driven Block** under [from library]{.smallcaps}: - Click on **VM Library** under [test-driven]{.smallcaps} in the left sidebar. diff --git a/site/training/validator-fundamentals/developing-potential-challengers.qmd b/site/training/validator-fundamentals/developing-potential-challengers.qmd index ff573ee0c7..9e378ca934 100644 --- a/site/training/validator-fundamentals/developing-potential-challengers.qmd +++ b/site/training/validator-fundamentals/developing-potential-challengers.qmd @@ -335,7 +335,7 @@ Log an artifact (finding) ::: ::: {.tc} -[Learn more ...](/notebooks/EXECUTED/validation/3-developing_potential_challenger.html#log-an-artifact){.button target="_blank"} +[Learn more ...](/guide/validation/assess-compliance.qmd#link-artifacts-to-reports){.button target="_blank"} ::: @@ -363,12 +363,13 @@ As we can observe from the output in our notebook, our champion model doesn't pa 1. Select the name of your model you registered for this course to open up the model details page. 2. In the left sidebar that appears for your model, click **Validation** under {{< fa file >}} Documents. -3. Locate the Data Preparation section and click on **2.2.2. Model Performance** to expand that section. -4. Under the Model Performance Metrics section, locate Artifacts then click **Link Artifact to Report** and select **Validation Issue**. -5. Click **{{< fa plus >}} Add Validation Issue** to add a validation issue type artifact. -6. Enter in the details for your validation issue and click **Save**. -7. Select the validation issue you just added to link to your validation report. -8. Click **Update Linked Artifacts** to insert your validation issue. +3. Click on **2.2.2. Model Performance** to expand that section. +4. Under the Model Performance Metrics guideline, click to expand the **Artifacts** panel. +5. Click **{{< fa link >}} Link Artifact** and select **Validation Issue** as the type of artifact. +6. Click **{{< fa plus >}} Add Validation Issue** and enter in the details for your validation issue. +7. Click **Add Validation Issue** to submit the validation issue. +8. Select the validation issue you just added to link to your validation report. +9. Click **Update Linked Artifacts** to insert your validation issue. When you're done, click [{{< fa chevron-right >}}]() to continue. diff --git a/site/training/validator-fundamentals/finalizing-validation-reports.qmd b/site/training/validator-fundamentals/finalizing-validation-reports.qmd index 19207cd60f..83efc88156 100644 --- a/site/training/validator-fundamentals/finalizing-validation-reports.qmd +++ b/site/training/validator-fundamentals/finalizing-validation-reports.qmd @@ -217,36 +217,20 @@ Automatically map & assess evidence :::: {.columns} ::: {.column width="30%" .pr4 .f2} -Map & assess evidence +Automatically map evidence ::: {.f5 .nt2 .pl2 .mb4} (Scroll down for the full instructions.) ::: -<br>Try it **live** on the next pages. {{< fa hand-point-right >}} +<br>Try it **live** on the next page. {{< fa hand-point-right >}} ::: ::: {.column width="70%" .bl .pl4 .f4} -1. In the left sidebar, click **{{< fa cubes >}} Inventory**. - -2. Select a record or find your record by [applying a filter or searching for it](/guide/inventory/working-with-the-inventory.qmd#search-filter-and-sort-records){target="_blank"}. - -3. In the left sidebar that appears for your record, click **Validation** under **{{< fa file >}} Documents**. - -::: {.panel-tabset} - -### Map evidence to guidelines - {{< include /guide/validation/_map-evidence-to-guidelines.qmd >}} -### Assess evidence for compliance - -{{< include /guide/validation/_assess-evidence-for-compliance.qmd >}} - -::: - ::: :::: @@ -254,82 +238,42 @@ Map & assess evidence :::: {.slideover--b .auto-collapse-10} ::: {.tc} -**Map evidence to guidelines** -::: - -1. Select the name of your model you registered for this course to open up the model details page. -2. On the left sidebar that appears for your model, click **Validation** under {{< fa file >}} Documents. -3. Navigate to a section and expand the **Evidence** panel. -4. Click **{{< fa wand-magic-sparkles >}} Map Evidence**. -5. Configure the mapping options, then click **Map Evidence** to run the AI mapping. -6. Review and approve the mapped evidence. - -When you're done, click [{{< fa chevron-right >}}]() to continue. - +**Map evidence to a guideline** ::: -## {background-iframe="https://app.prod.validmind.ai/model-inventory" background-interactive="true" data-preload="yes"} +:::: {.mt2 .f2 .embed} +Please note that test results and content blocks must be present on your model's associated default Development document for Developer Evidence to be automatically mapped to your validation report. -:::: {.slideover--b .auto-collapse-10} -::: {.tc} -**Assess evidence for compliance** ::: 1. Select the name of your model you registered for this course to open up the model details page. 2. On the left sidebar that appears for your model, click **Validation** under {{< fa file >}} Documents. -3. Navigate to a section that has linked evidence. -4. Expand the **Evidence** panel and click **{{< fa wand-magic-sparkles >}} Assess Evidence**. -5. After the AI analyzes the linked evidence and generates an **Evidence Assessment**, review and approve the evidence assessment. - +3. Click on **2.2.2. Model Performance** to expand that section. +4. Under the Model Performance Metrics guideline, locate Evidence then click **{{< fa wand-magic-sparkles >}} Map Evidence**. +5. Configure the mapping options, then click **Map Evidence** to run the AI mapping. +6. Review and approve the mapped evidence. When you're done, click [{{< fa chevron-right >}}]() to continue. ::: - ## {.scrollable} :::: {.columns} ::: {.column width="30%" .pr4 .f2} -Refine compliance assessments +Automatically assess evidence ::: {.f5 .nt2 .pl2 .mb4} (Scroll down for the full instructions.) ::: -::: {.tc} -[Learn more ...](/guide/validation/assess-compliance.qmd#provide-compliance-assessments){.button target="_blank"} - -::: <br>Try it **live** on the next page. {{< fa hand-point-right >}} ::: ::: {.column width="70%" .bl .pl4 .f4} -After {{< var vm.product >}} generates a draft compliance assessment for your review, you'll need to refine your assessments to ensure they are accurate and complete: - -::: {.panel-tabset} - -### Add risk assessment notes - -1. From the **{{< fa cubes >}} Inventory** in the {{< var validmind.platform >}}, go to the model you connected to earlier. - -2. In the left sidebar that appears for your model, click **Validation** under {{< fa file >}} Documents. - -3. Click on **2.2.1. Data Quality** to expand that section and locate the Class Imbalance Assessment sub-section. - -4. Click under **Risk Assessment Notes** to edit the content block using the content editing toolbar. - - For example, use **{{< fa diamond >}} <sup>[beta]{.smallcaps}</sup> (Generate Text with AI)** to create a draft summarizing the contents of the Class Imbalance Assessment sub-section. - - ![Sample compliance assessment](class-imbalance-assessment.png){width=90% fig-alt="A screenshot showing a sample compliance assessment" .screenshot} - -### Provide compliance assessments - -{{< include /guide/validation/_provide-compliance-assessments.qmd >}} - -::: +{{< include /guide/validation/_assess-evidence-for-compliance.qmd >}} ::: :::: @@ -338,93 +282,90 @@ After {{< var vm.product >}} generates a draft compliance assessment for your re :::: {.slideover--b .auto-collapse-10} ::: {.tc} -**Refine your compliance assessments** +**Assess mapped evidence for compliance** +::: + +:::: {.mt2 .f2 .embed} +Please note that evidence must already be mapped to be assessed automatically on your validation report. + ::: 1. Select the name of your model you registered for this course to open up the model details page. 2. On the left sidebar that appears for your model, click **Validation** under {{< fa file >}} Documents. -3. Click on **2.2.1. Data Quality** to refine the compliance assessment for the Class Imbalance Assessment sub-section. +3. Click on **2.2.2. Model Performance** to expand that section. +4. Under the Model Performance Metrics guideline, click **{{< fa wand-magic-sparkles >}} Assess Evidence** under the Evidence Assessment panel. +5. Review and approve the evidence assessment. When you're done, click [{{< fa chevron-right >}}]() to continue. ::: -{{< include /training/common-slides/_document-checker.qmd >}} -<br> -Learn how to **use the {{< var vm.checker >}} to review validation reports** on the next page. {{< fa hand-point-right >}} +# Manage artifacts {background-color="#083E44" background-image="/training/assets/home-hero.svg"} ## {.scrollable} :::: {.columns} ::: {.column width="30%" .pr4 .f2} -Use the {{< var vm.checker >}} +Add more artifacts ::: {.f5 .nt2 .pl2 .mb4} (Scroll down for the full instructions.) ::: +::: {.tc} +[Learn more ...](/guide/validation/add-manage-artifacts.qmd){.button target="_blank"} -<br>Try it **live** on the next page. {{< fa hand-point-right >}} ::: -::: {.column width="70%" .bl .pl4 .f4} +<br> -{{< include /guide/documentation/_check-documents.qmd >}} +Try it **live** on the next pages. {{< fa hand-point-right >}} ::: -:::: +::: {.column width="70%" .bl .pl4 .f4} -## {background-iframe="https://app.prod.validmind.ai/model-inventory" background-interactive="true" data-preload="yes"} +1. In the left sidebar, click **{{< fa cubes >}} Inventory**. -:::: {.slideover--b .auto-collapse-10} -::: {.tc} -**Check your validation report** -::: +2. Select a record or [find your record by applying a filter or searching for it](/guide/inventory/working-with-the-inventory.qmd#search-filter-and-sort-records){target="_blank"}. -1. Select the name of your model you registered for this course to open up the model details page. -2. On the left sidebar that appears for your model, click **Validation** under {{< fa file >}} Documents. -3. Locate {{< fa file-circle-check >}} Check Document on the right and click to expand the menu, then click **{{< fa check >}} Check Document**. -4. Select a **[regulation]{.smallcaps}** and an associated **[assessment]{.smallcaps}** from the drop-down menus to to check your report against. -5. Scroll to the bottom and click **Check Validation Document**. -6. After the {{< var vm.checker >}} has completed its analysis, expand individual questions or click **Expand All** to look through the observations. +3. Add additional artifacts from your validation report, or via during your review of documentation: -When you're done, click [{{< fa chevron-right >}}]() to continue. +::: {.panel-tabset} -:::: +### Autogenerate validation issues on reports -# Manage artifacts {background-color="#083E44" background-image="/training/assets/home-hero.svg"} +{{< include /guide/validation/_autogenerate-validation-issues.qmd >}} -## {.scrollable} +### Add artifacts on documentation -:::: {.columns} -::: {.column width="30%" .pr4 .f2} -Add more artifacts +Along with adding artifacts via validation reports, you can also add artifacts during your review of documentation: -::: {.f5 .nt2 .pl2 .mb4} -(Scroll down for the full instructions.) -::: +{{< include /guide/validation/_add-artifacts-on-documentation.qmd >}} -::: {.tc} -[Learn more ...](/guide/validation/add-manage-artifacts.qmd){.button target="_blank"} +::: ::: +:::: -<br> +## {background-iframe="https://app.prod.validmind.ai/model-inventory" background-interactive="true" data-preload="yes"} -Try it **live** on the next pages. {{< fa hand-point-right >}} +:::: {.slideover--b .auto-collapse-10} +::: {.tc} +**Autogenerate validation issues** ::: -::: {.column width="70%" .bl .pl4 .f4} - -Along with adding artifacts directly via validation reports, you can also add artifacts during your review of documentation: +1. Select the name of your model you registered for this course to open up the model details page. +2. On the left sidebar that appears for your model, click **Validation** under {{< fa file >}} Documents. +3. Click on **2.2.2. Model Performance** to expand that section. +4. Under the Model Performance Metrics guideline, locate Artifacts then click **{{< fa wand-magic-sparkles >}} Suggest Validation Issues**. +5. Approve or reject any suggested validation issues. -{{< include /guide/validation/_add-artifacts-on-documentation.qmd >}} +When you're done, click [{{< fa chevron-right >}}]() to continue. ::: -:::: ## {background-iframe="https://app.prod.validmind.ai/model-inventory" background-interactive="true" data-preload="yes"} @@ -432,12 +373,11 @@ Along with adding artifacts directly via validation reports, you can also add ar ::: {.tc} **Add an artifact on documentation** -<!-- **Add an artifact via overview** --> ::: 1. Select the name of your model you registered for this course to open up the model details page. 2. On the left sidebar that appears for your model, click **Development** under {{< fa file >}} Documents. -3. Click **{{< fa plus >}} Add Model Artifact** to add an artifact from the overview. +3. Click **{{< fa plus >}} Add Artifact** to add an artifact from the overview. When you're done, click [{{< fa chevron-right >}}]() to continue. @@ -526,6 +466,97 @@ When you're done, click [{{< fa chevron-right >}}]() to continue. # Submit report <br>for approval {background-color="#083E44" background-image="/training/assets/home-hero.svg"} +## {.scrollable} + +:::: {.columns} +::: {.column width="30%" .pr4 .f2} +Assigning compliance assessments + +::: {.f5 .nt2 .pl2 .mb4} +(Scroll down for the full instructions.) +::: + +::: {.tc} +[Learn more ...](/guide/validation/assess-compliance.qmd#assign-compliance-assessments){.button target="_blank"} + +::: + +:::: {.pa3 .mt3 .f5 .embed} +**Assessment options available for selection depend on configuration for individual validation report templates.** + +Your organization can customize these options to match its own assessment framework, or choose to remove them from templates entirely. + +::: + +Try it **live** on the next page. {{< fa hand-point-right >}} +::: + +::: {.column width="70%" .bl .pl4 .f4} +{{< include /guide/validation/_assign-compliance-assessments.qmd >}} + +::: +:::: + +## {background-iframe="https://app.prod.validmind.ai/model-inventory" background-interactive="true" data-preload="yes"} + +:::: {.slideover--b .auto-collapse-10} +::: {.tc} +**Assign your compliance assessments** +::: + +1. Select the name of your model you registered for this course to open up the model details page. +2. On the left sidebar that appears for your model, click **Validation** under {{< fa file >}} Documents. +3. Locate the Data Preparation section and click on **2.2.2. Model Performance** to assign the compliance assessment for the Model Performance Metrics guideline. + +When you're done, click [{{< fa chevron-right >}}]() to continue. + +::: + +{{< include /training/common-slides/_document-checker.qmd >}} + +<br> +Learn how to **use the {{< var vm.checker >}} to review validation reports** on the next page. {{< fa hand-point-right >}} + +## {.scrollable} + +:::: {.columns} +::: {.column width="30%" .pr4 .f2} +Use the {{< var vm.checker >}} + +::: {.f5 .nt2 .pl2 .mb4} +(Scroll down for the full instructions.) +::: + + +<br>Try it **live** on the next page. {{< fa hand-point-right >}} +::: + +::: {.column width="70%" .bl .pl4 .f4} + +{{< include /guide/documentation/_check-documents.qmd >}} + +::: + +:::: + +## {background-iframe="https://app.prod.validmind.ai/model-inventory" background-interactive="true" data-preload="yes"} + +:::: {.slideover--b .auto-collapse-10} +::: {.tc} +**Check your validation report** +::: + +1. Select the name of your model you registered for this course to open up the model details page. +2. On the left sidebar that appears for your model, click **Validation** under {{< fa file >}} Documents. +3. Locate {{< fa file-circle-check >}} Check Document on the right and click to expand the menu, then click **{{< fa check >}} Check Document**. +4. Select a **[regulation]{.smallcaps}** and an associated **[assessment]{.smallcaps}** from the drop-down menus to to check your report against. +5. Scroll to the bottom and click **Check Validation Document**. +6. After the {{< var vm.checker >}} has completed its analysis, expand individual questions or click **Expand All** to look through the observations. + +When you're done, click [{{< fa chevron-right >}}]() to continue. + +:::: + {{< include /training/common-slides/_work-with-workflows.qmd >}} {{< include /training/common-slides/_submit-for-approval.qmd >}} diff --git a/site/training/validator-fundamentals/running-data-quality-tests.qmd b/site/training/validator-fundamentals/running-data-quality-tests.qmd index 7140509d38..1108595116 100644 --- a/site/training/validator-fundamentals/running-data-quality-tests.qmd +++ b/site/training/validator-fundamentals/running-data-quality-tests.qmd @@ -295,7 +295,7 @@ When you're done, return to this page and click [{{< fa chevron-right >}}]() to Add test results to reporting ::: {.tc} -[Learn more ...](/notebooks/EXECUTED/validation/2-start_validation_process.ipynb#add-test-results-to-reporting){.button target="_blank"} +[Learn more ...](/guide/validation/assess-compliance.qmd#map-evidence){.button target="_blank"} ::: @@ -305,7 +305,40 @@ Add test results to reporting ::: {.column width="70%" .bl .pl4 .f4} #### Link validator evidence -{{< include /guide/validation/_link-validator-evidence.qmd >}} +With some test results logged, let's head to the record we connected to at the beginning of this notebook and insert our test results into the validation report as evidence. + +While the example below focuses on a specific test result, you can follow the same general procedure for your other results: + +::: {.panel-tabset} + +### 1. Link data quality test results +a. From the **{{< fa cubes >}} Inventory** in the {{< var validmind.platform >}}, go to the record you connected to earlier. + +b. In the left sidebar that appears for your record, click **Validation** under {{< fa file >}} Documents. + +c. Click **2.2.1. Data Quality** to expand that section. + +d. Under the Class Imbalance Assessment guideline, click **Evidence** to expand the evidence panel. + +e. Click **{{< fa link >}} Link Evidence**, then select **Validator Evidence**. + +f. Select the Class Imbalance test results we logged: **ValidMind Data Validation Class Imbalance** + + ![The ClassImbalance tests selected](/notebooks/tutorials/validation/selecting-class-imbalance-results.png){fig-alt="Screenshot showing the ClassImbalance tests selected" .screenshot} + +g. Click **Update Linked Evidence** to add the test results to the validation report. + +h. Confirm that the results for the Class Imbalance test you inserted has been correctly inserted into right section under **2.2.1. Data Quality** of the report. + +### 2. Review Class Imbalance test results + +- Once linked as evidence to section **2.2.1. Data Quality**, note that the ValidMind Data Validation Class Imbalance test results are flagged as **Requires Attention** — as they include comparative results from our initial raw dataset. +- Click **See evidence details** to review the LLM-generated description that summarizes the test results, that confirm that our final preprocessed dataset actually passes our test: + + ![ClassImbalance test generated description in the text editor](/notebooks/tutorials/validation/class-imbalance-results-detail.png){fig-alt="Screenshot showing the ClassImbalance test generated description in the text editor" .screenshot} + +::: + ::: @@ -320,10 +353,11 @@ Add test results to reporting 1. Select the name of your model you registered for this course to open up the model details page. 2. In the left sidebar that appears for your model, click **Validation** under {{< fa file >}} Documents. -3. Locate the Data Preparation section and click on **2.2.1. Data Quality** to expand that section. -4. Under the Class Imbalance Assessment section, locate Validator Evidence then click **Link Evidence to Report**. -5. Select the Class Imbalance test results we logged: **ValidMind Data Validation Class Imbalance** -6. Click **Update Linked Evidence** to add the test results to the validation report. +3. Click **2.2.1. Data Quality** to expand that section. +4. Under the Class Imbalance Assessment guideline, click **Evidence** to expand the evidence panel. +5. Click **{{< fa link >}} Link Evidence**, then select **Validator Evidence**. +6. Select the Class Imbalance test results we logged: **ValidMind Data Validation Class Imbalance** +7. Click **Update Linked Evidence** to add the test results to the validation report. When you're done, click [{{< fa chevron-right >}}]() to continue.