notebooks: Updated key concepts

[validbeck]

Pull Request Description

What and why?

sc-15992

I've updated the glossary / key concepts documentation side, so edits and improvements have been pulled into the notebooks:

Developer key concepts

Updated key concepts

**record**:  Tools tracked in the ValidMind inventory, such as models. Records include traditional statistical models, legacy systems, artificial intelligence/machine learning models, large language models (LLMs), agentic AI systems, and other documentable items that benefit from oversight, testing, and lifecycle management.

**model**: [SR 11-7](https://www.federalreserve.gov/supervisionreg/srletters/sr1107.htm) defines a model as a "quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates." Within ValidMind, a model is type of record tracked in the inventory.

**documentation, model documentation**: A structured and detailed record pertaining to a record (such as a model), encompassing key components such as its underlying assumptions, methodologies, data sources, inputs, performance metrics, evaluations, limitations, and intended uses. Within the realm of risk management, this documentation serves to ensure transparency, adherence to regulatory requirements, and a clear understanding of potential risks associated with the record's application.

**document template**: Lays out the structure of documents, segmented into various sections and sub-sections, and function as test suites specifying the tests that should be run, and how the results should be displayed. Document templates help automate your development, validation, monitoring, and other risk management processes. Document templates are available for default ValidMind document types as well as custom document types.

**documentation template**: A default ValidMind document type that serves as a standardized framework for developing and documenting records (such as models), including sections designated for record details, data descriptions, test results, and performance metrics. By outlining required documentation and recommended analyses, document templates ensure consistency and completeness across documentation and help guide developers through a systematic development process while promoting comparability and traceability of development outcomes.

**test**: A function contained in the ValidMind Library, designed to run a specific quantitative test on the dataset or record (such as a model). Test results are logged to the ValidMind Platform, where they are attached to documents. Tests are the building blocks of ValidMind, used to evaluate and document records and datasets, and can be run individually or as part of a suite defined by your templates.

**test suite**: A collection of tests designed to run together to automate and generate documentation end-to-end for specific use cases. (Learn more: [`test_suites`](https://docs.validmind.ai/validmind/validmind/test_suites.html))

**metric**: A subset of tests that do not have thresholds. In the context of this notebook, metrics and tests can be thought of as interchangeable concepts.

**custom test**: Functions that you define to evaluate your record (such as a model) or dataset. These functions can be registered with the ValidMind Library to be used in the ValidMind Platform.

**inputs**: Objects to be evaluated and documented in the ValidMind Library. They can be any of the following:

  - **model**: A single record (such as a model) that has been initialized in ValidMind with [`init_model()`](https://docs.validmind.ai/validmind/validmind.html#init_model). Despite the naming convention, model objects can be any type of record you want to test, document, validate, or monitor with ValidMind.
  - **dataset**: Single dataset that has been initialized in ValidMind with [`init_dataset()`](https://docs.validmind.ai/validmind/validmind.html#init_dataset).
  - **models**: A list of ValidMind records - usually this is used when you want to compare multiple records in your custom tests.
  - **datasets**: A list of ValidMind datasets - usually this is used when you want to compare multiple datasets in your custom tests. (Learn more: [Run tests with multiple datasets](https://docs.validmind.ai/notebooks/how_to/tests/run_tests/configure_tests/run_tests_that_require_multiple_datasets.html))

**parameters**: Additional arguments that can be passed when running a ValidMind test, used to pass additional information to a test, customize its behavior, or provide additional context.

**outputs**: Custom tests can return elements like tables or plots. Tables may be a list of dictionaries (each representing a row) or a pandas DataFrame. Plots may be matplotlib or plotly figures.

Files modified

• notebooks/code_sharing/r/r_custom_tests.Rmd
• notebooks/how_to/data_and_datasets/dataset_inputs/configure_dataset_features.ipynb
• notebooks/how_to/data_and_datasets/dataset_inputs/load_datasets_predictions.ipynb
• notebooks/how_to/metrics/log_metrics_over_time.ipynb
• notebooks/how_to/tests/custom_tests/implement_custom_tests.ipynb
• notebooks/how_to/tests/explore_tests/explore_test_suites.ipynb
• notebooks/how_to/tests/explore_tests/explore_tests.ipynb
• notebooks/how_to/qualitative_text/qualitative_text_generation.ipynb
• notebooks/how_to/tests/run_tests/1-run_dataset-based_tests.ipynb
• notebooks/how_to/tests/run_tests/2-run_comparison_tests.ipynb
• notebooks/how_to/tests/run_tests/configure_tests/enable_pii_detection.ipynb
• notebooks/how_to/tests/run_tests/configure_tests/run_tests_that_require_multiple_datasets.ipynb
• notebooks/how_to/tests/run_tests/documentation_tests/document_multiple_results_for_the_same_test.ipynb
• notebooks/how_to/tests/run_tests/documentation_tests/run_documentation_sections.ipynb
• notebooks/how_to/tests/run_tests/documentation_tests/run_documentation_tests_with_config.ipynb
• notebooks/quickstart/quickstart_documentation.ipynb
• notebooks/templates/about-validmind/_about-validmind-developers.ipynb
• notebooks/tutorials/development/1-set_up_validmind.ipynb
• notebooks/use_cases/agents/document_agentic_ai.ipynb
• notebooks/use_cases/capital_markets/quickstart_option_pricing_models.ipynb
• notebooks/use_cases/capital_markets/quickstart_option_pricing_models_quantlib.ipynb
• notebooks/use_cases/code_explainer/quickstart_code_explainer_demo.ipynb
• notebooks/use_cases/credit_risk/application_scorecard_executive.ipynb
• notebooks/use_cases/credit_risk/application_scorecard_full_suite.ipynb
• notebooks/use_cases/credit_risk/application_scorecard_with_bias.ipynb
• notebooks/use_cases/credit_risk/application_scorecard_with_ml.ipynb
• notebooks/use_cases/credit_risk/document_excel_application_scorecard.ipynb
• notebooks/use_cases/nlp_and_llm/prompt_validation_demo.ipynb
• notebooks/use_cases/time_series/quickstart_time_series_full_suite.ipynb
• notebooks/use_cases/time_series/quickstart_time_series_high_code.ipynb

Special case

• notebooks/how_to/data_and_datasets/use_dataset_model_objects.ipynb

The key concepts in this notebook is only focused on test/datasets, edited the expanded definitions for the dataset- & model-based tests as well.

Updated key concepts

**test**: A function contained in the ValidMind Library, designed to run a specific quantitative test on the dataset or record (such as a model). Test results are logged to the ValidMind Platform, where they are attached to documents. Tests are the building blocks of ValidMind, used to evaluate and document records and datasets, and can be run individually or as part of a suite defined by your templates.

 - **dataset-based test**

    ![Dataset based test architecture](./dataset_image.png)
    Dataset-based tests take VM dataset objects as inputs, can be configured with values passed in as parameters, and return outputs such as tables, plots, or images.

 - **model-based test**:

    ![Model based test architecture](./model_image.png)
    Similar to dataset-based tests, model-based tests take additional VM model objects as inputs alongside VM dataset objects. The VM model object can wrap any type of record and is used to obtain prediction values for entries in the dataset.

**test suite**: A collection of tests designed to run together to automate and generate documentation end-to-end for specific use cases. (Learn more: [test_suites](https://docs.validmind.ai/validmind/validmind/test_suites.html))

**metric**: A subset of tests that do not have thresholds. In the context of this notebook, metrics and tests can be thought of as interchangeable concepts.

**custom test**: Functions that you define to evaluate your record (such as a model) or dataset. These functions can be registered with the ValidMind Library to be used in the ValidMind Platform.

**inputs**: Objects to be evaluated and documented in the ValidMind Library. They can be any of the following:

  - **model**: A single record (such as a model) that has been initialized in ValidMind with [`init_model()`](https://docs.validmind.ai/validmind/validmind.html#init_model). Despite the naming convention, model objects can be any type of record you want to test, document, validate, or monitor with ValidMind.
  - **dataset**: Single dataset that has been initialized in ValidMind with [`init_dataset()`](https://docs.validmind.ai/validmind/validmind.html#init_dataset).
  - **models**: A list of ValidMind records - usually this is used when you want to compare multiple records in your custom tests.
  - **datasets**: A list of ValidMind datasets - usually this is used when you want to compare multiple datasets in your custom tests. (Learn more: [Run tests with multiple datasets](https://docs.validmind.ai/notebooks/how_to/tests/run_tests/configure_tests/run_tests_that_require_multiple_datasets.html))

**parameters**: Additional arguments that can be passed when running a ValidMind test, used to pass additional information to a test, customize its behavior, or provide additional context.

**outputs**: Custom tests can return elements like tables or plots. Tables may be a list of dictionaries (each representing a row) or a pandas DataFrame. Plots may be matplotlib or plotly figures.

Validator key concepts

Updated key concepts

**record**:  Tools tracked in the ValidMind inventory, such as models. Records include traditional statistical models, legacy systems, artificial intelligence/machine learning models, large language models (LLMs), agentic AI systems, and other documentable items that benefit from oversight, testing, and lifecycle management.

**model**: [SR 11-7](https://www.federalreserve.gov/supervisionreg/srletters/sr1107.htm) defines a model as a "quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates." Within ValidMind, a model is type of record tracked in the inventory.

**validation report:** A validation report is a comprehensive and structured review evaluating a record's accuracy, performance, and suitability for its intended purpose. A report follows established validation guidelines to ensure consistency and adherence to internal and regulatory standards — encompassing the process of risk assessment, identifying areas of potential error or risk within the record's components, supporting transparency, regulatory compliance, and informed decision-making by documenting the validator’s independent review and conclusions.

**document template**: Lays out the structure of documents, segmented into various sections and sub-sections, and function as test suites specifying the tests that should be run, and how the results should be displayed. Document templates help automate your development, validation, monitoring, and other risk management processes. Document templates are available for default ValidMind document types as well as custom document types.

**validation report template**: A default ValidMind document template that serves as a standardized framework for conducting and documenting validation, including sections designated for attaching test results, evidence, or artifacts (findings). By outlining required documentation, recommended analyses, and expected validation tests, validation report templates ensure consistency and completeness across validation reports and help guide validators through a systematic review process while promoting comparability and traceability of validation outcomes.

**artifacts (findings)**: Observations or issues identified during validation, including any deviations from expected performance or standards. Artifacts are organized by type — default types provided by ValidMind include Validation Issue, Policy Exception, and Limitation. Custom artifact types can be created to track other categories relevant to your organization.

**test**: A function contained in the ValidMind Library, designed to run a specific quantitative test on the dataset or record (such as a model). Test results are logged to the ValidMind Platform, where they are attached to documents. Tests are the building blocks of ValidMind, used to evaluate and document records and datasets, and can be run individually or as part of a suite defined by your templates.

**test suite**: A collection of tests designed to run together to automate and generate documentation end-to-end for specific use cases. (Learn more: [`test_suites`](https://docs.validmind.ai/validmind/validmind/test_suites.html))

**metric**: A subset of tests that do not have thresholds. In the context of this notebook, metrics and tests can be thought of as interchangeable concepts.

**custom test**: Functions that you define to evaluate your record (such as a model) or dataset. These functions can be registered with the ValidMind Library to be used in the ValidMind Platform.

**inputs**: Objects to be evaluated and documented in the ValidMind Library. They can be any of the following:

  - **model**: A single record (such as a model) that has been initialized in ValidMind with [`init_model()`](https://docs.validmind.ai/validmind/validmind.html#init_model). Despite the naming convention, model objects can be any type of record you want to test, document, validate, or monitor with ValidMind.
  - **dataset**: Single dataset that has been initialized in ValidMind with [`init_dataset()`](https://docs.validmind.ai/validmind/validmind.html#init_dataset).
  - **models**: A list of ValidMind records - usually this is used when you want to compare multiple records in your custom tests.
  - **datasets**: A list of ValidMind datasets - usually this is used when you want to compare multiple datasets in your custom tests. (Learn more: [Run tests with multiple datasets](https://docs.validmind.ai/notebooks/how_to/tests/run_tests/configure_tests/run_tests_that_require_multiple_datasets.html))

**parameters**: Additional arguments that can be passed when running a ValidMind test, used to pass additional information to a test, customize its behavior, or provide additional context.

**outputs**: Custom tests can return elements like tables or plots. Tables may be a list of dictionaries (each representing a row) or a pandas DataFrame. Plots may be matplotlib or plotly figures.

Files modified

• notebooks/quickstart/quickstart_validation.ipynb
• notebooks/templates/about-validmind/_about-validmind-validators.ipynb
• notebooks/tutorials/validation/1-set_up_validmind_for_validation.ipynb
• notebooks/tutorials/validation/4-finalize_validation_reporting.ipynb
• notebooks/use_cases/validation/validate_application_scorecard.ipynb

Monitoring key concepts

Updated key concepts

**record**:  Tools tracked in the ValidMind inventory, such as models. Records include traditional statistical models, legacy systems, artificial intelligence/machine learning models, large language models (LLMs), agentic AI systems, and other documentable items that benefit from oversight, testing, and lifecycle management.

**model**: [SR 11-7](https://www.federalreserve.gov/supervisionreg/srletters/sr1107.htm) defines a model as a "quantitative method, system, or approach that applies statistical, economic, financial, or mathematical theories, techniques, and assumptions to process input data into quantitative estimates." Within ValidMind, a model is type of record tracked in the inventory.

**ongoing monitoring report**: A comprehensive and structured periodic report assessing the record (such as a model)'s performance and compliance over time, ensuring it remains valid under changing conditions. Monitoring includes key elements such as data sources, inputs, performance metrics, and periodic evaluations, ensuring transparency and visibility of the record's performance in the production environment.

**document template**: Lays out the structure of documents, segmented into various sections and sub-sections, and function as test suites specifying the tests that should be run, and how the results should be displayed. Document templates help automate your development, validation, monitoring, and other risk management processes. Document templates are available for default ValidMind document types as well as custom document types.

**monitoring template, monitoring report template**: A default ValidMind document template that serves as a standardized framework for ongoing monitoring, including sections designated for test results, performance metrics, and drift analyses. By outlining required monitoring checks and expected routine tests, monitoring templates ensure consistency and completeness across monitoring reports and help guide owners through a systematic monitoring process while promoting early detection of performance degradation.

**test**: A function contained in the ValidMind Library, designed to run a specific quantitative test on the dataset or record (such as a model). Test results are logged to the ValidMind Platform, where they are attached to documents. Tests are the building blocks of ValidMind, used to evaluate and document records and datasets, and can be run individually or as part of a suite defined by your templates.

**test suite**: A collection of tests designed to run together to automate and generate documentation end-to-end for specific use cases. (Learn more: [`test_suites`](https://docs.validmind.ai/validmind/validmind/test_suites.html))

**metric**: A subset of tests that do not have thresholds. In the context of this notebook, metrics and tests can be thought of as interchangeable concepts.

**custom test**: Functions that you define to evaluate your record (such as a model) or dataset. These functions can be registered with the ValidMind Library to be used in the ValidMind Platform.

**inputs**: Objects to be evaluated and documented in the ValidMind Library. They can be any of the following:

  - **model**: A single record (such as a model) that has been initialized in ValidMind with [`init_model()`](https://docs.validmind.ai/validmind/validmind.html#init_model). Despite the naming convention, model objects can be any type of record you want to test, document, validate, or monitor with ValidMind.
  - **dataset**: Single dataset that has been initialized in ValidMind with [`init_dataset()`](https://docs.validmind.ai/validmind/validmind.html#init_dataset).
  - **models**: A list of ValidMind records - usually this is used when you want to compare multiple records in your custom tests.
  - **datasets**: A list of ValidMind datasets - usually this is used when you want to compare multiple datasets in your custom tests. (Learn more: [Run tests with multiple datasets](https://docs.validmind.ai/notebooks/how_to/tests/run_tests/configure_tests/run_tests_that_require_multiple_datasets.html))

**parameters**: Additional arguments that can be passed when running a ValidMind test, used to pass additional information to a test, customize its behavior, or provide additional context.

**outputs**: Custom tests can return elements like tables or plots. Tables may be a list of dictionaries (each representing a row) or a pandas DataFrame. Plots may be matplotlib or plotly figures.

Files modified

• notebooks/templates/about-validmind/_about-validmind-monitoring.ipynb
• notebooks/use_cases/ongoing_monitoring/application_scorecard_ongoing_monitoring.ipynb
• notebooks/use_cases/ongoing_monitoring/quickstart_customer_churn_ongoing_monitoring.ipynb

How to test

Open up the notebooks directory for this branch/PR and spot check some notebooks.

What needs special review?

Someone please double check that the information for the special case of notebooks/how_to/data_and_datasets/use_dataset_model_objects.ipynb is accurate.

Dependencies, breaking changes, and deployment notes

Notebooks pulled back into docs in validmind/documentation#1339.

Release notes

n/a

Checklist

• What and why
• Screenshots or videos (Frontend)
• How to test
• What needs special review
• Dependencies, breaking changes, and deployment notes
• Labels applied
• PR linked to Shortcut
• Unit tests added (Backend)
• Tested locally
• Documentation updated (if required)
• Environment variable additions/changes documented (if required)

[validbeck]

Comments from Steven to address:

• done

Grammar/copy (repeated ~40 files)
• a model is type of record → a model is a type of record
• and function as test suites → and functions as test suites
• record: Tools → single space after colon
• Confirm “Tools tracked in the ValidMind inventory” vs docs glossary wording

I did update "Tools tracked [...]" to "A tool tracked [...]" to make the singular entries consistent.

• done

Possible copy error In the documentation template definition, closing sentence says “document templates ensure consistency…” — confirm whether that should be “documentation templates” (sync with docs PR #1339)