Repository navigation
AI Verify Evals Toolkit Developer Guide to Create Custom Benchmark Tests
This guide explains how to plug a new benchmark test into the system, add a new metric (evaluation logic), and configure an evaluator (LLM-as-a-Judge) to judge outputs. It assumes that you are running moonshot-cicd from source. Follow below instructions to access our latest unreleased code:
git clone https://github.com/aiverify-foundation/moonshot-cicd.git
cd moonshot-cicd
git checkout dev_main| Term | Description | Location |
|---|---|---|
| Benchmark Test | Single Test, consist of single Dataset, Metric and Evaluator. It is a test configuration entry in YAML file. Benchmark Test must belong to a Test Bundle in order to run it. | A YAML entry under a test bundle in moonshot_core/data/test_config/shared.yaml. |
| Test Bundle (or Benchmark Test Bundle) | Contain a collection of Benchmark Tests. | A YAML entry in moonshot_core/data/test_config/shared.yaml. |
| Dataset | Contain prompt and target. JSON under moonshot_core/data/datasets/ listing examples (input / target) |
In moonshot_core/data/datasets directory. |
| Metric | Evaluate the prompt response; aggregate all evaluation results; implements Metric Port | In moonshot_core/src/adapters/metric directory. |
shared.yaml |
This YAML config file contains definition of Benchmark Test Bundles and Benchmark Tests. The Test Bundles are available to run on both CI/CD mode and web UI mode. | moonshot_core/data/test_config/shared.yaml |
moonshot_config.yaml |
This YAML config file contains metric & evaluator configuration. | moonshot_core/moonshot_config.yaml |
| Step | Action |
|---|---|
| 1. | Add dataset json file into moonshot_core/data/datasets directory |
| 2. | [OPTIONAL] Adding custom metric: 2.1. Add <metric_name>.py implementing MetricPort2.2. Add metric block with Evaluator configuration in moonshot_config.yaml
|
| 3. | Add YAML test under bundle in data/test_configs/shared.yaml
|
| 4. | Validate — Run test on Moonshot Web UI |
Directory: moonshot_core
Format: JSON file (https://www.json.org/)
- Add
data/datasets/<dataset_id>.json(file name without.jsonis thedataset_id). - Use the shape used by existing datasets, for example
test_sample_dataset.json:
{
"name": "Human-readable name",
"description": "",
"license": "",
"reference": "",
"examples": [
{"input": "Prompt text...", "target": "Reference label or rubric baseline"}
]
}-
inputbecomes the user/model prompt side;targetis stored and passed to metrics (e.g. for accuracy or refusal rubrics).
- Create
moonshot_core/src/adapters/metric/<metric_name>.py. - Implement
MetricPort(metric_port.py): at minimumget_metric_connectors,update_metric_params,async get_individual_result,async get_results. - Typical pattern (
refusal_adapter.py): in__init__, derivemetric_id = Path(__file__).stem, callget_metric_config(metric_id)so configuration is loaded frommoonshot_config.yamlmetrics list wherename:equals that stem. - Module loading:
MetricLoaderresolvesmoonshot_core/src/adapters/metric/<metric_name>.py. - Runtime:
AsyncioPromptProcessorcallsModuleLoader.load(metric["name"], ModuleTypes.METRIC), thenupdate_metric_paramsfromAppConfig().get_metric_config(met_id).params(so YAMLparamsreach your adapter).
Result contract: follow the docstring on MetricPort.get_individual_result for keys such as prompt, predicted_value, target, score, evaluated_response, etc., so persisted test result stay consistent.
Most non-trivial metrics use an Evaluator (LLM-as-a-Judge) to score the LLM/LLM App output.
- Open
moonshot_core/moonshot_config.yaml. - Under
metrics:, add an entry whosename:equals the metric module file name — the same stem assrc/adapters/metric/<name>.py(e.g.refusal_adapter,accuracy_adapter).
Example pattern (conceptually similar to refusal_adapter):
metrics:
- name: <metric_name>
connector_configurations:
connector_adapter: <connector_adapter_name> # e.g. openai_adapter
model: <model_name> # valid model supported by connector adapter / API Key
system_prompt: <Instruction for the judge>
params:
categorise_result: <True|False> # influences TaskManager result bucketing behaviour-
connector_adapter/model: pick a connector adapter name frommoonshot_core/src/adapters/connectorand provide a model name that is available to the selected connector adapter and provided API Key (via UI or environment variable). -
params.categorise_result: affects how summaries are built inTaskManagerwhen formatting results.
Add the metric's module name (file name) into _METRIC_AAJ_PROVIDER_BY_NAME so that Moonshot is able to locate/access the module file.
# Metric adapter module name (YAML metric.name / DB benchmark_test_metric.name).
AILUMINATE_SAFETY_CLASSIFIER_METRIC = "ailuminate_safety_classifier_adapter"
SG_UC_CLASSIFIER_METRIC = "sg_uc_classifier_adapter"
SG_FACT_CHECK_METRIC = "sg_fact_check_adapter"
REFUSAL_METRIC = "refusal_adapter"
CYBERSEC_REFUSAL_METRIC = "cybersecevalannotator2_adapter"
# Connector system names used by moonshot_config for each metric's evaluator.
AILUMINATE_JUDGE_CONNECTOR_SYSTEM_NAME = "openai_adapter"
REFUSAL_JUDGE_CONNECTOR_SYSTEM_NAME = "openai_adapter"
_METRIC_AAJ_PROVIDER_BY_NAME: dict[str, str] = {
AILUMINATE_SAFETY_CLASSIFIER_METRIC: AILUMINATE_JUDGE_CONNECTOR_SYSTEM_NAME,
SG_UC_CLASSIFIER_METRIC: AILUMINATE_JUDGE_CONNECTOR_SYSTEM_NAME,
SG_FACT_CHECK_METRIC: REFUSAL_JUDGE_CONNECTOR_SYSTEM_NAME,
REFUSAL_METRIC: REFUSAL_JUDGE_CONNECTOR_SYSTEM_NAME,
CYBERSEC_REFUSAL_METRIC: REFUSAL_JUDGE_CONNECTOR_SYSTEM_NAME,
}Add a new Test Bundle in moonshot_core/data/test_config/shared.yaml:
<Test Bundle key>:
category: <Category label/name on UI for filter and grouping>
name: <Display name of Test Bundle on UI>
description: <Description text shown on UI>
tests:
- <Benchmark Test A>
- <Benchmark Test B>To add a Benchmark Test into a Test Bundle, append the following data structure under the tests item:
- name: <Display name of Benchmark Test on UI>
description: <Description text shown on UI>
type: benchmark # default for Benchmark Test
dataset: <dataset file name (omit .json extension name)>
metric:
name: <Metric file name (omit .py extension name)>Conventions:
-
datasetmust match<dataset_id>.jsonunderdata/datasets/. -
metric.namemust be the metric module file name — the same stem assrc/adapters/metric/<name>.py(e.g.refusal_adapter,accuracy_adapter).
Follow this guide to run Moonshot with official released images: [Run Moonshot with official released images](https://aiverify.atlassian.net/wiki/external/MWUwMWY4MzI1MDkxNDcwMGIyMDhmY2JjZjZkMDc5ZjY)
Add custom benchmarks to Moonshot container on Docker Desktop.
From Docker Desktop, ensure the Moonshot container is running, click on the "…" button under Actions to open up the menu, then click "View files".
Navigate to the /app directory — this directory reflects the same directory structure as moonshot_core from source code. Add your custom benchmarks and metrics in this directory.
This command saves the current state, file changes, and settings of a container into a brand-new image.
-
Find your container ID or name. List your running containers to get the correct name or ID:
docker ps
-
Commit the container to a new image. Run the
docker container commitcommand, providing your container identifier and the desired name for your new image:docker commit my-running-container my-custom-image:v1
-
Verify the new image. Check your local image registry to confirm it was created successfully:
docker images
- AI Verify Evals Toolkit Installation Guide
- AI Verify Evals Toolkit User Guide
- AI Verify Evals Toolkit Developer Guide
- Onboarding Guide for CI/CD Deployment
- Basic User Guide - Moonshot Configurations
- Basic User Guide - Running Moonshot Tests
- Advanced User Guide - Create Custom Connectors for your Application Endpoint
- Advanced User Guide - Create Custom Evaluation Metrics for your Tests
- Expert User Guide - Create Custom Moonshot Tests