Skip to content

AI Verify Evals Toolkit Developer Guide to Create Custom Benchmark Tests

Harry Zhao edited this page Sep 22, 2026 · 1 revision

Introduction

This guide explains how to plug a new benchmark test into the system, add a new metric (evaluation logic), and configure an evaluator (LLM-as-a-Judge) to judge outputs. It assumes that you are running moonshot-cicd from source. Follow below instructions to access our latest unreleased code:

git clone https://github.com/aiverify-foundation/moonshot-cicd.git
cd moonshot-cicd
git checkout dev_main

Terminology

Term Description Location
Benchmark Test Single Test, consist of single Dataset, Metric and Evaluator. It is a test configuration entry in YAML file. Benchmark Test must belong to a Test Bundle in order to run it. A YAML entry under a test bundle in moonshot_core/data/test_config/shared.yaml.
Test Bundle (or Benchmark Test Bundle) Contain a collection of Benchmark Tests. A YAML entry in moonshot_core/data/test_config/shared.yaml.
Dataset Contain prompt and target. JSON under moonshot_core/data/datasets/ listing examples (input / target) In moonshot_core/data/datasets directory.
Metric Evaluate the prompt response; aggregate all evaluation results; implements Metric Port In moonshot_core/src/adapters/metric directory.
shared.yaml This YAML config file contains definition of Benchmark Test Bundles and Benchmark Tests. The Test Bundles are available to run on both CI/CD mode and web UI mode. moonshot_core/data/test_config/shared.yaml
moonshot_config.yaml This YAML config file contains metric & evaluator configuration. moonshot_core/moonshot_config.yaml

Steps to Add New Custom Benchmark Test

Step Action
1. Add dataset json file into moonshot_core/data/datasets directory
2. [OPTIONAL] Adding custom metric:
2.1. Add <metric_name>.py implementing MetricPort
2.2. Add metric block with Evaluator configuration in moonshot_config.yaml
3. Add YAML test under bundle in data/test_configs/shared.yaml
4. Validate — Run test on Moonshot Web UI

1. Add Dataset File

Directory: moonshot_core

Format: JSON file (https://www.json.org/)

  1. Add data/datasets/<dataset_id>.json (file name without .json is the dataset_id).
  2. Use the shape used by existing datasets, for example test_sample_dataset.json:
{
    "name": "Human-readable name",
    "description": "",
    "license": "",
    "reference": "",
    "examples": [
        {"input": "Prompt text...", "target": "Reference label or rubric baseline"}
    ]
}
  1. input becomes the user/model prompt side; target is stored and passed to metrics (e.g. for accuracy or refusal rubrics).

2. [Optional] Add Custom Metric

2.1. Add <metric_name>.py implementing MetricPort

  1. Create moonshot_core/src/adapters/metric/<metric_name>.py.
  2. Implement MetricPort (metric_port.py): at minimum get_metric_connectors, update_metric_params, async get_individual_result, async get_results.
  3. Typical pattern (refusal_adapter.py): in __init__, derive metric_id = Path(__file__).stem, call get_metric_config(metric_id) so configuration is loaded from moonshot_config.yaml metrics list where name: equals that stem.
  4. Module loading: MetricLoader resolves moonshot_core/src/adapters/metric/<metric_name>.py.
  5. Runtime: AsyncioPromptProcessor calls ModuleLoader.load(metric["name"], ModuleTypes.METRIC), then update_metric_params from AppConfig().get_metric_config(met_id).params (so YAML params reach your adapter).

Result contract: follow the docstring on MetricPort.get_individual_result for keys such as prompt, predicted_value, target, score, evaluated_response, etc., so persisted test result stay consistent.

2.2. Add metric block with Evaluator configuration in moonshot_config.yaml

Most non-trivial metrics use an Evaluator (LLM-as-a-Judge) to score the LLM/LLM App output.

  1. Open moonshot_core/moonshot_config.yaml.
  2. Under metrics:, add an entry whose name: equals the metric module file name — the same stem as src/adapters/metric/<name>.py (e.g. refusal_adapter, accuracy_adapter).

Example pattern (conceptually similar to refusal_adapter):

metrics:
  - name: <metric_name>
    connector_configurations:
      connector_adapter: <connector_adapter_name>   # e.g. openai_adapter
      model: <model_name>  # valid model supported by connector adapter / API Key
      system_prompt: <Instruction for the judge>
    params:
      categorise_result: <True|False>   # influences TaskManager result bucketing behaviour
  • connector_adapter / model: pick a connector adapter name from moonshot_core/src/adapters/connector and provide a model name that is available to the selected connector adapter and provided API Key (via UI or environment variable).
  • params.categorise_result: affects how summaries are built in TaskManager when formatting results.

2.3. Add metric module name to metric_aaj_requirements.py

Add the metric's module name (file name) into _METRIC_AAJ_PROVIDER_BY_NAME so that Moonshot is able to locate/access the module file.

# Metric adapter module name (YAML metric.name / DB benchmark_test_metric.name).
AILUMINATE_SAFETY_CLASSIFIER_METRIC = "ailuminate_safety_classifier_adapter"
SG_UC_CLASSIFIER_METRIC = "sg_uc_classifier_adapter"
SG_FACT_CHECK_METRIC = "sg_fact_check_adapter"
REFUSAL_METRIC = "refusal_adapter"
CYBERSEC_REFUSAL_METRIC = "cybersecevalannotator2_adapter"

# Connector system names used by moonshot_config for each metric's evaluator.
AILUMINATE_JUDGE_CONNECTOR_SYSTEM_NAME = "openai_adapter"
REFUSAL_JUDGE_CONNECTOR_SYSTEM_NAME = "openai_adapter"

_METRIC_AAJ_PROVIDER_BY_NAME: dict[str, str] = {
    AILUMINATE_SAFETY_CLASSIFIER_METRIC: AILUMINATE_JUDGE_CONNECTOR_SYSTEM_NAME,
    SG_UC_CLASSIFIER_METRIC: AILUMINATE_JUDGE_CONNECTOR_SYSTEM_NAME,
    SG_FACT_CHECK_METRIC: REFUSAL_JUDGE_CONNECTOR_SYSTEM_NAME,
    REFUSAL_METRIC: REFUSAL_JUDGE_CONNECTOR_SYSTEM_NAME,
    CYBERSEC_REFUSAL_METRIC: REFUSAL_JUDGE_CONNECTOR_SYSTEM_NAME,
}

3. Register a new Benchmark Test and Test Bundle in shared.yaml

Add a new Test Bundle in moonshot_core/data/test_config/shared.yaml:

<Test Bundle key>:
  category: <Category label/name on UI for filter and grouping>
  name: <Display name of Test Bundle on UI>
  description: <Description text shown on UI>
  tests:
  - <Benchmark Test A>
  - <Benchmark Test B>

To add a Benchmark Test into a Test Bundle, append the following data structure under the tests item:

  - name: <Display name of Benchmark Test on UI>
    description: <Description text shown on UI>
    type: benchmark  # default for Benchmark Test
    dataset: <dataset file name (omit .json extension name)>
    metric:
      name: <Metric file name (omit .py extension name)>

Conventions:

  • dataset must match <dataset_id>.json under data/datasets/.
  • metric.name must be the metric module file name — the same stem as src/adapters/metric/<name>.py (e.g. refusal_adapter, accuracy_adapter).

4. Additional Step if you installed via Docker

Follow this guide to run Moonshot with official released images: [Run Moonshot with official released images](https://aiverify.atlassian.net/wiki/external/MWUwMWY4MzI1MDkxNDcwMGIyMDhmY2JjZjZkMDc5ZjY)

Objective

Add custom benchmarks to Moonshot container on Docker Desktop.

Step 1: Access to files in a container

From Docker Desktop, ensure the Moonshot container is running, click on the "…" button under Actions to open up the menu, then click "View files".

image

Step 2: Add custom benchmarks and metrics to /app directory

Navigate to the /app directory — this directory reflects the same directory structure as moonshot_core from source code. Add your custom benchmarks and metrics in this directory.

image

Step 3: Restart container for Moonshot to ingest the changes to DB and reflect it on UI

image

[Optional] Step 4: Create a Docker image from an existing container using docker commit command

This command saves the current state, file changes, and settings of a container into a brand-new image.

  1. Find your container ID or name. List your running containers to get the correct name or ID:

    docker ps
  2. Commit the container to a new image. Run the docker container commit command, providing your container identifier and the desired name for your new image:

    docker commit my-running-container my-custom-image:v1
  3. Verify the new image. Check your local image registry to confirm it was created successfully:

    docker images

Clone this wiki locally