Skip to content

feat(core): support eval test vars templating#1255

Merged
christso merged 3 commits into
mainfrom
feat/1254-eval-vars
May 20, 2026
Merged

feat(core): support eval test vars templating#1255
christso merged 3 commits into
mainfrom
feat/1254-eval-vars

Conversation

@christso
Copy link
Copy Markdown
Collaborator

@christso christso commented May 19, 2026

Closes #1254

Manual verification

Red on main

Loader behavior

{
  "criteria": "Answers {{question}} correctly",
  "input": [
    {
      "role": "user",
      "content": "Answer clearly: {{question}}"
    }
  ],
  "expected_output": [
    {
      "role": "assistant",
      "content": "{{expected_answer}}"
    }
  ]
}

agentv import promptfoo --dry-run

# Converted from promptfoo config: /tmp/.../promptfooconfig.yaml
name: promptfooconfig
tests:
  - id: capital
    input: "Answer clearly: What is the capital of France?"
    assertions:
      - type: equals
        value: Paris
    metadata:
      promptfoo:
        vars:
          question: What is the capital of France?
        prompt_label: prompt-1
        prompt_source: inline

Green on this branch

Loader behavior

{
  "criteria": "Answers What is the capital of France? correctly",
  "input": [
    {
      "role": "user",
      "content": "Answer clearly: What is the capital of France?"
    }
  ],
  "expected_output": [
    {
      "role": "assistant",
      "content": "Paris"
    }
  ]
}

agentv import promptfoo --dry-run

# Converted from promptfoo config: /tmp/.../promptfooconfig.yaml
name: promptfooconfig
tests:
  - id: capital
    input: "Answer clearly: {{question}}"
    vars:
      question: What is the capital of France?
    assertions:
      - type: equals
        value: Paris
    metadata:
      promptfoo:
        vars:
          question: What is the capital of France?
        prompt_label: prompt-1
        prompt_source: inline

christso and others added 3 commits May 20, 2026 09:21
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@cloudflare-workers-and-pages
Copy link
Copy Markdown

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: ae13b94
Status: ✅  Deploy successful!
Preview URL: https://fd8a96b3.agentv.pages.dev
Branch Preview URL: https://feat-1254-eval-vars.agentv.pages.dev

View logs

@christso christso marked this pull request as ready for review May 20, 2026 04:59
@christso christso merged commit 91d1d21 into main May 20, 2026
4 checks passed
@christso christso deleted the feat/1254-eval-vars branch May 20, 2026 04:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: support per-test variable substitution in eval input

1 participant