Skip to content

feat(config): lower scenarios into tests#1693

Merged
christso merged 1 commit into
mainfrom
feat/av-kfik-50-scenarios-lowering
Jul 6, 2026
Merged

feat(config): lower scenarios into tests#1693
christso merged 1 commit into
mainfrom
feat/av-kfik-50-scenarios-lowering

Conversation

@christso

@christso christso commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

AgentV can now load Promptfoo-style scenarios as a pre-expansion matrix helper: each scenario config row is lowered with each scenario test row into ordinary AgentV tests before the normal prompts x tests x targets expansion.

This keeps scenarios as authoring sugar rather than a new run artifact or grouping model. providerPromptMap and provider_prompt_map remain rejected because prompt selection should stay explicit through AgentV prompts, targets, files, tags, and CLI selection rather than Promptfoo provider identity baggage.

Validation

After PR #1691 merged to main, this branch was rebased onto origin/main and validated with:

  • bun test packages/core/test/evaluation/validation/eval-file-schema.test.ts packages/core/test/evaluation/validation/eval-validator.test.ts packages/core/test/evaluation/loaders/jsonl-parser.test.ts packages/core/test/evaluation/validation/eval-schema-sync.test.ts
  • bun --filter @agentv/core typecheck
  • bun --filter @agentv/core lint
  • bun --filter @agentv/core build

The worker also regenerated skills-data/agentv-eval-writer/references/eval.schema.json and verified the pinned Promptfoo clone at 6bfc5a0c7f16f9c4717ac731d276b578e63d0769.

Live provider dogfood is deferred to the final docs/examples pass (av-kfik.16) because this PR changes deterministic schema/validation/loading and does not alter provider/grader adapter behavior or run artifact contracts.

Related

Related: av-kfik.50
Supersedes closed stacked PR #1692.


Compound Engineering
GPT--5.5_high

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 6, 2026

Copy link
Copy Markdown

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: c2d5e12
Status: ✅  Deploy successful!
Preview URL: https://275b7db3.agentv.pages.dev
Branch Preview URL: https://feat-av-kfik-50-scenarios-lo.agentv.pages.dev

View logs

@christso christso force-pushed the feat/av-kfik-50-scenarios-lowering branch from 7265c58 to c2d5e12 Compare July 6, 2026 06:23
@christso

christso commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

Coordinator review update: after rebasing onto merged main, I found and fixed one validation bug before merge. Scenario tests without their own prompts/provider_output now validate when every scenario config row supplies prompts/provider_output, matching runtime lowering. Added positive and mixed-config negative regression tests. Local focused tests now pass with 212 cases, plus core typecheck, lint, build, and git diff --check.

@christso christso merged commit a045834 into main Jul 6, 2026
8 checks passed
@christso christso deleted the feat/av-kfik-50-scenarios-lowering branch July 6, 2026 06:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant