@primer/agent-eval@0.4.0

Minor Changes

• bd4eb9a: Add support for installing Copilot plugins in sandbox treatments.

@primer/agent-eval@0.3.0

Minor Changes

• fdbb330: Add support for optional scenario browser tests powered by Vitest and Playwright.

@primer/agent-eval@0.2.0

Minor Changes

• a49df87: Add an optional description to scenario configuration.

@primer/agent-eval@0.1.0

Minor Changes

• c2388a9: Add support for running experiments with Claude Opus 5.

v0.0.1

What's Changed

• feat: add initial project setup by @joshblack in #1
• feat: add harness for evals, experiments, cli, and sandbox by @joshblack in #4
• feat: add table reporter by @joshblack in #8
• Add eval for Primer Octicons React usage by @joshblack with @Copilot in #5
• feat: add octicon eval to mcp experiment by @joshblack in #9
• Update CI workflow to include permissions by @joshblack in #10
• feat: add support for addAgentSkill, addAgentInstruction by @joshblack in #11
• Add eval: agent uses Primer form components for sign-up form by @joshblack with @Copilot in #12
• docs: update agent instructions and docs by @joshblack in #13
• refactor: move experiments into own folder by @joshblack in #14
• chore(deps-dev): bump turbo from 2.9.12 to 2.9.14 by @dependabot[bot] in #15
• ci: add run experiment workflow by @joshblack with @Copilot in #16
• chore(deps): bump actions/upload-artifact from 4.6.2 to 7.0.1 by @dependabot[bot] in #19
• chore(deps): bump the react group with 4 updates by @dependabot[bot] in #21
• chore(deps-dev): bump vitest from 4.1.6 to 4.1.7 in the vitest group by @dependabot[bot] in #22
• chore(deps): bump @primer/react from 38.23.0 to 38.24.0 by @dependabot[bot] in #23
• Support new Copilot message payloads by @joshblack with @Copilot in #27
• Expose experiment concurrency in workflow dispatch by @joshblack with @Copilot in #28
• Add experiment results to GitHub job summaries by @joshblack with @Copilot in #29
• chore(deps-dev): bump typescript-eslint from 8.59.3 to 8.60.0 by @dependabot[bot] in #34
• chore(deps): bump @primer/octicons-react from 19.25.0 to 19.27.0 by @dependabot[bot] in #33
• chore(deps): bump @primer/react from 38.24.0 to 38.25.0 by @dependabot[bot] in #32
• chore(deps-dev): bump vitest from 4.1.7 to 4.1.8 in the vitest group by @dependabot[bot] in #37
• chore(deps): bump the react group with 4 updates by @dependabot[bot] in #36
• chore(deps): bump actions/checkout from 6.0.2 to 6.0.3 by @dependabot[bot] in #35
• feat: Add experiment-level setup hook by @joshblack with @Copilot in #40
• chore(deps): bump @primer/octicons-react from 19.27.0 to 19.28.1 by @dependabot[bot] in #38
• chore(deps-dev): bump rolldown from 1.0.1 to 1.1.0 by @dependabot[bot] in #39
• feat: Add Docker image option for experiments by @joshblack with @Copilot in #41
• chore: update project to Node.js v26 by @joshblack with @Copilot in #52
• feat: Support inline eval definitions by @joshblack with @Copilot in #43
• feat: Prepare agent-eval for npm CLI usage by @joshblack with @Copilot in #42
• chore(deps-dev): bump typescript-eslint from 8.60.0 to 8.62.1 by @dependabot[bot] in #48
• chore(deps): bump @primer/react from 38.25.0 to 38.30.0 by @dependabot[bot] in #49
• chore(deps): bump actions/checkout from 6.0.3 to 7.0.0 by @dependabot[bot] in #50
• chore(deps): bump pnpm/action-setup from 6.0.8 to 6.0.9 by @dependabot[bot] in #51
• chore(deps): update deps to include cooldown period by @joshblack in #53
• chore(deps-dev): bump rolldown-plugin-dts from 0.25.2 to 0.26.0 in the rolldown group by @dependabot[bot] in #55
• feat: Add custom sub-agent sandbox support by @joshblack with @Copilot in #56
• feat: support additional files for agent skills by @joshblack with @Copilot in #57
• docs: reorganize README topics by @joshblack with @Copilot in #58
• chore(deps-dev): bump @types/node from 26.0.0 to 26.0.1 by @dependabot[bot] in #62
• chore(deps-dev): bump turbo from 2.9.18 to 2.10.0 by @dependabot[bot] in #61
• chore(deps): bump dockerode from 5.0.0 to 5.0.1 by @dependabot[bot] in #60
• chore(deps-dev): bump rolldown from 1.1.2 to 1.1.3 in the rolldown group across 1 directory by @dependabot[bot] in #59
• chore(deps-dev): bump the rolldown group across 1 directory with 2 updates by @dependabot[bot] in #68
• chore(deps-dev): bump turbo from 2.10.0 to 2.10.5 by @dependabot[bot] in #64
• chore(deps): bump next from 16.2.9 to 16.2.10 by @dependabot[bot] in #65
• chore(deps): bump @primer/octicons-react from 19.28.1 to 19.29.2 by @dependabot[bot] in #66
• [WIP] Fix failing GitHub Actions job build by @joshblack with @Copilot in #70
• refactor: rename evals to scenarios by @joshblack with @Copilot in #72
• test: add agent setup Next.js eval by @joshblack with @Copilot in #73
• chore(deps-dev): bump the eslint group across 1 directory with 3 updates by @dependabot[bot] in #67
• test: Add theme switching scenario by @joshblack with @Copilot in #76
• feat: Include agent logs in eval output by @joshblack with @Copilot in #77
• feat: include test metadata in agent results by @joshblack with @Copilot in #75
• refactor: use dynamic scenarios, add public exports to agent-eval by @joshblack in #80
• chore(deps): bump actions/checkout from 7.0.0 to 7.0.1 by @dependabot[bot] in #83
• chore(deps): bump actions/setup-node from 6.4.0 to 7.0.0 by @dependabot[bot] in #82
• feat: configure experiment reasoning effort by @joshblack with @Copilot in #84
• feat: add latest Anthropic, OpenAI, and Google models by @joshblack with @Copilot in #86
• chore: pin Copilot CLI version by @joshblack with @Copilot in #88
• chore: pin runner Node.js and npm versions by @joshblack with @Copilot in #89
• feat: add website package by @joshblack in #78
• release: publish 0.0.1 by @joshblack in #91
• chore: Add Changesets release workflow by @joshblack with @Copilot in #90
• chore: add Turbo task caching to CI by @joshblack with @Copilot in #94

New Contributors

• @joshblack made their first contribution in #1
• @joshblack with @Copilot made their first contribution in #5
• @dependabot[bot] made their first contribution in #15

Full Changelog: https://github.com/primer/agent-eval/commits/v0.0.1