feat: add ground truth reference inputs for on-demand evaluation

[jariy17]

Summary

Adds ground truth reference inputs to on-demand evaluation (run eval), allowing users to specify expected agent behavior for more precise evaluation.

New CLI flags

• -A, --assertion <text...> — Assertion the agent should satisfy (repeatable)
• --expected-trajectory <names> — Expected tool calls in order (comma-separated)
• --expected-response <text> — Expected agent response text

TUI changes

• New Ground Truth wizard step in run eval flow (appears when exactly 1 session is selected)
• Evaluator creation screen now shows reference input placeholders ({assertions}, {expected_tool_trajectory}, {actual_tool_trajectory}, {expected_response})
• Updated default evaluator instructions to include GT placeholders

Implementation

• evaluationReferenceInputs passed through to the evaluate() API call
• Results include referenceInputs summary in both CLI and TUI output
• Note: The second commit (temp: use middleware...) is a temporary workaround — the SDK does not yet include evaluationReferenceInputs in its model, so we inject it via Smithy middleware. Remove when SDK is updated.

Test plan

• npx tsc --noEmit — clean
• npm run lint — 0 errors
• 5 new unit tests for GT reference inputs (all pass)
• Existing 33 run-eval tests continue to pass

[github-actions]

Package Tarball

aws-agentcore-0.4.0.tgz

How to install

npm install https://github.com/aws/agentcore-cli/releases/download/pr-732-tarball/aws-agentcore-0.4.0.tgz

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	45.98%	6575 / 14297
🔵	Statements	45.56%	6986 / 15333
🔵	Functions	44.68%	1177 / 2634
🔵	Branches	46.18%	4362 / 9444

Generated in workflow #1538 for commit d5e84ae by the Vitest Coverage Report Action

[jariy17]

Testing Results

Verified all new GT features against a deployed runtime (TempAgent in EvalCheck project):

CLI flags tested

• -A, --assertion — single and multiple assertions with Builtin.GoalSuccessRate
• --expected-trajectory — comma-separated tool names with Builtin.TrajectoryExactOrderMatch
• --expected-response — with and without explicit -t trace ID, using Builtin.Correctness
• All three combined in a single run across multiple evaluators
• Baseline run (no GT inputs) — confirms referenceInputs is absent from output

Custom evaluators tested

• MyEvaluatorTraceGT (TRACE level) — {expected_response} placeholder populated correctly
• CustomEvaluatorSessionGT (SESSION level) — {assertions}, {expected_tool_trajectory}, {actual_tool_trajectory} placeholders populated correctly

SDK upgrade verified

• @aws-sdk/client-bedrock-agentcore@^3.1020.0 — evaluationReferenceInputs passed natively via EvaluateCommand (no middleware workaround needed)

Other checks

• npx tsc --noEmit — clean
• npm run lint — 0 errors
• All 38 unit tests pass (including 5 new GT tests)
• TUI run eval wizard shows Ground Truth step when exactly 1 session is selected

[Hweinstock]

Some nits and question on service behavior. Otherwise lgtm.

[Hweinstock]

Slightly confused on the customer experience, but thats likely because I'm new to evals. The changes themselves lgtm.