feat(eval): add eval evaluator CLI commands (CRUD + LLaJ/code-based)

[jariy17]

What

Adds the imperative agentcore eval evaluator command surface (CLI only). Covers LLM-as-a-Judge and code-based evaluator create/update plus type-agnostic get/list/delete, following the DevX Evaluations/Optimization refactor doc and the existing identity handler conventions.

Command structure

agentcore eval                              # evaluate and optimize AgentCore agents
└── evaluator                               # manage AgentCore evaluators
    ├── llm-as-a-judge                       # LLM-as-a-Judge evaluators
    │   ├── create
    │   └── update
    ├── code-based                           # code-based (Lambda-backed) evaluators
    │   ├── create
    │   └── update
    ├── get                                  # get an evaluator by id (type-agnostic)
    ├── list                                 # list evaluators (client-side --type filter)
    └── delete                               # delete an evaluator by id (requires --yes)

Flags

eval evaluator llm-as-a-judge create

Flag	Required	Notes
--name	✓	evaluator name
--level	✓	SESSION | TRACE | TOOL_CALL
--model	✓	Bedrock model id used to judge
--instructions	✓	source-aware: inline, file://<path>, or - (stdin)
--rating-scale	✓*	preset: 1-5-quality | 1-3-simple | pass-fail | good-neutral-bad
--rating-scale-json	✓*	raw RatingScale JSON (source-aware). Mutually exclusive with --rating-scale; exactly one required
--kms-key-arn		customer-managed KMS key ARN
--tags		JSON object (source-aware)
--client-token		idempotency token

eval evaluator llm-as-a-judge update

Flag	Required	Notes
--id	✓	evaluator id
--instructions		source-aware
--model
--rating-scale / --rating-scale-json		mutually exclusive
--kms-key-arn
--client-token

Fields left unset are preserved: UpdateEvaluator replaces the whole evaluatorConfig union, so the client does get-then-merge over the current definition.

eval evaluator code-based create

Flag	Required	Notes
--name	✓	evaluator name
--level	✓	SESSION | TRACE | TOOL_CALL
--lambda-arn	✓	Lambda that scores a session
--timeout		seconds (1–300). No CLI default; the service applies its own (60s)
--kms-key-arn
--tags		JSON object (source-aware)
--client-token

eval evaluator code-based update

Flag	Required	Notes
--id	✓	evaluator id
--lambda-arn
--timeout
--kms-key-arn
--client-token

eval evaluator get

Flag	Required	Notes
--id	✓	evaluator id

eval evaluator list

Flag	Required	Notes
--next-token		pagination token (server-side)
--max-results		max items (server-side)
--type		filters the returned page client-side: Builtin | code-based | llm-as-a-judge. The ListEvaluators API paginates only, so the filter is page-local

eval evaluator delete

Flag	Required	Notes
--id	✓	evaluator id
--yes	✓	confirm deletion (required in this headless/JSON-only path)

Notes / decisions

• Source-aware field values (--instructions, --rating-scale-json, --tags): inline, file://<path>, or - for stdin, following the AWS CLI file:// convention. One flag reads stdin per command.
• Rating scale offers presets (the common defaults) and a raw-JSON escape hatch (what the API supports directly).
• --timeout has no CLI default; the service default applies when omitted.
• Dependency inversion: CoreEvalClient is declared next to the handlers (src/handlers/eval/types.tsx) and implemented in src/core/eval.tsx.

Testing

• bun test — full suite green (356 tests).
• bun run typecheck, bun run lint:check, bun run format:check — clean.
• bun run build + bundle smoke (node dist/index.js eval evaluator --help) — works.
• New tests: handler behavior + validation/source/list-filter via TestCoreClient, EvalClient get-then-merge unit tests, and source-resolver tests.

---

TUI flows for these commands will come in a later PR.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.21%. Comparing base (ccd4024) to head (c0338a9).

Additional details and impacted files

@@             Coverage Diff              @@
##           refactor    #1822      +/-   ##
============================================
+ Coverage     94.92%   95.21%   +0.29%     
============================================
  Files           155      170      +15     
  Lines          7565     8026     +461     
============================================
+ Hits           7181     7642     +461     
  Misses          384      384              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[AlexanderRichey]

Really good work here. Left a few comments. Main blocking things are using types vs interfaces appropriately and handling duplication of source reading.

[aidandaly24]

A few things:

[aidandaly24]

A few more comments

[Hweinstock]

one thing I noticed testing e2e is that the help pages no longer tell us whats required since we aren't using commander's required option:

Usage: agentcore eval evaluator code-based create [options]

create a code-based (Lambda-backed) evaluator

Options:
  --name <name>                  the name of the evaluator
  --level <level>                evaluation level (SESSION | TRACE | TOOL_CALL)
  --lambda-arn <lambda-arn>      ARN of the Lambda function that scores a session
  --timeout <timeout>            Lambda timeout in seconds (1-300)
  --kms-key-arn <kms-key-arn>    customer managed KMS key ARN for evaluator data
  --tags <tags>                  tags to apply (JSON object of key/value strings; inline, file://<path>, or - for stdin)
  --client-token <client-token>  idempotency token
  -h, --help                     display help for command

I think this is an issue with a lot of the work we've done though, not specific to here.

the apis are definitely a little awkward, but the commands themselves work great!

Addressed his comments and two other reviewers reviewed it