fix: allow code-based evaluators in online eval configs

[jariy17]

Summary

• Remove restrictions that blocked code-based evaluators from being used in online evaluation configs
• The service now supports code-based (Lambda-backed) evaluators for online evaluation
• Companion CDK change: https://github.com/aws/agentcore-l3-cdk-constructs/pull/162

Changes

• OnlineEvalConfigPrimitive: Remove code-based evaluator block in createOnlineEvalConfig()
• agentcore-project.ts: Remove code-based evaluator validation in schema superRefine
• AddOnlineEvalFlow.tsx: Remove code-based evaluator filter in TUI evaluator picker

Test plan

• agentcore add online-eval with a code-based evaluator succeeds (CLI and TUI)
• agentcore deploy with code-based evaluator in online eval config succeeds
• Existing LLM-as-a-Judge and Builtin evaluators still work in online eval configs
• Schema validation passes with code-based evaluators in online eval configs

[github-actions]

Package Tarball

aws-agentcore-0.10.0.tgz

How to install

npm install https://github.com/aws/agentcore-cli/releases/download/pr-947-tarball/aws-agentcore-0.10.0.tgz

[jesseturner21]

Release coordination concern: the companion CDK PR (aws/agentcore-l3-cdk-constructs#162) is still open and the Lambda-permission fix hasn't been published to npm yet.

Rendered CDK projects depend on @aws/agentcore-cdk from the npm registry (see src/assets/cdk/package.json). If this CLI change ships in a release before a @aws/agentcore-cdk version containing the permission fix is on npm, users who configure a code-based evaluator in an online eval config will hit a deploy-time failure:

does not have permission to access the specified Lambda functions for the Evaluation

A couple of options to mitigate:

1. Hold this PR until [P0][CLI] Text not cleared when entering second API key #162 is merged and a new @aws/agentcore-cdk version is published, then bump the @aws/agentcore-cdk version pin in src/assets/cdk/package.json as part of (or immediately after) merging this PR so rendered projects pick up the fix.
2. Land both PRs together and cut a coordinated release (CDK published first, then CLI), again making sure the version pin in src/assets/cdk/package.json requires the fixed CDK version.

Either way, the version pin on @aws/agentcore-cdk should be updated to a version that includes the Lambda-permission change, otherwise existing users on older rendered projects will also silently miss the fix when they regenerate lockfiles.

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	42.05%	7245 / 17229
🔵	Statements	41.47%	7679 / 18516
🔵	Functions	40.25%	1288 / 3200
🔵	Branches	40.49%	4824 / 11912

Generated in workflow #1938 for commit 7c1025c by the Vitest Coverage Report Action

[jariy17]

Hey @jesseturner21, we are performing a coordinated release (CDK first, then CLI) since this PR is part of a launch.

[jesseturner21]

cookies