feat: evo preview features — config bundles, batch evaluation, recommendations, AB testing

[notgitika]

Summary

Adds preview support for the Evo feature set: config bundles, batch evaluation, recommendations, and AB testing.

Config Bundles [preview]

• add config-bundle — add versioned runtime configuration bundles
• cb versions — list version history for a bundle
• cb diff — diff two versions of a bundle
• cb create-branch — create a new branch on an existing bundle
• --with-config-bundle flag on agent creation auto-wires config bundle support
• Config bundle baggage passed on invoke for runtime config injection

Batch Evaluation [preview]

• run batch-evaluation — run evaluators across all agent sessions in CloudWatch
• stop batch-evaluation — stop a running batch evaluation
• Ground truth support (assertions, expected trajectory, turns)
• Name validation against API pattern [a-zA-Z][a-zA-Z0-9_]{0,47}

Recommendations [preview]

• run recommendation — optimize system prompts or tool descriptions using agent traces
• Supports inline, file, and config bundle input sources
• Config bundle integration: reads current prompt, writes optimized version back
• JSONPath resolution from --runtime flag for multi-component bundles

AB Testing [preview]

• Target-based AB test routing
• AB test detail screen with p-value significance display

Other

• TUI routing fixes for agentcore add config-bundle and agentcore add ab-test
• Documentation for all preview features (docs/config-bundles.md, docs/batch-evaluation.md, docs/recommendations.md)
• README updated with preview commands and doc links

Companion PR

• CDK constructs: aws/agentcore-l3-cdk-constructs (separate PR)

Test plan

• Unit tests passing
• Manual CLI testing: batch eval (valid name, hyphens rejected, fake evaluator error, ground truth, multiple evaluators, lookback days, stop)
• Manual CLI testing: recommendations (inline, file, config bundle, tool descriptions, nonexistent agent)
• Manual CLI testing: config bundles (versions, diff, create-branch, add)
• Manual CLI testing: status shows config bundles
• Validate command passes on current schema

[github-actions]

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	42.91%	8934 / 20817
🔵	Statements	42.18%	9480 / 22475
🔵	Functions	39.66%	1537 / 3875
🔵	Branches	39.89%	5744 / 14397

Generated in workflow #2250 for commit 90939c2 by the Vitest Coverage Report Action

[jariy17]

Re-reviewed at HEAD 6e085f4. All previously flagged regressions are resolved:

• ✅ --days flag restored on run eval
• ✅ onListCloudWatchTraces / onGetCloudWatchTrace restored in agentcore dev
• ✅ RESOURCE_SUFFIX isolation restored in e2e import tests
• ✅ Version and agent-inspector dep back to 0.12.2 / 0.3.0
• ✅ PRIVATE_DEV_DISTRO config reverted

No regressions against the private repo baseline. The 4 issues flagged by agentcore-cli-automation (hardcoded amazonaws.com in recommendation/config-bundle wrappers, stale JSON schema, silent agentcore.json mutation on deploy, config bundle/AB test teardown leak) are separate functional issues worth addressing but not regressions from this PR's changes.

[jariy17]

Updating review — the 4 issues flagged by agentcore-cli-automation are blocking and need to be addressed before merge.

---

1. Hardcoded amazonaws.com breaks non-commercial partitions

Files:

• src/cli/aws/agentcore-recommendation.ts:228
• src/cli/aws/agentcore-config-bundles.ts:181

Both hardcode https://bedrock-agentcore..amazonaws.com / https://bedrock-agentcore-control..amazonaws.com. The sibling wrappers in this same PR (agentcore-ab-tests.ts, agentcore-batch-evaluation.ts, agentcore-http-gateways.ts) correctly use dnsSuffix(region) from ./partition. Recommendations and config bundles will silently fail in GovCloud and China partitions.

Fix: import dnsSuffix from ./partition and replace the hardcoded literal in both files.

---

2. schemas/agentcore.schema.v1.json is stale

The Zod schemas now include configBundles, abTests, and httpGateways as top-level fields on AgentCoreProjectSpecSchema, but the committed JSON schema was not regenerated. Users whose editors validate agentcore.json against the published schema (VS Code, etc.) will see false "property not allowed" errors on every new preview field.

Fix: run npm run build:lib && npm run build:schema and commit the regenerated schemas/agentcore.schema.v1.json.

---

3. validateProject() silently rewrites agentcore.json on every deploy

File: src/cli/operations/deploy/preflight.ts

The deploy preflight injects type: "ConfigurationBundle" into config bundle entries and writes the file back with JSON.stringify(rawJson, null, 2). This runs on every agentcore deploy, producing surprise git diffs for users and clobbering their file's original formatting (tabs, trailing newlines, key order). The Zod ConfigBundleSchema already applies this default in-memory, so the write-back is unnecessary.

Fix options:

1. Fix the CDK side to consume the Zod-parsed spec with defaults applied, and drop this patching code.
2. Inject type only into the in-memory object the CDK reads at synth time, without persisting to disk.
3. If persisting is truly required: only write when patched === true (already done), preserve trailing newline, and print a warning to the user that their file was modified.

---

4. Config bundles and AB tests are leaked on stack teardown

performStackTeardown explicitly calls deleteHttpGatewayWithTargets for HTTP gateways before destroying the CFN stack, but there is no equivalent cleanup for config bundles or AB tests. When a user runs agentcore remove all + deploy, the CFN stack is destroyed but any config bundles and AB tests in deployed-state remain orphaned in AWS — silently accumulating charges with no CLI surface to clean them up.

Fix: extend performStackTeardown to iterate deployedState.targets[target].resources.configBundles and .abTests and delete them, mirroring what it already does for httpGateways.

[notgitika]

Follow up PR for addressing:

1. validateProject() silently rewrites agentcore.json on every deploy

2. Config bundles and AB tests are leaked on stack teardown

stale