E2E test failures in shards 3/6 and 5/6 — regression since evo preview features commit

[Hweinstock]

Summary

The E2E Tests (Full Suite) workflow has been failing on main since April 30. Shards 3 and 5 fail consistently across both install methods (main and npm).

Latest failing run: https://github.com/aws/agentcore-cli/actions/runs/25228574758
First failing run: https://github.com/aws/agentcore-cli/actions/runs/25193120248
Last passing run: https://github.com/aws/agentcore-cli/actions/runs/25189513210

Bisection

Run	Commit	Title	Date	Result
25189513210	bd6f841	fix: resolve e2e import test concurrency races (#1067)	Apr 30	✅ Pass
25193120248	9ccf802	feat: evo preview features — config bundles, batch evaluation, recomm…	Apr 30	❌ Fail

Every run on main since 9ccf802 has failed (8 consecutive failures as of May 1).

Failed Jobs (latest run)

Job	Shard	Test Files	Tests
e2e (main, 3/6)	3	1 failed, 3 passed	3 failed, 26 passed
e2e (npm, 3/6)	3	1 failed, 3 passed	3 failed, 26 passed
e2e (main, 5/6)	5	2 failed, 1 passed	13 failed, 21 passed
e2e (npm, 5/6)	5	2 failed, 1 passed	13 failed, 21 passed

Shards 1, 2, 4, 6 and browser-tests all pass.

Failing Test Files & Errors

Shard 3 — http-gateway-targets.test.ts

• deploy gateway: expected 1 to be 0 (exit code) — deploy command fails
• status shows gateway deployed: expected 'local-only' to be 'deployed' — gateway never reaches deployed state
• invokes the deployed agent directly: expected 1 to be 0 — invoke fails since deploy did not succeed

Shard 5 — ab-test-target-based.test.ts

• deploys project: expected 2 to be 0 — deploy exits with code 2
• status shows all resources deployed: expected undefined to be defined — HTTP gateway missing from status
• pauses/resumes/promotes AB test: all fail as cascading failures from failed deploy

Shard 5 — config-bundle-eval-rec.test.ts

• status shows deployed config bundle: expected undefined to be defined — config bundle not found in status
• batch evaluation (multiple tests): 403 — the E2E role is not authorized to perform bedrock-agentcore:StartBatchEvaluation
• recommendation (multiple tests): 403 — the E2E role is not authorized to perform bedrock-agentcore:StartRecommendation

Potential Root Causes

Two distinct issues appear likely:

1. Deploy/gateway failures (shard 3 + part of shard 5): Gateway deployments are failing or stuck in local-only state. This may be a backend service issue or a regression in the deploy command introduced by the evo preview features commit (9ccf802).

2. Missing IAM permissions (shard 5): The E2E GitHub Actions role appears to lack bedrock-agentcore:StartBatchEvaluation and bedrock-agentcore:StartRecommendation permissions. The evo preview features commit added tests for these APIs, but the CI role policy may not have been updated to match.

Additional Note

All jobs emit a deprecation warning: aws-actions/aws-secretsmanager-get-secrets@v2 runs on Node.js 20, which will be forced to Node.js 24 starting June 2, 2026.

Next Steps

• Add bedrock-agentcore:StartBatchEvaluation and bedrock-agentcore:StartRecommendation to the E2E IAM role policy
• Investigate why gateway deploys are failing (shard 3 / AB test deploy)
• Upgrade aws-actions/aws-secretsmanager-get-secrets to a Node.js 24-compatible version

[Hweinstock]

Upgrade aws-actions/aws-secretsmanager-get-secrets to a Node.js 24-compatible version

marking this OOS on this ticket.