Skip to content

test: add e2e test for evaluations lifecycle#628

Merged
tejaskash merged 1 commit intoaws:mainfrom
Hweinstock:feat/e2e-evals-lifecycle
Mar 24, 2026
Merged

test: add e2e test for evaluations lifecycle#628
tejaskash merged 1 commit intoaws:mainfrom
Hweinstock:feat/e2e-evals-lifecycle

Conversation

@Hweinstock
Copy link
Copy Markdown
Contributor

@Hweinstock Hweinstock commented Mar 24, 2026

Description

The evaluations feature (custom evaluators, online eval configs, on-demand evals, pause/resume) is missing e2e test coverage. This PR adds a standalone e2e suite that tests the full lifecycle against real AWS infrastructure.

Also refactor e2e test setup to be build consumable utility functions rather than a single mono-function.

Known limitation: logs evals not included

Online eval log processing is async and the CloudWatch log group (/aws/bedrock-agentcore/evaluations/results/{configId}) was not populated within 8+ minutes of polling across multiple test runs. This makes logs evals assertions unreliable for e2e testing.

Related Issue

Closes #

Documentation PR

N/A — test-only change.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update
  • Other (please describe): e2e test coverage for evaluations lifecycle (user stories 2.4, 2.5, 10.1, 10.3, 10.4)

Testing

How have you tested the change?

  • I ran npm run test:unit and npm run test:integ
  • I ran npm run typecheck
  • I ran npm run lint
  • If I modified src/assets/, I ran npm run test:update-snapshots and committed the updated snapshots

Verified against dev account (us-east-1). All 6 tests pass in ~4 minutes:

Checklist

  • I have read the CONTRIBUTING document
  • I have added any necessary tests that prove my fix is effective or my feature works
  • I have updated the documentation accordingly
  • I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the
terms of your choice.

@github-actions github-actions bot added the size/m PR size: M label Mar 24, 2026
@Hweinstock Hweinstock force-pushed the feat/e2e-evals-lifecycle branch from 6eed8a7 to 61b6f9f Compare March 24, 2026 21:29
@github-actions github-actions bot added size/m PR size: M and removed size/m PR size: M labels Mar 24, 2026
@Hweinstock Hweinstock force-pushed the feat/e2e-evals-lifecycle branch from 61b6f9f to 94f181f Compare March 24, 2026 22:10
@github-actions github-actions bot added size/m PR size: M and removed size/m PR size: M labels Mar 24, 2026
@Hweinstock Hweinstock force-pushed the feat/e2e-evals-lifecycle branch from 94f181f to 2c8784e Compare March 24, 2026 22:19
@github-actions github-actions bot added size/m PR size: M and removed size/m PR size: M labels Mar 24, 2026
@Hweinstock Hweinstock force-pushed the feat/e2e-evals-lifecycle branch 2 times, most recently from ec089b6 to 0eae153 Compare March 24, 2026 22:26
@github-actions github-actions bot added size/m PR size: M and removed size/m PR size: M labels Mar 24, 2026
@Hweinstock Hweinstock force-pushed the feat/e2e-evals-lifecycle branch from 0eae153 to 8599456 Compare March 24, 2026 22:36
@github-actions github-actions bot added size/m PR size: M and removed size/m PR size: M labels Mar 24, 2026
@Hweinstock Hweinstock force-pushed the feat/e2e-evals-lifecycle branch from 8599456 to 256bc9f Compare March 24, 2026 22:37
@github-actions github-actions bot added size/m PR size: M and removed size/m PR size: M labels Mar 24, 2026
@Hweinstock Hweinstock force-pushed the feat/e2e-evals-lifecycle branch from 256bc9f to 9419081 Compare March 24, 2026 22:41
@github-actions github-actions bot added size/m PR size: M and removed size/m PR size: M labels Mar 24, 2026
@Hweinstock Hweinstock force-pushed the feat/e2e-evals-lifecycle branch from 9419081 to 81529cb Compare March 24, 2026 22:44
@github-actions github-actions bot added size/m PR size: M and removed size/m PR size: M labels Mar 24, 2026
@Hweinstock Hweinstock marked this pull request as ready for review March 24, 2026 22:57
@Hweinstock Hweinstock requested a review from a team March 24, 2026 22:57
Copy link
Copy Markdown
Contributor

@tejaskash tejaskash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Clean refactoring of e2e helpers and solid evals lifecycle test coverage.

@tejaskash tejaskash merged commit ec3d007 into aws:main Mar 24, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/m PR size: M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants