fix(eval): surface batch-evaluation result truncation as a warning - #1964
Conversation
…ntly returning partial results When readEvaluationResults hits MAX_RESULT_PAGES it now throws instead of warning to the file logger and returning the accumulated list. getBatchEvaluation already catches that into resultsError, which the CLI surfaces as a stderr warning (stdout metadata stays clean) — the same customer-visible path as any other CloudWatch read failure. Previously 'get --json' succeeded with partial 'results' and no customer-visible signal that they were incomplete. Follow-up to #1924 (aidandaly24 review comment).
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## refactor #1964 +/- ##
============================================
+ Coverage 96.71% 96.74% +0.02%
============================================
Files 296 296
Lines 16209 16213 +4
============================================
+ Hits 15677 15685 +8
+ Misses 532 528 -4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Live demo against the exploratory accountVerified the new warning path end-to-end. To force truncation on a real job I temporarily set 1. Truncation now warns (stderr) while the job metadata still prints (stdout): $ bun run src/index.ts eval batch-evaluation get --id BugBash3P_BugBash3P_1785443436425-b1c354f3b9 --region us-east-1 | head -12
warning: could not retrieve CloudWatch results (batch-evaluation results exceed 1 CloudWatch pages; retrieved 5 results are incomplete). Job status is unaffected. See CloudWatch: region us-east-1, log group /aws/bedrock-agentcore/evaluations/batch-evaluations/results/default, stream run-BugBash3P_BugBash3P_1785443436425-b1c354f3b9.
{
"batchEvaluationId": "BugBash3P_BugBash3P_1785443436425-b1c354f3b9",
"batchEvaluationArn": "arn:aws:bedrock-agentcore:us-east-1:725476964917:batch-evaluate/BugBash3P_BugBash3P_1785443436425-b1c354f3b9",
"batchEvaluationName": "BugBash3P_BugBash3P_1785443436425",
"status": "COMPLETED",
...2. The warning goes to stderr only — stdout stays clean, machine-readable JSON, and $ bun run src/index.ts eval batch-evaluation get --id ... --region us-east-1 2>/dev/null | python3 -c 'import sys,json; d=json.load(sys.stdin); print("stdout is valid JSON. status:", d["status"], "| results key present:", "results" in d)'
stdout is valid JSON. status: COMPLETED | results key present: FalseThis is the intended behavior: truncation travels the same customer-visible |
…e cap Replace the bare Error with a new AgentCoreCLIError subclass, ResultTruncationError (source: INTERNAL — the page cap is ours, not a user or service fault). Carries the log group/stream, cap, and retrieved count in meta for telemetry. Slots into the existing errors hierarchy alongside NetworkingError/FileWriteError; getBatchEvaluation's try/catch and the stderr warning path are unchanged.
The message already carries the page count and retrieved count; the log group/stream are in the handler's stderr warning. No separate meta payload.
What
Follow-up to #1924. Addresses @aidandaly24's non-blocking review comment on
src/core/batchEvaluationResults.tsx:Change
readEvaluationResultsnow throws when it exhaustsMAX_RESULT_PAGESwith the forward token still advancing (i.e. the stream has more pages than we read), instead of logging to the file logger and returning the partial list.getBatchEvaluationalready wraps the call in atry/catchthat routes any throw intoresultsError, and thegethandler already turnsresultsErrorinto a stderr warning viawarnCloudWatchFailurewhile leaving stdout metadata clean. So truncation now travels the same customer-visible path as every other CloudWatch read failure — no new plumbing.Before:
get --jsonon a very large job returned partialresultsthat read as complete, with only a file-logger line.After: the job metadata still prints on stdout; a stderr warning states the results are incomplete.
Test
Added a unit test with an ever-advancing token so the loop runs into the cap; asserts
readEvaluationResultsrejects rather than returning a partial list. Existing pagination/parse tests unchanged and green (bun test src/core/batchEvaluationResults.test.ts→ 6 pass; handler tests → 12 pass).