W-22203426: feat: move eval normalizer, formatter, YAML translator, and runner into library#276
Open
andresrivas-sf wants to merge 3 commits intomainfrom
Open
W-22203426: feat: move eval normalizer, formatter, YAML translator, and runner into library#276andresrivas-sf wants to merge 3 commits intomainfrom
andresrivas-sf wants to merge 3 commits intomainfrom
Conversation
…nd runner into library Extracts evalNormalizer, evalFormatter, yamlSpecTranslator, and the new agentEvalRunner module from plugin-agent into the @salesforce/agents library so any consumer can use them directly without depending on the CLI plugin. All four modules are exported from the package index. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
4 tasks
- Export EvalOutput, EvalResult, TestError, TestResult from evalFormatter so callers get proper types without casting - Rewrite buildResultSummary using the exported types, eliminating all Record<string,unknown> casts - Type AgentEvalRunResult.tests[].evaluations and .outputs as EvalResult[]/EvalOutput[] - Improve resolveAgent error messages to include actionable suggestions - Fix evalCounter to only increment when an id is auto-generated - Replace as-any stub cast with as-unknown StreamPromise in tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
@W-22203426@
evalNormalizer,evalFormatter,yamlSpecTranslatorfromplugin-agentinto this shared library so any consumer can use them without depending on the CLI pluginagentEvalRunnermodule that encapsulatesresolveAgent,executeBatches, andbuildResultSummary(API execution logic previously embedded in the run-eval command)index.tsevalNormalizer,evalFormatter,yamlSpecTranslator, andagentEvalRunnerTest plan
yarn compilepasses in agents repoyarn testpasses in agents repo (all new test files: evalNormalizer, evalFormatter, yamlSpecTranslator, agentEvalRunner)plugin-agentresolves correctly🤖 Generated with Claude Code