Eval metadata

[miguelg719]

why

To help make sense of eval test cases and results

what changed

Added metadata to eval runs, cleaned deprecated code

test plan

[changeset-bot]

⚠️ No Changeset found

Latest commit: 904bc53

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[greptile-apps]

Greptile Summary

This PR enhances the evaluation system with comprehensive error handling and metadata improvements. The changes introduce a structured ErrorType enum for better error categorization, expand the EvalOutput interface with additional metadata fields, and improve task loading with better error reporting. The PR also adds process-level error handlers and enhances resource cleanup with safer stagehand closing logic.

Key improvements:

• Enhanced error categorization with specific error types (timeout, network, parsing, etc.)
• Better task loading logic with fallback paths for .js and .ts files
• Improved metadata tracking with category information in testcases
• Safer resource cleanup with proper stagehand instance management
• More comprehensive error reporting with stack traces and error details

Confidence score: 4/5

• This PR is generally safe to merge with proper testing
• Score reflects solid improvements to error handling and metadata tracking, with only one minor issue around deterministic file loading that should be addressed
• evals/index.eval.ts needs attention for the task loading logic

_{2 files reviewed, 1 comment}

_{Edit Code Review Bot Settings | Greptile}