Fix AppWorld evaluation and improve prompt #20
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 5 commits
October 27, 2025 15:47
- Fix: Save model_hashes.json to enable proper change detection * AppWorld's evaluator relies on model hash counters to detect DB changes * Without saving these hashes, evaluation fails even when agent completes * Added save_model_hashes=True in mcp_server.py:234 - Improve: Remove turn limit from agent prompt * Removed max_steps parameter and references from system instruction * Eliminates artificial pressure on agent to be overly conservative * Improves task completion rate by reducing unnecessary batching - Increase test timeout from 5 to 15 minutes * Accounts for bearer token expiration issues with some tasks * Prevents premature timeout on complex tasks Verified with successful test runs on tasks 692c77d_2 and 22cc237_3.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix: Save model_hashes.json to enable proper change detection
Improve: Remove turn limit from agent prompt
Increase test timeout from 5 to 15 minutes
Verified with successful test runs on tasks 692c77d_2 and 22cc237_3.