Skip to content

Feature: Add tools metadata for eval viewer#767

Merged
willccbb merged 3 commits intoPrimeIntellect-ai:mainfrom
d42me:feature/tools-metadata
Jan 23, 2026
Merged

Feature: Add tools metadata for eval viewer#767
willccbb merged 3 commits intoPrimeIntellect-ai:mainfrom
d42me:feature/tools-metadata

Conversation

@d42me
Copy link
Collaborator

@d42me d42me commented Jan 22, 2026

Description

Add tools field to evaluation metadata to expose available tool definitions (name, description, parameters) in saved results. Tools are stored in metadata when uniform across all samples, or null when tools vary per sample (e.g., mcp_universe, fh_aviary).

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Test improvement

Testing

  • All existing tests pass when running uv run pytest locally.
  • New tests have been added to cover the changes

Checklist

  • My code follows the style guidelines of this project as outlined in AGENTS.md
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

Note

Exposes available tool definitions in evaluation outputs for downstream viewers.

  • Adds tools field to GenerateMetadata in types.py
  • In environment._prepare_rollout_results, aggregates oai_tools from states, detects variance by tool function names, and sets metadata.tools to the shared set or None
  • Persists tools alongside other metadata for saved results

Written by Cursor Bugbot for commit 95cccb3. This will update automatically on new commits. Configure here.

Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 3 potential issues.

Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.

@d42me d42me requested a review from willccbb January 23, 2026 00:10
@willccbb willccbb merged commit e3f67b9 into PrimeIntellect-ai:main Jan 23, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants