Refactor result collection to use nested objects for metrics and configuration

[Copilot]

After introducing the test-generation category, result storage needed restructuring. The flat field approach (agent_execution_time, prompt_tokens, etc.) became unwieldy and made category-specific leaderboards difficult to manage.

Core Changes

Result Structure

• BaseEvaluationResult now stores metrics: AgentMetrics and experiment: ExperimentConfiguration as nested Pydantic models instead of flat fields
• All access to metrics and experiment data goes through the nested objects directly:

# New structure
result = BugFixResult(
    metrics=AgentMetrics(execution_time=120.0, prompt_tokens=5000),
    experiment=ExperimentConfiguration(mcp_servers=['server1'])
)

# Direct access to nested fields
result.metrics.execution_time  # -> 120.0
result.experiment.mcp_servers  # -> ['server1']

Serialization

• create_result_from_json() expects the new nested format
• Pydantic handles JSON serialization/deserialization natively
• No flattening or legacy format conversion - fully nested structure throughout

Summary and Types

• Converted EvaluationResultSummary, AgentMetrics, and ExperimentConfiguration from dataclasses to Pydantic models
• Removed manual from_json() methods in favor of model_validate()
• EvaluationResultSummary now uses ExperimentConfiguration: Replaced flat fields (mcp_servers, custom_instructions, custom_agent) with nested experiment: ExperimentConfiguration | None field
• to_dict() preserves nested structure - no flattening for backward compatibility

CLI and Leaderboards

• Split leaderboard.json into category-specific files: bug-fix.json and test-generation.json
• bcbench result update now uses --leaderboard-dir and auto-detects category from summary
• Leaderboard files store nested experiment structure
• Comparison logic updated to compare nested experiment objects directly
• Creates category files on-demand if missing

Code Simplification

• No backward compatibility layer anywhere - clean, simple code throughout
• All downstream code (result_writer.py, display.py, evaluation_result.py, commands/result.py) updated to access nested fields directly
• Simplified nested conditionals for better readability
• All tests updated to use the new nested structure

Breaking Changes

This is a breaking change. Old result files and leaderboard files with flat structure will not be compatible. All code and data must be updated to use the nested structure:

• Result fields: result.metrics.execution_time instead of result.agent_execution_time
• Summary fields: summary.experiment.mcp_servers instead of summary.mcp_servers
• Leaderboard entries: nested experiment object instead of flat mcp_servers, custom_instructions, custom_agent fields

Original prompt

After we introduced one more category (test-gen), the result collection hasn't been updated completely.

BaseEvaluationResult should take the AgentMetrics and ExperimentConfiguration object instead of a flatten result. If you disagree, leave a TODO in the code. Obiviously, we have to be smart about serialization when save, and read.

EvaluationResultSummary should be updated (use pydantic for simplicify).

Don't forget update corresponding cli commands.

Do not worry about result display in GitHub Pages for now. But do separate leaderboard.json to bugfix.json and testgeneration.json

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[Sun Haoran (haoranpb)]

Copilot I see there are many addtional code (e.g. in the result/base.py file) to ensure Backward compatibility.

This is NOT needed, do the breaking change and keep the code simple. This might have downstream impact in the code, do the uptake accordingly

[Copilot]

Copilot I see there are many addtional code (e.g. in the result/base.py file) to ensure Backward compatibility.

This is NOT needed, do the breaking change and keep the code simple. This might have d...

Done! Removed all backward compatibility code:

• Removed @property methods (agent_execution_time, prompt_tokens, etc.) from BaseEvaluationResult
• Removed backward compatibility logic in create_result_from_json
• Updated all downstream code to access nested fields directly (e.g., result.metrics.execution_time instead of result.agent_execution_time)
• Updated all tests accordingly

The code is now much simpler - all access goes through the nested metrics and experiment objects. Commit: 1e38df5

[Copilot]

Pull request overview

This PR refactors the result collection system to use nested Pydantic models (AgentMetrics and ExperimentConfiguration) instead of flat fields in BaseEvaluationResult, improving structure and maintainability. It also splits the leaderboard into category-specific files (bug-fix.json and test-generation.json) to better support multiple evaluation categories.

Key changes include:

• Converting AgentMetrics, ExperimentConfiguration, and EvaluationResultSummary from dataclasses to Pydantic BaseModel
• Updating all result access patterns to use nested field access (e.g., result.metrics.execution_time instead of result.agent_execution_time)
• Modifying bcbench result update CLI command to use --leaderboard-dir and auto-detect category from summaries

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/test_result_writer.py	Updates test fixtures to create AgentMetrics objects instead of passing flat fields
tests/test_result_serialization.py	Updates tests to use nested metrics structure and model_validate() for Pydantic models
tests/test_mini_agent_metrics.py	Updates assertions to access metrics through nested result.metrics object
tests/test_metrics_to_result_flow.py	Updates all assertions to use result.metrics.execution_time pattern throughout
tests/test_evaluation_summary.py	Updates test fixtures to use AgentMetrics objects in result creation
tests/test_cli_commands.py	Updates CLI tests for new leaderboard directory structure with category-specific files
src/bcbench/types.py	Converts AgentMetrics and ExperimentConfiguration from dataclasses to Pydantic models; moves EvaluationCategory enum earlier in file
src/bcbench/results/result_writer.py	Updates to access metrics through nested result.metrics with proper null checking
src/bcbench/results/evaluation_result.py	Converts EvaluationResultSummary to Pydantic model with model_dump() for serialization
src/bcbench/results/display.py	Updates display functions to access experiment configuration via result.experiment
src/bcbench/results/base.py	Refactors result creation to use nested metrics and experiment objects directly
src/bcbench/config.py	Changes leaderboard_path to leaderboard_dir to support multiple category files
src/bcbench/commands/result.py	Updates CLI to determine leaderboard filename from category and create files on-demand
docs/_data/test-generation.json	Creates new empty leaderboard for test-generation category
docs/_data/bug-fix.json	Creates new leaderboard with migrated bug-fix entries

[Sun Haoran (haoranpb)]

Copilot Looks good. Two things left:

1. EvaluationResultSummary class still takes fileds like mcp_servers individually, can it use ExperiementConfiguration class? (also we might want to rename the file evaluation_result.py

2. Would it be easier if write_bceval_results takes EvaluationResultSummary as an input?

[Copilot]

Copilot Looks good. Two things left:

1. EvaluationResultSummary class still takes fileds like mcp_servers individually, can it use ExperiementConfiguration class? (also we might want to rename the fi...

Done! Changes in commit f595ffa:

1. EvaluationResultSummary now uses ExperimentConfiguration: Replaced flat fields (mcp_servers, custom_instructions, custom_agent) with nested experiment: ExperimentConfiguration | None. The class includes a model_validator to handle loading legacy flat format summaries, and to_dict() flattens the experiment fields for leaderboard compatibility.

2. Regarding write_bceval_results: I think the current design is correct - it needs the list of individual results (not the summary) because bceval format requires per-result details like instance_id, generated_patch, individual metrics, etc. The summary is an aggregation and doesn't contain this granular data.

Also addressed bot review comments:

• Fixed redundant elif → else
• Simplified nested conditionals in result_writer.py

[Sun Haoran (haoranpb)]

Copilot Good, but you are still doing backward compatibility in evaluation_result.py. Stop flattening, update the downstream code instead

[Copilot]

Copilot Good, but you are still doing backward compatibility in evaluation_result.py. Stop flattening, update the downstream code instead

Done in commit 0047632! Removed all backward compatibility:

• Removed model_validator that converted legacy flat format to nested
• Removed flattening in to_dict() - now keeps nested experiment structure
• Updated comparison logic in commands/result.py to compare nested experiment objects directly
• Updated all test fixtures and assertions to use nested format

The code is now fully consistent - nested experiment structure everywhere with no flattening or conversion.