Question: How to log the trace that openevolve iterates and improve?

[zigzagcai]

Hi,

We want to log the trace that openevolve iterate over multiple steps, and logs how it improves over the previous best programs.

The trace can be useful for RL training.

So, does openevolve already support it? Or how can we modify the framework code to get it supported?

[codelion]

OpenEvolve already captures significant evolution data that could be used for RL training:

What's Already Available:

• Full Lineage Tracking: Each program has parent_id, allowing complete lineage reconstruction
• Performance History: Programs store parent_metrics in metadata, showing previous performance
• Temporal Information: iteration_found tracks when programs were discovered
• Persistent Storage: All programs are saved to disk in JSON format at checkpoints
• Visualization Support: The visualizer can reconstruct the full evolution tree from this data

You can access this data today by:

1. Loading checkpoint data from openevolve_output/checkpoints/checkpoint_*/programs/*.json
2. Each program JSON contains parent-child relationships and metrics
3. The metadata.json file contains the overall evolution state

What's Missing for RL Training:

However, for RL training, you likely need a more structured, real-time trace format that logs:

1. State-action-reward transitions at each iteration
2. Improvement trajectories over the global best
3. Prompt-response pairs for understanding LLM decisions
4. Complete evolution trajectories in a format suitable for RL training

Proposed Implementation

1. Create Evolution Trace Module (openevolve/evolution_trace.py)

• New EvolutionTrace class to track state transitions
• Store: iteration, parent state, action (code changes), child state, reward (improvement)
• Support multiple formats: JSON lines for streaming, HDF5 for efficient storage
• Track both local improvements (parent→child) and global improvements (vs best)

2. Modify Process Controller (openevolve/process_parallel.py)

• Add trace recording after each successful iteration
• Record: parent metrics, child metrics, improvement delta, prompts, LLM responses
• Track improvements against both parent and global best program
• Store trace in real-time for online RL training compatibility

3. Add Configuration Options

evolution_trace:
  enabled: true
  format: 'jsonl'  # or 'hdf5', 'both'
  include_code: false  # Can be large
  include_prompts: true
  output_path: 'evolution_trace.jsonl'

4. Create Trace Export Utilities

• Export traces to common RL formats (e.g., D4RL, RLlib)
• Convert traces to state-action-reward-next_state tuples
• Support filtering traces by improvement thresholds

Benefits for RL Training

The trace would provide:

• States: Program metrics, complexity, diversity scores
• Actions: Code modifications (diffs or rewrites)
• Rewards: Improvement in metrics (both immediate and cumulative)
• Trajectories: Complete evolution paths from initial to best programs
• Context: LLM prompts and responses for understanding decision-making

This would enable:

• Training RL agents to guide evolution more efficiently
• Learning from successful evolution trajectories
• Understanding which code modifications lead to improvements
• Potentially replacing or augmenting LLM-based evolution with learned policies

Interim Solution

While this feature is being developed, you can extract traces from existing checkpoints:

import json
import os
from pathlib import Path

def extract_evolution_trace(checkpoint_dir):
    """Extract evolution trace from checkpoint data"""
    traces = []
    programs_dir = Path(checkpoint_dir) / 'programs'
    
    # Load all programs
    programs = {}
    for prog_file in programs_dir.glob('*.json'):
        with open(prog_file) as f:
            prog = json.load(f)
            programs[prog['id']] = prog
    
    # Build parent-child traces
    for prog_id, prog in programs.items():
        if prog.get('parent_id') and prog['parent_id'] in programs:
            parent = programs[prog['parent_id']]
            trace = {
                'iteration': prog.get('iteration_found', 0),
                'parent_id': prog['parent_id'],
                'child_id': prog_id,
                'parent_metrics': parent.get('metrics', {}),
                'child_metrics': prog.get('metrics', {}),
                'improvement': calculate_improvement(parent, prog),
                'generation': prog.get('generation', 0)
            }
            traces.append(trace)
    
    return sorted(traces, key=lambda x: x['iteration'])

def calculate_improvement(parent, child):
    """Calculate improvement between parent and child"""
    parent_score = parent.get('metrics', {}).get('combined_score', 0)
    child_score = child.get('metrics', {}).get('combined_score', 0)
    return child_score - parent_score

[zigzagcai]

OpenEvolve already captures significant evolution data that could be used for RL training:

What's Already Available:

• Full Lineage Tracking: Each program has parent_id, allowing complete lineage reconstruction
• Performance History: Programs store parent_metrics in metadata, showing previous performance
• Temporal Information: iteration_found tracks when programs were discovered
• Persistent Storage: All programs are saved to disk in JSON format at checkpoints
• Visualization Support: The visualizer can reconstruct the full evolution tree from this data

You can access this data today by:

1. Loading checkpoint data from openevolve_output/checkpoints/checkpoint_*/programs/*.json
2. Each program JSON contains parent-child relationships and metrics
3. The metadata.json file contains the overall evolution state

What's Missing for RL Training:

However, for RL training, you likely need a more structured, real-time trace format that logs:

1. State-action-reward transitions at each iteration
2. Improvement trajectories over the global best
3. Prompt-response pairs for understanding LLM decisions
4. Complete evolution trajectories in a format suitable for RL training

Proposed Implementation

1. Create Evolution Trace Module (openevolve/evolution_trace.py)

• New EvolutionTrace class to track state transitions
• Store: iteration, parent state, action (code changes), child state, reward (improvement)
• Support multiple formats: JSON lines for streaming, HDF5 for efficient storage
• Track both local improvements (parent→child) and global improvements (vs best)

2. Modify Process Controller (openevolve/process_parallel.py)

• Add trace recording after each successful iteration
• Record: parent metrics, child metrics, improvement delta, prompts, LLM responses
• Track improvements against both parent and global best program
• Store trace in real-time for online RL training compatibility

3. Add Configuration Options

evolution_trace:
enabled: true
format: 'jsonl' # or 'hdf5', 'both'
include_code: false # Can be large
include_prompts: true
output_path: 'evolution_trace.jsonl'

4. Create Trace Export Utilities

• Export traces to common RL formats (e.g., D4RL, RLlib)
• Convert traces to state-action-reward-next_state tuples
• Support filtering traces by improvement thresholds

Benefits for RL Training

The trace would provide:

• States: Program metrics, complexity, diversity scores
• Actions: Code modifications (diffs or rewrites)
• Rewards: Improvement in metrics (both immediate and cumulative)
• Trajectories: Complete evolution paths from initial to best programs
• Context: LLM prompts and responses for understanding decision-making

This would enable:

• Training RL agents to guide evolution more efficiently
• Learning from successful evolution trajectories
• Understanding which code modifications lead to improvements
• Potentially replacing or augmenting LLM-based evolution with learned policies

Interim Solution

While this feature is being developed, you can extract traces from existing checkpoints:

import json
import os
from pathlib import Path

def extract_evolution_trace(checkpoint_dir):
"""Extract evolution trace from checkpoint data"""
traces = []
programs_dir = Path(checkpoint_dir) / 'programs'

# Load all programs
programs = {}
for prog_file in programs_dir.glob('*.json'):
    with open(prog_file) as f:
        prog = json.load(f)
        programs[prog['id']] = prog

# Build parent-child traces
for prog_id, prog in programs.items():
    if prog.get('parent_id') and prog['parent_id'] in programs:
        parent = programs[prog['parent_id']]
        trace = {
            'iteration': prog.get('iteration_found', 0),
            'parent_id': prog['parent_id'],
            'child_id': prog_id,
            'parent_metrics': parent.get('metrics', {}),
            'child_metrics': prog.get('metrics', {}),
            'improvement': calculate_improvement(parent, prog),
            'generation': prog.get('generation', 0)
        }
        traces.append(trace)

return sorted(traces, key=lambda x: x['iteration'])

def calculate_improvement(parent, child):
"""Calculate improvement between parent and child"""
parent_score = parent.get('metrics', {}).get('combined_score', 0)
child_score = child.get('metrics', {}).get('combined_score', 0)
return child_score - parent_score

@codelion, Thanks for your detailed explanation!

[zigzagcai]

@codelion

Hi, just another question, if we also want to get the prompt for each improvement, how can we make the saved program information include the prompt, and finally the post-process script can include the prompt in each improvement.
We just want the action information in each improvement.

Thanks in advance!

[codelion]

OpenEvolve already has built-in support for logging prompts with each program. The prompts (which represent the "action" in RL terms) are automatically saved when log_prompts: true is set in your configuration.

How to Enable Prompt Logging

Add this to your config file (or verify it's present - it's true by default):

database:
  log_prompts: true  # Default is true, so prompts are logged automatically

What Gets Saved

When prompt logging is enabled, each program in the checkpoint includes:

{
  "id": "program_id",
  "code": "/* the evolved code */",
  "parent_id": "parent_program_id",
  "metrics": {
    "score": 0.95
    // other metrics
  },
  "metadata": {
    "changes": "Description of what was changed",
    "parent_metrics": {...}
  },
  "prompts": {
    "diff_user": {  // or "full_rewrite_user" depending on evolution mode
      "system": "System prompt used",
      "user": "Full user prompt with context about parent program",
      "responses": ["LLM's response including the code changes"]
    }
  }
}

Post-Processing Script to Extract Evolution Trace with Actions

Here's a script to extract the complete evolution trace including the prompts (actions):

import json
import os
from pathlib import Path

def load_checkpoint_programs(checkpoint_path):
    """Load all programs from a checkpoint"""
    programs_dir = Path(checkpoint_path) / "programs"
    programs = {}
    
    for program_file in programs_dir.glob("*.json"):
        with open(program_file) as f:
            program = json.load(f)
            programs[program["id"]] = program
    
    return programs

def extract_evolution_trace(checkpoint_path, output_file="evolution_trace.json"):
    """Extract complete evolution traces with prompts/actions"""
    programs = load_checkpoint_programs(checkpoint_path)
    
    # Build lineage for each program
    traces = []
    
    for program_id, program in programs.items():
        # Build lineage chain
        lineage = []
        current = program
        
        while current:
            lineage.append(current)
            parent_id = current.get("parent_id")
            current = programs.get(parent_id) if parent_id else None
        
        # Reverse to get chronological order
        lineage.reverse()
        
        # Extract improvement steps with actions
        improvements = []
        for i in range(len(lineage) - 1):
            parent = lineage[i]
            child = lineage[i + 1]
            
            # Extract the action (prompt and response)
            prompts = child.get("prompts", {})
            action = None
            
            # Get the prompt used (could be diff_user or full_rewrite_user)
            for template_key, prompt_data in prompts.items():
                action = {
                    "template": template_key,
                    "system_prompt": prompt_data.get("system", ""),
                    "user_prompt": prompt_data.get("user", ""),
                    "llm_response": prompt_data.get("responses", [""])[0] if prompt_data.get("responses") else ""
                }
                break
            
            improvement = {
                "step": i,
                "parent_id": parent["id"],
                "child_id": child["id"],
                "parent_metrics": parent.get("metrics", {}),
                "child_metrics": child.get("metrics", {}),
                "improvement": {
                    metric: child.get("metrics", {}).get(metric, 0) - parent.get("metrics", {}).get(metric, 0)
                    for metric in child.get("metrics", {}).keys()
                },
                "changes_description": child.get("metadata", {}).get("changes", ""),
                "action": action  # The prompt that led to this improvement
            }
            improvements.append(improvement)
        
        if improvements:  # Only add traces that have improvements
            traces.append({
                "final_program_id": program_id,
                "final_metrics": program.get("metrics", {}),
                "generation_depth": len(lineage),
                "improvement_steps": improvements
            })
    
    # Save to file
    with open(output_file, "w") as f:
        json.dump(traces, f, indent=2)
    
    print(f"Saved {len(traces)} evolution traces to {output_file}")
    return traces

# Usage
if __name__ == "__main__":
    # Point to your checkpoint directory
    checkpoint_path = "openevolve_output/checkpoints/checkpoint_100"
    traces = extract_evolution_trace(checkpoint_path)
    
    # Example: Print first improvement with its action
    if traces and traces[0]["improvement_steps"]:
        first_improvement = traces[0]["improvement_steps"][0]
        print("\nFirst improvement action:")
        print(f"Changes: {first_improvement['changes_description']}")
        if first_improvement["action"]:
            print(f"LLM Response Preview: {first_improvement['action']['llm_response'][:200]}...")

For RL Training Format

If you need the data in a specific format for RL training, here's an example that extracts state-action-reward tuples:

def extract_rl_training_data(checkpoint_path):
    """Extract data in RL-friendly format"""
    programs = load_checkpoint_programs(checkpoint_path)
    
    training_data = []
    
    for program in programs.values():
        if not program.get("parent_id"):
            continue  # Skip initial programs
            
        parent = programs.get(program["parent_id"])
        if not parent:
            continue
            
        # State: parent program + metrics
        state = {
            "code": parent["code"],
            "metrics": parent.get("metrics", {})
        }
        
        # Action: the prompt and response
        action = {}
        prompts = program.get("prompts", {})
        for template_key, prompt_data in prompts.items():
            action = {
                "prompt": prompt_data.get("user", ""),
                "response": prompt_data.get("responses", [""])[0] if prompt_data.get("responses") else ""
            }
            break
        
        # Reward: improvement in metrics
        reward = 0
        if "combined_score" in program.get("metrics", {}):
            parent_score = parent.get("metrics", {}).get("combined_score", 0)
            child_score = program.get("metrics", {}).get("combined_score", 0)
            reward = child_score - parent_score
        
        # Next state: child program + metrics
        next_state = {
            "code": program["code"],
            "metrics": program.get("metrics", {})
        }
        
        training_data.append({
            "state": state,
            "action": action,
            "reward": reward,
            "next_state": next_state,
            "metadata": {
                "parent_id": program["parent_id"],
                "child_id": program["id"],
                "changes": program.get("metadata", {}).get("changes", "")
            }
        })
    
    return training_data

Important Notes

1. If prompts are missing: Make sure you're using a recent version of OpenEvolve. The prompt logging feature was added recently.

2. For existing runs: If you have older runs without prompts, you'll need to re-run the evolution with log_prompts: true to capture this data.

3. Storage overhead: Prompts can be large, so they do increase checkpoint size. This is why it's configurable via log_prompts.

Let me know if you need help adapting the extraction script for your specific RL training framework!

[zigzagcai]

Thank you @codelion , it is awesome that this feature is supported!!!