Questions in using artifacts

[ypwang61]

Just for double checking, if we want the model to change with previous results (suboptimal result, compilation errors, etc, we need to :

1. In config.yaml, add enable_artifacts: true to evaluator, include_artifacts: true \n max_artifact_bytes: 32768 # Include larger artifacts (32KB) \n artifact_security_filter: true to prompt
2. In evaluator, add artifacts output in examples/xxx_problem/evaluator.py as circle_packing_artifacts

Is this understanding correct? (like both are necessary?)

[codelion]

Your understanding is partially correct, but there's an important correction:

• ❌ There is NO enable_artifacts: true under evaluator: in config.yaml
• ✅ You only need include_artifacts: true under prompt: in config.yaml
• ✅ You DO need to modify your evaluator to return EvaluationResult objects

Both changes are necessary for artifacts to work properly.

Correct Configuration

1. Config.yaml Settings

# config.yaml
prompt:
  include_artifacts: true             # Enable artifacts in LLM prompts
  max_artifact_bytes: 32768          # Include larger artifacts (32KB)
  artifact_security_filter: true     # Apply security filtering

# Note: There is NO enable_artifacts under evaluator section

2. Evaluator Implementation

# evaluator.py
from openevolve.evaluation_result import EvaluationResult

def evaluate(program_path):
    # ... your evaluation logic ...
    
    # Return EvaluationResult instead of dict
    return EvaluationResult(
        metrics={
            "score": 0.95,
            "combined_score": 0.95,  # Required for scoring
            # ... other metrics
        },
        artifacts={
            "stdout": "Program output...",
            "stderr": "Error messages if any...",
            "execution_time": "12.45s",
            "validation_report": "Details about what went wrong...",
            # ... any debugging info that helps LLM understand failures
        }
    )

How Artifacts Work

1. Evaluator returns EvaluationResult with both metrics and artifacts
2. OpenEvolve automatically collects and stores these artifacts
3. PromptSampler includes artifacts in prompts when include_artifacts: true
4. LLM sees the artifacts and can fix issues based on error messages

Example: Circle Packing Problem

# From examples/circle_packing_with_artifacts/evaluator.py
return EvaluationResult(
    metrics={
        "sum_radii": 45.2,
        "target_ratio": 0.89,
        "validity": 1.0,
        "combined_score": 0.89,
    },
    artifacts={
        "stderr": "IndexError: list index out of range",
        "failure_stage": "circle_generation",
        "debug_info": "Failed at circle 15 of 26",
        "execution_time": "2.3s",
    }
)

Key Points

1. Both changes are necessary:

   • Config: include_artifacts: true under prompt:
   • Evaluator: Return EvaluationResult objects

2. Backward compatible: Old evaluators returning dicts still work (just no artifacts)

3. Size limits: Artifacts over max_artifact_bytes are truncated in prompts

4. Security: artifact_security_filter removes sensitive data like API keys

5. Storage: Small artifacts (<10KB) stored in database, large ones on disk

Common Mistakes

• ❌ Adding enable_artifacts under evaluator: (doesn't exist)
• ❌ Returning artifacts in the metrics dict
• ❌ Forgetting to import EvaluationResult
• ❌ Not including combined_score in metrics (required for scoring)

Environment Variable

You can globally disable artifacts with:

export ENABLE_ARTIFACTS=false

This overrides all config settings and is useful for debugging.