Different handling of programs that fail to run?

[benjaminy]

I'm not sure how common this is, but in the experiments I've been doing (relatively big program, mostly using Gemini 2.5 pro and Claude 4 Opus), in most iterations the LLM gives back a program that at least works. But non-functional programs are somewhat common too (using nonexistent API, syntax errors). As I understand it, the common practice in this case is to give the iteration some made up low combined_score. That's ok, but it seems a bit of a waste. I wonder if anyone else thinks it might be useful to build in an additional special case iteration with the LLM where a prompt is generated that says something like "You gave me this code XXX, but it failed with this error YYY. Try to fix the error without significant logic changes".

[codelion]

Thanks for raising this interesting point! Actually, OpenEvolve already implements a similar mechanism through its artifacts side-channel system. When programs fail with errors, the system captures detailed error information and passes it to the LLM in the next iteration, effectively achieving what you're suggesting.

How It Works

1. Error Capture in Evaluators

When evaluation fails, evaluators can return detailed error information as artifacts. Here's an example from examples/circle_packing_with_artifacts/evaluator.py:331-349:

except Exception as e:
    error_msg = f"Evaluation failed completely: {str(e)}"
    traceback.print_exc()
    return EvaluationResult(
        metrics={
            "sum_radii": 0.0,
            # ... other metrics set to 0
        },
        artifacts={
            "stderr": error_msg,
            "traceback": traceback.format_exc(),
            "failure_stage": "program_execution", 
            "suggestion": "Check for syntax errors, import issues, or runtime exceptions",
        },
    )

2. Artifacts in Next Iteration's Prompt

These artifacts are automatically included in the next generation's prompt through openevolve/prompt/sampler.py:476-479:

# Format artifacts section if enabled and available
artifacts_section = ""
if self.config.include_artifacts and program_artifacts:
    artifacts_section = self._render_artifacts(program_artifacts)

The artifacts appear in the prompt as:

## Last Execution Output

### stderr
NameError: name 'np' is not defined

### traceback
Traceback (most recent call last):
  File "...", line 15, in <module>
    centers = np.zeros((26, 2))
NameError: name 'np' is not defined

### failure_stage
program_execution

### suggestion
Check for syntax errors, import issues, or runtime exceptions

3. Real-World Example

The examples/circle_packing_with_artifacts/README.md:89-112 shows a concrete example:

Generation N: Program fails with overlapping circles

centers[5] = [0.3, 0.3]  
centers[6] = [0.3, 0.3]  # Same position - error!

Artifacts captured:

Circles 5 and 6 overlap: dist=0.000000, r1+r2=0.200000

Generation N+1: LLM sees the overlap details and fixes it

centers[5] = [0.3, 0.3]
centers[6] = [0.5, 0.3]  # Different position - fixed!

Configuration

To enable artifacts (enabled by default):

# In config.yaml
include_artifacts: true
max_artifact_bytes: 10000

Or via environment variable:

export ENABLE_ARTIFACTS=true

Benefits Over Additional Error-Fixing Iteration

The current approach has several advantages:

1. No extra LLM calls - Error context is naturally included in the next evolution iteration
2. Maintains evolution flow - Programs still compete based on fitness, even failed ones
3. Rich context - The LLM sees not just the error but also performance metrics and evolution history
4. Flexible - Evaluators can provide domain-specific error guidance

For Your Use Case

If you're seeing many syntax errors with Gemini 2.5 Pro and Claude Opus, you might want to:

1. Ensure artifacts are enabled in your config
2. Enhance your evaluator to capture more detailed error information
3. Consider adding domain-specific error suggestions in the artifacts

The system already provides the error feedback mechanism you're looking for - it just needs to be leveraged through proper evaluator implementation!

Let me know if you'd like examples of how to enhance your evaluator to provide better error feedback through artifacts.

[benjaminy]

Thank you for your detailed response. I am already using artifacts. (Of course maybe the details matter and I could be doing something better.) One sanity check: include_artifacts defaults to True, right? So this mechanism is enabled if no value is given in the config file?

Is there any special logic/handling for the keys in the artifacts dict in your example (stderr, traceback, failure_stage, suggestion)?

I wouldn't say I'm seeing many errors. Maybe 5% of iterations (rough estimate with high variance). But when I see an iteration fail because the generated code refers to a nonexistent variable or function it feels like I would rather not include that version in the archive at all. Either give the LLM a chance to fix the uninteresting mistake(s) or "skip" that version entirely. I can see how that would be bad if the system was struggling to generate any good versions; I can see the importance of error message feedback in that case. But that's not what I'm seeing.

Your "Maintain evolution flow" point feels off to me, but I'm not sure. Say I'm trying to optimize cars and in one test a car gets a nail in its tire and completely fails. That doesn't seem like an interesting test to include. Or maybe a better analogy would be the test driver shows up drunk and crashes to no particular fault of that car design.