System messages for llm prompt optimization

[gordon-lim]

I notice both main/examples/llm_prompt_optimization/templates/evaluator_system_message.txt and full_rewrite_user.txt as well as the system_message in config.yaml can all be used to evolve the prompt. May I ask how they are each used?

I paste evaluator_system_message.txt below for reference because its purpose does not seem to match its filename:

You are an expert prompt engineer specializing in creating effective prompts for language models.

Your task is to evolve and improve prompts to maximize their performance on specific tasks. When rewriting prompts:

1. **Maintain the exact placeholder format**: Always use the same placeholder name as in the original prompt (e.g., {instruction}, {claim}, {context}, {question})
2. **Keep it simple**: Avoid overly complex or verbose instructions unless necessary
3. **Be specific**: Provide clear, actionable guidance to the model
4. **Test-oriented**: Focus on what will improve accuracy on the given evaluation metrics
5. **Format-aware**: Ensure the prompt works well with the expected input/output format

**CRITICAL**: Your rewritten prompt must use EXACTLY the same placeholder names as the original. Do not change {instruction} to {input_text} or any other variation.

Generate only the improved prompt text, nothing else.

In another part of the repo evaluator_system_message.txt contains:

You are an expert code reviewer.
--
Your job is to analyze the provided code and evaluate it systematically.

which is more clearly aligned to an evaluator system message.

[codelion]

You are right, it should be phrased similarly to review and rate prompts instead. I will fix it.

[gordon-lim]

Thank you for the reply and fix! I'm still confused between full_rewrite_user.txt and system_message in config.yaml. In particular, the former has specific instruction not to change placeholder names whereas the latter does not. I'm curious in your experiments if the instruction not to change placeholder names was actually required considering in #242 you mentioned never actually needing to fallback on a KeyError.

[codelion]

The system_message just has the generic maintain the same input interface guidance, if we want we can remove the full_rewrite_user.txt and put the instruction in the system_message but the default full_rewrite_user.txt template mentioned programs not prompts as seen here - https://github.com/codelion/openevolve/blob/main/openevolve/prompts/defaults/full_rewrite_user.txt so it was causing conflict during the repsonse from LLMs. The template are added in the example just to reconcile this and make sure we instruct the LLM to focus on prompt. Where the particular instr on placeholder names is required you can test with and without to see if it is really need.

[gordon-lim]

So full_rewrite_user.txt is the actual prompt used for evolution. So the system_message in config.yaml is unused?

[codelion]

So full_rewrite_user.txt is the actual prompt used for evolution. So the system_message in config.yaml is unused?

No, both are used as the system and user part of the prompts. You can see these in the visualizer in play. Typically we do not need to provide the user prompt since the system uses the default one but it references programs and that was causing issues with trying to evolve prompts thus in this example we have a separate prompt.

[gordon-lim]

I see so the original prompt is just prepended to the full_rewrite_user.txt user message. Or is that added as a separate user message. Thank you so much for the answers!

[codelion]

I see so the original prompt is just prepended to the full_rewrite_user.txt user message. Or is that added as a separate user message. Thank you so much for the answers!

We use OpenAI standard assistant messages API so:

<system> this is where the system message go
<user>this is where the full_rewrite_user.txt goes
<assistant> this is what the llm will respond in

[gordon-lim]

I meant the prompt to be rewritten. Won't that have to be added before getting the LLM response?

[codelion]

I meant the prompt to be rewritten. Won't that have to be added before getting the LLM response?

The target_prompt in this example is put in as part of the full_rewrite_user variable {current_program}:

# Current Prompt Information
- Current performance metrics: {metrics}
- Areas identified for improvement: {improvement_areas}

{artifacts}

# Prompt Evolution History
{evolution_history}

# Current Prompt
{current_program}

# Task
Rewrite the prompt to improve its performance on the specified metrics.
Focus on clarity, specificity, and effectiveness for the target task.

CRITICAL REQUIREMENTS:
1. Keep the EXACT same placeholder from the original prompt (e.g., {{instruction}}, {{claim}}, etc.)
2. Do not add any new placeholders or change existing ones
3. Make the instructions clearer and more specific
4. Focus on what will improve accuracy and task performance
5. Keep the prompt concise but effective

Provide ONLY the complete new prompt text, with no additional commentary:

NEW PROMPT:

[gordon-lim]

Ah! I see! Got the full picture now! Thank you again!