[Question] Process for creating system prompt.

[amueller]

When looking at the system prompts in the examples, say for mlx_metal_kernel_opt or attention_optimization these seem very specific and involved.

Can you elaborate on how these were generated? Since meta-prompt evolution is not implemented yet #170, I assume these are either hand-written, or more likely llm-generated? Either way, do you have advice on how to design these prompts, and how to iterate on them? How many different iterations of the system prompt did you create before landing on these?

Thanks!

[codelion]

Prompts were hand written with assistance from an LLM but outside of the openevolve system. Depending on the problem it requires several iterations. You can see this in the circle packing example (https://github.com/codelion/openevolve/tree/main/examples/circle_packing) where we have 2 different config for 2 phases.

The way I worked on most of the examples was to start with an initial prompt and then see if we make progress with a few iterations of openevolve. Then keep updating on the prompt to see how much further we can push the improvements. Many of the details in the prompt around what library/framework to use or specific things to avoid were added after observing if the system gets stuck on some error during evolution. With the artifacts feedback enabled it eventually does fix those errors but this helps in speeding up the convergence.

The prompts (specially the system message for each example can easily be evolved as well using prompt optimization using openevolve. It is not yet implemented directly but you can create an evaluator that runs another instance of openevolve and measures the improvements over combined score while updating the prompt similar to what is done in the prompt optimization example.

[amueller]

Thank you for your quick response and the details! I think it would be great if you could add some of that to the docs, since that (together with writing the evaluation function) is probably one of the main tasks to do when trying out the system on a new task.

[codelion]

That's a good idea I will update the documentation.