The issue of the temperature parameter #162
Description
I’ve encountered something confusing while using the OpenEvolve framework, and I’d appreciate some clarification.
In the OpenAILLM class in your openai.py file, there is this code block when setting up generation parameters:
if self.api_base == "https://api.openai.com/v1" and str(self.model).lower().startswith("o"):
# For o-series models
params = {
"model": self.model,
"messages": formatted_messages,
"max_completion_tokens": kwargs.get("max_tokens", self.max_tokens),
}
else:
params = {
"model": self.model,
"messages": formatted_messages,
"temperature": kwargs.get("temperature", self.temperature),
"top_p": kwargs.get("top_p", self.top_p),
"max_tokens": kwargs.get("max_tokens", self.max_tokens),
}
It seems that when the model name starts with "o" (e.g., "o4-mini"), the temperature parameter is not passed to the API.
I’ve also tested an example based on your function_minimization. I set temperature values like 0.3 vs 0.7 with o4-mini(just set one model) and observed significant behavioral differences in the output, which suggests that these parameters are indeed supported and have an effect.
This is my config.yaml
max_iterations: 10
checkpoint_interval: 5
log_level: "INFO"
primary_model: "o4-mini"
primary_model_weight: 1.0
temperature: 0.3
api_base: "XXXX"
api_key: "XXXX"
max_tokens: 4096
prompt:
include_artifacts: true
system_message: "You are an expert programmer specializing in optimization algorithms. Your task is to improve a function minimization algorithm to find the global minimum of a complex function with many local minima. The function is f(x, y) = sin(x) * cos(y) + sin(x*y) + (x^2 + y^2)/20. Focus on improving the search_algorithm function to reliably find the global minimum, escaping local minima that might trap simple algorithms."
evaluator_system_message: "You need to ensure each round score is improved by the last best one"
num_top_programs: 1
max_artifact_bytes: 4096
use_template_stochasticity: true
artifact_security_filter: true
database:
population_size: 1
archive_size: 1
num_islands: 1
elite_selection_ratio: 0.3
exploitation_ratio: 0.7
evaluator:
timeout: 60
cascade_evaluation: true
enable_artifacts: true
diff_based_evolution: false
allow_full_rewrites: true
Testing result
When Temperature = 0.7
0.767818. 0.954938 0.945888 0.952813 0.951586 0.779630 0.952998 0.797274 0.787243 0.986230 0.974237
Temperature = 0.3
0.767818 0.706419 0.784706 0.777148 0.988748 0.994253 0.702351 0.797274 0.999715 0.981818 0.999708
three questions:
Q1 Why are the two results different? If the temperature cannot be passed, the result should be the same.
Q2 Can OpenEvolve guarantee that each successive round’s score improves over the previous one and demonstrates a steadily converging trend?
Q3 Whether or not they can set elite_selection_ratio=1.0 and exploitation_ratio=1.0 at the same time?
Could you please provide some explanations? Thank you so much.