How to dynamically change API parameters through oaieval CLI tool?

[leodiegues]

I'm trying to run an eval based on evals.elsuite.basic.match:Match class but I can't find a way to modify API parameters such as temperature and max_tokens through the CLI. What is the current solution that the framework offers to this problem?

[andrew-openai]

It's a good idea! I added a --modelspec_extra_options that would support setting those values from the CLI, e.g.

oaieval gpt-3.5-turbo test-match --modelspec_extra_options max_tokens=50,temperature=1

Edit: Although that said, I am curious why you want nonzero temperature for evals, usually we want to set that value to 0 for determinism across runs.

[leodiegues]

Wow! Thank you so much for your help @andrew-openai . About the temperature, it doesn't make any sense changing it at all for match evals. You're completely right. Actually, I was just trying to mess around with API parameters in general to better understand how to work with Portuguese prompts and responses. For a moment I thought temperature could be something to that I should change but I wasn't thinking straight anymore I guess.

Again, thank you so much @andrew-openai !