update(openai): Support "oaieval" and "oaievalset" as executables within the "openai" shell plugin

[arunsathiya]

This PR introduces a Shell Plugin for OpenAI's new Evals framework that was announced as part of the GPT-4 language model.

• GPT-4 homepage: https://openai.com/product/gpt-4
• Evals framework homepage: https://github.com/openai/evals

This is a fairly straightforward shell plugin as the Evals CLI expects a OPENAI_API_KEY environment variable with the OpenAI API Key which can be obtained on the OpenAI API Keys page. There's no CLI flag that accepts the API Key, nor a configuration file.

There's a way to set the API Key within an individual Evals module code though, using openai.api_key or a path to the configuration file using openai.api_key_path but neither are applicable to Shell Plugins because that's within the context of the Evals module code:

No API key provided. You can set your API key in code using 'openai.api_key = <API-KEY>', or you can set the environment variable OPENAI_API_KEY=<API-KEY>). If your API key is stored in a file, you can point the openai module at it with 'openai.api_key_path = <PATH>'. You can generate API keys in the OpenAI web interface. See https://onboard.openai.com for details, or email support@openai.com if you have any questions.

Here's how to test

• Clone the Evals repository and set it up as outlined under Making evals section.
• Checkout the current PR branch and run make evals/build
• Set up API Key on 1Password by running op plugin init oaieval (API Key can be obtained from this page)
• Future runs for oaieval will use the API Key stored on 1Password.
• To test, run an example eval like oaieval gpt-3.5-turbo test-match and ensure it runs successfully.

Blockers

• The Evals framework project introduces not only oaieval CLI command but also oaievalset to test eval sets.
• As my understanding stands (from the flyctl PR), there isn't currently a way to introduce more than one executable command for a shell plugin without changes to the op CLI code. So, I am not sure how to proceed here.

[hculea]

Hey Arun, thank you for your contribution!

I am wondering, what similarity does this plugin bear with the openai plugin? https://github.com/1Password/shell-plugins/blob/main/plugins/openai

Would it make sense to add another executable to the same plugin, or perhaps at least reuse the credential definition?

[arunsathiya]

Hi @hculea, great question! OpenAI Evals is a different project from OpenAI's openai CLI project. The latter is access to the OpenAI API, while Evals is only a framework/benchmark registry for OpenAI models and allows the general public to contribute evaluations that help understand where GPT 3.5 (current ChatGPT model) and GPT 4 (the newest model) stand.

Also, OpenAI CLI's executable is openai while Evals framework's executable is oaieval, which means we cannot include both of them in the same Shell Plugin.

[hculea]

Thanks for clarifying! 😄 Is the credential definition the same between the two, though?

I see that both use an envvar provisioner with OPENAI_API_KEY, not sure if the credential composition is the same across the two of them?

edit: just to clarify, the rest looks good to me. This comment only aims to reduce some code duplication.

[arunsathiya]

I haven't compared the credential composition so far, but the main blocker is that we cannot configure a Shell Plugin with two executables. I've started a conversation here:

• Ability to support more than one executable #230

Will mark this PR as a draft until that's a possibility.

[arunsathiya]

@hculea Following the same guidance as in Flyctl PR, evals shell plugin has been completely dropped, and oaieval and oaievalset executables are now supported within the openai shell plugin. After all, those two executables and openai use the same credential composition.

Please let me know how things look now.

[accraw]

Tested, it worked for me!