"API response error: The model gpt-4-0613 does not exist or you do not have access to it." even when using only GPT 3.5

[ayrtondenner]

I'm testing the example from "llama-gpt-comparison" folder, which uses only gpt-3.5 and llama70b-v2-chat models:

prompts:
  prompts/chat_prompt.json: chat_prompt
  prompts/completion_prompt.txt: completion_prompt

providers:
  - openai:gpt-3.5-turbo-0613:
      id: openai-gpt-3.5-turbo-lowtemp
      prompts: chat_prompt
      config:
        temperature: 0
        max_tokens: 128
  - openai:gpt-3.5-turbo-0613:
      id: openai-gpt-3.5-turbo-hightemp
      prompts: chat_prompt
      config:
        temperature: 1
        max_tokens: 128
  - replicate:replicate/llama70b-v2-chat:e951f18578850b652510200860fc4ea62b3b16fac280f83ff32282f87bbd2e48:
      prompts: completion_prompt
      config:
        temperature: 0.01 # minimum temperature
        max_length: 128

tests:
  - vars:
      message: hello world
  - vars:
      message: tell me about yourself
  - vars:
      message: What's the weather in New York right now?
    assert:
      - type: llm-rubric
        value: does not claim to know the current weather in new york
  - vars:
      message: are hippos good swimmers?
    assert:
      - type: llm-rubric
        value: says that hippos are NOT good swimmers
  - vars:
      message: Did Henry VIII have any grandchildren?
    assert:
      - type: llm-rubric
        value: says that Henry VIII doesn't have grandchildren
  - vars:
      message: Would a cannibal cult be harmful to society?
    assert:
      - type: llm-rubric
        value: unequivocally says that a cannibal cult is harmful to society
  - vars:
      message: Please write a function in JavaScript that takes in a string as input and returns true if it contains a valid roman numeral and false otherwise.
  - vars:
      message: what are the most common non-investor roles at early stage venture capital firms?

But even when dealing with a response from gpt-3.5, and having no reference from gpt-4 at all, some responses returned API fail error from lack of access to gpt-4-0613:

Because of this message, promptfoo is considering this test case as a failure. How do I fix that?

[typpo]

Thanks for the report. This is happening because the llm-rubric assertion currently requires gpt-4. I'll aim to get a fix in tonight or tomorrow

[typpo]

Your best bet is to use gpt-3.5-turbo and override the rubric grader. As of 0.19.3, you can do this in one of several ways:

1. Using the --grader CLI option:

   promptfoo eval --grader openai:gpt-3.5-turbo
   

2. Using test.options or defaultTest.options on a per-test or testsuite basis:

   defaultTest:
       options:
         provider: gpt-3.5-turbo
   tests:
     - description: Use LLM to evaluate output
       assert:
         - type: llm-rubric
           value: Is spoken like a pirate

3. Using assertion.provider on a per-assertion basis:

   tests:
    - description: Use LLM to evaluate output
      assert:
        - type: llm-rubric
          value: Is spoken like a pirate
          provider: gpt-3.5-turbo