-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ChatGPT API with model gpt-4 is not using GPT4. It's completely different from CHATGPT PLUS GPT4 #381
Comments
Better comparison would be the OpenAI Playground, since ChatGPT (Plus) is pre-fed some system context we don't know. |
I believe ChatGPT with model gpt-4 (within the browser) is using GPT-4 with additional context. I also believe the Chat API (not within the browser) with model In short: It's incorrect that it reports using GPT-3 when it really uses some version of GPT-4. My justification is that I am running a bunch of evals and And also my experience in using ChatGPT in the browser: The GPT-4 configuration outperforms the GPT-3 configuration and outputs better (more correct) responses. |
My only concern and question mark is training data cut off date. I know as an end user when I install or use a consumer facing app that claims gpt4 and answers as gpt3 is a UX problem. |
The issue appears to be: When GPT-4 is configured, is GPT-4 used when the output says it is not GPT-4? I think the best analogy I can make is: If I trained a parrot to bark like a dog, it does not mean that it is not a parrot. If the configuration is set to GPT-4, GPT-4 is used even though the output states that it is not GPT-4. For example, in your query: ozgur@Ozgurs-MacBook-Pro ~ % curl https://api.openai.com/v1/chat/completions
-H "Content-Type: application/json"
-H "Authorization: Bearer [REDACTED]"
-d '{
"model": "gpt-4",
"messages": [{"role": "user", "content": "What is your model number!"}]
}'
{"id":"chatcmpl-6wVWzn95O6iDc1Z9W1kJNIZZtZZLh","object":"chat.completion","created":1679402249,"model":"gpt-4-0314","usage":{"prompt_tokens":12,"completion_tokens":45,"total_tokens":57},"choices":[{"message":{"role":"assistant","content":"As an AI language model, I do not have a model number like a physical device or product would. I am powered by OpenAI's GPT-3, which stands for Generative Pre-trained Transformer 3."},"finish_reason":"stop","index":0}]} the query is set to use "model": "gpt-4" and the prompt is: "What is your model number!" and the output is: "I am powered by OpenAI's GPT-3, which stands for Generative Pre-trained Transformer 3." Here, there appears to be a factual error: The output says it's powered by GPT-3 but the fact is that it's powered by GPT-4. Although the output is inaccurate, it is still GPT-4. A model can be wrong about self-identifying questions about itself, but it does not change what model is being used. In the GPT-4 Technical Report, OpenAI states: Here, OpenAI acknowledges that GPT-4 "is not fully reliable" and "makes reasoning errors". In this case, it made a reasoning error about what model it is. OpenAI does not make any guarantees or promises about accuracy. When you use ChatGPT or OpenAI's API, you agree to OpenAI's Terms of use:
In Section 7(b) of the Terms of use, the Disclaimer states that they do not warrant that their services will be accurate or error free:
|
Might be an interesting eval to verify if a model is capable of knowing about itself without additional context. |
You would think when the platform was written that it would include the ID for the updated iteration. Not so much self aware but embedded in the program. |
As mentioned by @jonathanagustin, our models tend to hallucinate facts and this case was most likely an instance of hallucination. However, it may be possible (though the issue you noticed doesn't point to it) that the UI is actually wrong! If you suspect that might be the case, please contact our support team. |
Not right place for this issue but this is more important than any issue in any repo you have because it's literally lying. The api request result says it uses GPT-3 when I query this
ozgur@Ozgurs-MacBook-Pro ~ % curl https://api.openai.com/v1/chat/completions
-H "Content-Type: application/json"
-H "Authorization: Bearer [REDACTED]"
-d '{
"model": "gpt-4",
"messages": [{"role": "user", "content": "What is your model number!"}]
}'
{"id":"chatcmpl-6wVWzn95O6iDc1Z9W1kJNIZZtZZLh","object":"chat.completion","created":1679402249,"model":"gpt-4-0314","usage":{"prompt_tokens":12,"completion_tokens":45,"total_tokens":57},"choices":[{"message":{"role":"assistant","content":"As an AI language model, I do not have a model number like a physical device or product would. I am powered by OpenAI's GPT-3, which stands for Generative Pre-trained Transformer 3."},"finish_reason":"stop","index":0}]}
vs CHATGPT PLUS GPT4 says
![Screenshot 2023-03-21 at 13 08 56](https://user-images.githubusercontent.com/3254799/226615591-744c8f43-0191-471f-982c-44f00c57b3a1.png)
The text was updated successfully, but these errors were encountered: