-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
By default, the OpenAI connector should not add any system messages. #5544
Comments
Relevant OpenAI documentation: https://platform.openai.com/docs/guides/text-generation/chat-completions-api However note that the system message is optional and the model’s behavior without a system message is likely to be similar to using a generic message such as "You are a helpful assistant." |
@markwallace-microsoft May I ask if any serious testing has been done on the difference between no system message at all and "You are a helpful AI assistant" and "You are an AI assistant created by OpenAI". I've seen on more than one occasion someone reference the fact that using the "created by OpenAI" was observably better than just "You are a helpful AI assistant". But they do not cite any test results. On another but similar note, I have seen and used myself the addition of offering some find of monetary reward for providing a well thought out answer and although I could not tell from just a few interactions, the demonstrator showed results using something like weights and biases or some other prompt testing software like Rivet's variant testing and there was an observable quantitative and qualitative improvement in the answers! Would there have been a reward model during the building of the GPT that has created a "instinctual" need to be rewarded? When I get excited and start telling GPT it did a great job, I do find it gets into the project more and really begins to get "primed"as they say! Anyway, just wondering and since you work at Microsoft with the Semantic Kernel group I thought maybe you could lend some insight into this? |
@markwallace-microsoft And as for the original question, I noticed you moved it to being an issue? As a user of the OpenAI API since Sept 2022, and having played with and test all kinds of software, models, llmware per sey, and I really think if there is no intriincic harm in leaving it there, then you should keep the system message as a placeholder. Without it, it may not be obvious to all new users that a system message can go there and that modifying it will change the GPTs behavior. Much like when adding a system message became possible for the retail users it opened a huge door and was like a godsend from you guys. But the average user doesn't call it a system message because they see Custom Message... food for thought.... sorry I get on --verbose mode sometimes! :D |
### Motivation and Context Closes #5544 ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [ ] The code builds clean without any errors or warnings - [ ] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [ ] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄
### Motivation and Context Closes microsoft#5544 ### Description <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [ ] The code builds clean without any errors or warnings - [ ] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [ ] All unit tests pass, and I have added new tests where possible - [ ] I didn't break anyone 😄
Today, the OpenAI connector adds "Assistant is a large language model" as a system message if none is provided. This should be removed.
The text was updated successfully, but these errors were encountered: