Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ShareGPT format support #720

Merged
merged 16 commits into from
Mar 31, 2024
Merged

ShareGPT format support #720

merged 16 commits into from
Mar 31, 2024

Conversation

wheresmyhair
Copy link
Collaborator

@wheresmyhair wheresmyhair commented Mar 30, 2024

Add ShareGPT format data support.
Currently only field "messages" works. In other word, users need to include their system prompt, tool prompt, instruction template into the first message like:

  {
     "conversation_id": 1,
     "system": "",
     "tools": [""],
     "messages": [
        {
           "role": "user",
           "content": "[INST] <<SYS>>\nYou are a helpful assistant.\n<</SYS>>\n\nHello! [/INST]"
        },
        {
           "role": "assistant",
           "content": "Hi, how are you?"
        },
        {
           "role": "user",
           "content": "[INST] Good. [/INST]"
        },
        {
           "role": "assistant",
           "content": "Glad to hear that."
        }
     ]
  }

We will add auto formatting for system prompt, tool prompt based on different model input requirements ASAP.

@wheresmyhair wheresmyhair marked this pull request as ready for review March 31, 2024 07:54
Copy link
Contributor

@research4pan research4pan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@research4pan research4pan merged commit f96aae2 into main Mar 31, 2024
@wheresmyhair wheresmyhair deleted the sharegpt-support branch April 1, 2024 02:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants