-
Notifications
You must be signed in to change notification settings - Fork 65
Description
What is the feature request? What problem does it solve?
Antoni has already completed a POC on this.
Members of the team should test Antoni's solution and provide feedback on how it can be made easier to work with.
We also need to make sure that the datasets we are creating are as most LLMs will expect them.
This is very important. We are not going to get a single solution which matches the format of all LLMs but most popular LLM probably expect the dataset to be in a similar format.
Definition of Done
Others on the team have tested generating datasets with this and there are a list of issues ppl had with it.
After the dataset is created we have also fine tuned an LLM to make sure it is actually simple from after the dataset is created. (This is core because it is something we didn't do last time. )