There are many people working nowadays with synthetic data to further improve the quality of deep learning models, e.g., Llama-3.1 was trained on a large fraction of synthetically generated data.
Quality and diversity are the main ingredients to build good synthetic data, and they do often conflict. In words of Nathan Cooper, former AI Lead Researcher at Anthropic: "A random string generator has high diversity but low quality. The Encyclopedia Britannica has high-quality content but its scope is limited, e.g., it doesn’t have all the amazing Star Trek fan fiction people have created. Our challenge is to balance these factors."
This repo implements techniques to foster quality and diversity according to Nathan's recommendations (https://x.com/ncooper57/status/1846612127911760261 and his blog https://www.answer.ai/posts/2024-10-15-how-to-synthesize-data.html):
- In-context examples: to steer the language model to generate data that is similar to yours
- Role playing: conditioning the language model to generate data about a topic or mimicking a random persona or writing style
- Quality filtering: use LLMs to critique synthetic data using rubriques and requesting a quality score for later filtering
You can use this repo to create synthetic data for almost any task you want (see the Examples section), and has been tested for topic ABSA, translation, semantic similarity, and sentiment analysis.
Some related papers:
Strikingly, replacing up to 90% of the training data only marginally decreases performance, but replacing the final 10% leads to severe declines. We find that models trained on purely synthetic data can be reliably improved by including as few as 125 human generated data points.
Requirement: python >= 3.10
$ pip install -e .You will need an OPENAI_API_KEY environment variable to use GPT in DataMaker.
You can create topics by calling get_topic_generator to get the generator and call its generate method. The topic_class argument specifies which generator to use.
from datamaker import *
num_topics = 10
topic_config = TopicConfig(
topic_class="LLMUnsupervisedTopics",
num_topics=num_topics,
instruction=f"Write {num_topics} very specific topics (1-3 words) about restaurants and laptops.",
topics=["dell vostro", "Pekin Muralla restaurant"],
)
openai_config = OpenAIConfig(model_name="gpt-4o")
config = Config(task="sentiment_analysis", topic=topic_config, openai=openai_config)
topic_generator = get_topic_generator(config)
print(topic_generator.generate())The predefined topics in topics will be extended with topics generated by an LLM following the instruction. You can use just the predefined ones using the DummyTopics topic generator:
from datamaker import *
topic_config = TopicConfig(
topic_class="DummyTopics",
topics=["dell vostro", "Pekin Muralla restaurant"],
)
config = Config(task="sentiment_analysis", topic=topic_config)
topic_generator = get_topic_generator(config)
print(topic_generator.generate())You can create personas by calling get_persona_generator to get the generator and call its generate method. The persona_class argument specifies which generator to use.
from datamaker import *
persona_config = PersonaConfig(
persona_class="PersonaHub",
num_personas=10,
personas=["An LLM expert", "A deep learning practitioner"],
)
config = Config(task="sentiment_analysis", persona=persona_config)
persona_generator = get_persona_generator(config)
print(persona_generator.generate())The predefined personas will be extended with personas extracted from the PersonaHub dataset. You can use just the predefined ones using the DummyPersonas persona generator:
from datamaker import *
persona_config = PersonaConfig(
persona_class="DummyPersonas",
num_personas=10,
personas=["An LLM expert", "A deep learning practitioner"],
)
config = Config(task="sentiment_analysis", persona=persona_config)
persona_generator = get_persona_generator(config)
print(persona_generator.generate())You can infer the style of the texts by calling get_style_generator to get the generator and call its generate method. The style_class argument specifies which generator to use:
from datamaker import *
style_config = StyleConfig(
style_class="LLMSupervisedStyle",
instruction=f"""Describe the writing style of the following texts. Do not focus on the topics or specific terms, just describe the style features.
You must describe the following features using one or two sentences:
- Writing style: a summary of the writing style used in the texts.
- Word count range: the range of word counts. Verbalize it.
- Sentence count range: the range of sentence counts. Verbalize it.
""",
kwargs={
"texts": [
"the tucks have a secret , they 're immortal . they",
"this could be lizzy 's only chance to start a new life and recreate the family she tragically lost as a child .",
"the book tells of murray , the old scot patriot , who has had his eyes torn out and his house taken away during the english invasion ."
],
"num_texts": 3
}
)
openai_config = OpenAIConfig(model_name="gpt-4o-mini")
config = Config(task="topic_classification", style=style_config, openai=openai_config)
style_generator = get_style_generator(config)
print(style_generator.generate())In case you do not want to specify your own style or just not include it for synthesizing, you can use DummyStyle:
from datamaker import *
style_config = StyleConfig(
style_class="DummyStyle",
style_description="- straightforward and concise, often employing a conversational tone that engages the reader directly" # Remove this to not use style in synthesizers
)
openai_config = OpenAIConfig(model_name="gpt-4o-mini")
config = Config(task="topic_classification", style=style_config, openai=openai_config)
style_generator = get_style_generator(config)
print(style_generator.generate())You can evaluate a set of samples by retrieving the evaluator with get_evaluator and calling its run method with a set of examples. These examples are instances of a task schema.
from datamaker import *
from datamaker.synthesizers.schemas import SentimentAnalysis
evaluator_config = EvaluatorConfig(
evaluator_class="LLMThreeScaleCategoricalEvaluator",
filtering_criterion=["high"],
instruction="Evaluate how good is this example for a sentiment analysis task. Explain your decision with just one sentence.",
)
openai_config = OpenAIConfig(model_name="gpt-4o")
config = Config(
task="sentiment_analysis",
evaluator=evaluator_config,
openai=openai_config,
)
evaluator = get_evaluator(config)
inputs = [
SentimentAnalysis(text="I hate the mountains", completion="positive"),
SentimentAnalysis(text="I love the mountains", completion="positive"),
]
print(evaluator.run(inputs))In the example, we are using the LLMThreeScaleCategoricalEvaluator to compute a rationale and an score for each example in inputs, and then filtering those with an score matching the items in filtering_criterion (high in our case).
While the previous examples were aimed to be familiar with topics, personas, evaluators, and configurations, we are missing the main function of the repo: generate synthetic data.
To generate synthetic data we can call the synthesizer as follows:
from pydantic import BaseModel
from typing import Literal
from datamaker import *
class SentimentAnalysis(BaseModel):
text: str
completion: Literal["positive", "negative", "neutral"]
register_task_schema("sentiment_analysis", SentimentAnalysis)
synthesizer_config = SynthesizerConfig(
synthesizer_class="LLMSynthesizer",
num_samples=10,
instruction="Create a sample for a sentiment analysis task",
examples=[
{"text": "I love the mountain", "completion": "positive"},
{"text": "I hate climbing", "completion": "negative"},
],
)
topic_config = TopicConfig(
topic_class="LLMUnsupervisedTopics",
num_topics=10,
topics=["dell vostro", "Pekin Muralla restaurant"],
instruction="Write 10 very specific topics (1-3 words) about laptops and restaurants.",
)
persona_config = PersonaConfig(persona_class="PersonaHub", num_personas=50)
style_config = StyleConfig(
style_class="LLMSupervisedStyle",
instruction=f"""Describe the writing style of the following texts. Do not focus on the topics or specific terms, just describe the style features.
You must describe the following features using one or two sentences:
- Writing style: a summary of the writing style used in the texts.
- Word count range: the range of word counts. Verbalize it.
- Sentence count range: the range of sentence counts. Verbalize it.
""",
kwargs={
"texts": [
"the tucks have a secret , they 're immortal . they",
"this could be lizzy 's only chance to start a new life and recreate the family she tragically lost as a child .",
"the book tells of murray , the old scot patriot , who has had his eyes torn out and his house taken away during the english invasion ."
],
"num_texts": 3
}
)
evaluator_config = evaluator_config = EvaluatorConfig(
evaluator_class="LLMThreeScaleCategoricalEvaluator",
filtering_criterion=["high"],
instruction="Evaluate how good is this example for a sentiment analysis task. Explain your decision with just one sentence.",
)
openai_config = OpenAIConfig(model_name="gpt-4o-mini")
config = Config(
task="sentiment_analysis",
languages=["English", "Spanish"],
synthesizer=synthesizer_config,
topic=topic_config,
style=style_config,
persona=persona_config,
openai=openai_config,
evaluator=evaluator_config,
)
synthesizer = get_synthesizer(config)
print(synthesizer.generate())Here, we specified all the configs that the synthesizer needs: topics, personas, and evaluator. But, not all of them are required, i.e., you can do things like skip the evaluation/filtering pass, or not conditioning the LLM with topics or personas.
from datamaker import *
from pydantic import BaseModel
class Translation(BaseModel):
source_text: str
completion: str
register_task_schema("translation", Translation)
synthesizer_config = SynthesizerConfig(
synthesizer_class="LLMSynthesizer",
num_samples=10,
instruction="Create a sample for a translation task, where the source language is English and the target one is Spanish",
examples=[
{"source_text": "I love the mountain", "completion": "Amo las montañas"},
{
"source_text": "This weekend was funny",
"completion": "Este fin de semana fue divertido",
},
],
)
topic_config = TopicConfig(
topic_class="LLMUnsupervisedTopics",
num_topics=10,
topics=["dell vostro", "Pekin Muralla restaurant"],
instruction="Write 10 very specific topics (1-3 words) on the healthcare domain.",
)
persona_config = PersonaConfig(persona_class="PersonaHub", num_personas=50)
style_config = StyleConfig(
style_class="LLMSupervisedStyle",
instruction=f"""Describe the writing style of the following texts. Do not focus on the topics or specific terms, just describe the style features.
You must describe the following features using one or two sentences:
- Writing style: a summary of the writing style used in the texts.
- Word count range: the range of word counts. Verbalize it.
- Sentence count range: the range of sentence counts. Verbalize it.
""",
kwargs={
"texts": [
"the tucks have a secret , they 're immortal . they",
"this could be lizzy 's only chance to start a new life and recreate the family she tragically lost as a child .",
"the book tells of murray , the old scot patriot , who has had his eyes torn out and his house taken away during the english invasion ."
],
"num_texts": 3
}
)
evaluator_config = evaluator_config = EvaluatorConfig(
evaluator_class="LLMThreeScaleCategoricalEvaluator",
filtering_criterion=["high"],
instruction="Evaluate how good is this example for a translation task (from English to Spanish). Explain your decision with just one sentence.",
)
openai_config = OpenAIConfig(model_name="gpt-4o-mini")
config = Config(
task="translation",
languages=["Spanish"],
synthesizer=synthesizer_config,
topic=topic_config,
style=style_config,
persona=persona_config,
openai=openai_config,
evaluator=evaluator_config,
)
synthesizer = get_synthesizer(config)
print(synthesizer.generate())from pydantic import BaseModel
from typing import Literal
from datamaker import *
class QQP(BaseModel):
question_a: str
question_b: str
completion: Literal["0", "1", "2", "3", "4", "5"]
register_task_schema("qqp", QQP)
synthesizer_config = SynthesizerConfig(
synthesizer_class="LLMSynthesizer",
num_samples=10,
instruction="Create a sample for a semantic similarity task between two sentences. Similarity goes from 0 (not similar at all), to 5 (exactly the same question)",
examples=[
{
"question_a": "How can I reduce the fat of my body?",
"question_b": "What sports should I do to reduce the fat of my body?",
"label": 2,
},
{
"question_a": "What are dual encoders?",
"question_b": "What are siamese networks?",
"label": 4,
},
],
)
topic_config = TopicConfig(
topic_class="LLMUnsupervisedTopics",
num_topics=10,
topics=[],
instruction="Write 10 very specific topics (1-3 words) of deep learning architectures.",
)
persona_config = PersonaConfig(persona_class="PersonaHub", num_personas=50)
style_config = StyleConfig(
style_class="LLMSupervisedStyle",
instruction=f"""Describe the writing style of the following texts. Do not focus on the topics or specific terms, just describe the style features.
You must describe the following features using one or two sentences:
- Writing style: a summary of the writing style used in the texts.
- Word count range: the range of word counts. Verbalize it.
- Sentence count range: the range of sentence counts. Verbalize it.
""",
kwargs={
"texts": [
"the tucks have a secret , they 're immortal . they",
"this could be lizzy 's only chance to start a new life and recreate the family she tragically lost as a child .",
"the book tells of murray , the old scot patriot , who has had his eyes torn out and his house taken away during the english invasion ."
],
"num_texts": 3
}
)
evaluator_config = evaluator_config = EvaluatorConfig(
evaluator_class="LLMThreeScaleCategoricalEvaluator",
filtering_criterion=["high"],
instruction="Evaluate how good is this example for a semantic similarity task. Explain your decision with just one sentence.",
)
openai_config = OpenAIConfig(model_name="gpt-4o-mini")
config = Config(
task="qqp",
languages=["English"],
synthesizer=synthesizer_config,
topic=topic_config,
persona=persona_config,
style=style_config,
openai=openai_config,
evaluator=evaluator_config,
)
synthesizer = get_synthesizer(config)
print(synthesizer.generate())from pydantic import BaseModel
from typing import Literal
from datamaker import *
class AspectTopicSentiment(BaseModel):
aspect: str
sentiment: Literal["negative", "neutral", "positive"]
topic: str
class TopicABSA(BaseModel):
text: str
aspects: list[AspectTopicSentiment]
register_task_schema("topic_absa", TopicABSA)
synthesizer_config = SynthesizerConfig(
synthesizer_class="LLMSynthesizer",
num_samples=10,
instruction="Create a sample for an topic aspect-based sentiment analysis. Aspects are words or phrases from the text.",
examples=[
{
"text": "I love the mountain, but I hate climbing",
"aspects": [{"mountain", "positive", "geography"}, {"climbing", "negative", "sports"}]
}
],
)
topic_config = TopicConfig(
topic_class="LLMUnsupervisedTopics",
num_topics=10,
topics=[],
instruction="Write 10 very specific topics (1-3 words) about mountains and climbing",
)
persona_config = PersonaConfig(persona_class="PersonaHub", num_personas=50)
evaluator_config = evaluator_config = EvaluatorConfig(
evaluator_class="LLMThreeScaleCategoricalEvaluator",
filtering_criterion=["high"],
instruction="Evaluate how good is this sample for a topic aspect-based sentiment analysis task. Explain your decision with just one sentence.",
)
style_config = StyleConfig(
style_class="LLMSupervisedStyle",
instruction=f"""Describe the writing style of the following texts. Do not focus on the topics or specific terms, just describe the style features.
You must describe the following features using one or two sentences:
- Writing style: a summary of the writing style used in the texts.
- Word count range: the range of word counts. Verbalize it.
- Sentence count range: the range of sentence counts. Verbalize it.
""",
kwargs={
"texts": [
"the tucks have a secret , they 're immortal . they",
"this could be lizzy 's only chance to start a new life and recreate the family she tragically lost as a child .",
"the book tells of murray , the old scot patriot , who has had his eyes torn out and his house taken away during the english invasion ."
],
"num_texts": 3
}
)
openai_config = OpenAIConfig(model_name="gpt-4o-mini")
config = Config(
task="topic_absa",
languages=["English"],
synthesizer=synthesizer_config,
topic=topic_config,
style=style_config,
persona=persona_config,
openai=openai_config,
evaluator=evaluator_config,
)
synthesizer = get_synthesizer(config)
print(synthesizer.generate())Please install and use dev-tools for correctly formatting the code when contributing to this repo.