Skip to content

rasinmuhammed/misata

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

21 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

🧠 Misata

The Intelligent Synthetic Data Engine

PyPI version Python versions License Downloads

Stop writing fake data scripts.
Generate production-grade datasets from natural language.

Quick Start β€’ Features β€’ Python API β€’ Enterprise


πŸš€ Why Misata?

Misata isn't just a random data generator. It's an intelligent engine that understands your business logic, relationships, and constraints. Whether you need 50 rows for unit tests or 10 million rows for load testing, Misata delivers statistically realistic data that looks and behaves like the real thing.

Feature Faker SDV Misata
Natural Language Input ❌ ❌ βœ…
Auto Schema Generation ❌ ❌ βœ…
Relational Integrity ❌ βœ… βœ…
Business Constraints ❌ ❌ βœ…
No Training Data Needed βœ… ❌ βœ…
Streaming (10M+ rows) ❌ ❌ βœ…

⚑ Quick Start

1. Install

pip install misata

2. Generate

Describe what you need in plain English. Misata handles the rest.

# Basic generation (Rule-based, instant)
misata generate --story "A SaaS platform with 50K users, monthly subscriptions, and a 20% churn rate in Q3"

# Intelligent generation (LLM-powered)
export GROQ_API_KEY=gsk_...
misata generate --story "E-commerce store with seasonal trends and customer segments" --use-llm

3. Result

Misata creates a relational schema, generates the data, and saves it to ./generated_data.

πŸ“‹ Schema: SaaS_Platform
   Tables: 4 (users, subscriptions, payments, events)
   Relationships: 3
   Events: 1 (Churn Spike Q3)

πŸš€ Performance: 385,000 rows/second
πŸ’Ύ Data saved to: ./generated_data

πŸ”₯ New in v0.5.2 β€” The Realism Engine

Every column is now aware of every other column. Misata generates data that is mathematically consistent, not randomly independent.

What makes this different from Faker?

                 Faker/Random              Misata v0.5.2
─────────────────────────────────────────────────────────
order.total      $847.23 (random)          $847.23 = $798.50 + $29.99 + $18.74
product.cost     $96.00 (> price!)         $41.20 (43% of price $95.81)
line_total       $3,291.00 (random)        $3,291.00 = 5 Γ— $662.00 βˆ’ $19.00
user.email       luke.ri@wanadoo.co.uk     emma.chen@gmail.com (from name)
rating           137 (wat?)                4 β˜… (J-curve weighted)
categories       "Hypothyroidism"          "Electronics"
delivered_at     2021-01-03 (before order) 2024-03-15 (+7 days after order)
─────────────────────────────────────────────────────────
Row counts       100 Γ— every table         15 categories, 500 order_items

Smart Row Proportions

Misata analyzes your FK graph to size tables realistically:

misata generate --db-url sqlite:///shop.db --smart --rows 100

# categories:    15   (reference β€” fewer, no duplicates)
# users:        100   (entities β€” your base count)
# products:     250   (entities with variety)
# orders:       250   (transactions β€” more than users)
# order_items:  500   (line items β€” most rows)
# reviews:      150   (activity β€” subset of orders)

Seed Any Existing Database

# PostgreSQL, MySQL, SQLite β€” just point and seed
misata generate \
  --db-url postgresql://user:pass@localhost:5432/mydb \
  --smart --rows 10000 --db-truncate

πŸ’» Python API

Seamlessly integrate Misata into your test suites and CI/CD pipelines.

Standard Generation

from misata import DataSimulator
from misata.llm_parser import LLMSchemaGenerator

# 1. Design schema with AI
llm = LLMSchemaGenerator(provider="groq")
config = llm.generate_from_story(
    "Healthcare app with patients, doctors, and appointments"
)

# 2. Generate data
simulator = DataSimulator(config)
for table_name, df in simulator.generate_all():
    print(f"Generated {len(df)} rows for {table_name}")
    df.to_csv(f"{table_name}.csv", index=False)

SQLAlchemy Seeding (Powerful!)

Directly seed your SQLAlchemy models without writing factories.

from misata import seed_from_sqlalchemy_models
from myapp.models import Base, engine

# Automatically analyzes your models and foreign keys
report = seed_from_sqlalchemy_models(
    engine, 
    Base, 
    default_rows=10_000, 
    create=True, 
    smart_mode=True  # Infers realistic values from column names
)

print(f"Seeded {report.total_rows} rows in {report.duration_seconds}s")

🎯 Business Constraints

Define complex rules that simple random generators can't handle.

from misata import Constraint, Table

timesheets = Table(
    name="timesheets",
    row_count=10000,
    constraints=[
        Constraint(
            name="max_daily_hours",
            type="sum_limit",
            group_by=["employee_id", "date"],
            column="hours",
            value=8.0,
            action="redistribute"  # Automatically fixes violations
        )
    ]
)

πŸ”Œ Providers

Misata supports multiple LLM providers for schema generation.

Provider Env Var Tier Best For
Groq GROQ_API_KEY Free Speed (Recommended)
OpenAI OPENAI_API_KEY Paid Quality
Ollama None Free Privacy (Local)

🏒 Enterprise

Building a platform? Misata Studio is our commercial offering for teams.

  • πŸ–₯️ Visual Schema Editor: Drag-and-drop schema design.
  • πŸ”’ Privacy Filters: PII scanning and masking.
  • πŸ“¦ One-Click Deploy: Docker & Kubernetes ready.
  • 🀝 Support: Dedicated support and custom integration.

Contact Sales for a demo.


Built with ❀️ by Muhammed Rasin

About

High-performance open-source synthetic data engine. Uses LLMs for schema design and vectorized NumPy for deterministic, scalable generation.

Topics

Resources

License

Contributing

Stars

Watchers

Forks

Packages

 
 
 

Languages