AutoData

Local, strict implementation of the AutoData / Agentic Self-Instruct inner loop described in Meta's RAM AutoData work.

This project focuses on the core data-mining loop:

1. Generate a grounded QA candidate and a positive rubric.
2. Validate the candidate and rubric.
3. Run a weak solver 3 times.
4. Run a strong solver 3 times.
5. Use a rubric judge to score each answer without seeing the reference answer.
6. Compute weak_avg, strong_avg, and gap = strong_avg - weak_avg.
7. Accept only examples that expose a strong/weak capability gap.
8. Persist the full trajectory as JSONL.

Chinese documentation: README.zh-CN.md

Paper Idea

AutoData treats data generation as a search problem rather than a one-shot synthetic-data prompt. The goal is to find grounded tasks where a stronger solver can reliably do better than a weaker solver under the same context and the same rubric. In other words, useful data is not just "a plausible question and answer"; useful data is a question, answer, and evaluation rubric that exposes a measurable capability gap.

The loop has three important constraints:

1. The task must be grounded in the source context.
2. The rubric must describe positive, answer-checkable criteria.
3. The judge must score solver answers from the rubric, without seeing the reference answer.

This makes the accepted examples usable as training data: each accepted record contains a context-grounded question, a reference answer, a rubric, weak/strong solver attempts, judge scores, and the acceptance decision.

Algorithm Architecture

The implementation follows the paper-style AutoData mining architecture: a generator proposes grounded candidates, weak and strong solvers answer the same task, a rubric judge scores those answers, and a gap filter accepts only capability-separating data.

flowchart TD
    A["Source corpus<br/>papers / markdown / domain text"] --> B["Chunker<br/>c"]
    B --> C["Candidate generator<br/>x = (q, a_ref, R)"]
    C --> D{"Candidate validator<br/>grounded? rubric valid?"}
    D -- "reject" --> X["Rejected trajectory<br/>validation failed"]
    D -- "pass" --> E["Weak solver<br/>N sampled answers"]
    D -- "pass" --> F["Strong solver<br/>N sampled answers"]
    E --> G["Rubric judge<br/>score weak answers"]
    F --> H["Rubric judge<br/>score strong answers"]
    G --> I["Aggregate<br/>weak_avg"]
    H --> J["Aggregate<br/>strong_avg"]
    I --> K["Gap filter<br/>gap = strong_avg - weak_avg"]
    J --> K
    K -- "passes thresholds" --> L["Accepted AutoData record<br/>training/evaluation data"]
    K -- "fails thresholds" --> M["Rejected trajectory<br/>too easy / too hard / low gap"]
    L --> N["JSONL dataset<br/>candidate + attempts + scores + decision"]
    M --> N

Loading

The optional outer loop can mutate prompts or mining strategies using accepted yield as feedback:

flowchart LR
    H0["Prompt / harness variants"] --> H1["Run inner-loop miner"]
    H1 --> H2["Measure mining utility S(h)"]
    H2 --> H3["Boltzmann sampling<br/>choose next variants"]
    H3 --> H0

Loading

Formulas

For one source chunk c, the generator proposes a candidate:

$$ x = (q, a_{\mathrm{ref}}, R) $$

where q is the question, a_ref is the reference answer, and the rubric is:

$$ R = {(r_i, w_i)}_{i=1}^{m}, \qquad w_i \in {1,2,3,4,5,6,7} $$

r_i is one positive scoring criterion and w_i is its importance weight.

The weak solver and strong solver each produce N answers:

$$ y_j^{\mathrm{weak}} = W(c, q), \qquad y_j^{\mathrm{strong}} = S(c, q), \qquad j = 1,\dots,N $$

$$ N = 3 \quad \text{by default} $$

For a solver answer y, the judge assigns each criterion a normalized score:

$$ g_i(y) \in [0,1] $$

The weighted answer score is:

$$ \mathrm{score}(y; R) = \frac{\sum_{i=1}^{m} w_i g_i(y)} {\sum_{i=1}^{m} w_i} $$

The weak and strong averages are:

$$ \mathrm{weak_avg} = \frac{1}{N}\sum_{j=1}^{N} \mathrm{score}(y_j^{\mathrm{weak}}; R) $$

$$ \mathrm{strong_avg} = \frac{1}{N}\sum_{j=1}^{N} \mathrm{score}(y_j^{\mathrm{strong}}; R) $$

$$ \mathrm{gap} = \mathrm{strong_avg} - \mathrm{weak_avg} $$

A candidate is accepted only when:

$$ \mathrm{accept}(x) = \mathbb{1}\left[ \mathrm{validation_passed} \land \mathrm{strong_avg} \ge 0.65 \land \mathrm{weak_avg} < 0.50 \land \mathrm{gap} \ge 0.20 \right] $$

Rejected examples are expected and useful. Common rejection reasons are:

weak_avg is too high    -> the task is too easy
strong_avg is too low   -> the task/rubric may be bad or too hard
gap is too small        -> the task does not separate model capability
validation failed       -> the candidate is malformed or not grounded enough

The paper also discusses an outer optimization loop over prompts or data generation strategies. A simple abstraction is to score each harness or prompt variant h_k with a mining utility S(h_k) and sample future variants with a temperature-scaled distribution:

$$ P(h_k) = \frac{\exp(S(h_k) / T)} {\sum_l \exp(S(h_l) / T)} $$

This repository focuses on the strict inner-loop implementation above. The older prompt-mutation scaffold under src/autodata/meta/ is kept as a starting point for outer-loop experiments, but it is not the primary validated path.

Alignment With Meta AutoData

The strict local miner follows the AutoData CS research-task acceptance shape:

strong_avg >= 0.65
weak_avg < 0.50
gap >= 0.20
weak_samples = 3
strong_samples = 3

The judge is rubric-based and self-contained. It receives:

context
question
rubric
solver answer

It does not receive reference_answer. The reference answer is kept in the candidate record for dataset provenance, not for judging solver outputs.

This repository is an engineering MVP aligned with the AutoData inner loop. It does not claim to reproduce Meta's model scale, training scale, or outer meta-optimization results.

Implementation Architecture

The validated implementation lives in src/autodata/local_miner/ and is kept separate from the original multi-agent scaffold. The runtime path is:

flowchart TD
    CLI["scripts/run_local_miner.py"] --> PARSE["cli.py<br/>arguments, thresholds, paths"]
    PARSE --> RUNNER["runner.py<br/>LocalMiner orchestration"]
    RUNNER --> LOAD["data_module<br/>paper loading + chunking"]
    RUNNER --> PROMPTS["prompts.py<br/>candidate / solver / judge prompts"]
    RUNNER --> RUNTIME["runtime.py<br/>local HF / PEFT model inference"]
    RUNNER --> VALIDATOR["validator.py<br/>candidate + rubric checks"]
    RUNNER --> SCORER["scorer.py<br/>judge JSON -> weighted scores"]
    RUNNER --> SCHEMAS["schemas.py<br/>typed records and decisions"]
    RUNNER --> SERIAL["serialization.py<br/>JSONL records + summary"]
    SERIAL --> OUT["outputs/*.jsonl<br/>ignored by git"]

Loading

Module responsibilities:

cli.py            parses inputs, thresholds, model paths, and output paths
runner.py         owns the AutoData loop and trajectory events
runtime.py        wraps local Hugging Face / PEFT causal-LM inference
prompts.py        builds candidate, solver, and judge prompts
schemas.py        defines candidate, rubric, score, attempt, and decision models
validator.py      rejects malformed or weak rubrics before solver sampling
scorer.py         converts judge JSON into weighted rubric scores
serialization.py  appends records and writes run summaries

Data flow:

paper / markdown chunks
  -> candidate generation
  -> candidate + rubric validation
  -> weak solver sampling
  -> strong solver sampling
  -> rubric-only judge scoring
  -> weak_avg / strong_avg / gap decision
  -> JSONL trajectory and summary

The implementation intentionally avoids a multi-role agent workflow. It uses one deterministic local pipeline with role-specific prompts because the strict AutoData behavior depends on reproducible scorer inputs, auditable thresholds, and a complete trajectory for every candidate.

Repository Layout

.
├── conf/                         # Hydra configs for the original scaffold
├── scripts/
│   ├── run_local_miner.py        # strict local AutoData CLI
│   ├── run_inner_loop.py         # original multi-agent scaffold entrypoint
│   └── train_*.py                # optional training utilities
├── src/autodata/
│   ├── local_miner/              # strict local inner-loop implementation
│   │   ├── cli.py
│   │   ├── prompts.py
│   │   ├── runner.py
│   │   ├── runtime.py
│   │   ├── schemas.py
│   │   ├── scorer.py
│   │   ├── serialization.py
│   │   └── validator.py
│   ├── data_module/              # markdown paper loading/chunking
│   ├── pipeline/                 # original Agentic Self-Instruct scaffold
│   ├── llm/                      # LLM schemas/client abstraction
│   └── meta/                     # prompt mutation scaffold
└── tests/
    ├── test_local_miner.py
    ├── test_local_miner_runtime.py
    └── test_local_miner_cli.py

Install

Python 3.11+ is recommended.

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

For local model inference:

pip install -e ".[serve]"

On the tested target machine, the implementation was run inside an existing conda environment with torch, transformers, peft, and trl already installed.

Run The Strict Local Miner

Single-file smoke run:

PYTHONPATH=src python scripts/run_local_miner.py \
  --context-file data/local_miner_sample.md \
  --output outputs/local_miner/strict_qwen_0p6_1p7.jsonl \
  --summary-output outputs/local_miner/strict_qwen_0p6_1p7_summary.json \
  --weak-model /home/charles/llm_models/Qwen3-0.6B \
  --strong-model /home/charles/llm_models/Qwen3-1.7B \
  --generator-model /home/charles/llm_models/Qwen3-1.7B \
  --judge-model /home/charles/llm_models/Qwen3-1.7B \
  --weak-samples 3 \
  --strong-samples 3 \
  --candidate-max-new-tokens 1024 \
  --answer-max-new-tokens 180 \
  --judge-max-new-tokens 512

Directory-of-papers run:

PYTHONPATH=src python scripts/run_local_miner.py \
  --papers-dir data/papers \
  --papers-limit 10 \
  --chunks-per-paper 2 \
  --output outputs/local_miner/records.jsonl \
  --summary-output outputs/local_miner/summary.json \
  --weak-model /path/to/weak-model \
  --strong-model /path/to/strong-model \
  --judge-model /path/to/judge-model

Output Format

Each JSONL row contains:

candidate       generated question/reference/rubric
validation      candidate/rubric validation result
weak_attempts   3 weak answers and judge scores
strong_attempts 3 strong answers and judge scores
decision        accepted/rejected plus weak_avg/strong_avg/gap
trajectory      event log for auditability
input_metadata  source file/chunk metadata

Accepted examples satisfy all strict thresholds. Rejected examples are still useful for debugging prompts, rubric quality, model capability, and data difficulty.

Validation

Local checks:

PYTHONPATH=src PYTHONDONTWRITEBYTECODE=1 pytest -q -p no:cacheprovider \
  tests/test_local_miner.py \
  tests/test_local_miner_runtime.py \
  tests/test_local_miner_cli.py

PYTHONDONTWRITEBYTECODE=1 python -m compileall -q \
  src/autodata/local_miner \
  scripts/run_local_miner.py

Current local validation:

20 passed
compileall passed

Target-machine validation:

weak      = Qwen3-0.6B
strong    = Qwen3-1.7B
generator = Qwen3-1.7B
judge     = Qwen3-1.7B
samples   = 3 weak / 3 strong

The strict smoke run completed with errors = 0. The sample was rejected because it was too easy for the weak model:

weak_avg   = 0.8933
strong_avg = 0.9533
gap        = 0.0600
accepted   = false

That is expected behavior: AutoData should reject samples where the weak model already performs well.

Notes

• A stronger judge model is recommended for production-quality mining.
• The current local strict miner aligns with the AutoData inner loop, not the full model scale or outer-loop optimization reported by Meta.
• outputs/, caches, local secrets, and .omx/ runtime files are ignored and should not be committed.