nftt — naive Fine Tune Tooling

Config-driven fine-tuning pipeline for LLM projects: pre-prepared jsonl data → QLoRA training → GGUF export → ollama serving → judged evaluation. Installed once on the machine, pointed at many project directories. A consuming project owns its config, domain module (validators/judge rubric), and data; artifacts (adapters, exports, logs, caches) land in that project's tree.

Status: acceptance passed and the external-dogfood era closed — all nine dogfood findings plus the follow-up queue shipped as archived changes (see STATUS.md and openspec/changes/archive/). Spec of record: openspec/specs/. Durable decisions: adr/ (0001–0016).

Design-rationale map (BYOD is deliberately citation-free — the consuming-project skill bundles it verbatim): system/serve parity 0008, declared domain checks 0009, auto-generated config reference 0011, ground-truth scoring 0012, serve terms as scoring currency 0013, artifacts declare completeness 0014, eval row extras 0015, chat-template identity 0016; data-pipeline rationale: 0004–0007 (see adr/).

Install (tool, once per machine)

Base install is pure stdlib (instant, no ML stack):

python3 -m venv .venv
.venv/bin/pip install -e .

With the pinned training stack (AMD ROCm). Torch wheels come from the PyTorch rocm index, so install is two stages — torch first, then the rest:

python3 -m venv ~/.venvs/nftt
~/.venvs/nftt/bin/pip install "torch==2.11.0+rocm7.2" "torchvision==0.26.0+rocm7.2" \
  --index-url https://download.pytorch.org/whl/rocm7.2
~/.venvs/nftt/bin/pip install -e ".[train]"
ln -s ~/.venvs/nftt/bin/nftt ~/.local/bin/nftt   # bare `nftt` on PATH

The training stack is pinned exactly (see pyproject.toml and ADR-0001) — never upgrade it casually; a stack bump is a deliberate change with a smoke test. To install into a venv other than the repo's .venv, pass the repo path: pip install -e "/path/to/nFTT[train]".

Requirements: Python ≥ 3.12. Training: AMD GPU + ROCm (tested on RX 7900 XT, 20GB). Serving: ollama installed and running. Judge / teacher: an OpenRouter API key in OPENROUTER_API_KEY, a key file in OPENROUTER_API_KEY_FILE, or dropped at nftt/secrets/openrouter.key in each project (auto-loaded). Providers can also point at any OpenAI-compatible endpoint in the project config.

Layout of a consuming project

Everything nFTT owns or scaffolds lives under one namespaced parent — your project's own data//logs/ never collide:

your-project/
├── nftt.json            # the config (one file drives every stage)
├── nftt_domain.py       # OPTIONAL escape hatch (not scaffolded) — only for
│                        #   domain semantics beyond the declared vocabulary
└── nftt/
    ├── data/            # training rows you provide (jsonl)
    │   └── pipeline/    #   per-stage pipeline outputs (<id>.jsonl)
    ├── eval/            # eval sets you provide (jsonl)
    ├── prompts/         # your prompt texts + programmatic templates
    ├── pipeline.json    # the declared data-pipeline chain
    ├── models/<run>/    # adapters, checkpoints, nftt_provenance.json
    ├── exports/<run>/   # GGUF + Modelfile
    ├── logs/            # per-run tee logs + eval_<set>_<label>.jsonl
    ├── cache/           # judge cache + per-stage pipeline caches
    └── secrets/         # openrouter.key (auto-loaded, self-ignoring in git)

nftt.json still references sources relative to the project root, so you can point at data outside nftt/ if you prefer. Provider auth resolves in order: OPENROUTER_API_KEY → OPENROUTER_API_KEY_FILE → nftt/secrets/openrouter.key.

Usage tour

The whole loop is: scaffold → declare → check → run stages → read results → iterate.

1. Scaffold (once per project)

cd ~/Documents/your-project
nftt init

Creates nftt.json and the nftt/ tree (no domain stub — domain checks are declared in config). Refuses to overwrite anything; re-running reports and writes nothing.

2. Declare your task

Edit nftt.json — every key is validated, and a typo names the accepted keys:

{
  "task":  { "system_prompt": "…", "row_schema": "flat", "input_prefix": "Query: " },
  "data":  { "sources": ["nftt/data/train.jsonl"] },
  "model": { "id": "unsloth/Qwen3-4B-bnb-4bit",   // base swap = edit this line
             "lora_r": 8, "holdout": 500, "think": "strip" },
  "providers": { "judge": { "model": "deepseek/deepseek-v4-flash-0731" } },
  "eval":  { "sets": ["nftt/eval/heldout.jsonl"], "model": "myproj-v1" },
  "domain": { "validators": ["shell_syntax", "danger_patterns"],  // declared
              "rubric": "nftt/prompts/rubric.txt" }               // checks
}

Declare your domain checks in the domain section (ADR-0009) — the closed vocabulary (shell_syntax, binary_on_path, danger_patterns, {"regex","reason"} ban-patterns), a rubric text file, and parse_output: identity|strip_fences. Zero Python for known domains; nftt_domain.py (hooks: validate, judge_rubric, parse_output) remains the optional escape hatch for genuinely novel semantics — declared checks and module hooks compose, and a module hook can never silence a declared check.

Drop your rows into nftt/data/train.jsonl (jsonl; shape per task.row_schema) and optionally your key into nftt/secrets/openrouter.key.

3. Check yourself

nftt status        # config valid? sources exist? domain resolves? inventory + latest run

4. Run stages — primitives or the chain

nftt train                        # QLoRA -> nftt/models/<run>/ + provenance + tee log
nftt train --name v1 --epochs 3 --resume nftt/models/v1/checkpoint-250
nftt export --name v1             # GGUF + Modelfile -> nftt/exports/v1/, registers with ollama
nftt export --no-register --quant q4_k_m
nftt eval --name eval-v1 --limit 20 --json
nftt pipeline --dry-run        # inspect the declared data chain
nftt run --name v2             # the whole chain (pipeline first when declared)

What every run gives you for free: the resolved config echoed before work starts, a timestamped line-flushed log under nftt/logs/, a GPU guard (a second GPU job refuses; --force overrides), exit codes 0/1/2, and --json summaries.

5. Read results

• nftt/logs/train_<run>_<timestamp>.log — provenance header (seeds, stack versions, data identity), holdout evals, best-checkpoint selection
• nftt/logs/eval_<set>_<label>.jsonl — a meta line (judge identity — the scoring currency) then one record per row: generated output, validator verdict, judge verdict
• Terminal summary — validator pass rate and judge correct/partial/wrong reported separately (two layers, neither subsumes the other)
• Re-evaluating unchanged (input, output, judge) pairs costs nothing — the cache under nftt/cache/ is pinned to the judge identity

Artifacts are fixed conventions under nftt/ (not configurable). Tool state (the GPU-guard lockfile) lives under .nftt/ at the project root. Train/export refuse to run beside another live GPU consumer (--force overrides).

Data pipeline

Data-creation is declared in nftt/pipeline.json and executed by nftt pipeline (or as the first stage of nftt run). Five verbs:

verb	teacher?	what it does	reads
transform	yes	rewrite/enrich existing rows (paraphrase fan-out, think-traces)	seeds or previous stage
generate	yes	propose new rows from a spec table	table file
programmatic	no	deterministic rows from template × table (byte-identical re-runs; deep {slot} render through nested lists/dicts — messages-schema rows; tables may be .jsonl, JSON array, or {categories})	template + table
gate	judge	cheapest-first filter: validators → judge → decontam; min_pass breach ABORTS the chain	previous stage
assemble	no	merge → dedupe (canonical identity) → decontam-verify → train file; clean-slate every build	declared stage ids

Stage outputs land at nftt/data/pipeline/<id>.jsonl; a stage's input defaults to the previous stage's output. Teacher stages keep resumable caches (nftt/cache/pipeline/) — an interrupted run loses nothing. Judge verdicts inside gates use their own namespaced cache, never the eval cache.

Worked example (one of each verb):

// nftt/pipeline.json
{
  "stages": [
    // programmatic: template × table, no teacher, deterministic
    { "id": "prog", "verb": "programmatic",
      "template": "nftt/prompts/prog_rows.json",   // JSON array of row templates
      "table": "nftt/data/table.json" },           // JSON array of entries ({slot} fill)

    // transform: teacher paraphrases seed rows in batches
    { "id": "para", "verb": "transform",
      "prompt": "nftt/prompts/paraphrase.txt",     // prompt TEXT is yours
      "protocol": "batched-id-tsv",                // output FORMAT is nFTT's
      "seeds": ["nftt/data/seeds.jsonl"], "n": 2 },

    // gate: cheapest first; survival below min_pass aborts the chain
    { "id": "gate1", "verb": "gate",
      "checks": { "validators": true, "judge": "nftt/prompts/rubric.txt",
                  "decontam": true },
      "min_pass": 0.85 },

    // assemble: clean-slate rebuild of the training file
    { "id": "traindata", "verb": "assemble",
      "inputs": ["gate1", "prog"],
      "output": "nftt/data/train.jsonl" }
  ]
}

nftt pipeline --dry-run    # resolved chain: verb, id, input -> output
nftt pipeline              # run it (per-stage counters, stop at first failure)
nftt pipeline --stage para # re-run one stage (cache makes it cheap)

Parser protocols (closed vocabulary, paired with your prompt): batched-id-tsv (id-echoed batch lines — amortizes API cost), pair-per-line (input<TAB>output per line), verbatim-echo (enrichment; source item must be echoed verbatim, drift is dropped), json-rows (one JSON object per line). Prompt text and protocol are validated together at load time.

Bringing your own data

nFTT never models where data comes from — everything enters as jsonl files in your project. Two questions decide your shape:

1. Row schema — declare it in task (flat {in, out} fields or a messages array; prefixes like "Query: " are config, and think-trace fields are supported). Training and eval read rows through the same contract, so serve-time prompt shape must match what you train on.
2. Serving format — if your serving prompt is more than a user message (context blocks, retrieved snippets, tool lists), synthesizing training rows in that format is YOUR project's job: write them by hand, or use a programmatic stage with a template that emits the full serving shape. nFTT deliberately does not model serving-format variation.

The complete data contract — row shapes with worked examples, seed/prompt/ table file conventions, protocol prompt requirements, decontamination expectations, and a new-project checklist — lives in docs/BRINGING-YOUR-OWN-DATA.md and its task-shaped topic files under docs/topics/ (rows, pipeline, markup-masking, eval-logs, exports, decontam-gates — one fetch answers one question).

Quality doctrine (from docs/LESSONS.md): judge sources before fanning out (gates are cheapest-first for exactly this reason), never train on judged-wrong rows, and decontaminate against eval sets on every assembly — the assembler fails the build on any hit, structurally.

Teacher choice matters (throughput varies 3–5×): canary a new teacher on a small fixed set before bulk generation, and never mix teachers in one stage cache (each stage's cache is separate; changing teacher = new cache).

Config reference

One nftt.json per project. The authoritative annotated reference is generated from the validation schema (ADR-0011):

nftt config           # every key: type, default, accepted values, required
nftt config --json    # the same, machine-readable
nftt docs             # era gotchas shipped with the installed version

It works in any directory (no project needed) and always matches the installed binary. nftt status prints a project's effective resolved values. Unknown keys anywhere are rejected with the accepted keys listed (typo detection).

A minimal config to orient by:

{
  "task":    { "system_prompt": "what the model does", "row_schema": "flat" },
  "data":    { "sources": ["nftt/data/train.jsonl"] },
  "model":   { "id": "unsloth/Qwen3-4B-bnb-4bit" },
  "providers": { "judge": { "model": "deepseek/deepseek-v4-flash-0731" } },
  "eval":    { "sets": ["nftt/eval/heldout.jsonl"], "model": "myproj-v1" }
}

The domain section declares validators, rubric, and parse behavior (ADR-0009) — zero lines of Python for known domains; declared checks and optional module hooks compose additively (declared first; a module hook can never silence a declared check). See nftt config for the full key listing and docs/BRINGING-YOUR-OWN-DATA.md for the data contract.

Secrets never enter the config (the loader warns on token-shaped values). Auth resolves: OPENROUTER_API_KEY → OPENROUTER_API_KEY_FILE → nftt/secrets/openrouter.key.

Development

.venv/bin/pip install -e ".[dev]"
.venv/bin/pytest          # unit tests (pure functions; no GPU, no ML imports)
.venv/bin/ruff check .    # lint

Tests are pure-function at the tool layer; GPU-touching paths get manual smoke tests (documented in the change's apply notes).

Repository layout

src/nftt/        the tool (CLI, config, stages; ML imports stay lazy)
openspec/        change workflow (proposals, specs, designs, tasks)
adr/             architecture decision records (durable decisions)
docs/            data contract (BRINGING-YOUR-OWN-DATA.md), doctrine
                 (LESSONS.md), dormant triggers (WATCHLIST.md)

Parked decisions (deterministic serve defaults, ollama jinja-TEMPLATE support, docs packaging) live in docs/WATCHLIST.md — each with the evidence threshold that would reopen it. Check the trigger conditions there before proposing related work.

License

nFTT is free software, licensed under the GNU Affero General Public License v3.0 or later (SPDX: AGPL-3.0-or-later) — see the LICENSE file for the full text.

The AGPL was chosen because nftt is tooling you may run behind your own web frontend or server-side pipeline: section 13 closes the service loophole, so anyone operating a modified version over a network must offer its source to the users of that network service. Your training data, fine-tuned model weights produced by running nftt on that data, and projects generated with nftt init remain yours; the license covers the nftt code itself.