v1.0.0 - the execution loop, the flag as the reward

The closed execution loop is proven end-to-end: the LLM emits source-derived facts, augure decides the technique and proves it, lictor builds and runs against the live lab, and the flag read from the running service is the ground truth. Five of the six lab targets land their own flags (2026-10-29 run, the sixth being the designed handoff).

The GRPO wiring carries the execution-verified flags into the trainer: dataset ingest-trajectories merges the pilot's trajectories by id and the grpo_flags.jsonl sidecar carries the flag the service read. The first GRPO run closed nine steps on the student merged from the SFT best run: the reward curve opened at 1.0, breathed between 0.42 and 1.0 where a generation missed the format, and closed at 1.0.

Companion lictor releases carry the layout and pinning fixes this loop demanded: the shellcode ret slot riding the banner's leak (!10), run --technique pinning the decided technique across attempts (!11), the orw stub's flag path precedence (!12).

v0.9.0

Fixed

• SFT training fails fast when max_length would truncate a completion (TRL cuts the sequence tail, so the model never saw the answer terminator; the default 1024 cut every 1145-2168 token KD-GLM completion).
• train task-eval thinking default is now --no-thinking: the training render carries the empty think block into the prompt (per-row chat_template_kwargs), so the inference prompt matches it byte-for-byte.
• train task-eval warns when the token budget cannot fit the longest ground-truth completion, and --max-new-tokens defaults to 2048 (was 512).
• CTF archive cases run the same bounded static recon on ELF attachments as the tier-A cases (the archive corpus used to emit only binary/image, not excerpted placeholders).

Changed

• Benchmark doc: third run results - the union SFT run (16 records, holdout ids excluded) scores the first non-zero task-eval accuracy (0.1923, FPR 0.7308) while perplexity regresses (19.77 vs baseline 12.61); both metrics now sit side by side with the explanation.

v0.8.0 - served-model task-eval, KD guards, CLI wave complete

Everything since v0.4.0 (details in the CHANGELOG 0.8.0 section). Highlights:

Training science (documented, including failures)

• KD loss vectorized ~16x (28 min first step -> ~1.7 min/step)
• KD negative result measured and documented: a cross-tokenizer teacher drove the student to 20.07 ppl (worse than the 12.61 baseline); bimodal loss curve
• Tokenizer-family guard warns before KD training; both KD volume locks measured (tokenizer alignment AND the verified-flag gate)
• GRPO backend + verifiable-reward registry; ADR-0019 OPST

New CLI surface (replaces the ad-hoc host scripts)

• ops status / ops log, dataset filter --sidecar, dataset split --sidecar
• pipeline kd-pass (resume, per-case timeout, --workers parallel), config set / set-model
• train evaluate --base, train runs, providers check-logprobs
• train task-eval --endpoint: score served models (llama.cpp) without loading weights, with thinking control (Qwen3 chat_template_kwargs)

Ecosystem

• GGUF export to llama-server on the training host; multi-event CTF corpus; ground-truth arbitration; pipeline state tracing.

v0.7.0 - ops/CLI wave, KD tokenizer guard, benchmark negative results

Retroactive cut of the Oct 28 afternoon wave.

New CLI surface (replaces the ad-hoc host scripts)

• brainforge ops status / ops log: run-state and log inspection with encoding fallback (#64)
• brainforge dataset filter --sidecar (#64) and dataset split --sidecar (#65)
• brainforge pipeline kd-pass: per-case KD generation with resume, per-case timeout, sidecar-aligned assembly (#66)
• brainforge config set / config set-model (#67)
• brainforge train evaluate --base: base-model perplexity, comparable numbers (#64)
• brainforge providers check-logprobs: KD teacher signal probe (#72)
• brainforge train runs: summary table over every run (#73)

KD science + guard

• Benchmark negative results documented: a cross-tokenizer teacher drove the student to 20.07 ppl (worse than the 12.61 baseline); bimodal loss curve; tokenizer-family guard warns before training (#68, #69, #70).

v0.6.0 - KD loss vectorized (~16x), ADR-0019 OPST

Retroactive cut of the Oct 28 morning wave.

Training

• KD loss fully vectorized: first optimizer step went from 28+ minutes (issue #57) to ~1.7 min/step - a ~16x real-world speedup on the gpu3080 workload. Vocabulary cached once, targets as tensors, batched loss (#63).
• Tokenizer vocabulary cache in the KD loop (#58).

Docs / decisions

• ADR-0019: on-policy self-training (OPST) accepted (#62).
• Benchmark documentation v1: perplexity table + runbook; README moves to 1.0-rc status (#59-#61).

v0.5.0 - pipeline and training hardening

Retroactive cut of the Oct 26-27 wave (the CHANGELOG was frozen at 0.8.0; these notes group the corresponding merges).

Pipeline

• Multi-event CTF corpus ingestion, 30 new tier-B challenges (verified-tier flags)
• Self-tracing pipeline: atomic state file next to the dataset + brainforge pipeline status
• Ground-truth arbitration: teacher/judge flag disputes are resolved by verification, not discarded
• Provider timeout/deadline hardening (wall-clock guard against hung requests)

Training

• GRPO backend (TRL GRPOTrainer) + verifiable-reward registry (register_reward/compose_reward)
• Auto-fit preflight integration, resume on fresh installs (zero run dirs), CPU memory-fit fixes
• KD dataset/target fixes (mixed-type rows, eager dataset, cached logprobs)

Full change list: the v0.5.0..v0.4.0 merge range.

0.4.0

[0.4.0] - 2026-10-24

Added

• Portable training: brainforge train check prefights hardware, model
  compatibility (architecture, chat template, pre-quantized and remote-code
  rejection, gated-model HF_TOKEN guidance) and a memory estimate without
  downloading model weights; --json and --strict modes for CI. Device
  auto-detection covers CUDA/ROCm, Intel XPU, Apple MPS and CPU; TPU/NPU
  fail fast with a --device cpu escape. training.device,
  training.quantization and training.backend (new auto defaults)
  resolve a per-run plan recorded in train_summary.json and eval.json.
  make train-check, make install-rocm and make install-xpu targets.
  Docs: new docs/reference/hardware.md; README, tutorial, training,
  configuration, CLI and architecture pages updated. CUDA/ROCm/XPU paths are
  implemented and unit-tested but not hardware-validated.
• huggingface-hub>=0.26 is now a direct dependency (preflight model
  metadata checks; previously available only transitively).
• Graceful failure: PipelineDef.fallback_examples (default false) turns a
  rejected case into an abstention training record (assistant target is a
  canonical insufficient_information verdict plus the gate reasons) instead
  of dropping it. Metadata carries fallback and fallback_reason; dataset
  validation accepts those records without a passing quality gate. The
  rejected-case audit trail is unchanged. ADR-0016.
• train task-eval treats abstention as a third outcome class and reports
  abstain_rate and abstain_recall (of the records the judge could not
  resolve, how many the model refused too).
• Critic escalation: PipelineDef.retry_with_critic (default null) names a
  critic role that runs, followed by a second judge pass, when the first pass
  is rejected. The critic never sees the first judge verdict. Records from a
  retry carry critic_retry and first_pass_reasons in metadata.

Changed

• Training and evaluation no longer hard-require CUDA: the device plan
  resolves automatically and qlora degrades to plain LoRA where 4-bit is
  unavailable. train evaluate and train task-eval default to
  --quantization auto; train run accepts --device and
  --quantization overrides.
• Rewrite the README for evaluators: dataset-first quickstart, verified
  mechanics, explicit limits, and intent-based docs routing. The docs hub and
  first-model tutorial use the same commands, vocabulary, and qualifiers.
• AirLLM evaluated and not adopted (ADR-0017): layer streaming helps models
  that do not fit VRAM, but its custom training API is incompatible with the
  PEFT/TRL, KD, merge and GGUF contracts, and throughput rules out local
  dataset generation.
• Provider failures are isolated per case: the case is recorded with a
  provider_error reason and the batch continues instead of aborting. Run
  summaries gain fallback and errors counters.
• run_pipeline writes the dataset and the KD sidecar once, atomically, at
  the end of the run (temp file plus rename). A crash plus rerun can no
  longer duplicate lines, and a rerun after a config or prompt tweak no
  longer keeps stale records. The dataset file always exists after a run,
  empty when nothing is accepted.
• New docs/reference/references.md maps each research paper and standard to
  what it informs (judge independence, contamination, QLoRA, KD, RAG,
  graceful failure). Stale links fixed: ADR-0008 pointed at the retired
  GitLab host, README advertised a make docs-serve target that does not
  exist.

Fixed

• Force pillow>=13.0.0 to clear its known CVEs despite fastembed's <13
  cap, and bump certifi to 2026.10.19.
• Bump filelock to 4.2.4 in the lock, clearing CVE-2026-112907,
  CVE-2026-112913 and CVE-2026-112940 (fixed upstream in 4.2.1).
• Audit the uv lock with pip-audit --no-deps --disable-pip so pip's
  resolver cannot reject the Pillow override.