De-voice writer SFT pairs and widen the ship-gate holdout - #48
Conversation
Writer SFT briefs were mined verbatim from their own target, so the input sat inside the output (5-gram copy ratio median 1.0) and copying the prompt forward was a winning strategy for the objective. Add a deterministic de-voicing operator so rows are (D(y), y): entities, figures and claim vocabulary are preserved while second-person address, contractions, emphasis, discourse markers, fragment rhythm and connective scaffolding are removed. Pair construction is gated rather than merely built. Every row is measured against the shipped pair gate for cadence movement, checked for entities or figures the operator invented, and dropped unless what the brief shares with the post falls under a cap. Copy ratio drops from median 1.0 to 0.15. Also here: - select-writer-holdouts: deterministic, pinned-compatible holdout carve sized as a share of the briefable pool. A three-item gate cannot reach any significance threshold; a sign test needs more comparisons than that. - eval-writer-adapter: committed ship gate for RAG+adapter vs RAG-alone, loading each arm's weights once. Keeps an adapter only on a majority win that clears a one-sided sign test and does not raise the disqualification rate. - index-voice --from-carve: read holdout ids from the carve file instead of retyping them as flags. - train --detach: portable detached launch via start_new_session, so an unattended run does not depend on a shell staying open (and does not depend on setsid, which macOS does not ship). - Writer train recipe: 16 layers, LoRA rank 16, lr 3e-5, 10 epochs, with rank and learning rate plumbed through the chunk worker and recorded in the checkpoint meta. - release_mlx_memory now honours the MLX opt-in gate; a Metal-less session aborts in C++ where except Exception cannot catch it. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Ran the full pass end to end on a real corpus. Pairs — 60 gated Train — 600 steps in 12 chunks at 16 layers / rank 16 / lr 3e-5. Peak Metal Gate (n=20, adapter vs RAG-alone)
p = 0.98 one-sided, so all three keep-conditions fail. The adapter was archived Worth recording from the disqualification breakdown, since it is what the next
That hypothesis could not be tested from this run: the chunk trainer passes The gate result does not change the case for this PR — the pair construction, |
Persist adapters/latest/checkpoints/step_* after each MLX chunk so a gate can evaluate early weights. Default writer training to 3 epochs. Add eval-writer-adapter --sweep-checkpoints and Contoso-safe archive basenames. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Rebased onto main (merged #49). Added durable per-chunk checkpoints, writer recipe at 3 epochs, and |
Summary
The writer SFT built each row's brief by extracting sentences from the post it
was trying to teach the model to write. Measured on the corpus, the share of
brief words sitting inside a 5-gram of their own target had a median of 1.0,
and 126 of 129 rows would themselves have tripped the
brief_echo_rejectlimitthe gate uses to disqualify a draft. Copying the prompt forward was a winning
strategy for that objective.
This PR makes the pairs
(D(y), y)and gates them on that property.De-voicing operator (
devoice.py)Deterministic and inspectable — no model in the loop, since an LLM flattener is
exactly the component that can leak cadence back into the input.
shouted words, sentence-initial conjunctions, discourse markers, one-line
fragment rhythm, and the article/copula/preposition scaffolding that separates
a written sentence from a jotted note
Negation and comparison words are deliberately never dropped: removing
notwould not de-voice a claim, it would reverse it.
The operator ships with its verifier. Each pair is measured on the existing
gate_paircadence axes, checked for entities or figures the operator itselfintroduced, and dropped unless the brief's overlap with the post stays under a
cap. Fail-closed — a row that cannot be moved is not trained on.
build-writer-sftnow reports the ratio distribution and a per-reason dropbreakdown instead of one opaque skip count.
Widened holdout carve (
select-writer-holdouts)The previous gate ran on three holdouts. Three paired comparisons cannot reach
any significance threshold — a clean sweep is still p=0.125 — so the result
could not distinguish a regression from a coin flip.
The carve reserves a share of the briefable pool (25%, clamped to 12–24),
orders candidates by a stable digest of the piece id so it is reproducible, and
keeps previously carved ids pinned so runs stay comparable. Pieces that cannot
produce a brief are excluded: neither arm could be scored on them.
index-voice --from-carvereads the ids from the carve file, since retypingtwenty flags is how a holdout quietly re-enters retrieval.
Committed ship gate (
eval-writer-adapter)Both arms see the same de-voiced brief and the same exemplars; only the adapter
differs. An adapter is kept only when all three hold:
Weights are loaded once per arm rather than per draft.
--archive-on-failmovesa rejected adapter aside so the write path resolves back to
adapter=none.Also
train --detachruns a train in its own session viastart_new_session.Shell-level
setsidis util-linux and absent on macOS, where a launcher thatreaches for it fails before Python starts.
rank 16, lr 3e-5, 10 epochs. Rank and learning rate are plumbed through the
chunk worker and recorded in the checkpoint meta.
release_mlx_memorynow honours the MLX opt-in gate. A Metal-less sessionaborts inside
metal::load_devicevia C++terminate, which the surroundingexcept Exceptioncannot catch.mine_brief_from_holdoutderives its minimum source length from the caller'soverlap budget instead of a fixed constant.
Test plan
pytest— 317 passing, Contoso fixtures onlyruff check src/ tests/build-writer-sfton a real corpus: pair copy ratio median 1.0 → 0.147select-writer-holdouts+index-voice --from-carve: holdouts confirmedabsent from retrieval
train --writeron the de-voiced SFT, theneval-writer-adapteron thewidened holdout — adapter kept only if it clears the bar
No corpus, posts, or adapters are committed; profile data stays under
~/.personality-protect/.Made with Cursor