feat(agent): --distill flag recording LLM calls as a ChatML NDJSON fine-tuning dataset - #136
Merged
Conversation
…ne-tuning dataset
Add a boolean --distill (-d, -dt) flag to tablassert agent that appends
every LLM call of the run (inner agent, judge, reflexion) as one ChatML
record per line to <state-dir>/distill/records.ndjson. The file is
append-only across invocations so a corpus accumulates run over run, and
loads directly in Unsloth Studio / datasets.load_dataset("json").
- tablassert.distill: zero-dependency DistillRecorder + ChatML serializers;
recording never raises into a batch
- agent.make_distilling_model: Model wrapper around generate() (the single
capture seam), tagged per-article with pmc_id for state.json filtering
- tablassert distill-export: convert the NDJSON to an on-disk HF dataset
(new [distill] extra shipping datasets)
- docs (cli.md, agent.md) + docs-coverage mapping; CLI/recorder/wrapper/
supervisor end-to-end tests
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Team Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
SkyeAv
added a commit
that referenced
this pull request
Sep 4, 2026
Cut 16.6.0 and bump the package version in pyproject.toml, uv.lock,
and CITATION.cff.
Minor: two capabilities and one activated release filter. The agent
gains --distill (-d / -dt): every LLM call of a run (inner agent,
judge, reflexion) is appended as one ChatML record per line to
<state-dir>/distill/records.ndjson, tagged with the article's
pmc_id, for fine-tuning in Unsloth Studio or via
datasets.load_dataset("json"); distill-export converts the corpus
to an on-disk HF dataset behind the new [distill] extra (#136).
uuid_on_collision: merge now recomputes number_of_cases as the exact
union of a build-internal supporting_case_ids carrier when present,
replacing first-wins under-reporting and its scalar-conflict report,
and strips the carrier before write in both dedup paths (916f509).
The release-mode applied_to_treat filter is renamed
drop_low_number_of_cases and now reads the number_of_cases column
the pipeline actually emits; the old case_count column never
survived coercion to the significance phase, so the filter had been
a silent no-op since 16.1.0 — release builds now actually drop
treatment edges backed by fewer than 25 cases (#137). Also repairs
a stale gepa_metric test fixture that CI's missing dspy extra had
masked (2dcfcb3, tests only).
Changelog:
- Versioned the Unreleased section as 16.6.0 - 2026-09-04.
- Added the missing --distill entry under Added with its PR link
(#136).
- Added the missing drop_low_number_of_cases entry under Fixed with
its PR link (#137), stating the prior no-op honestly.
- Kept the supporting_case_ids union entry; it was committed
directly to main and carries no PR link.
- Normalized that entry's double-hyphen to the house em dash.
Docs: docs/configuration/graph.md documents the number_of_cases
exception to merge first-wins (8b669d8); #136 already updated
docs/cli.md and docs/agent.md.
Testing:
- uv run pytest -q -> 1197 passed, 3 skipped (96% coverage)
- uv run ruff check . && uv run ruff format --check . && uv run pyright -> clean / 0 errors
- uv lock --check -> up to date
- uv run mkdocs build --strict -> clean
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a boolean
--distill(-d,-dt) flag totablassert agentthat records every LLM call of the run — inner CodeAgent conversations, plus judge and tier-2 reflexion when enabled — as one ChatML JSON object per line in<state-dir>/distill/records.ndjson.The dataset is append-only across invocations, so a fine-tuning corpus accumulates run over run, and the
messagescolumn is plain ChatML, which Unsloth Studio auto-detects on JSONL upload (verified against unsloth.ai docs) — no column mapping orstandardize_sharegptstep.Record format
{"messages": [{"role": "system", ...}, ..., {"role": "assistant", ...}], "purpose": "agent|judge|reflexion", "model_id": "...", "pmc_id": "PMC...", "call_index": 0, "timestamp": "...", "token_usage": {...}}pmc_idjoins againststate.jsonto keep only MAPPED runs as training targets;call_indexlets you keep each run's final (most complete) conversation.How
agent.make_distilling_modelwrapsmodel.generate(modeled on the existingFakeModel) — one central seam covering agent, judge, and reflexion; recording failures log and never break a batch.tablassert.distill.DistillRecorder(pure JSONL append).[distill]extra (datasets) gating the newtablassert distill-export --distill-dir ... --out ...command, which converts the NDJSON to an on-disk HF dataset (save_to_disk).--distillis rejected with--optimize(the GEPA path bypasses the recording seam).Tests & docs
tests/test_distill.py(12 tests: serializers, append-across-invocations, never-raises, wrapper overFakeModel, HF export round-trip), CLI flag/forwarding tests, and a FakeModel-driven supervisor end-to-end test.docs/cli.md(flag table +distill-exportsection),docs/agent.md("Distilling a fine-tuning dataset"), docs-coverage mapping updated.Note
tests/test_agent_eval.py::test_gepa_metric_returns_predictionfails in a venv withdspyinstalled — pre-existing and unrelated (gepa_metric/quality_scoreare byte-identical to HEAD; the test was always skipped before because theoptimizeextra was not installed locally). Worth a separate fix.