Skip to content

Releases

v1.5.0

Choose a tag to compare

@mcp-tool-shop mcp-tool-shop released this 08 Nov 02:45
f473476

This release changes the example tooling and the tests. The installed aspire package has no code changes.

Added

  • Verifier-plan smoke test S1 (examples/verifier-distill/s1_nli_critic.py): an NLI critic run on the Intel NPU and iGPU,
    with its pre-registration and receipts. It failed its pass line on both devices
    (see the run note in docs/runs/receipts/2026-10-10-smoke/).
  • Verifier-plan smoke test S2 (examples/verifier-distill/s2_grpo_smoke.py): a GRPO memory-and-speed smoke on one GPU,
    with its pre-registration, an exclusive-GPU guard that works on Windows,
    memory flags, per-phase allocator logging, a probe mode, a hash deny-list and CPU tests.
    • Prompt draws are recorded (s2_draw.py, --draw), and --rollout-only counts correct items per prompt.
    • Logging: save-all output (written crash-safe), logging and post-pass modes, --stop-after-new, and S2 sign diagnostics
      with a pre-pass baseline and seed.
    • GRPO loss masks, truncation handling, --stress-pool {selected,tune} with width telemetry, and the offrig quote rule
      (quote_rule.py), including its edge cases.
    • A CT interview pipeline (s2_interview.py, s2_interview_tag.py).
  • examples/verifier-distill/s2_grpo_smoke.py: the S2 GRPO training run, launchable in chunks.
    • --arm mech|critic, --train-seed, --trunc-window-updates and --trunc-halt-rate name a training run.
      All four are required, with no defaults, once any training flag is given.
    • --stop-after-updates N ends a chunk cleanly. The train state (LoRA tensors, AdamW state, RNG states,
      global step, CT guard, truncation window, and the length and sha256 of every archive) is saved every
      --state-every-updates updates (default 10) and at the chunk's end; the last 2 saves are kept.
      --resume-from <chunk dir> continues into a fresh --out from the latest save. After a kill it cuts a
      copy of each archive back to the save point and records the cut.
    • A resume marks the chunk it consumed (CONSUMED_BY.json). Resuming a consumed chunk, or any state that is
      not the run's newest, is refused (RESUME_ALREADY_CONSUMED, RESUME_NOT_LATEST). The one escape,
      --resume-override-reason, is recorded, and every later chunk of that run carries forked_or_rewound: true
      and a verdict line ending FORKED_OR_REWOUND. The claim (CONSUMED_BY.json) is created exclusively
      just before the resuming chunk's first save: of two resumes racing for one chunk, the later halts with
      S2-TRAIN-HALT-RESUME_RACE_LOST and saves no state. A cut re-checks the saved prefix, hashes its copy, and leaves
      CUT.json in the killed chunk naming the canonical copies.
    • Resume refuses any difference in the run header: settings, seeds, holdout and draw shas, aspire commit,
      and the sha256 of the script, quote_rule.py and s2_draw.py, model revision, frozen stack.
    • Resume is state-exact on the CPU. On CUDA it restores all state, but the kernels are not
      bitwise-deterministic; nothing switches deterministic mode on, and the header records
      cuda_deterministic: false with the cuDNN and cuBLAS settings.
    • Training is seeded: the fresh LoRA's init, and each rollout by (seed, step, group). Save-all records the
      seed that was applied, and an unseeded smoke run records none.
    • A rolling truncation halt beside the CT collapse guard, sharing its halt path. Every guard window (S1, S2,
      U1, U2 and the truncation window) is appended to s2-hf-guard-history.jsonl.gz.
    • The critic arm: each all-wrong group gets one retry with a mechanical tool hint (offrig's quote rule and
      lexical retrieval; NLI stays off because S1 failed). Retries are scored on gold and archived in
      s2-hf-critic-retries.jsonl.gz. They never enter the loss, the guards or the training save-all, and
      they draw from a forked RNG. The registered runs are mechanical only; this arm is a diagnostic.
    • Training verdicts: S2-TRAIN-CHUNK-OK and S2-TRAIN-COMPLETE (exit 0), S2-TRAIN-HALT-<rule> (exit 2),
      S2-TRAIN-FAIL (exit 1).

Changed

  • Coverage now also counts examples/verifier-distill, and CI's floor of 90% applies to it.
  • Public files no longer name personal desktop apps: the canned desktop is OS-only, and a test keeps the names out.
  • The site's lockfile moves undici to 8.12.3 and zod to 4.6.7 (4 Dependabot alerts).

Fixed

  • Two S1 label counts in the docs were counted on a truncated key and are corrected.

aspire-si 1.4.0

Choose a tag to compare

@mcp-tool-shop mcp-tool-shop released this 24 Oct 01:51
4b755df

Added

  • examples/sft-experiment/sft.py: training options for distillation ladders (#168): --chat-template-kwargs, --overlong drop with --max-dropped-fraction, --gradient-checkpointing, and --ledger / --record-package.
  • Handbook: a page on the fine-tuning experiment and the distillation options.

Changed

  • LICENSE and the forge integration's metadata name mcp-tool-shop as the copyright holder and author (#169).
  • The release workflow pins pypa/gh-action-pypi-publish to v1.14.3 (#166).
  • README translations refreshed (#171).

Fixed

  • The 1.3.0 changelog entry listed options that weren't in 1.3.0; they're under 1.4.0.

Full notes: CHANGELOG.md

v1.3.0

Choose a tag to compare

@mcp-tool-shop mcp-tool-shop released this 15 Oct 00:12
b8ab54d

Added

  • examples/sft-experiment/: the fine-tune-then-ASPIRE experiment (#11) and its
    run report. It holds:

    • a dataset builder that keeps evaluation prompts out by word overlap and embeddings;
    • a judge set of one-sentence planted errors;
    • a cleaning step that records every drop;
    • QLoRA fine-tuning;
    • probes with schema 1.1 drift exports;
    • the held-out and judge measures, with intervals that resample prompts;
    • pod scripts that check the rented host's CUDA driver, GPU memory and download speed before
      installing anything.

    Result, over three training seeds (run 1 report):
    fine-tuning the student before ASPIRE makes no reliable difference to how well the critic spots
    planted errors, with either teacher. A first single-seed result said it hurt the composite
    critic (0.65 against 0.87); two more seeds did not bear that out. The critic's accuracy varies
    much more between training runs than between conditions (the composite control ranged from
    0.43 to 0.87). The hidden-state trajectory replicates: the fine-tune moves the student about
    30 times as far as ASPIRE does, always in the same direction, and ASPIRE's own drift opposes
    that direction in every seed, with or without the fine-tune.

  • Judges read order-averaged (report): fresh_pairs.py
    plants errors on new prompts, judge_kev.order_averaged averages a judge's probabilities over
    both answer orders, and judge_logprob.py reads the teacher's A/B log-probabilities. By single
    choices every judge tried is position-biased and below 0.75. Order-averaged, on 149 fresh pairs,
    Kev-4B reaches 0.973 (CI 0.946 to 0.993), Kev-9B 0.859 and the 32B teacher 0.886. Kev-4B,
    read this way, is the reference judge for later comparisons.

  • Step 2 (report): critics trained on 128 prompts instead of 32
    score higher on planted errors (mean 0.745 against 0.635, every seed up by 0.10 to 0.13), but
    differ between seeds exactly as much (a range of 0.181 at both). The fine-tune comparison waits
    for a critic setup that varies less.

  • Critic-init test (report): one draw of the critic's
    initial weights, with nothing else changed, produced a critic that prefers the flawed answer
    (0.315, interval entirely below 0.5). Under the committed rule both the critic's initial weights
    and the run's seed carry the spread between runs; the ranges rest on three runs each and one
    noise-floor pair.

  • critic.init_seed (config, default unset): seeds the critic's initial weights apart from the
    run's seed, leaving the run's own random stream (student adapter, data order, sampling) as if no
    critic had been built. Unset, nothing changes. Used by the critic-init test
    (plan).

  • Critics with chosen attributes (report): the critic found by
    chance that prefers flawed answers reads as an auditor on fresh pairs (0.678 [0.582, 0.763]).
    Fostered Auditor and Advocate heads on three feature sources trained, but the shuffled-label
    control failed by design (a learnable edit takes a random sign), so no role reading is made. The
    descriptive numbers show heads that separate a pair (up to 0.97) but barely flag a lone flawed
    answer (AUC about 0.55).

Changed

  • aspire.teachers.local: the scoring request and its parser are module functions
    (evaluation_request, parse_evaluation, extract_score, extract_improved), so other code
    scores exactly as the local teacher does. The request text and the parsed scores are unchanged.

Security

  • Workflow jobs run with a read-only token by default. ci.yml had jobs with no permissions block, so they inherited the repository default token. A top-level permissions: contents: read now applies to every job that does not set its own (the jobs that publish or write keep theirs). Closes the actions/missing-workflow-permissions code-scanning alerts.

aspire-si 1.2.0

Choose a tag to compare

@mcp-tool-shop mcp-tool-shop released this 07 Oct 03:55
c3130f2

The first release tested end to end on real models. A 1.5B student was trained against 32B teachers on a rented GPU; the run report is in docs/runs/2026-10-06-pod-run.md.

Fixed

  • Twelve bugs that the real runs exposed, each with a test:
    • Checkpoints could not be loaded, and aspire evaluate ignored the trained LoRA.
    • The critic read the prompt without the response it was judging.
    • The student's input was cut from the wrong end.
    • Teacher JSON broke on LaTeX escapes, or was cut off mid-answer.
    • --config lost its settings.
  • Security: checkpoints load with weights_only=True, so a crafted checkpoint can no longer run code.

Added

  • aspire judge and aspire.judge score responses with a trained critic. No teacher is needed.
  • Local and composite teachers work from the config and the CLI. The Claude teacher defaults to claude-sonnet-5-5, and the device defaults to cuda only when CUDA is available.
  • Geometry export schema 1.1: step_axis and scalar_source tell ScalarScope how to read the steps.
  • The export is written after every epoch.
  • examples/local-run holds a run that needs no API key. examples/pod-run holds the configs, probes and drift scripts behind the ScalarScope fixtures.

Honest docs: the critic learns the teacher's score, but the student is not yet trained toward better answers. That work is tracked in #11. The README, the handbook and all seven translations say so.

Full details are in CHANGELOG.md.

aspire-si 1.1.0

Choose a tag to compare

@mcp-tool-shop mcp-tool-shop released this 07 Oct 00:42
99b7505

What's Changed

  • Training-dynamics export for ScalarScope; trainer tests run again by @mcp-tool-shop in #6
  • Full treatment: 99% coverage held at 90%, structured CLI errors, ~40 bug fixes by @mcp-tool-shop in #8
  • ci: bump the actions group across 1 directory with 5 updates by @dependabot[bot] in #9
  • Release 1.1.0 prep: new logo, release.yml for PyPI Trusted Publishing by @mcp-tool-shop in #10

New Contributors

Full Changelog: https://github.com/mcp-tool-shop-org/aspire-si/commits/v1.1.0