Repository navigation
solvi 1.2.0 — stricter defaults, mutants for your cases, the extractor without torch
solvi 1.2.0 — stricter defaults, mutants for your cases, the extractor without torch
Three defaults move to the safer side, the test runner learns to test its own cases, and the document extractor
predicts without torch. Everything else is opt-in. Full details, with every measurement and its conditions, are in the
CHANGELOG.
Highlights
solvi mutants PATHmakes small faults in your task file — a comparison flipped or moved by its edge, a constant
nudged, an if/else swapped, a check dropped — runs your cases against each and reports the share they catch, with the
survivors by line. Passing cases now say whether they would fail if the catalog were wrong.- Cases from what you already have:
testing.cases_from_storeturns stored decisions into regression cases,
testing.boundary(cases, system=)writes inputs on your rules' own thresholds,testing.metamorphicchecks relations
such as a unit conversion.solvi testgains--fast,--isolate --max-mem,--timeout,--rerunsand a status per
case;solvi.testing.historykeeps verdicts across runs (stable / flaky / regression / fixed, known bugs). LongSpanExtractorpredicts without torch:load(..., backend="onnx")onpip install "solvi[onnx]"reads the
checkpoint's ONNX export; training stays on torch.- Learn-then-test answers more at the same promise with
calibrate(..., procedure="fallback")(a sequential
procedure over the threshold grid; the default is unchanged). Labels with several right answers:AnyOf(a, b).
A rule'swhycan name the branch it took and its numbers (because,missing). - Decisions you can account for: provenance follows every hard check that ran; a remote model whose server did not
answer is counted apart from the model's own doubt (no_answer);KnowledgeStore.forget(source)takes back what a source
told and re-decides what rested on it; knowledge facts with a validity interval resolve by time; loading a large
knowledge store is linear. - Experimental, all opt-in: laws that follow a changing stream (a windowed buffer, suspension instead of retirement,
probes under answer noise, history atoms, per-law confidence, two missing laws at once);AuditWatchfor a check run on
a random share of decisions;ShiftWatchnext to Platt calibration; anInjectionDetectoryou fit on your own tool
outputs; narrows / widens of a rule change proved with z3 (solvi[smt]);ShipWatch,SliceGuardand a labelled
Shadowfor changes after they ship;System(cache=)for pure catalogs; batch saves on every store; awriters=
registry for the knowledge store;Guard(allowed_tools=), signed approvals and signed user messages.
Breaking changes and how to migrate
- Several failed hard checks: the strictest answer decides. Before, the first check declared in the catalog decided.
List a question's options from mildest to strictest; where they give no order, the first declared still decides and a
warning names the checks.Catalog(combine="first")restores the old rule. Every catalog fingerprint built on the
default changes once, so a decision stored under 1.1 re-decides against the new default as if another catalog had made
it; an explicitcombine="first"keeps 1.1's fingerprint.Guardkeeps "first" on its own catalog. solvi.buildsplits examples by a hash (split="hash"): each example's share — fit, guarantee, dispatcher — is set
by a hash of its state salted byseed=, so a refit on more examples leaves the old ones where they were. A system
rebuilt on 1.2 is therefore fitted and calibrated on other examples than under 1.1.split="shuffle"rebuilds 1.1's
split exactly.- Drift is watched by default (
build(monitor="auto")) once the guarantee has at least 500 calibration examples —
about 2 000 examples with the default split, 4 000 with a slow path. After a flag, System 1's answers are checked by the
slow path or go to a person untilreset_drift().monitor=Noneturns it off,monitor=Trueturns it on at any size,
Dispatcher(on_drift="accept")keeps the old routing. Below 500 nothing changes. Certificate's alarm bets on breaks of a stated size: two CUSUMs against failure rates 1.5× and 2× the promise at
level 50. The old alarm fired on streams running well inside their promise.Certificate(alternatives=None)restores
it;alarm=still sets the level.- A failed
System.guaranteecall keeps the guarantee in force. The call still raises; aGuaranteeWarningsays the
old guarantee stays andguard_historylogs the attempt. Before, the failed call had already removed it. For the old
outcome remove it first:system.guarantee(question, False). LongSpanExtractor.loaddefaults tobackend="auto": torch when torch and transformers are installed (nothing
changes), else the ONNX export instead of an error. The backends have different model fingerprints, so a decision stored
on one does not replay as the same model on the other;backend="torch"pins the old behaviour.
Other behaviour changes, each with a switch back, are marked Behaviour change in the CHANGELOG: a drift flag with no
slow path sends answers to a person; a retracted knowledge item told again is a hypothesis until re-tested; laws count a
candidate's support where it would apply; admit_by_probes keeps witnesses when more than one law may be missing; a change
pipeline stage that raises is an error, not a rejection; testing.boundary writes a different case list.
pip install -U solvi