Skip to content

solvi 1.2.0 — stricter defaults, mutants for your cases, the extractor without torch

Choose a tag to compare

@mxkuzn mxkuzn released this 17 Oct 00:17
· 452 commits to main since this release

solvi 1.2.0 — stricter defaults, mutants for your cases, the extractor without torch

Three defaults move to the safer side, the test runner learns to test its own cases, and the document extractor
predicts without torch. Everything else is opt-in. Full details, with every measurement and its conditions, are in the
CHANGELOG.

Highlights

  • solvi mutants PATH makes small faults in your task file — a comparison flipped or moved by its edge, a constant
    nudged, an if/else swapped, a check dropped — runs your cases against each and reports the share they catch, with the
    survivors by line. Passing cases now say whether they would fail if the catalog were wrong.
  • Cases from what you already have: testing.cases_from_store turns stored decisions into regression cases,
    testing.boundary(cases, system=) writes inputs on your rules' own thresholds, testing.metamorphic checks relations
    such as a unit conversion. solvi test gains --fast, --isolate --max-mem, --timeout, --reruns and a status per
    case; solvi.testing.history keeps verdicts across runs (stable / flaky / regression / fixed, known bugs).
  • LongSpanExtractor predicts without torch: load(..., backend="onnx") on pip install "solvi[onnx]" reads the
    checkpoint's ONNX export; training stays on torch.
  • Learn-then-test answers more at the same promise with calibrate(..., procedure="fallback") (a sequential
    procedure over the threshold grid; the default is unchanged). Labels with several right answers: AnyOf(a, b).
    A rule's why can name the branch it took and its numbers (because, missing).
  • Decisions you can account for: provenance follows every hard check that ran; a remote model whose server did not
    answer is counted apart from the model's own doubt (no_answer); KnowledgeStore.forget(source) takes back what a source
    told and re-decides what rested on it; knowledge facts with a validity interval resolve by time; loading a large
    knowledge store is linear.
  • Experimental, all opt-in: laws that follow a changing stream (a windowed buffer, suspension instead of retirement,
    probes under answer noise, history atoms, per-law confidence, two missing laws at once); AuditWatch for a check run on
    a random share of decisions; ShiftWatch next to Platt calibration; an InjectionDetector you fit on your own tool
    outputs; narrows / widens of a rule change proved with z3 (solvi[smt]); ShipWatch, SliceGuard and a labelled
    Shadow for changes after they ship; System(cache=) for pure catalogs; batch saves on every store; a writers=
    registry for the knowledge store; Guard(allowed_tools=), signed approvals and signed user messages.

Breaking changes and how to migrate

  • Several failed hard checks: the strictest answer decides. Before, the first check declared in the catalog decided.
    List a question's options from mildest to strictest; where they give no order, the first declared still decides and a
    warning names the checks. Catalog(combine="first") restores the old rule. Every catalog fingerprint built on the
    default changes once, so a decision stored under 1.1 re-decides against the new default as if another catalog had made
    it; an explicit combine="first" keeps 1.1's fingerprint. Guard keeps "first" on its own catalog.
  • solvi.build splits examples by a hash (split="hash"): each example's share — fit, guarantee, dispatcher — is set
    by a hash of its state salted by seed=, so a refit on more examples leaves the old ones where they were. A system
    rebuilt on 1.2 is therefore fitted and calibrated on other examples than under 1.1. split="shuffle" rebuilds 1.1's
    split exactly.
  • Drift is watched by default (build(monitor="auto")) once the guarantee has at least 500 calibration examples —
    about 2 000 examples with the default split, 4 000 with a slow path. After a flag, System 1's answers are checked by the
    slow path or go to a person until reset_drift(). monitor=None turns it off, monitor=True turns it on at any size,
    Dispatcher(on_drift="accept") keeps the old routing. Below 500 nothing changes.
  • Certificate's alarm bets on breaks of a stated size: two CUSUMs against failure rates 1.5× and 2× the promise at
    level 50. The old alarm fired on streams running well inside their promise. Certificate(alternatives=None) restores
    it; alarm= still sets the level.
  • A failed System.guarantee call keeps the guarantee in force. The call still raises; a GuaranteeWarning says the
    old guarantee stays and guard_history logs the attempt. Before, the failed call had already removed it. For the old
    outcome remove it first: system.guarantee(question, False).
  • LongSpanExtractor.load defaults to backend="auto": torch when torch and transformers are installed (nothing
    changes), else the ONNX export instead of an error. The backends have different model fingerprints, so a decision stored
    on one does not replay as the same model on the other; backend="torch" pins the old behaviour.

Other behaviour changes, each with a switch back, are marked Behaviour change in the CHANGELOG: a drift flag with no
slow path sends answers to a person; a retracted knowledge item told again is a hypothesis until re-tested; laws count a
candidate's support where it would apply; admit_by_probes keeps witnesses when more than one law may be missing; a change
pipeline stage that raises is an error, not a rejection; testing.boundary writes a different case list.

pip install -U solvi