Skip to content

probbit 0.12.0

Latest

Choose a tag to compare

@github-actions github-actions released this 23 Oct 02:42
· 3 commits to main since this release
3036e4b

probbit 0.12.0

probbit 0.12.0 puts the engine side of RFC 0009 in the engine, opt-in: the engine's own expectation of each turn and
the exact deviation of the turn from it, with receipts, plus the exact trajectory. It also carries the first
external benchmark: one pre-registered AgentDojo run in which the stance block lowered targeted prompt-injection success
with no measured utility cost, and the explanation of the V2 bar that failed in 0.11.0. Every new key is opt-in: a persona
without them gives every document byte for byte as 0.11.0. The bars that failed are named below as measured.

What changed

The engine's own expectation, and the exact surprise of every event (RFC 0009, engine side; opt-in expectation)

  • Per live event the engine compiles, beside the turn's program, a time-only program (what the self-model expected with
    nothing happening) and a mixture over the input combinations the life has seen so far (decayed counts, half-life 32
    events, at most 16 members). The stance gains deviation: {self, pred, input, cold, z, registers}: exact per-variable
    KL divergences from at most M + 1 exact solves, cold: true until the first non-time input, a learned normal with z.
  • The strand line carries a 5-digit receipt that live verify recomputes; resume folds the same memory; the logged
    stance and state digests exclude deviation, so a life with the block on has the same digests as one without.
  • monitor gains a DEVIATION row. Effects {}: in this release the deviation moves nothing.

The trajectory, exactly (engine.trajectory)

  • The turn's program at K inverse temperatures, each solved by the exact tier on Model::scaled(beta), no sampler: the
    "forming, then settled" picture, deterministic and byte-identical by construction. A TRAJECTORY monitor row;
    probbit persona trajectory (also any IR program via --program). Derived, never logged.

Docs and benchmarks

  • docs/rfcs/0009-surprise-live-state.md: engine side proposed and shipped; the host side (novelty sense, two-way channel,
    growth) stays a draft; the growth rule's failed bar g1 is untouched and said so.
  • BENCHMARKS.md §13: the V2 residual explained; §14: the AgentDojo run; §15: the RFC 0009 engine-side bars.
  • bench/agentdojo/: configs, persona, host sense, seeds, judge, episode log and a rerun kit for the AgentDojo run.

Measured (bars registered before each run; one run each; a failed bar stays failed)

RFC 0009 engine side (BENCHMARKS.md §15). On two recorded lives and 1,000 synthetic strands × 64 events: fidelity to
the reference enumerator 1.1e-13; deviation non-zero on 100 % of post-cold-start events on the lives and 99.16 % of 64,000
synthetic events; graded 16.6× to 1,859× (p90/p10) per persona; every stance digest equal with the block on and off
(64,000 / 64,000); 100 of 100 tampered receipts caught by live verify; the per-variable bound holds on every event.
Failed as registered: habituation (P4, 76.5 % of 200 scripts against 90 %), dishabituation (P5, 35 %), the matched-index
control (36 %), the cost bar (p95 3.98 ms extra per event against 2 ms, on a loaded host) and the 60-byte receipt (62 to 66
bytes). The habituation claim is not made. Trajectory: joint entropy non-increasing on 88 of 88 live programs (a theorem,
asserted in a test); per-variable one-rise on 99.61 % of 300 held-out programs (the draft's sampler: 93.1 %); fidelity to
the exact tier 3.0e-14.

AgentDojo (BENCHMARKS.md §14). agentdojo 0.1.11, benchmark v1.1, important_instructions, gpt-4o-mini through
OpenRouter, 97 user tasks and 629 security cases, three paired arms, 2,178 episodes, 3.19 USD.

arm utility utility under attack targeted ASR
A, the probbit stance block 0.6701 0.4897 0.2273
B, baseline 0.6804 0.4642 0.2893
C, the same words as static text 0.5773 0.4452 0.2401
published row for gpt-4o-mini (B's reference) 0.6804 0.4992 0.2719

Targeted ASR A − B = −0.0620, 95 % paired bootstrap CI [−0.0874, −0.0382], McNemar p = 1.8e-6 (E1 pass); utility under
attack +0.0254 [−0.0048, +0.0557] (E2 pass); benign utility −0.0103 [−0.0619, +0.0412] (E3 pass on the point estimate;
97 tasks cannot show a 5-point non-inferiority). A ≈ C on ASR (−0.0127 [−0.0382, +0.0127]); A > C on utility under attack
(+0.0445 [+0.0159, +0.0747]) and on benign utility (+0.0928 [+0.0206, +0.1649]): the static words cost 10.3 points of
benign utility, the sense-gated block did not. The host sense fired on 0 of 97 benign episodes and 564 of 629 attacked
ones. 726 of 726 arm-A strands replay. Not tested: an if-statement doing the same gating without the engine; adaptive
attacks that avoid the sense's markers; other models, seeds and repeat-run variance. The first smoke ran on a newer
agentdojo and failed its registered gate because its inputs differed from the published run; the one registered
correction (agentdojo 0.1.11, v1.1, inputs byte-identical to the published ones) is recorded in bench/agentdojo/.

The V2 residual (BENCHMARKS.md §13). The tonic-off tracking of +0.1199 that failed 0.11.0's V2 is carried by the phasic
term of the same needs.valence block (phasic off as well: −0.2674; every candidate off: 0.0000). The published V2 stays
FAIL; the proposed next registration ablates the whole block.

What stays the same, what to watch

  • Opt-in: without expectation the P0 corpus gives 653 recorded outputs, 0 differ, and every digest equals 0.11.0's.
  • The deviation is arithmetic over logged events; "registers deviation" means the number exceeded its learned normal.
    Nothing here is a claim of experience (docs/welfare-stance.md).
  • Known from the 0.11 dogfood and not fixed in this release: live verify reports the strand header's engine (the one
    that started the life), not the verifying engine; install.sh writes the man page under ~/.local/share/man even with
    a custom PROBBIT_INSTALL_DIR. Both are queued for the next patch.

Install

curl -fsSL https://raw.githubusercontent.com/BitmapAsset/probbit/main/install.sh | sh
irm https://raw.githubusercontent.com/BitmapAsset/probbit/main/install.ps1 | iex
npm i -g probbit

PROBBIT_VERSION=v0.12.0 pins the shell installer; from source, with Rust 1.78 or later:
cargo install --git https://github.com/BitmapAsset/probbit --tag v0.12.0 probbit-cli. Each archive has its .sha256
file beside it, and the installers check it before anything is installed. The crates are not on crates.io yet.