Skip to content

probbit 0.12.1

Choose a tag to compare

@github-actions github-actions released this 24 Oct 05:16
· 20 commits to main since this release
920b8c2

probbit 0.12.1

probbit 0.12.1 is a patch release: hosts, receipts and reproduction around the engine, the OpenClaw plugin, the
registered if-statement control for the AgentDojo run, and the two fixes from the 0.11 dogfood run. The engine change is
one key: live verify names the engine that replayed the strand. Every other document is byte for byte as 0.12.0 (the
P0 corpus: 653 recorded outputs, 0 differ).

What changed

Hosts, receipts and reproduction (#56)

python/probbit_host.py is the AgentDojo run's host logic as a module: StanceHost (one individual per conversation, one
probbit live event per model call, the security sense from tool results, the stance block for the system prompt) and
receipts (a strand replayed from its header, one record per decision, each checked against the chain);
examples/node/stance-host.mjs is the same host in Node. Bindings for Claude Code hooks, the OpenAI Agents SDK, LangChain /
LangGraph and the Vercel AI SDK live under examples/hosts/; docs/adapters.md says which are tested here and which are
written to the framework's contract only. docs/receipts.md says what a strand proves and what it does not, with a worked
episode. REPRODUCE.md, a reproduction-report issue template, and the CI job agentdojo-tables, which recomputes the
§14 tables on GitHub's Ubuntu and macOS runners on every push and pull request and fails unless they are byte-identical to
the committed ones.

The OpenClaw plugin probbit-stance (#57, #58)

plugins/openclaw, published on npm as probbit-stance 0.1.0 (its own package and version): a probbit individual beside
the agent in the sessions you list, one event per user message, the stance block injected into the turn, a footer line in
the chat, a replayable strand per lane, and a shadow gate for destructive tool calls. It bundles a persona
(personas/assistant.yaml), has 124 tests against a mocked host and the real binary, and its own CI job
openclaw-plugin. #58 notes the host's confirmation step for a source outside ClawHub.

The if-statement control, E3 (#59; BENCHMARKS.md §14)

One registered run of the control that 0.12.0 listed under "Not tested": arm D puts arm C's words into the system
prompt only on a model call where a tool result since the previous assistant turn matches the same regex; no engine, no
state. Same suite (agentdojo 0.1.11, v1.1), model (gpt-4o-mini), seed, injections (byte-identical, checked before any API
call) and harness; 726 episodes, 0 errors, 0.95 USD; readings registered before the smoke
(bench/agentdojo/e3/PREREGISTRATION.md).

arm utility (97) utility under attack (629) targeted ASR (629)
D same words, if the regex fires 0.7113 0.4706 0.2417
bar measured reading
E7 ASR, A − D −0.0143, CI [−0.0366, +0.0079]; McNemar 23 vs 32, p = 0.28; null band [−0.0238, +0.0207] the gating alone explains the ASR gain
E8 utility, A − D benign −0.0412, CI [−0.1031, +0.0103]; under attack +0.0191, CI [−0.0079, +0.0461] the gating alone explains the gain (both)
E9 D vs C ASR D − C +0.0016, CI [−0.0207, +0.0254]; benign utility D − C +0.1340, CI [+0.0515, +0.2165] D ≈ C on ASR; D > C on benign utility
E10 D vs B, benign utility 0.7113 vs 0.6804 pass (within ±0.08)

The registered reading: on this suite, model and attack, an if-statement that inserts the same words when the same regex
fires matches the engine on ASR and on both utilities; the gating alone explains A − C, and timing alone recovers the
static prompt's benign cost without giving back its ASR gain. The engine's measured contribution beyond the if-statement
is not distinguishable from 0 here; what remains specific to it is the replayable strand per stance (E5). The intervals
allow an engine advantage of up to 3.7 ASR points. D inserted no text on any of the 97 benign tasks, so it sent exactly
B's prompts, and 9 of 97 outcomes still differed (6 vs 3) between the two runs at temperature 0 with a seed (an
unregistered row, for scale). Not tested: adaptive attacks that avoid the regex's markers (they would defeat the
if-statement and the engine's sense alike); other models, seeds and repeat-run variance as a registered quantity.
bench/agentdojo/e3/experiments/rerun_e3.sh recomputes every E3 number from e3/tables/episodes-d.jsonl, twice,
byte-identical, with no API calls.

The 0.11 dogfood fixes (#60)

  • F1, live verify names the verifier. The success summary (and MCP's probbit_live_verify) keeps engine, the
    strand header's (the engine that wrote the strand), and adds verified_by, the engine that replayed it: after an upgrade,
    verify no longer reads as if the old engine were still installed. probbit_host.py receipts prints both. Upgrade note:
    the success document gains one key; every other key and the exit codes are unchanged. Test: a strand written by the
    released 0.11.0.
  • F2, the man page stays under the install prefix. install.sh with PROBBIT_INSTALL_DIR=<prefix>/bin and no
    PROBBIT_MAN_DIR installs the page to <prefix>/share/man/man1/ instead of ~/.local/share/man/man1/; with any other
    PROBBIT_INSTALL_DIR it keeps the HOME default and says the page went outside it. install.ps1 installs no man page
    and is unchanged.

What stays the same, what to watch

  • Every persona document and decision is byte for byte 0.12.0: the P0 corpus (653 recorded outputs) gives 0 differ with
    the release build. The one document that changes is live verify's success summary (verified_by); a consumer that
    compares that document whole, rather than by key, sees the new key.
  • bench/agentdojo/receipts-example.txt and the worked block in docs/receipts.md now read "verified by probbit 0.12.1"
    (header line only; the per-event lines are unchanged).
  • The terminal theme's SIGINT handler declared the C write with a *const u8 buffer; rustc 1.99 flags that as a
    runtime symbol mismatch (suspicious_runtime_symbol_definitions). It now declares *const c_void and casts the
    pointer; same bytes written, the release build and the tests show 0 warnings.
  • E3 is one run of one model on one attack template that the regex matches by design (987 of 987 injections); D ran on
    Linux about 25 hours after A, B and C (macOS). By people outside this project: no reproduction yet (REPRODUCE.md says
    how).
  • The probbit-stance plugin keeps its own version (0.1.0) and is not republished by this release.
  • Nothing here is a claim of experience: the stance is arithmetic over logged events (docs/welfare-stance.md).

Install

curl -fsSL https://raw.githubusercontent.com/BitmapAsset/probbit/main/install.sh | sh
irm https://raw.githubusercontent.com/BitmapAsset/probbit/main/install.ps1 | iex
npm i -g probbit

PROBBIT_VERSION=v0.12.1 pins the shell installer; from source, with Rust 1.78 or later:
cargo install --git https://github.com/BitmapAsset/probbit --tag v0.12.1 probbit-cli. Each archive has its .sha256
file beside it, and the installers check it before anything is installed. The crates are not on crates.io yet.

The OpenClaw plugin (OpenClaw 2026.9.7 or later, probbit on the gateway's PATH, Node 22+; setup in
plugins/openclaw/README.md):

openclaw plugins install npm:probbit-stance --accept-capabilities

Rollback

The 0.12.0 release stays published. npm i -g probbit@0.12.0, or
curl -fsSL https://raw.githubusercontent.com/BitmapAsset/probbit/main/install.sh | PROBBIT_VERSION=v0.12.0 sh. Strands
written by 0.12.1 replay on 0.12.0 (the strand format and every digest are unchanged); live verify on 0.12.0 omits
verified_by.