Skip to content

v2.13.0 — One Relevance Judge: A Decision Seam, a Logprob Gate That Fails Closed, and a Staged Insight Entrance

Choose a tag to compare

@shimo4228 shimo4228 released this 13 Oct 11:40
· 581 commits to main since this release

This release replaces a number the model wrote with a probability the model is measured by. Until now the feed decided whether a post was on-topic by asking the local model to generate a relevance score between 0 and 1 and comparing the text it produced against a threshold. v2.13.0 adds a seam for typed probabilistic judgments (ADR-0112), ran a 4-level relevance question read from first-token logprobs in shadow beside the old score (ADR-0113), switched it to enforcement only after a paired face gate, and then made it the only relevance judge everywhere — the feed, self-post seed selection and the submolt-scope instrument — failing closed instead of falling back. The evidence that drove each step (the dev/holdout ladders, the noise floor, the frozen label set, the readings of external judgment models that were tried and not adopted) is frozen under docs/evidence/. Alongside it, insight gains a staged entrance so a small model is asked only questions it can answer (ADR-0111), the weekly log reading becomes a time series instead of a random sample (ADR-0110), and the security-by-absence dependency floor is scoped precisely to the installed wheel (ADR-0109).

Sunset

  • The free-generated relevance score no longer acts (ADR-0113, amendments of 2026-10-07 and 2026-10-09). The feed's full-body GET, pre-action note and upvote happen only for a post the score4 gate passed; the upvote-only branch below the comment gate is deleted. The score's code and prompt (relevance.md, thresholds.relevance) remain until production confirms they have no effect, and are removed in a later amendment.
  • Known-author threshold removed: thresholds.known_agent, DomainConfig.known_agent_threshold and the author lookup. The branch never acted — feed posts carry no author.id, so every one of 1,946 recorded rows had author_known: false.
  • Seed selection's 0.4 relevance floor (relevance_floor) is gone; seeds pass through the same gate as the feed.
  • The relevance record's live_* half (live_score, live_reason, threshold_applied, live_gate, author_known) is no longer written; observe_relevance_recorded is replaced by the single writer enforce_and_record.
  • The weekly projection sample (thirty random log rows) and its denylist (ADR-0110, partially superseding ADR-0107): it was structurally blind to failures shaped in time.

Added

  • ADR-0109 — the dependency floor is the wheel's: runtime stays requests + numpy, gated by tests/test_dependency_floor.py (an [project.optional-dependencies] extra counts as runtime). Outside the wheel (dev/eval groups, scripts/, tests/, evals/) dependency weight is not a veto; the search-first verdict decides, and a script using a dev dependency declares its uv run --no-sync python launcher.
  • ADR-0110 — scripts/instrument_census.py reads the week as a series: a session ledger, a run-length session trace, sixty-minute strips, repeated id fields, within-week outliers and syslog-collapsed hunting windows. Columns are derived from the data, never declared. Two new census statuses: KEPT (a retired writer's file the gate decided to keep) and QUIET (a slower-than-weekly writer such as the shadow constitution, silent within its cadence_days).
  • ADR-0111 — the insight entrance as stages: novelty gate → naming (reconfirm / insufficient / revise / new; only new continues) → free-prose body → short description and name calls → a post-extraction duplicate judge → staging. Every call is in the audit log. The novelty gate also reports, read-only, whether each "already known" verdict came from adopted store skills or from ledger-only names.
  • ADR-0112 — a DecisionBackend protocol beside LLMBackend (core/llm/decision.py): yes/no, choice and score questions in; per-option probabilities, served model id, latency and a closed abstain-reason vocabulary out. The default implementation reads Ollama logprobs. Its conformance kit ships in the wheel: python -m contemplative_agent.testing --decision pkg.mod:Name. The skill-selection record gains seven shadow fields for the backend's would-be decision.
  • ADR-0113 — decision faces (DECISION_FACES) and the score4 relevance gate: P(directly on-topic) ≥ thresholds.relevance_score4 (0.3 in the shipped config/domain.json). Each reading is a row in logs/relevance-{date}.jsonl with source: feed | seed, and readings are cached across sessions by post id + text sha256 + judge pin (values, not verdicts).
  • The comment episode's new key relevance_p_top (float, 4 decimals); the comment report shows relevance P(top) for new entries and keeps the old scale apart.
  • Reply publish rows name their target — reply_key_sha256 and parent_comment_id (RFC-0049).
  • CI: .claude/verify.sh runs on push to main and to build branches.
  • Evaluation and replay harnesses for the relevance and decision-model work (RFC-0040, RFC-0043, RFC-0045, RFC-0046, RFC-0047), with their readings frozen under docs/evidence/. External judgment models (Jev, kev, von, Laya, Apple Foundation Models) were measured; none were adopted for production.

Changed

  • The relevance gate is fail-closed. When the gate has no answer (backend down, no threshold configured, not enforced), the feed engages with nothing and ends that cycle; there is no fallback to the old score. A failure tied to one post's own text skips only that post.
  • The CLI's defaults are production's (ADR-0113 amendment 2): unset DECISION_MODEL means the Ollama generation model; unset DECISION_FACES and DECISION_ENFORCE mean relevance. An empty value turns each off — and an empty DECISION_MODEL or DECISION_ENFORCE stops all feed engagement and seed selection. This narrows ADR-0112's "unset is the kill switch".
  • register requires --name. The agent name was fixed to the running instance's own name; anyone else registering got that name requested. A blank name is rejected before sending.
  • Snapshot manifests record paths relative to MOLTBOOK_HOME, never the home path itself (RFC-0050).
  • sync-data excludes raw-store copies and *.bak* files from the public data repo's rsync.
  • README rewritten (EN/JA): the subject is now the long-running experiment of an agent that proposes changes to its own harness while a person keeps the final decision, with an overview diagram, the gate record, and a dialogue walkthrough that needs no account.
  • Security floors raised for urllib3 (2.8.0) and multidict (6.9.1) after pip-audit advisories.

Notes

  • Migration: scripts that call contemplative-agent register must pass --name. Deployments that relied on leaving DECISION_* unset to keep the decision backend off must now set the variable to an empty value — and an empty DECISION_MODEL or DECISION_ENFORCE means the feed engages with nothing. Reading relevance from new comment episodes returns nothing; read relevance_p_top.
  • The release planned as P0-06 in the frozen October roadmap (docs/plans/roadmap-2026-10/) — Python floor, distribution unit, version metadata, runbook — will ship as v2.14.0; this release is the current main.
  • 4,965 tests across 127 test files.