v2.7.0 — Seven New Ways to Annotate

Release Notes: v2.7.0 — Seven New Ways to Annotate

The largest Potato release yet: 342 files changed, ~50,000 lines added, 33 commits since v2.6.2. It has two halves. The first is a wave of seven new opt-in features — Psychometrics, Multiplayer Rooms, Boundary Lab, Truth Serum, Think-Aloud Mode, Paper Mode, and Pocket Mode — built around one idea: an annotation tool should not just collect labels, it should measure how labels come to be, and the process itself should produce more than point labels. None of the seven requires an LLM. The second half is a broad workflow-and-infrastructure wave: cross-document event annotation, turn-level annotation, CoT process-reward labeling, PDF cross-page linking, a living-document codebook, RBAC roles with per-cohort schemas, localized dashboards in 10 languages, and a much lighter core install.

Psychometrics Engine — labels with error bars

A live item-response-theory layer (multiclass GLAD; no gold labels, no LLM) that refits in milliseconds as labels stream in, jointly estimating each annotator's ability and each item's difficulty from agreement patterns alone.

• Every exported label carries a posterior and confidence interval — p = 0.94 [0.88–0.97] instead of a bare majority vote (/psychometrics/api/export).
• assignment_strategy: psychometric routes each item to whichever annotator's judgment carries the most expected information, and items past the confidence threshold stop consuming annotation budget (the dashboard shows judgments and dollars saved).
• Codebook-bug detector — items where your best annotators disagree most (negative discrimination) are flagged: that pattern means the item is broken, not the people.
• Pre-study power analysis — python -m potato.psychometrics.design recommends annotators-per-item for a target Krippendorff-α precision via Monte Carlo, with cost estimates.
• Live dashboard at /psychometrics/dashboard; example in examples/advanced/psychometrics/. See Psychometrics.

Multiplayer Rooms — live norming sessions, measured

Synchronized group annotation at /rooms: the calibration meetings teams already run over screen-share, made part of the tool and instrumented. Rooms sync by cursor-polling an event stream — no WebSockets, no new dependencies.

• Norming rooms — everyone votes blind (enforced server-side: pre-reveal, nobody can see another member's label, only who has voted) → the host reveals → the group discusses in a side chat → anyone may change their vote. Blind votes are immutable after the reveal; every post-reveal change is logged with the majority at that moment — a per-member conformity record.
• Live agreement meter — Krippendorff's α computed twice over revealed items: blind votes vs current votes. The gap is the session's norming lift — whether the meeting was worth it, on screen while it happens.
• Huddles — one click seeds a room from every item the team currently disagrees on, with the original annotations shown as context: adjudication as a conversation.
• Shadow rooms — trainees join as observers and watch the host's votes, reveals, and text selections live.
• Event-sourced — every room appends to rooms/room-<CODE>.jsonl; state is a pure replay, so live rooms survive restarts and the log is the session's audit trail. Final votes persist into members' regular annotation state. See Multiplayer Rooms.

Boundary Lab — annotate the boundary, not the point

The moment an annotator commits a label, Potato shows minimal counterfactual edits of the text and asks: you said Polite — would that survive this edit? One click per probe. Ordinary annotation then produces contrast sets for free (every answered probe is a labeled original/counterfactual pair, à la Gardner et al. 2020), boundary rationales that pinpoint codebook ambiguity, and invisible quality control (annotators who flip on meaning-preserving paraphrases get flagged — no fake gold items planted). Rule-based generation, no LLM. See Boundary Lab.

Truth Serum — surprisingly-popular scoring

After each label, one micro-question: what percentage of other annotators will agree with you? Those predictions power the surprisingly-popular estimator (Prelec et al. 2017, Nature): the label whose actual popularity most exceeds its predicted popularity beats majority vote exactly where annotation is hard — no gold labels required. The dashboard's "where the crowd is likely wrong" queue surfaces items to review first, plus per-annotator calibration scores. To our knowledge the first annotation tool to ship peer-prediction scoring. See Truth Serum.

Think-Aloud Mode — voice rationales, zero LLM

Annotators just talk while they work. Speech-to-text runs fully locally (faster-whisper, CPU real-time on the 39 MB tiny.en model); the verbatim transcript is stored as the rationale — deliberately un-summarized, because paraphrasing a think-aloud protocol contaminates the artifact. Labels can be committed by voice ("I'd call it neutral", "final answer: polite") via a rule-based parser. Nothing leaves the machine. See Think-Aloud Mode.

Paper Mode — the dataset that writes its own methods section

python -m potato.paper config.yaml -o paper_export

One command turns a Potato project into a compilable LaTeX dataset report: dataset description, label distributions, annotator table, agreement statistics with the right citations (paper.bib included), timing, and a limitations paragraph. Every section sits between cut-paste markers; tables also export as CSV and every number as JSON. Fully offline, no LLM, no server. See Paper Mode.

Mobile annotation (Pocket Mode)

Potato now supports phones and tablets as first-class annotation devices. With pocket: {enabled: true}, touch devices that open the task are detected and routed automatically to a mobile interface — a card stack with thumb-zone label buttons, swipe navigation, offline annotation with automatic sync, and home-screen installation as a PWA. For single-radio tasks, one tap = one labeled item. Detection is two-layer (server-side User-Agent plus a client-side primary-pointer check that catches iPads masquerading as Macs), with a sticky "Desktop site" opt-out.

Not every annotation type is mobile friendly: tasks that include desktop-only schemas (spans, bounding boxes, video...) are never degraded onto touch — phones keep the desktop UI and see a dismissible warning that a desktop browser is recommended. Either way, a new Devices tab in the admin dashboard shows who is annotating from what (mobile/tablet/desktop visit counts per user, tracked whether or not Pocket Mode is enabled), so you can spot phone annotators on desktop-only tasks. See Mobile Annotation.

Try them together

examples/advanced/showcase/ runs one politeness task with five of the new features enabled at once — probes, peer predictions, voice rationales, live psychometrics, and a one-command paper export.

Cross-document event annotation + corpus map

Annotate events that span many documents. Instead of labeling one document at a time, annotators build events — first-class records that aggregate evidence from across a corpus into a shared template (event type, participants, location, time), the way cross-document event corpora (ECB+, GVC, ERE) are actually constructed.

• New multi_document_event schema wired through config and blueprints, backed by a cross-document event registry.
• Corpus map — an ingest pipeline plus an annotator navigation API for moving through a corpus by event rather than by document.
• Annotator UI for citing evidence into event slots; unit, server, and Selenium coverage; a runnable disaster-news corpus example. See Multi-Document Events.

Turn-level annotation & multi-agent traces

Apply any annotation scheme per turn of a conversation or agent trace instead of once per instance. Schemes bind to declarative filters — by speaker, agent, step type, tool name, or turn range — and render inline inside every matching turn, with values stored under stable turn ids. This generalizes the dialogue display's per_turn_ratings to multiple display types and all widget types. Alongside it: a multi-agent discussion display with agent identity tracking and a run tree, plus a review workflow for session-level annotation. See Turn-Level Annotation.

Chain-of-thought process-reward annotation

Turn a long CoT reasoning trace into a vertical list of reviewable steps, have an external LLM pre-score each step (correct / neutral / incorrect), and let a human confirm or override each one — the standard "LLM labels, human verifies" pattern for producing Process Reward Model (PRM) training data much faster than labeling from scratch. Includes automatic CoT segmentation, the new cot_trace display, a process_reward scheme with per-step provenance (which labels came from the model, which were verified or overridden), and a pre-label API. See Process Reward CoT.

PDF: cross-page linking + OCR

A new PDF link mode: place text-span and region-box anchors on any page and draw typed links between them across pages — anchors are unified span objects carrying page/bbox geometry. Optional OCR (false/true/"auto") builds a text layer for scanned PDFs. PDF.js is now vendored, so PDF tasks work offline.

Living codebook document

The codebook grows a full living-document form: a typed-block markdown document (definitions, examples, decision rules) with a full-page editor at /codebook and a distiller that keeps the document and the code list in sync. Codes remain first-class — add, rename, recolor, move, and delete without touching YAML. See Codebook.

Teams, access control & assignment

• RBAC roles — role/permission-based access control; admin blueprints now route through it uniformly.
• Per-cohort schemas — different annotator cohorts can see different annotation schemes (or subsets), bound to the same cohorts used for batch assignment; the global scheme list remains the default and fallback. See Per-Cohort Schemas.
• Automatic batch assignment for unknown annotators (#163) — annotators who show up without a pre-assigned batch get one automatically, and auto-assigned cohort pins now persist across restarts.
• Scaling guide — a new deployment doc on how Potato actually behaves with large datasets and which knobs matter. See Scaling.

Localization

Admin and annotator dashboards are localized, with 10 bundled languages: Arabic, German, Spanish, French, Hindi, Japanese, Korean, Portuguese, Russian, and Chinese.

LLM endpoint plumbing

• ${ENV_VAR} expansion inside LLM endpoint config blocks — keep API keys out of YAML.
• OpenAI-compatible base_url normalization for local/vLLM endpoints, and bare-host URL normalization before Playwright navigation in web-agent tasks.

Packaging: lighter core install

• AI SDKs are now imported lazily. A core pip install potato-annotation no longer requires (or imports) any AI SDK: endpoints register as lazy specs and the SDK is imported only when a config selects that endpoint_type. Previously a core install could fail to boot with a missing-ollama error; now a missing SDK produces a clear message naming the exact pip install to run.
• The [ai] extra now covers all first-party LLM endpoints — anthropic and google-genai join ollama and openai (pip install "potato-annotation[ai]").
• Server boot import time drops from ~2.0s to ~0.7s with SDKs installed, since none of them load at boot anymore.

Paper & citation

Potato 2.0 has an official paper at ACL 2026 (demo track). The README badge, official citation, and a CITATION.cff are included — if you use Potato in research, there is now a current reference to cite.

Fixes & maintenance

• Repaired 7 pre-existing unit-test failures; the full suite (7,482 tests) passes.
• Upgraded the bundled vite from 5 to 7, resolving 5 Dependabot alerts.
• cot_trace added to the static display-type fallback list in config validation.
• Corrected the Gemini SDK package name in docs (google-genai, not the legacy google-generativeai).

Upgrade

pip install --upgrade potato-annotation==2.7.0