Skip to content

CiteVahti 0.46.0

Latest

Choose a tag to compare

@heidihelena heidihelena released this 03 Aug 08:11
ed1b8d1

Added

  • ai_connection.think — an operator-visible chain-of-thought switch for a local
    Ollama model.
    The OpenAI-compatible /v1 shape carries no such switch, so setting
    it routes the rating call over Ollama's native /api/chat (same host, same ceiling
    via options.num_predict); done_reason: length still reads as a truncation and a
    shapeless answer is still a transport failure, never a rating. This is a latency
    control, not a correctness fix
    , and it is a measured trade (2026-07-27, qwen3:14b,
    44-pair prescreen corpus): think: false never truncates and ran ~4.4x faster — and
    lost agreement with the anchor exactly on items whose anchor is unclear (40/44 vs
    35/44; exact McNemar on the 11 discordant pairs p = 0.227, not resolvable at
    n = 44
    ). The default (null) therefore leaves a thinking model thinking; api
    mode rejects the option at connection-resolve time rather than silently ignoring it.
    Locked by tests/test_think_control.py.

Fixed

  • A fresh pip install "citevahti[mcp]" no longer gets an MCP server that cannot
    start.
    The extra pinned mcp>=1.27.2 with no upper bound; mcp 2.0.0 removed
    mcp.server.fastmcp, so a clean install picked it up and citevahti demo / run
    reported "the 'mcp' package is required to serve" — telling the user to install the
    extra they had just installed. The pin is now mcp>=1.27.2,<2, and the serve-time
    error distinguishes a missing package (install the extra) from an incompatible one
    (pin the version), so a future API break says the truth. Found by running the
    fresh-user path in a clean venv. Locked by two contract tests in
    tests/test_mcp_server_transport.py.
  • Product copy says check, not verify. House trust doctrine: CiteVahti records
    claim↔source support, never truth, so public copy says check / test / assess. The
    README's FullVahti section claimed it "writes CiteVahti's verified results back as
    tags"; the same word had spread to the site copy, llms.txt, ROADMAP, INTEGRATION,
    QUICKSTART, a design note, and skills. The site and OG-image generator also still
    labelled the [oo] chip "verified" — that state was renamed to accepted in 0.16.0
    and is frozen in schemas/report.py, so the public site contradicted the shipped
    vocabulary. Left alone on purpose: the hash-chain internals (verify-audit,
    verifyChain/verifyAudit — they check the log, not the evidence), CLI/tool
    identifiers, and doctrine-affirming negations ("not a guarantee").
  • The published pair counts count judged pairs, not rating files. A (claim,
    candidate) pair can carry several claim-support records — one per panel rater by
    design, plus duplicates from ledgers written before opening became idempotent (below).
    Two published surfaces counted the files instead of the pairs they name: the
    evidence-basis sentence read "Of 118 rated claim-candidate pair(s)" on a real ledger
    holding 61 pairs of which 30 were rated, and the PRISMA flow's assessed box reported
    118 against 61 staged — more pairs assessed than staged, an impossible funnel in a
    figure meant for publication, with 88 of those records holding no rating at all. Both
    now collapse to pairs through one shared selector and count only pairs carrying a
    committed human value, matching their own labels. Locked by
    tests/test_methods_counts_pairs.py.
  • A stale candidate record on re-import is reported, and repaired only under audit.
    Dedupe matches on identifier, so re-importing a corrected record found the candidate
    already on file, reported already_in_prior_intake, and stopped — the stale metadata
    stood however many times it was re-imported (a real ledger carried Platt 1964's DOI
    under a different paper's title, with no way out short of txn-undo). Two halves,
    deliberately separate: linking now reports the divergence per field and changes
    nothing — a candidate's title is what a rater read, so refreshing it silently would
    rewrite the record of what was judged — and a new candidate-refresh applies the
    correction explicitly as an audited candidate.correct event carrying every field's
    old and new value. Descriptive fields only; how the paper entered consideration is not
    rewritable by a later import. Locked by tests/test_candidate_metadata_repair.py.
  • Linking candidates hands back the candidates, not a count. claim-link-candidates --json returned {"linked": <int>} while candidate-list --json returned the
    candidate objects — the same concept shaped two ways across two commands, so a loader
    reading .get("candidates") from the link result crashed. The link report now carries
    the candidates themselves, keyed and shaped as in candidate-list, and reports matches
    as well as new links so a re-run describes the same set instead of collapsing to
    linked: 0. Additive: the existing count fields are unchanged. The agent's
    link_candidates gains the same list in compact identity-only form, since its next
    step (start_support_rating) needs a candidate_id. Locked by
    tests/test_link_candidates_returns_objects.py.
  • The de-identified corpus export keeps the rated record, not a blank duplicate. On
    a ledger where pairs carried several claim-support records, _support_rating_for
    picked whichever record sorted last in listing order — on a real ledger it disagreed
    with the shared selector on 24 of 61 pairs and chose a blank started record over the
    rated one on 13, exporting human_support_rating: null where a human had rated. Worse
    than exporting nothing: a dropped rating reads downstream as "no human rated this
    pair", so the corpus understated its own human coverage. The export now uses
    select_support_rating, the one deterministic picker the panel, claim report, and
    agent provenance already share. Locked by tests/test_validation_warehouse.py.
  • Opening a rating twice no longer forks the pair in two. claim-support-start
    minted a fresh rating on every call, so an agent loader that retried a step opened a
    second record for the same pair — a real corpus load left 118 support records for 61
    pairs, and every duplicate was a second, unrated record of a judgement made once.
    Opening is now idempotent: a pair with an open human-unrated rating gets that same
    rating back, and the repeat writes nothing — no record, no audit entry. The panel's
    guarantees hold: a rating holding a human value is never handed out again (reviewer 2
    still gets a record of their own; N is never deflated), and force_new (CLI --new)
    still opens an extra slot for concurrent raters. Blinding: the returned rating may
    already carry an AI value, so claim-support-start --json emits identity fields only.
    Locked by tests/test_support_start_idempotent.py and tests/test_rating_selection.py.
  • import-results now reads the abstract, and says when a record arrived without one.
    The RIS parser handled TY/TI/AU/JO/DO/PY/AN/ER and had no branch for AB (or N2, the
    other common abstract tag); the CSV aliases had no abstract column; BibTeX read no
    abstract field. Everything downstream was ready for it — IntakeHit.abstract,
    ClaimPaperCandidate.abstract, and IntakeService._build_hits passing it through — so
    the field was dropped at the one place it entered, and the command reported status: ok / hits staged: 41 either way. Measured 2026-07-27 on a real 41-record export
    (intelligence-decomposition/sources.ris): 26 records carried an AB abstract and 0
    reached the ledger.
    That is not a cosmetic loss. claims/ai.py::_build_prompt builds
    the blinded rating prompt from the paper's title and abstract and falls back to "(no
    abstract available)" rather than refusing, so on a manual-intake project the AI rater was
    judging support from titles alone, the human panel was shown no abstract either, and the
    resulting ratings — the ones the agreement report and κ are computed from — looked exactly
    like ratings made on the full text of an abstract. Now all three parsers read it (RIS AB
    and N2, including the untagged continuation lines a wrapped abstract spills onto, and
    without doubling an abstract an exporter emits under both tags; CSV abstract /
    Zotero's Abstract Note / EndNote's summary; BibTeX abstract = {...}), and the same
    file now stages 26 of 41 with an abstract. Because coverage is a property of the source
    that the reviewer needs before rating and not an error, the import always reports it: a
    batch where any staged record lacks an abstract carries the warning "26 of 41 staged
    records carry an abstract; 15 have a title only and will be rated from the title alone",
    which import-results prints. Silence that reads as a complete import was the actual
    defect; the parser gap was how it happened. Locked by tests/test_import_results.py.
  • A claim that cites several sources is no longer decided by one accept. Support is a
    property of a (claim, source) pair, but the report read the first accepting decision as
    the whole claim's verdict: the claim turned green, dropped out of the pending queue and
    out of triage, and the papers cited alongside it were left with no human support rating
    and no decision at all. Measured in a real session ledger (2026-07-28,
    intelligence-decomposition): 30 claims, 61 linked candidates, 30 decisions — 31
    candidates never judged
    , while the reviewer's own account was "I accepted all the
    claims, few had several citations that got accepted at the same time". They were not
    accepted together; they were skipped, and the human's blind per-pair rating — the anchor
    everything else is compared against (ADR-0007) — was quietly never collected for half the
    corpus. Nothing invented a value on their behalf (the untouched pairs hold null, and
    DecisionService._derive still refuses to synthesise a support status), but silence that
    reads as completion is the same loss. Now a claim reaches accepted only once every
    linked candidate carries a trusted decision; until then it stays needs_support, which is
    literally true. The row carries decided_count, so the panel's cited-source picker marks
    each source judged (✓) or not (○) and names how many are left, the write/done steps say
    the decision covers this source and offer the next unjudged one as the primary action
    (↵), triage says "2 of this claim's 3 cited sources have no judgement yet" instead of "no
    accepted supporting citation yet", and claim-report prints n/N judged. This also
    reconciles the panel with the CLI, which was already per-pair
    (claim-support-commit-human, decide) — the two disagreed only about what finishing a
    claim means. Switching sources in the panel now also clears the in-session write/undo
    state, so u can no longer undo the previous source's Zotero write. Locked by
    tests/test_multi_candidate_claim_decision.py and the panel navigation tests.
  • The local reply-token ceiling is now sized from the whole corpus, and a truncation says
    which ceiling it hit.
    PR #298 set the local budget to 2048 from the 12 items that had
    truncated at 300 — a sample of the items that had already proved slow, so the tail was
    measured from the wrong end. Re-measured 2026-07-27 by instrumenting the provider's own
    usage.completion_tokens over all 44 prescreen pairs with the ceiling set deliberately
    non-binding (8192), qwen3:14b: median 269, p90 393, one answered reply at 2396 — above
    the shipped ceiling — and one item, C09, that never stopped at all: it spent all 8192
    tokens over 489 s and returned zero characters, exactly as it had spent all 2048 before.
    So the local default rises to 4096, sized to clear the largest reply that actually
    reached an answer, and explicitly not sized as a fix for a model that will not stop —
    no ceiling is. Headroom is not free either: it is also the time a stuck item burns before
    failing, and at the ~17 tok/s measured here the default 60 s request_timeout_s runs out
    near 1000 tokens, so a bigger ceiling does nothing unless the operator also allows the
    time. ai_connection.max_reply_tokens still overrides. A truncated_reply now records
    the budget in force and what the reply spent, so the operator is not left hunting for the
    setting — worded to state that a cut-off reply spent exactly the ceiling and therefore
    shows no shortfall, because two numbers that look like a measured gap get read as one.
    Truncation is still deliberately not retried: the ceiling is a setting, so the same
    call is cut off again. Locked by tests/test_ai_reply_truncation.py.
  • A transient AI-call failure is retried before it costs a rating — and a judgement is
    never re-asked.
    Following #302, which stopped a failed call being recorded as the model
    abstaining, a no-answer that is merely flaky is now retried (up to
    ai_connection.retry_attempts, default 3, identical prompt each time) instead of
    discarding a recoverable rating: provider_error, unparseable_reply, and a transient
    raised error (timeout / connection / 429 / 5xx), which previously escaped the rater
    entirely so nothing could recover it. A deterministic HTTP 4xx — bad key, unknown
    model, wrong path — is not retried: it fails identically every time, and in api mode
    retrying would triple the billed requests for every item in a batch. If any attempt
    classified a failure, that failure is recorded even when a later attempt raises: a typed
    failure in the ledger is better provenance than an exception that records nothing.
    The other half of the policy matters more: a rating, an abstention, an off-scale answer
    and a reply cut off at the token ceiling all return on the first attempt.
    Re-asking a
    judgement until it changes would select on the outcome and could manufacture a rating the
    model never gave; re-asking a token ceiling just reproduces a misconfiguration. A failure
    that survives its retries says so in its recorded reason, separating one unlucky call from
    a consistently broken adapter. Terminal behaviour is unchanged: if every attempt raises,
    the exception still propagates rather than being quietly written into the ledger.
    Honest sizing — this recovered nothing on the corpus that motivated it. Of qwen3:14b's
    15 lost ratings on the 2026-07-26 prescreen corpus, 13 were the 300-token ceiling cutting
    a reasoning model off mid-thought (reported as "unparseable AI reply" before truncation
    had its own label); the 2048-token headroom in #298 already fixed those. The same 44 pairs
    were re-run 2026-07-27 with and without this retry, at temperature 0 (every item that
    produced a value produced the same value in both runs): 42/44 rated and 0 abstentions
    in both runs, 2 failures in both
    . In the with-retry run both failures were
    truncated_reply — the token ceiling, which retry deliberately does not touch — so
    retry fired zero times and recovered nothing. (The baseline's two were one
    truncated_reply and one raised read-timeout; that timeout was the same item, which
    under retry reached the token ceiling instead, so it was never a recoverable transient.)
    This is therefore insurance against a failure class that corpus did not exhibit, not a
    measured recovery; the remaining loss there is the reply-token ceiling, which needs its
    own fix. The insurance is cheap: a
    successful rating still costs exactly one call and mean latency was unchanged
    (24.6s → 24.0s). Cost is stated where an operator meets it — a timeout has already spent
    its full budget before it can be retried, so the worst case for one item is
    retry_attempts × request_timeout_s + (retry_attempts − 1) × retry_backoff_s. Both are
    operator-settable on ai_connection; retry_attempts: 1 disables retrying. The
    never-re-ask-a-judgement guarantee stands on its own regardless. Locked by
    tests/test_ai_retry.py.
  • The agreement report and methods statement now see the claim-support ledger.
    AgreementReportService loaded store.list_ratings() only — the study-quality (GRADE /
    RoB) records — so a ledger of compared claim–candidate support pairs, which is CiteVahti's
    core asset and the path the prescreen work runs, produced an all-zero agreement report
    and an empty model scoreboard
    . Built on the same service, the auto-filled methods
    paragraph then said "Of 0 comparable human–AI pairs, 0 were concordant and 0 discordant"
    and carried a Before you submit note reading "No comparable human–AI pairs yet" — in a
    document whose own evidence-basis line said "Of 6 rated claim–candidate pair(s)" and whose
    PRISMA table counted the same 6. One generated document, two contradictory answers, and
    the paragraph's every sentence is about claim support. Claim-support ratings are now
    loaded and reported under the reserved scheme id claim_support, so counts, raw
    agreement, Cohen's κ and the per-model complementary-catch scoreboard (which feeds the
    model advisor) describe the work that was actually done. Two guards keep that honest:
    the two instruments are never pooled into one κ — agreement on claim support and
    agreement on study quality measure different things on different units, and the existing
    mixed-scheme refusal now covers this — and ordinal weighted κ is refused for
    claim_support
    (error: no_ordinal_scale) because the support vocabulary has no
    defined ordinal ranking (overstated and unclear are not points on a strength
    continuum); ordering it to satisfy the statistic would invent a scale the instrument does
    not have, so Cohen's κ (nominal) is reported instead. The methods statement is scoped to
    claim support to match its prose, and a ledger that also holds study-quality ratings is
    told so with a pointer to agreement_report — neither pooled nor silently dropped.
    Locked by tests/test_agreement_sees_claim_support.py.
  • A broken AI adapter no longer reads as the model exercising judgement. The raters
    collapsed transport and parse failures into epistemic abstention: a reply the endpoint
    never delivered, a reply with no readable JSON, and a value outside the controlled
    vocabulary all recorded abstained = true, landed as ai_abstained in the ledger, and
    reached a journal reader through the auto-filled methods paragraph as "AI abstentions (n)
    were excluded from the agreement denominator" — a sentence a reader takes to mean the model
    read the item and declined. Measured 2026-07-26 on the 44-pair prescreen corpus: qwen3:14b
    recorded 15 abstentions of which zero were semantic (13 unparseable replies, 2 timeouts);
    on the 5 items where the adapter worked it scored 5/5. The events are now separate:
    ai_rating.failure carries a typed kind (provider_error / truncated_reply /
    unparseable_reply / out_of_vocab_value), abstained is reserved for the model reading
    the item and declining, the two are mutually exclusive (validators refuse a record claiming
    both), and the comparison status ai_failed sits beside ai_abstained. An off-vocabulary
    answer is its own recorded event rather than a silent abstention — it is a prompt-compliance
    defect, and it is still never mapped onto a nearby in-vocabulary value. Downstream, both are
    excluded from the agreement denominator exactly as before, but they are counted and
    described apart
    : the agreement report breaks failures out by kind, the model scoreboard
    tallies failed separately from abstained (a model whose calls keep failing no longer
    accrues a record that reads as caution), the methods statement names abstentions as "pairs
    the AI rater read and declined to rate" and discloses failed calls in their own sentence —
    omitted entirely when there are none — plus a Before you submit note, and the panel's
    Reveal & decide card names each failure kind with the fix it actually calls for. Records
    written before failure existed are not reclassified retroactively; the panel still
    reads the legacy configuration: reason prefix so an old truncation does not now read as a
    clean abstention. Locked by tests/test_ai_failure_vs_abstention.py. A retry policy for the
    transient kinds (the 2026-07-26 evidence shows most recover on a second attempt) is a
    separate follow-up.
  • An AI that ran and declined is no longer re-offered as an AI that was never asked.
    The rest of the card had learned to say "abstained — no rating given", but the heading and
    the button under it had not: they still read "Decide now, or get an AI second opinion" and
    "✦ Get AI second opinion", so the one actionable control on the card described a run that
    had already happened as one that had not. Both now name the prior attempt — an abstention
    offers "✦ Ask the AI again" and says the same model asked the same question will likely land
    the same way (switch model, or supply full text); a cut-off reply offers "✦ Get the second
    opinion again", pointing at the setting to change first. The run stays reachable in both
    cases; only the claim that it never happened is gone. Behind it, workflow.reveal_ready
    now takes has_ai_value rather than has_ai_rating: an abstention is a record with no
    value, so there is nothing to reveal. That is a separate question from whether the
    abstention is shown — it still is, through ai_abstained — and the one blinding rule in
    rating/blinding.py is untouched. Locked by tests/test_workflow.py,
    tests/test_panel_wizard.py, and frontend-tests/units.test.mjs.
  • The review panel now shows when the AI was cut off instead of when it declined to judge.
    A truncated reply was recorded as a configuration problem, but nothing rendered it: the
    Reveal & decide card showed a bare "not recorded yet", which reads as "no second opinion
    has been run" — the exact misreading the recorded reason exists to prevent. The card now
    tells three states apart: no AI run yet, a genuine abstention (neutral, "abstained — no
    rating given"), and a cut-off reply, which gets a marked AI column plus a separated warning
    row naming the fix — pick a model that answers directly, or raise
    ai_connection.max_reply_tokens. The stepper stops ticking "AI second opinion ✓" for a run
    that produced no value. The panel view exposes only derived flags (ai_abstained,
    ai_config_issue) behind the same reveal predicate as the AI value itself, so this is not
    a second blinding surface and the AI's own reasoning text is never shipped to the page.
    Locked by tests/test_panel_api.py + frontend-tests/units.test.mjs.
  • A reasoning ("thinking") local model no longer quietly drops a quarter of its ratings.
    The AI raters capped every reply at 300 tokens. A model that reasons before answering
    (the qwen3 family) spends reply tokens on its chain of thought, so it was cut off before
    it answered; the rater then abstained with "unparseable AI reply" — honest, but
    indistinguishable from a genuine "cannot judge", so an operator saw abstentions where the
    real event was a misconfiguration. Measured on a 44-pair corpus (2026-07-26): qwen3:14b
    abstained on 12/44 (27%) through the product path versus 0/44 for two non-reasoning
    models on the identical items. Two changes: local mode now sends a 2048-token reply
    budget (the 12 affected items needed 302–596; re-running the corpus through the fixed path
    gave 0/44 abstentions, 44/44 in vocabulary, and no change to any previously-rated value),
    overridable via ai_connection.max_reply_tokens; and a reply the provider reports as
    cut off (finish_reason: length / stop_reason: max_tokens) is now recorded as a
    configuration problem, not a rating, with is_truncation_reason() as the one predicate
    a surface uses to tell the two apart. (The reason lands in the record's domain_reasoning;
    the review panel now reads that predicate — see the entry above.) No rating was ever
    corrupted and blinding was never affected: a
    truncated reply still abstains and still never invents a value. The advisory chat turn was
    capped the same way and now gets 1024 tokens plus a note when a reply is cut off. Locked by
    tests/test_ai_reply_truncation.py.
    Known trade-off: letting a reasoning model finish costs time — mean 55.4s per item on the
    measured corpus (max 142.6s) versus 18.5s when it was being cut off. request_timeout_s
    still defaults to 60s, so on a thinking model some items will now raise a visible timeout
    instead of quietly abstaining. That is the better failure (loud, not silent) and the timeout
    is configurable, but a thinking model is not yet comfortable at the default; sending Ollama's
    native think: false is the follow-up that makes it so.
  • Closing the window no longer takes the app away (pilot finding: "Apple takes the
    app away always when I close it"
    ). CiteVahti.app is a menu-bar app: the window's
    close button now hides the window — sidecars keep running, and "Open Review Panel"
    brings it back instantly with the researcher's place intact (no reload unless the
    window had stopped showing the panel). Real quit (menu Quit / Cmd+Q) still stops the
    agent server, then the engine; during a quit the close is allowed through so
    termination is never blocked. make_close_handler + open_panel show/reload logic
    locked by tests/test_desktop_app.py.
  • The desktop webview can no longer serve stale project state (pilot finding:
    "cache makes it stale"). JSON API and HTML responses carried no cache headers, so
    WKWebView heuristically cached them and the panel showed yesterday's manuscripts.
    Every response now says Cache-Control: no-store (static assets already sent
    no-cache/no-store). Locked by tests/test_panel_api.py::test_every_response_kind_forbids_caching.

Added

  • claims-import — bulk-load a corpus from one JSONL file. Loading a real corpus
    (30 claims / 61 candidate pairs) took ~90 lines of scripting across import-results,
    claim-add, claim-link-candidates and claim-support-start, plumbing ids between
    every step — and agents are how most corpora will be loaded, so every id hop was
    somewhere to drop one. citevahti claims-import --jsonl corpus.jsonl [--dry-run]
    stages each row's sources as one intake batch, creates the claim, links its sources as
    candidates, and opens a support-rating slot per pair; every id comes back in the
    report. Resumable, not atomic — CiteVahti has no general ledger transaction, so this
    does not claim all-or-nothing; every step is idempotent instead, and re-running the
    same file converges (checked on a real ledger: two full runs leave the same claims and
    ratings, and the audit chain checks out). Locked by tests/test_claims_import.py.
  • Recent manuscripts — reopen the paper you were on, in one click. Pilot feedback
    ("impossible to change manuscript") + docs/design/working-file-selection.md idea 3:
    the researcher's unit of work is a manuscript, not a folder+ledger. Opening a
    manuscript now records it on a cross-root recents list (rootcfg.remember_recent_manuscript,
    capped at 8, deduped, temp-leak-guarded, dropped when its project's ledger disappears);
    /api/context surfaces the list; the Manuscripts surface shows Recent manuscripts
    above "Reviews on this Mac"; a new handler route POST /api/recents/open switches the
    ledger and marks that manuscript active in one step, so the reload lands directly in
    the paper. remember_root now preserves the state file's other keys instead of
    rewriting it. Locked by tests/test_rootcfg.py (recents) + tests/test_panel_api.py
    (context + record-on-open); driven end-to-end against a live panel (CSRF handshake,
    two projects, cross-project reopen).
  • getting_started — low-friction onboarding on the AI surface (agent + engine tool,
    read-only) — the third better-science improvement. CiteVahti's state-aware "where am I,
    what's my single next step" guide (workflow.project_status) already powered the panel's
    one-banner-one-button, but a student on the primary distribution path — connecting
    CiteVahti to their AI (Claude Desktop / Code) — couldn't reach it: triage is a
    risk-first list that says nothing on an empty or uninitialized ledger, exactly where a
    newcomer starts. getting_started surfaces the guide as the entry point: it speaks to
    every state and always names one concrete next action — create the ledger → paste a
    paragraph → review the flagged claims → export the report
    — grounded in the real ledger
    state, plus any soft blockers (e.g. Zotero write not set up) that never block progress.
    Read-only, degradation-safe offline; on the safe agent surface with readOnlyHint. Locked
    by tests/test_getting_started.py and the read-only/annotation guards.
  • model_advisor — pick a good AI second opinion (agent + engine tool, read-only) —
    the second better-science improvement. A student (or their AI) can now ask which
    identifiable model should I trust as a second rater?
    and get an answer from this
    project's own record of validated divergences
    (ADR-0009 §3b): it ranks models by
    complementary value — catches (divergences the human adopted) over resolved
    divergences — never by agreement, so a model that only echoes the human ranks
    nowhere. It applies an evidence floor (stays silent on any model without enough
    resolved divergences to judge) and, given a named model that rates low, suggests a
    better-evidenced alternative — the maintainer's "if a model has a low rating, suggest
    another."
    Writes nothing (no exports/, no audit entry); it is a derived view over
    existing rating records. Registered on the safe agent surface with readOnlyHint; locked
    by tests/test_model_advisor.py and the read-only/annotation guards.
  • Biomedical synonym normalization in claim matching (text.py) — the first
    better-science improvement: the lexical layer was synonymy-blind, so "heart attack"
    didn't match "myocardial infarction" and a genuinely-supporting source was returned as
    no-support — risking a student dropping a good citation. A small, curated,
    high-precision map folds unambiguous biomedical equivalents (heart attack ≈ myocardial
    infarction, hypertension ≈ high blood pressure, type-2 diabetes ≈ T2DM, stroke ≈ CVA,
    COPD, hand hygiene ≈ handwashing, physical activity ≈ exercise) to a canonical token in
    the matching path only (content_tokens stays raw for display). Measured: support
    recall 0.882 → 0.943, precision held at 1.000
    . Deliberately narrow — the long tail of
    paraphrase (e.g. meditation ≈ mindfulness) stays the AI-model layer's job (ADR-0009).
    Locked by tests/test_synonyms.py; docs/EVALUATION.md regenerated.
  • .zenodo.json — Zenodo archive metadata (title, Apache-2.0, creators, keywords) so
    the next GitHub release archives with correct metadata instead of GitHub's autoguess.
    The DOI + README badge follow at that release: Zenodo only archives releases published
    after the integration is enabled, and none has been cut since, so there is no DOI to
    wire yet.
  • Published evaluation pagedocs/EVALUATION.md, generated by
    validation/eval_report.py from the frozen lexical-layer baseline. It assembles the
    numbers CiteVahti actually has (the automatic claim-lexicon detectors) and describes the
    mechanisms whose numbers accrue from use (the per-model complementary-catch scoreboard)
    and later from the pooled corpus — and states plainly what is not yet measured (no
    whole-system accuracy benchmark; the human ground-truth ledger is unfilled). Deterministic
    and golden-file-tested (tests/test_eval_report.py) so the published numbers can never
    silently drift from the measured ones. This is the "we will publish our evaluation
    results" surface the launch copy promised — honest by construction.
  • Test-coverage visibility in CI. The offline suite now runs under pytest-cov
    (--cov=citevahti, term-missing report) with a conservative 75% floor — a ratchet
    against regression, not a strict gate (current coverage is ~79%). This surfaces which
    paths are untested, notably tools.py (the large agent-surface façade, ~66%) and the
    Zotero read layer — exactly the runtime-heavy code where "tests green" was least
    reassuring. Coverage config lives in pyproject.toml; the panel's static web assets are
    excluded. No behaviour change.
  • Certainty / overclaim flag (text.py, claimcheck service) — the last engine cue:
    the claim asserts a relation plainly while the source supports it only weakly or
    correlationally ("associated with", "modest", "nonsignificant") — the "overstated"
    support value. A high-precision cue set (correlational + weak-effect terms only —
    not bare "may/could", which attach to sub-clauses and over-fire) raises an inspectable
    advisory warning (certainty_cue); it does not change the support status, and it
    stays silent when the claim is itself hedged. Measured: precision 0.833 / recall
    1.000 — the one false positive is a deliberately-kept hard control (a hedge word on a
    covariate, not the relation), the honest limit of a lexical approach. It's the
    lowest-precision flag, named as such; the AI/human layer adjudicates. Locked by
    tests/test_certainty_overclaim.py.
  • Section-aware, multi-sentence passage selection (retrieval) — the passage the human
    is shown to rate. Selection used to score single sentences, so support spanning 2–3
    sentences ("X was lower. This reached significance.") was under-selected. It now scores
    sentence windows (up to 3) and keeps the tightest high-coverage, non-overlapping
    passage per region, so multi-sentence support surfaces as one quote without redundant
    variants. It also detects section headers and labels each passage
    (RetrievedPassage.section), preferring Results/Conclusions on a coverage tie — a
    tiebreak that never overrides the coverage score. Deterministic, no new dependency; the
    engine already reads Zotero's indexed full text. Locked by tests/test_passage_selection.py.
    (docs/KNOWN_LIMITATIONS.md updated: the old "shows the abstract" wording was stale.)
  • Per-model complementary-catch scoreboard in agreement_report — the local precursor
    to ADR-0009's Atlas model scoreboard, and the first piece of the model-rating
    differentiator. For each identifiable model it tallies catches (the model diverged
    from the human and the human's adjudicated final matched the AI — a validated divergence,
    the AI right where the human's first take wasn't), overruled, pending,
    concordances, and a catch-rate. Derived entirely from existing rating records — no new
    write path, no record-schema change
    , and nothing written when persist=False (the
    read-views-don't-mutate invariant holds, tests/test_readonly_tools_dont_mutate.py).
    Agreement scores nothing here, by design (the cheese-hole principle: a model that only
    agrees adds no defence). Surfaced in the report JSON + markdown; locked by
    tests/test_model_scoreboard.py.
  • Population / PICO mismatch flag (text.py, claimcheck service) — the highest-value
    citation error the engine didn't catch: a source that supports the claimed relation but
    in a different population ("works in adults" cited for a paediatric claim; a mouse study
    cited for a human claim). A deterministic cue set across three axes (age, sex, species)
    raises an inspectable, advisory warning (population_cue, e.g. "children ≠ adults")
    on an otherwise-supporting citation — it does not change the support status; the
    human/AI layer adjudicates (ADR-0009 "floor flags, AI confirms"). Deliberately
    conservative: it fires only when both sides name a different population, and stays
    silent when a population is implicit or the two sides use the same pole ("patients",
    women≈female). A new population detector in the eval scores precision 1.000 / recall
    1.000
    with 0 false flags on the controls. Locked by tests/test_population_fit.py.
  • Conservative inflectional stemmer in lexical coverage (text.py) — the eval's next
    target. Coverage matching was inflection-blind ("antidepressants" ≠ "antidepressant",
    "increases" ≠ "increased"). A shallow, deterministic stemmer (iterated to a fixed point,
    with a _MIN_STEM floor so short words like "less"/"loss" are never mangled) now folds
    common inflections in the matching path only — content_tokens stays raw for display.
    Measured precision-safe by the eval: support recall 0.688 → 0.812, precision held at
    1.000
    ; the remaining gap is true synonymy (the AI-model layer's job). Locked by
    tests/test_text_stemming.py; baseline re-frozen.
  • ADR-0009 — evaluation & model-quality
    architecture (defence in depth).
    Fixes the cheese-hole principle: the human, the
    lexical detector, and the AI model(s) are independent layers, and safety comes from
    layers whose holes don't line up — so a model that merely agrees with the human adds no
    defence. Three separate evaluation objects: (1) an automatic claim-lexicon eval we
    run, (2) model rating by complementary catches, not agreement, (3) a pooled Atlas
    scoreboard + divergence maps
    later. Supersedes the earlier "human-gold release gate"
    framing.
  • A runnable claim-lexicon evaluationvalidation/claimcheck/eval_lexicon.py over
    curated, author-labelled (claim, passage, expected) cases (lexicon_cases.jsonl),
    scoring the real text.py and naming its holes per phenomenon. Frozen baseline
    (lexicon_baseline.json) + CI regression guard (tests/test_lexicon_eval.py, --check).
  • Direction-aware polarity guard (text.py) — the first fix the new eval drove. The
    eval found that antonym contradictions with no negation cue ("X increased mortality" vs
    a claim of "reduced") slipped through as support, dragging support precision to 0.714.
    polarity_conflict now also flips on opposite direction — two independent axes
    (magnitude, quality) XOR-combined with negation, so a double flip ("did not increase" vs
    "reduced") correctly cancels — with an inspectable direction_cue ("reduced ≠
    increased"). Result: support precision 0.714 → 1.000, contradiction recall 0.500 →
    0.889
    , still 0 negation leaks; 6 held-out antonym pairs confirm it generalizes.
    Locked by new units in tests/test_claimcheck_polarity.py. Paraphrase/synonymy stays the
    AI-model layer's job (ADR-0009).
  • citevahti-models skill — choose/compare the AI second-rater model, run a topic
    through several models, the 3-model guideline pre-check (Layer-0 screening — leads,
    not verdicts). Registered in .claude-plugin/plugin.json.

Changed

  • acceptance-thresholds.md repurposed from a human-gold release gate to the
    eval_lexicon.py regression policy (precision floored, recall published — the
    inverted-U; negated-contradiction leaks must stay 0; known holes reported, not gated).
    citevahti-eval rewritten around ADR-0009's three objects; the human blinded-rater
    ledger is retained as optional calibration, not the gate. BETA_TO_PRODUCTION.md
    kill criterion retied to lexical-layer regression. Launch copy: "evidence-tiered
    decision" → "reasoned decision, per claim" (tiers come from multiple assessors, not one
    run).

Added

  • The beta → production skill set — five maintainer-facing Claude Code skills in
    .claude-plugin/plugin.json, per the plan of record in
    docs/BETA_TO_PRODUCTION.md: citevahti-eval (the
    production gate — pre-registered ground-truth ledger, κ before metrics,
    precision/recall per release, kill criterion; wraps validation/claimcheck/),
    citevahti-release (eval gate + surface parity + DOI + rollback notes around the
    existing secure-release gates), citevahti-claims (audits every public artifact
    against the must-not-claim list and the house trust language before publication),
    citevahti-support (triage template, known-issues register, immediate escalation for
    data-loss / false-final-value reports), and citevahti-onboarding (per-channel
    quickstart generation from the shipped truth). All five pass the
    tests/test_plugin_skills.py guards, including the trust-language check.
  • Production launch copy, repo-checked v1
    docs/marketing/launch-copy-v1.md. All six draft
    [VERIFY]/mismatch flags are now resolved, with a resolution log recording each fix:
    tier labels rewritten to the shipped vocabulary (accepted/caution/review/rejected +
    untestable, no "verified"); PRISMA-trAIce/Cochrane "aligned" softened to
    "RAISE-style transparency reporting" per docs/METHODS.md; the "web app" clarified as
    the local loopback panel (ADR-0007); the quotation-error rate sourced to Jergas &
    Baethge (2015, PeerJ 3:e1364 — pooled total error 25.4%, 95% CI 19.5–32.4), figures taken from the paper's meta-analytic estimates and checked against the
    paper; and the eval-results promise moved to future tense pointing at the pre-registered
    protocol. Still gated by citevahti-claims before any public use.
  • Pre-registered claim-check acceptance thresholds
    validation/claimcheck/acceptance-thresholds.md,
    the release gate citevahti-eval enforces: a ground-truth-validity gate (κ ≥ 0.60,
    N ≥ 50, per-class minimums), hard precision floors (mismatch-detector precision ≥ 0.80 —
    the kill-criterion metric; support-detector precision ≥ 0.85; zero contradictions
    leaking into support), published-not-gated recall targets, and an append-only change
    log. Proposed v0 — floors confirmable before the first scored run, frozen after.
  • CITATION.cff — Citation File Format metadata (entity author Vahtian,
    Apache-2.0), scaffolding the Zenodo–GitHub DOI wiring. Intentionally carries no
    version field so it is not a seventh version-lockstep file; Zenodo derives the version
    from the release tag. The one remaining setup step (enabling the repo on Zenodo) is
    documented in the citevahti-release DOI gate.
  • Three downstream Claude Code skillscitevahti-screen (sweep a reference list for
    retractions + claim–source mismatches before a reviewer does), citevahti-review (a
    read-only peer-review/editor pass over someone else's manuscript, library-free via
    claim-verify), and citevahti-report (package a finished audit: methods paragraph,
    integrity summary, evidence appendix, audit-chain check). They join citevahti-dev and
    citevahti-writing in .claude-plugin/plugin.json. Each references only real commands
    (verified end-to-end against the demo ledger), states the honest claim vocabulary
    ([oo] accepted, not "verified"), and keeps the house trust-language + safety boundaries
    (check/assess never verify/prove; not a medical device; the human is the decider). Guarded
    by tests/test_plugin_skills.py (manifest↔disk parity, frontmatter-name match, and a
    trust-language check on the new skills).
  • Published privacy policy + reviewer examples (toward Anthropic's software directory).
    A full docs/PRIVACY.md (canonical home
    https://vahtian.com/citevahti/privacy) states plainly what CiteVahti handles: no
    telemetry, no account, everything local; the only egress is the literature services you
    search, your Zotero if connected, and the optional PyPI update check. Both .mcpb
    manifests now carry a privacy_policies link, and the README's privacy section points at
    the hosted policy. A new docs/DIRECTORY_EXAMPLES.md gives
    three runnable example prompts against the zero-setup citevahti demo ledger, so the
    workflow can be evaluated with no Zotero, no key, and no network.
  • Every MCP tool now advertises a title and safety hints. The 19 agent tools carry a
    human-readable title and MCP annotations (readOnlyHint / destructiveHint /
    idempotentHint / openWorldHint), so a Claude Desktop-style host can tell a read
    (search, report, preview) from a write and can confirm before the one external write —
    commit_write, the decision-gated Zotero write-back, marked destructive + open-world.
    The read-only labels are honest by test: any tool marked read-only is cross-checked
    against the ones proven to leave the ledger byte-identical. (These are advisory hints
    for clients; CiteVahti's real enforcement stays the constrained surface and the
    token-confirmed write path.) A hard requirement for Anthropic's software directory.
  • Opt-in launch-time update check. A new Settings checkbox — default off — makes
    the panel ask PyPI once, when it opens, whether a newer CiteVahti exists; a quiet ⬆
    badge appears in the header when one does (clicking it opens Settings with the update
    steps). Off, nothing changes: the documented no-launch-time-phone-home posture holds,
    and the manual Check for updates button keeps working as before. The choice is
    stored per-project (panel.json), the check sends nothing about you, and a failed
    check stays silent — a courtesy lookup must never disturb boot.

Fixed

  • Three silent panel dead-ends now say what's wrong and what works (each cost a real
    reviewer real time before being reported):
    • a manuscript whose file can't be found names the file and the folder the panel
      looked in ("move or copy it there, or open your document") instead of only a tiny
      "document not open" chip — while chat-saved §-style claims keep the gentler
      "open your document" guidance;
    • a real manuscript with zero claims explains that nothing has been extracted yet
      and names both paths that work (run_claim_tests in your chat client, or + Claim)
      instead of rendering inert prose;
    • pressing a review keyboard shortcut with no claims to act on shows a one-time
      hint instead of silently doing nothing ("my beloved keyboard shortcuts are all off").

Added

  • Every frozen artifact is now smoke-RUN before it ships. A new cross-platform
    smoke (desktop-extension/smoke_frozen_panel.py, stdlib-only) drives the frozen
    citevahti-mcp over MCP stdio through the real open_review_panel path — and the
    app's sidecars the way the shell runs them — then fetches /api/ping, index.html,
    app.js, styles.css and reconnect.js expecting 200s, on an ephemeral port with a
    throwaway root (it can never touch a live CiteVahti instance) and a hard watchdog.
    Wired into all four release-build jobs (win/linux/mac .mcpb + CiteVahti.app,
    after codesign and before packing/notarizing) and the local build-binary.sh /
    build-app.sh / sign-notarize.sh, so the scripts and the workflow can't drift
    apart on this again. "It froze" is not "it works": the 0.44.x series shipped three
    consecutive builds that only failed in a user's hands — this gate catches all three.