v0.3.0

Reading very long sources in full, a check that sb ask answers say what the pages say, prompt-injection hardening, machine-readable --json output for editor plugins, optional OCR with a guided model choice, a ChatGPT-plan provider (codex-cli), consolidated concept summaries (off by default), hourly backlog draining, a USD cost cap, and no AGPL dependency any more (PDFs are read with PDFium).

Security

• Prompt injection hardening, with a corpus and tests. A hostile source (a web page, a mail, a PDF, a clip, a transcript) can tell the model to write anything, and a small model often obeys, so the worker now checks what the model wrote instead of trusting the delimiters around the source. tests/fixtures/injection/ holds 15 inert hostile sources (instructions in the body, in fake system turns, in a fake </source>, in a clip's title, in an image alt text, in a mail subject, in a PDF line, in a transcript; exfiltration through an image or a link; links to missing or private pages; page names that escape their folder; fake contradictions; raw HTML; a mail that asks to be sent to the cloud) and tests/test_injection.py plays a model that obeys all of them. What it found and fixed: (1) the one-line summary of a note, a contradiction note and a PDF figure caption could still carry an image or HTML (the index page loaded the image when opened: the address could carry other pages' titles out); (2) a Markdown link or a bare address added by the model could carry the same data (clicked, not loaded); (3) [[links]] the model wrote to pages that do not exist, or, in a public note, to email pages; (4) a public source could link to, or put a contradiction callout on, an email page, and a mail could write its text into a public page through a contradiction callout, which a cloud model reads later; (5) a concept named Home, index, SCHEMA, log, lint or a date shadowed the real page in [[links]]; (6) a quote spanning lines could forge a section heading; (7) a clip's source: could carry line breaks or HTML into the note; (8) a source could close its own </source> block in the prompt, and a title could add lines to the system prompt of the long-source reader. Now all model text is plain prose (no HTML, no image syntax); an address survives only if the source itself contains it, a link only if it names an existing page that a public note may point at, a contradiction only lands on the same side of the email line, and the prompt builders make a source's look-alike tags and chat markers inert. A note says when it removed addresses. sb ask --save gets the same treatment, and its answer can no longer cite an email page when the model sends text out. Still not guaranteed: that a model resists an injection (it often does not: the worker only limits what the output can do), that an invented but plausible summary or contradiction is caught, and that the images and links a web page itself contains (kept as linked images, and in raw/) do not phone home when the note is opened.

Added

• sb info --json has sends_text_out, a boolean per task saying whether the text of a source leaves this machine for that model (a model that cannot be built counts as yes), so the Obsidian plugin does not have to guess from the provider name. Additive: contract 1 stays.
• A very long source is read in full. Before, a source of more than [run].max_chunks chunks (16 of 8,000 characters by default) was sampled: the first three chunks, the last two and a few evenly spaced ones, so a book or a 60,000-word document gave a note about a third of it. Now every chunk gets its notes and the notes of each group of five chunks are merged into a section note (level by level if needed) until at most max_chunks remain; the plan and the detailed summary are written from those, never from the raw text, so no prompt outgrows the model's context. The terms, quotes and relations of every chunk are carried over by code and still checked against the source. A source of at most max_chunks chunks is read exactly as before. sb run --json reports the new step as {"event":"step","name":"merging section","index":2,"total":8}, like the chunk step.
• [run].max_calls_per_source (default 60, at least 12): a hard cap on the model calls for one source, retries included. Calls are kept back for merging what was read and for the plan, the summary and the connections; when the cap stops the reading early the note starts with a warning callout saying how many parts and what share of the text were not read and where the unread part starts, and the run prints the same as a warning. Raise it for books; a source of 40 chunks takes about 55 calls.
• sb ask checks that the answer says what the pages say, not only that its citations are real. Every sentence with facts in it must find its numbers, its proper names and most of its key words (accents, case, plurals and endings ignored) in the pages the model was given or cited; no second model call. A sentence that fails gets a ⚠ after it and one footer line ("Not found in the cited pages: ..."), and an answer where most of the sentences fail is refused like an uncited one. A sentence that says the pages do not have the answer is not checked, and neither are pages written in another language than the answer. It is a vocabulary check: it catches an invented number, name or topic, not a wrong relation between words that are all in the pages. Measured on 197 sentences of real llama3.2 answers over a Spanish wiki of 15 notes: it flagged all 51 hand-labelled unsupported sentences and wrongly flagged none of the 119 supported sentences in answers that cite a page (5 of 146 over all answers, all of them in answers with no valid citation, which were refused anyway). sb ask --json gains an unsupported list ([{"sentence": ..., "missing": [...]}]; contract 1 is unchanged). [run].check_answers = false switches it off.
• A backlog drains hour by hour. When a scheduled run stopped only because it hit [run].max_sources_per_run and sources are still queued, the next hourly tick (sb run --if-due) runs the next batch instead of waiting for the next night. One batch per tick; never after an outage, an interruption or a budget stop, and never while another run is active.
• Unticking undoes a tick. Unticking a read source's box in the daily note of the day it was processed puts it back to processed (the read date is cleared) the next time sb index or sb run registers ticks. A source with no box there, or ticked anywhere in that note, is left as it is. Daily notes are regenerated from state, so a read source always shows ticked and an unticked box is always your doing.
• A model-server outage is visible. When the latest run stopped because the model could not be reached, the daily index says so first in "To review" (with how many sources are waiting), and sb doctor shows a WARN last run line with the fix (start Ollama or check the model, then sb run). A later run that reached the model clears both.
• [run].max_usd_per_run (a number, unset by default = no cap) stops a run once its estimated cost reaches that many USD, like max_tokens_per_run does for tokens: the rest stays queued, with no attempt counted, and the run is recorded as usd_budget (also the stopped_by of sb run --json's finished event, a new value of an existing field). The estimate is the tokens a model used times its price in [bench.prices] (USD per million tokens, the table sb bench uses). Only models that send text out count: local models are 0, and so are claude-cli/ and codex-cli/, which are flat-rate subscriptions. A model without a price counts as 0.
• The smoke test that every wheel passes before a release (scripts/smoke.sh) now reads a real PDF with the installed package, and a test fails if an AGPL package becomes a dependency again.
• --json on sb version, status, info, doctor, add, ask and run, for editor plugins and scripts (contract version 1, reported by sb version --json as contract). Each command prints one JSON object on stdout; sb run --json prints JSON Lines (started, source_started, step, source_done, source_failed, finished), flushed line by line, with the human progress lines dropped. A failure prints {"contract":1,"error":...,"code":...} and exits 1, with no traceback; exit codes are the same as without --json.
• Reading images is a choice you can make, see and change. sb ocr status shows whether it is on, the model, whether Ollama has it, its download size, this machine's memory and whether Ollama is new enough; sb ocr enable [--model NAME] [--pull] turns it on (and downloads the model only with --pull or a typed yes); sb ocr disable turns it off and keeps the model choice. [llm.ocr] enabled = false does the same from the config.
• sb init asks "Read images and scanned PDFs (jpg, png, ...)?" and, on yes, shows the vision models with their size and the one recommended for this machine's memory, lets you type another Ollama model, says whether it is installed and asks "Download it now?" (progress is shown; an interrupted download resumes). Flags for scripts: --ocr/--no-ocr, --ocr-model NAME, --pull-ocr-model. Nothing is downloaded without that flag or a yes.
• DeepSeek-OCR (deepseek-ocr:3b, 6.7 GB) is a known model for machines with 16 GB of memory or more (never the default: qwen3-vl:2b-instruct is the recommended one), asked with the instruction it needs (not measured above 8 GB: on an 8 GB Mac it loads at about 10 GB and takes minutes per image); any other Ollama vision model gets a general one. sb doctor warns when this machine has less memory than a known model needs, or Ollama is older than it requires, and fails with the exact ollama pull command when the model is missing.
• codex-cli/<model>: use your ChatGPT plan through OpenAI's official codex tool (codex login once), the twin of claude-cli/. The call runs read-only in an empty scratch folder with the tools and your Codex settings switched off, and the text of each source goes to OpenAI, so email is never shown to it. sb doctor checks that codex is installed and logged in. Checked once against the real tool (codex-cli 0.146.0, ChatGPT plan): it needs about 8.5k tokens of overhead per call, so it suits the nightly run rather than many small calls.
• Concept and entity pages get a consolidated summary. A ## Summary section (## Resumen in a Spanish vault) now sits above the per-source From sections, written by the synthesis model (else the summarize one) from those sections only. Anything it states that the sections do not (a number, a name, a quote), a link, a wrong language or a bad length fails the check; it is retried once, and a second failure leaves the page as it was and writes the proposal to wiki/review/. A page that email helped write is summarised only by [llm.private] (or a local model), and its summary is never shown to a model that sends text away. sb ask reads the summary first.
• sb consolidate [--all] [--only TITLE ...] [--dry-run] writes the summaries on demand. The nightly run and sb ingest can do it by themselves for a page that just reached three sources, or two more than at its last summary, up to the new [run].max_consolidations_per_run. It is 0 (off) by default: with llama3.2 (3B) the summaries read well but sometimes state things the sections do not say, so run sb consolidate and read them first, or set 5 together with a stronger [llm.synthesize] model.
• When a page is consolidated, pages that look like the same idea (same name ignoring case and accents, plural and singular, an acronym and its expansion, or very similar titles) get a merge suggestion in wiki/review/. Nothing is merged for you.
• sb lint finds the same plural/singular and acronym duplicates ("Red"/"Redes", "IA"/"Inteligencia artificial").

Changed

• PDFs are read with PDFium (pypdfium2, BSD-3-Clause or Apache-2.0) and esbi-cli depends on no AGPL package. pymupdf4llm and pymupdf (AGPL-3.0 or a paid Artifex licence) are gone, and with them onnxruntime, numpy, networkx and the layout model. The text is rebuilt from PDFium's characters: lines are reflowed into paragraphs (by the gaps between lines and by indents), a hyphen at the end of a line is dropped inside a word and kept in a compound (position-wise), running headers and page numbers are dropped, and headings are found by font size and weight and numbering (## 1 Introduction, ### 3.1 Method). The text has no emphasis markers (**, _x_, <sup>), so quotes copied from it match the source word for word more often. Tables are no longer turned into Markdown tables: their rows stay one per line (Base 65 27.3 3.3), numbers intact. Text printed over a picture or inside a chart is left out. The PDF part of the install goes from 214 MB to 8 MB, and a 15 to 50 page paper is read in about a second or less instead of 8 to 60 seconds, with about a third of the memory (measured on 10 papers of 8 to 51 pages). PDFium's wheels need macOS 13 or later (or Linux).
• Figures drawn as vector graphics (diagrams, flow charts) are now found when they have a Figure N caption, as the images always were; they are rendered at the same 130 to 150 dpi cap, at most 8 figures per PDF.

Fixed

• Clips of GitHub, LinkedIn, X/Twitter, Reddit and YouTube no longer feed the page chrome to the model. Navigation, "Sign in", cookie and consent lines, GitHub's repository sidebar (Languages, Releases, Contributors, issue Assignees/Labels), reaction and share bars, "People also viewed", related-post lists and footers are removed before the clip is read, so the note summarises the article instead of the menu. The rule is conservative: only a whole line that is known chrome goes (a bare word like "Share" only as a link on the same site), never a sentence that contains it, code blocks, plain bullets or YouTube transcript rows, and hosts other than those five are left alone. The note records stripped_lines: N in its frontmatter, the run says how many lines went, and the original clip in raw/inbox/ is untouched.