v0.3.0 — datasets engine, finance advisory, hybrid pre-retrieval

Added

• Dataset engine — a domain-neutral datasets/ capability: tabular data
  files the assistant queries through an opt-in query_dataset tool, kept
  separate from the curated wiki so numbers come from the data, never the model.
• Argentine finance advisory (finance_argentina) — a deterministic
  estimar_alternativas tool that ranks investment alternatives for a given
  amount and horizon with code-computed gains (never LLM-estimated), flags
  non-estimable instruments (equities), and carries a nominal-vs-real inflation
  disclaimer.
• finanzas-argentinas demo — a pre-ingested Spanish finance wiki with
  datasets/, a tuned wiki_config.toml, a demo guide, and its own live
  acceptance UAT (scripts/uat_finanzas.py).
• Ingest-time vocabulary subsystem — the assistant generates data/concept
  aliases while ingesting; a vocabulary linter + auto-repair keeps the alias map
  honest (collisions dropped, stale/covered/ambiguous surfaced); a coverage
  roster decides what the wiki actually covers; and a thin-page detector
  flags source chunks the wiki leaves uncovered.
• Hybrid pre-retrieval (opt-in per wiki) — code retrieves and injects wiki
  context before the model answers, gated on the coverage roster so an
  off-topic or uncovered question is refused deterministically instead of leaking
  from a tangential chunk; tiered curated-then-raw sources with answer-vs-source
  verification; and a live toggle in the read app.
• Pluggable citation/grounding guardrail and an opt-in JSONL chat trace
  (one row per turn) for offline diagnosis.
• A tabbed read app ships alongside the grid one (marimo/read_app_tabs.py).
  It is not the default and not yet documented: quickstart.py, both READMEs
  and the programmer manual still launch marimo/read_app.py, and the E2E suite
  still covers that one. Both carry the same chat, including the pre-retrieval
  toggle, so any change to the chat currently has to be made twice. Promotion —
  and the removal of the grid app — waits on a parity test proving nothing was
  lost in the move.
• Two walkthrough documents, generated from real runs. The ingestion
  walkthrough follows one small corpus through
  its whole lifecycle — first document, second, a no-op re-ingest, an edited
  source, a deletion — and the query walkthrough
  follows a spectrum of questions through both chat modes. Each is split so a
  reader with an ordinary wiki can stop half way, and each is paired with a
  capture script (scripts/capture_*_walkthrough.py) that regenerates its
  appendix, so the figures they quote come from running the pipeline rather
  than from memory.

Changed

• The ingest end-to-end test was rewritten for the current app
  (tests/e2e/test_ingest_app_v2.py, replacing test_ingest_app.py): it drives
  the wiki picker, ingest form, Activity Log, vocabulary lint lines, scan
  idempotency and cross-links through the real Marimo UI. The slower cases stay
  opt-in behind E2E_FULL=1 and E2E_DESTRUCTIVE=1.

Fixed

• Chunk breadcrumbs named the wrong section. header_breadcrumb — the
  heading path a search hit is traced back to, and what a citation names beyond
  a page number — was serialised when a chunk was closed, from the heading stack
  as it then stood. Because a heading is pushed before the size check, a chunk
  flushed at a heading boundary was labelled with the section that starts after
  it: in the shipped demo one chunk was named "Principales riesgos
  estructurales" while containing nothing of it but the heading line, and that
  section's actual text sat in the next chunk under a different name. A
  breadcrumb naming the wrong section is worse than none — it aims a citation at
  a passage the fragment does not contain.

  Two further cases came out of the same work: a chunk that merely ends on a
  heading no longer takes that section's name, and the heading outline now
  carries across page breaks, so a section running past the bottom of a PDF page
  keeps its name (and its document title) instead of restarting. The column had
  no tests; it has seven now.

• Pre-retrieval now actually retrieves. The FTS query reached SQLite FTS5 raw,
  so any natural question crashed the search and silently returned no hits — the
  gate then refused valid, covered questions. Queries are now sanitized; both
  tiers are gated on the coverage roster; a data/advisory question routes to the
  tools before any raw-doc fallback; and a generic advisory question (an amount +
  horizon, no named instrument) reaches the advisory tool instead of being
  refused.

• Citation detection recognizes the real format. A Referencia:/Fuente:
  line (what the prompt asks for and the app emits) and a source-document
  citation (.docx/.pdf) now count as grounded/cited in the chat trace and the
  eval graders, not only an inline (wiki/….md).

• Lint & repair labels advisory-only vocabulary findings clearly instead of
  reporting them as "Unknown check type".

What's Changed

• feat: dataset engine + Argentine finance advisory + finanzas-argentinas demo by @Clod in #12

Full Changelog: v0.2.3...v0.3.0

v0.2.3

Fixed

Demo wikis now cite every fact in the read app. Both bundled demos (fairy-tales, cuentos-de-hadas) shipped a wiki_config.toml system prompt with only a single soft citation line — much weaker than the tuned default the model-validation eval runs under. Because the read app uses each wiki's own prompt, a synthesis/comparison answer came back grounded but uncited, breaking the project's core "every fact carries a citation" promise.

Both demo prompts were brought up to the default's rigor — an explicit grounding mandate, a mandatory-citation section with format examples, the "synthesis and comparisons must cite each point/row" rule, and a worked cited-comparison example — in each demo's own language. Verified end-to-end: the exact comparison that failed now cites every point in both English and Spanish (real retrieval, no fabricated citations).

Full changelog: v0.2.2...v0.2.3

v0.2.2

A focused fix to the model-validation eval.

Fixed

• The off-topic check no longer false-fails strong models. The "refuses off-topic questions" check requires a real retrieval tool call, but the production system prompt both names the wiki's subject (through its worked example) and explicitly permits declining obvious trivia without searching — so a capable model like gpt-4o correctly refused "what is the capital of France?" with zero tool calls and was intermittently marked as failing.

  The off-topic check now runs under a domain-blind, strict-search prompt: with the subject hidden and the decline-without-searching shortcut removed, the model must retrieve before it may decline, so the tool-call requirement is meaningful and non-flaky. The two citation checks still run under the real production prompt (whose worked example is what makes citation reliable). Verified: gpt-4o now passes 3/3 across runs, with 1–2 tool calls on the off-topic refusal.

Full changelog: https://github.com/Clod/llmwiki-marimo/blob/v0.2.2/CHANGELOG.md

v0.2.1

Two fixes surfaced while dogfooding the v0.2.0 quick-start installer against OpenRouter.

Fixed

• Ingestion now cross-links generated pages. A final crosslink_wiki_pages pass injects a localized "See also" section into every concept/summary page after all documents are ingested, so a page written early can still link to a concept extracted from a later document. Previously inject_see_also was wired only into the chat "Save to wiki" path, so pipeline-generated concept pages never linked to one another. Deterministic and idempotent; runs from both scan_and_ingest and the ingest app. Both demo wikis were regenerated — concept cross-links went 0/11 → 13/13 (cuentos-de-hadas) and 7/14 → 14/14 (fairy-tales).
• OpenRouter openai/* models silently ran at the provider's default temperature. pydantic-ai mis-profiled the OpenRouter-namespaced OpenAI models (e.g. openai/gpt-4o) as reasoning models and dropped the pinned temperature=0, making grounding non-deterministic (the model-validation eval flapped between pass and fail on the same model). The chat agent now routes OpenRouter endpoints through pydantic-ai's dedicated OpenRouterProvider; other OpenAI-compatible endpoints (OpenAI, LM Studio, Ollama) are unchanged. Ingestion was unaffected.
• Quick-start docs now list git as a prerequisite (needed to clone the repo).

Full changelog: https://github.com/Clod/llmwiki-marimo/blob/v0.2.1/CHANGELOG.md

v0.2.0

One-command quick-start installer, an advisory model-validation eval, a Spanish demo wiki, and grounding fixes that make cite-or-refuse hold with local models.

Added

• One-command quick-start installer (quickstart.py) — stdlib-only, Python 3.12+ the only prerequisite. Builds an isolated venv from a lock-pinned requirements.txt, drops in a pre-ingested demo, runs a provider wizard, writes .env, validates the model, and launches the read app.
• Advisory model-validation step — scripts/eval_chat_model.py validates every configured model and verifies it actually called a retrieval tool (a citation fabricated from memory, with zero tool calls, fails).
• Spanish demo wiki (examples/cuentos-de-hadas/, language = "es") exercising the multilingual path end-to-end.
• Per-wiki multilingual content (en/es, extensible) — [wiki] language governs generated pages, headers, and chat answers, independent of source-document language.

Changed

• Provider-wizard default is now explicit in the prompt and both READMEs.
• Chat agent pins ModelSettings(temperature=0.0) for deterministic grounding.

Fixed

• Demo wikis ship a real cite-or-refuse system prompt (the old test prompt could hallucinate on a new user's first question).
• Guard None LLM message content; log previously-swallowed backend errors; render delete feedback and surface notebook errors.

Full changelog: https://github.com/Clod/llmwiki-marimo/blob/v0.2.0/CHANGELOG.md

v0.1.0 — first public release

First public release of llmwiki-marimo — a personal, local-first wiki that ingests your documents, builds a structured and interlinked knowledge base, and lets you read and chat with it, entirely on your machine.

▶ 1-minute demo

Highlights

• Local-first, no vector DB — sources, chunks, wiki pages, and the citation graph all live in one SQLite file; retrieval is FTS5 plus reading whole wiki pages.
• Citations as a contract — the chat assistant can only answer through tools that return wiki pages or source chunks, so every fact cites the page it came from.
• Human-in-the-loop growth — promote a good chat answer to a permanent wiki page with one form.
• Self-maintaining — six lint checks (orphans, stale summaries, missing cross-refs, missing concepts, contradictions, data gaps) with auto-repair of the safe ones.
• Provider-agnostic — any OpenAI-compatible endpoint, including a fully local Ollama setup; chat and ingestion models configurable independently.

Engineering

• Framework-agnostic core (base/) with marimo only at the UI edges
• 301 tests across three layers (fake-LLM unit, frozen golden-corpus regression, Playwright E2E)
• Apache-2.0; partly derived from Lucas Astorian's llmwiki (see NOTICE), inspired by Karpathy's LLM Wiki idea

Ships with a sample wiki, so the read/chat side works without ingesting anything. See the README to get started.