Skip to content

v0.16.0 — Restricted content

Choose a tag to compare

@phiweger phiweger released this 07 Sep 21:28
· 62 commits to main since this release

Some wikis hold material only part of an organisation may see. outmem now gates
that deterministically, at the store layer, before anything becomes prompt
text — no instruction, no system-prompt rule, and no tool description anywhere in
the enforcement path.

Content stays open by default. A wiki that never takes a view behaves exactly as
it did before: every enforcement point tests for a view first, so an unrestricted
store never reaches the label index at all.

The deployment assumed is the one this was built for — outmem runs server-side,
one company, one repository, employees with no clone rights. The repository is
dark to them, so filtering at the store layer is genuine access control rather
than a display convention. Full contract, threat boundary and rollout order in
docs/restricted-content.md.

store = WikiStore.open("/srv/wiki")             # operator, unfiltered
view  = store.as_viewer(mode={"hr"}, grants=g)  # per request
tools = wiki_read_tools(view)                   # the model never sees `store`

An item is visible when its labels are a subset of the session's mode, so
open-by-default falls out of the subset relation rather than being a special
case. A write requires labels equal to the mode, which confines whatever a
session read to the compartment it read from. A hidden page answers exactly as a
nonexistent one does — a distinguishable error is an existence oracle.

Added

  • restricted: in config.yaml — a declared label vocabulary plus optional
    path rules for pages and for ingested sources. Absent by default. Unlike every
    other config block this one is not forgiving: a malformed block refuses to open
    the wiki, and unparseable YAML in a file carrying a top-level restricted: key
    is fatal, because the forgiving load would otherwise open a restricted wiki with
    access control silently off. Withdrawing a label declaration hides the content
    carrying it rather than releasing it — one deleted line must not publish a corpus.

  • store.as_viewer(mode=…, grants=…) — the object a served request should
    hold. A view shares the store's registries, vector store, alias map, write lock
    and label index, so every view sees the same rows and the process opens one set
    of SQLite connections rather than one per request. It holds no reference back to
    the unrestricted store.

  • outmem ingest --restricted hr — label a source at the moment someone is
    holding the document. Every page later compiled from it inherits the label,
    which is the highest-leverage rule in the design. Orthogonal to --local: that
    tree is about redistribution rights, this is about secrecy.

  • outmem restrict <slug> --label hr [--cascade] and outmem sources restrict <path> --label hr — the operational verbs. Restricting is a graph
    operation, not a field edit: a page that visible pages already link to is
    refused until those links are resolved. Neither will remove a label that a path
    rule or a cited source immediately reapplies — reporting success for a
    declassification that did not happen is worse than refusing.

  • Compartment hints. When a search returns nothing, search_wiki tells a user
    who holds a label but is not scoped to it that the compartment exists and how
    much it holds. Counts only, per label, never for a label they do not hold. The
    count is a property of the corpus rather than of the question, and the hint
    takes no query at all: counting what this question matched would be a content
    oracle, since the model writes the question.

  • Six lint kinds — restricted-link-violation,
    restricted-provenance-violation, restricted-slug-mentioned,
    restricted-label-unknown, restricted-frontmatter-unparseable,
    restricted-chain-inconsistent — verifying the invariants against content
    already on disk, which is the case write-time enforcement cannot reach and the
    state a wiki is in the day it turns restrictions on.

  • Grants, RestrictionError, LabelError, RestrictedSettings as
    top-level exports. Every caller of a view needs the first two.

Breaking

  • restricted: is now a reserved frontmatter key. It previously round-tripped
    through extra as an opaque value. It is now parsed as a list of restriction
    labels, so a page using the key for something else changes shape (a bare string
    becomes a one-item list, an empty value is dropped) and a value that is not a
    label list makes the page unparseable — reported by outmem lint as
    frontmatter-invalid — and hidden from every view until it is fixed. A
    restricted: [a, b] value whose entries are not declared labels is the quiet
    one: the page parses and is then hidden from every view. Grep for
    restricted: under wiki/pages/ before upgrading if you used the key.

  • rename_page and restrict_page are operator-only — refused to a view, as
    import_vault and repair_pages already were. Both write files the caller did
    not name: rename rewrites inbound links across the corpus with the new slug as
    content, and --cascade picks its targets from the backlink graph. Neither was
    ever in a model-facing palette, so this affects only a downstream app that
    called them through a view; hold a bare store for administrative operations.

Changed

  • .sources.db schema 3 → 4, adding a restricted column. Migrated in place
    on open; NULL on existing rows reads as open, the same answer the wiki gave
    before the column existed. 0.15.0 still reads a v4 registry — but do not run
    mixed versions against one wiki, since a 0.15.0 write drops the column's value.

  • Tool calls are traced without their content on a wiki that declares labels.
    Bodies, titles and log topics are withheld (lengths survive), the item names
    with them — a slug can be as disclosing as a filename — and a refusal's message
    is reduced to its exception type, since IncompleteBodyError quotes the body it
    refused. A wiki that declares no labels keeps byte-identical traces, so eval
    recorders reading record.tool_call are unaffected.

  • build_consult_wiki accepts an open store, so a caller can pass a view. It
    previously always opened its own from a path, discarding a caller's mode and
    grants.

  • Caches derived from page content are keyed on the corpus, not on HEAD. BM25
    snapshots page bodies at construction, so a retriever built before a page
    changed kept answering from the text it had — a staleness bug that predates this
    work. HEAD alone is not enough: a local-tree ingest, and a re-ingest that only
    sets labels, both write the registry and commit nothing.

  • HEAD is read from .git/HEAD rather than by forking git rev-parse. A
    visibility check on a view was paying ~2 ms of subprocess per call; it is now
    two stats (exists() on a 300-page wiki: 1.96 ms → 0.08 ms).

  • page_history and topic_evolution leave the served palette when a view is
    in play
    , and the store refuses them. Both are answered by git, which knows
    nothing about labels. The system prompt and the injected evolution skill
    follow the palette, so a restricted session is no longer told to call tools it
    does not have.

  • A restricted session writes log/<label-set>/<date>.md. The open mode is
    unpartitioned, so a wiki with no restrictions has nothing to migrate.

  • add_source takes the write lock, like every other commit-producing path.
    Two concurrent ingests interleaving between git add and git commit raced on
    the index and one of them failed.

Upgrading

  1. pip install -U outmem — nothing changes until you declare labels.
  2. If any page uses restricted: in its frontmatter for something else, rename
    that key first (see Breaking).
  3. Follow the rollout in docs/restricted-content.md.
    Its first four steps are safe to run while still serving the unrestricted store.