Storage and Data Model

Storage and Data Model

Language: English · 简体中文

LWC stores durable knowledge in SQLite and keeps filesystem projections explicitly subordinate to it. This page describes the ownership model, table families, identities, and migration rules behind the public CLI.

Read it before changing schema, backup behavior, source identity, Page replacement, or filesystem layout.

Filesystem layout

project/
`-- .lwc/
    |-- wiki.db                 canonical project Wiki
    |-- config.json             project deployment configuration
    |-- wiki/                   generated Markdown
    |-- work/                   live durable Work
    |-- checkpoints/            named SQLite checkpoints
    |-- changesets/
    |   |-- <name>.db           sparse draft database
    |   `-- draft-<name>/       draft Work and graph sidecar
    |-- graph-grafeo/           selected derived document graph
    |-- graph-surrealdb/
    `-- codegraph/              project-local code index

user home/
`-- .lwc/
    |-- wiki.db                 global Wiki
    |-- config.json             global defaults
    |-- runtime/codegraph/      pinned runtime cache by version and target
    `-- agent-installs/         integration ownership receipts

The exact graph sidecar present depends on configuration. None of these directories is a supported manual editing surface.

Canonical table families

Family	Main records	Invariant
Identity and governance	meta	Store ID is stable; revision changes with canonical state; Purpose and Schema are non-empty
Evidence	sources, source_path_revisions	Source content is immutable and SHA-256 deduplicated; path observations are revisioned
Maintained knowledge	pages, page_sources, page_provenance, links	Page replacement updates body, grounding, provenance, and links together
Ingest	ingest_jobs	One durable state machine per Source
Strong context	tags, page_tags	Membership is explicit and priority ordered
Semantic graph	semantic_relations	Explicit typed relation keeps provenance, reason, and cited Source IDs
Retrieval control	retrieval_weights, retrieval_feedback	Audited adjustments never create lexical candidates
Search projection	search_fts, search_spans, span_fts	Derived rows correspond to active canonical documents and fingerprints
Audit and recovery	operations, changesets and inverse metadata	State-changing behavior remains attributable and recoverable

SQLite foreign keys and CHECK constraints enforce supported status values, provenance classes, weight ranges, tag budgets, and relation ownership where possible. Higher-level invariants that cross projections are enforced in Store transactions and lint.

Source identity

A Source has two complementary identities:

1. content_hash identifies immutable UTF-8 bytes globally within one Wiki.
2. source_path_revisions records that a tracked path pointed to a Source at revision N.

Adding identical content reuses the Source instead of storing a second copy. A file that changes receives a new Source ID and advances its path head; historical revisions remain frozen.

Removing the current tracked revision untracks the path rather than revealing an older revision as current. Deleting Source evidence is constrained when Pages or history still depend on it.

Page identity and replacement

The Page slug is its stable identity and Wiki filename. A replacement preserves created_at, updates updated_at, and replaces the maintained fields as one logical operation:

• title, kind, summary, and body;
• cited Source set;
• explicit provenance set;
• parsed outgoing [[wikilinks]];
• document and span search rows;
• operation record and Store revision;
• dirty graph document key when graph projection is enabled.

Callers must read the current Page and preserve still-valid knowledge. Replacement semantics are intentional: omitted citations or provenance do not linger invisibly.

Search documents and spans

Document FTS rows are contentless projections keyed by document type and identifier. Separate active span records point to exact byte ranges in the current body and carry:

• passage or sentence type;
• parent and ordinal;
• byte start and end;
• content fingerprint;
• segmenter version.

When a document changes, old locators become inactive. span get refuses stale fingerprints instead of guessing a nearby replacement.

Operations and revision fingerprints

Canonical mutations append an operations row with action, target, structured detail, and timestamp. Search and lint remain private reads unless the caller explicitly requests recording.

Store revision is a content fingerprint, not a monotonically increasing integer. Changesets capture base fingerprints for touched entities; checkpoint restore also checks that live revision did not advance between safety snapshot and replacement.

Do not infer chronology from the revision string. Use operation IDs and timestamps for audit order.

Transaction model

LWC uses WAL-mode SQLite with foreign keys enabled and a bounded busy timeout. Mutations use an immediate transaction when they must reserve the write path before reading and changing related rows.

The transaction covers canonical state and SQLite search projections. External graph Work and filesystem materialization happen after canonical commit because they live outside the database transaction.

That split produces an explicit partial-success contract instead of a distributed transaction illusion.

Generated Markdown

Materialization renders Purpose, Schema, index, Pages, and Source-facing navigation from SQLite. Files are staged and replaced as owned artifacts. Extra or obsolete generated files are removed only through LWC's manifest-aware projection logic.

Edits made directly under .lwc/wiki/ are overwritten or ignored. To change knowledge, use Page, Source, tag, Purpose, or Schema commands.

Draft storage

Sparse changesets store only touched entities, base observations, staged operations, and merge metadata. The live store is attached read-only for overlay queries and lint.

Every draft has a dedicated runtime directory derived from its validated name. Work and graph sidecars are scoped there so draft A cannot query or cancel draft B's Work or see its projected graph nodes.

Commit persists a checksummed inverse payload before applying the live merge. Optional fields serialize compatibly so older inverse patches remain verifiable after upgrades.

Checkpoints

Named checkpoints use SQLite's online backup API and are stored as regular .db files under checkpoints/. Restore validates a private candidate before writing live data and creates a pre-restore-* safety checkpoint.

Checkpoint files contain the complete SQLite database, including derived FTS tables, but not external filesystem state. Restore re-materializes Markdown and queues a fresh document-graph projection.

Schema versioning

PRAGMA user_version and meta.format_version identify the Store format. The tokenizer identifier is also stored because changing normalization invalidates search projections.

Supported migrations run in explicit ordered steps. Long schema migrations execute as Work through a shadow database and safety checkpoint rather than mutating an old database in place without recovery.

An unknown future version returns unsupported_store_version. LWC never guesses how to downgrade a newer store.

Integrity rules

Do not:

• open wiki.db in a GUI and edit rows;
• copy only wiki.db while a WAL write may be active;
• create fake checkpoint files in the checkpoint directory;
• move a draft database without its validated live binding;
• edit FTS, Work, graph, or materialized files to repair canonical knowledge;
• commit .lwc runtime state to source control.

Use CLI mutations, lint, checkpoint, changeset recovery, maintenance Work, and graph verify instead.

Next: Search, ranking, and indexing