Graph Projection and Performance

Graph Projection and Performance

Language: English · 简体中文

LWC projects canonical knowledge into document-graph sidecars and keeps graph, recall, and derived work bounded. It does not load every document, token, or edge into memory. The design targets local Agent workflows with one canonical SQLite writer and rebuildable indexes.

Read this page before raising a limit, adding a background task, or treating LWC as a hosted multi-user database.

Intended operating model

LWC is optimized for:

• one user or Agent process coordinating writes to a local Wiki;
• many bounded reads between occasional mutations;
• project and global knowledge stored separately;
• explicit background Work for expensive projections;
• deterministic retrieval without model or network latency;
• indexes that can be rebuilt from canonical state.

It is not a horizontally distributed knowledge service. If independent machines need simultaneous writes and remote tenancy, LWC's local Store is the wrong synchronization layer.

Bound every read

Public list, search, context, graph, log, lint, Work, and tag commands require or enforce limits. Candidate generation commonly overfetches only to a hard ceiling, then filters and truncates deterministically.

--scope all merges at most project and global read results; it does not scan arbitrary projects. MCP applies a stricter bounded response budget so an Agent cannot pull the complete Wiki through one tool call.

When a response says has_more or truncated, paginate or narrow the query. Raising a global ceiling is not the first response to a poorly scoped question.

Sparse changesets

changeset begin stores touched entities and base fingerprints instead of copying the live database. Draft lint and search attach the live baseline read-only and overlay the delta.

This keeps tag-only or small multi-Page updates proportional to their changed surface. Each draft still owns isolated Work and graph sidecars; isolation is not traded away for storage savings.

Durable Work and coalescing

Schema migration, maintenance, and document-graph projection run outside the initiating terminal. One active state-changing Work per Wiki avoids competing heavy jobs.

Graph mutations append document keys to a deduplicated pending set. An active projector drains successive batches, so a burst of Page writes does not spawn one process per Page. Failures put unprocessed keys back into the queue.

This design optimizes throughput while keeping canonical reads available. It does not remove the need to watch the final Work and verify the graph.

Word Graph density control

Token membership is the densest relationship in the product. LWC therefore never persists or returns one global word network.

Each request is bounded by:

• at most 8 query terms;
• 25 candidate documents per sample page;
• 30 selected terms;
• 200 nodes and 500 edges;
• 4 MiB inspected content;
• at most 4 inspected spans per document;
• pagination offset capped at 10,000.

Candidates come from FTS first. Tokenization then runs only over the selected metadata and spans, with byte, node, edge, and term budgets. The response reports diagnostics and truncation reasons instead of silently claiming completeness.

Search cost control

Document and span search allow at most 64 normalized query terms. Candidate overfetch is 4× or 8× the requested limit, capped at 1,000. Filters, adjustments, graph reranking, grouping, and final sorting operate on that bounded candidate set.

Markdown segmentation caps individual passage and sentence bytes and limits total spans per document. Exact UTF-8 ranges avoid copying a second body for every span.

FTS rows and spans update with the canonical Page or Source transaction. Routine search never performs opportunistic full reindexing.

Graph projection cost control

Document graph mutation is incremental by dirty document key. Full projection is reserved for engine enablement, checkpoint restore, repair, or explicit configuration changes.

Graph query commands bound depth and result count. Viewer further caps visual graphs at 1,000 nodes and 5,000 edges. graph verify compares fingerprints and key sets rather than trusting a successful query as proof of completeness.

Grafeo and SurrealDB are alternative derived engines; enabling both simultaneously would duplicate cost without adding canonical durability.

CodeGraph reuse

The pinned CodeGraph runtime is cached once per user, version, and target. A second project reuses the verified runtime but builds its own index.

Index writes commit one owner file completely before moving to the next, preserving a readable prior graph during updates. Viewer reads the existing database directly and never wakes indexing.

Runtime reuse removes repeated downloads; it does not merge project indexes or weaken project path containment.

SQLite concurrency

WAL mode allows readers while a writer commits. Write paths use transactions and a bounded busy timeout rather than waiting forever.

Expect database_busy when another writer owns the relevant boundary. Retry after the reported delay; do not add unbounded loops or bypass revision checks. Changeset commit and checkpoint restore deliberately reject stale observations instead of optimizing away correctness.

Filesystem cost

Generated Markdown and state files use staged replacement. Maintenance compaction checkpoints and truncates the WAL through durable Work. Checkpoints are full database copies, so retention cost grows with database size and checkpoint count.

Do not schedule a checkpoint for every write. Keep named recovery points that correspond to real operational boundaries and prune them only under an explicit retention policy.

Diagnose before tuning

Use existing evidence:

lwc --scope project work list
lwc --scope project graph status
lwc --scope project graph verify
lwc --scope project search "slow query terms" --limit 10 --explain
lwc --scope project lint

For Word Graph, inspect diagnostics, limits, and truncation_reasons. For CodeGraph, distinguish runtime installation from project initialization. For a slow mutation, identify its Work phase before changing a system-wide bound.

Scaling practices

• Keep project-specific facts in the project Wiki and reusable practices in global scope.
• Consolidate shared knowledge into maintained Pages instead of accumulating duplicate source summaries.
• Use stable titles, summaries, and terminology so lexical retrieval needs fewer candidates.
• Use strong tags only for a small deterministic core.
• Prefer incremental graph projection; trigger full rebuild only for recovery or engine changes.
• Paginate dense Word Graph and list views.
• Keep Source files curated; LWC is not a raw log warehouse.
• Measure a demonstrated ceiling before changing constants.

Known ceiling

The architectural ceiling is a local SQLite writer and local derived indexes. This keeps installation, audit, backup, and recovery simple. It also means that high-write multi-tenant service workloads require a different canonical store and coordination protocol, not a larger --limit.

Next: Configuration reference