systems storage and search

Storage and search

Active contributors: Bryan

SQLite is Lode's hot path. src/lode/storage.py defines the schema, writes parsed facts, supports FTS search, resolves symbols, returns graph neighbors, and stores embedding metadata.

Directory layout

src/lode/storage.py # schema, writes, queries, embedding persistence
src/lode/config.py  # data directory and repository ID helpers
tests/test_lode.py  # storage-backed CLI and indexing tests

Key abstractions

Name	File	Description
connect()	src/lode/storage.py	Opens SQLite, sets WAL mode, enables foreign keys, and initializes schema.
init_db()	src/lode/storage.py	Creates repos, files, nodes, edges, FTS, embedding queue, and embeddings tables.
replace_file_index()	src/lode/storage.py	Atomically replaces all facts for one file.
search_nodes()	src/lode/storage.py	Uses FTS5 with fallback matching.
find_symbol()	src/lode/storage.py	Finds exact and partial qualified-name matches.
get_neighbors()	src/lode/storage.py	Returns incoming and outgoing graph neighbors.

Schema overview

Table	Purpose
repos	Indexed repository roots and timestamps.
files	File metadata, language, size, hash, and indexed timestamp.
nodes	Symbols, files, routes, docs, config nodes, and external placeholders.
edges	Relationships between nodes.
node_fts	SQLite FTS5 index for names, signatures, docs, and paths.
embedding_queue	Nodes that should be embedded after indexing.
embeddings	Persisted vectors and model metadata.

How it works

replace_file_index() deletes old facts for a file, upserts file metadata, inserts de-duplicated nodes, updates node_fts, queues embeddable nodes, and inserts de-duplicated edges. This keeps re-indexing cheap because unchanged files are skipped by src/lode/indexer.py.

graph LR
    FileIndex[src/lode/model.py] --> Replace[replace_file_index]
    Replace --> Files[(files)]
    Replace --> Nodes[(nodes)]
    Replace --> Edges[(edges)]
    Replace --> FTS[(node_fts)]
    Replace --> Queue[(embedding_queue)]

Loading

Integration points

• src/lode/context.py calls search_nodes() and get_neighbors().
• src/lode/graph.py reads nodes and edges, then writes resolved edges.
• src/lode/cli.py and src/lode/daemon.py use repo_filter() to scope queries.
• src/lode/embeddings.py pairs with pending_embedding_nodes() and upsert_embedding().

Entry points for modification

Schema changes start in init_db() in src/lode/storage.py. Query behavior changes should be covered by tests/test_lode.py, and graph-neighbor behavior should be covered by tests/test_graph.py.

Key source files

File	Purpose
src/lode/storage.py	SQLite schema and query layer.
src/lode/config.py	Data paths and repository IDs.
src/lode/model.py	Stored fact shape.
tests/test_lode.py	Storage-backed behavior tests.

Related pages

• Data models
• Search and context
• Embeddings and Kuzu