systems parsing and indexing

Parsing and indexing

Active contributors: Bryan

The indexing system turns repository files into graph facts. src/lode/indexer.py discovers source files, hashes content, dispatches to parsers, and writes FileIndex records through src/lode/storage.py.

Directory layout

src/lode/indexer.py # source discovery, parsers, index orchestration
src/lode/model.py   # Node, Edge, FileIndex dataclasses
tests/test_lode.py  # indexing and search regression tests
tests/test_graph.py # parser facts used by graph tests

Key abstractions

Name	File	Description
IndexStats	src/lode/indexer.py	Counts scanned, indexed, skipped, removed, and resolved facts.
index_repo()	src/lode/indexer.py	Top-level repository indexing flow.
iter_source_files()	src/lode/indexer.py	Walks supported files and skips generated/cache paths.
parse_file()	src/lode/indexer.py	Creates a file node and dispatches by language.
parse_python()	src/lode/indexer.py	Uses Python AST for modules, classes, functions, routes, imports, calls, and inheritance.
parse_ts_js()	src/lode/indexer.py	Uses regex heuristics for TS/JS definitions, imports, routes, and calls.
FileIndex	src/lode/model.py	Per-file nodes and edges written to SQLite.

How it works

graph TD
    Walk[iter_source_files] --> Hash[hash_file]
    Hash --> Cache{unchanged?}
    Cache -->|yes| Skip[skip]
    Cache -->|no| Parse[parse_file]
    Parse --> Py[parse_python]
    Parse --> JS[parse_ts_js]
    Parse --> MD[parse_markdown]
    Parse --> Config[parse_config]
    Py --> Store[replace_file_index]
    JS --> Store
    MD --> Store
    Config --> Store
    Store --> Resolve[resolve_graph]

Loading

parse_file() creates a File node for every indexed source file, then language-specific parsers add functions, methods, classes, routes, document sections, config nodes, and unresolved external symbol placeholders. Import bindings are stored on the file node so resolve_graph() in src/lode/graph.py can build precise IMPORTS and CALLS edges later.

Integration points

• src/lode/cli.py and src/lode/daemon.py both call index_repo().
• src/lode/storage.py persists each FileIndex and queues embeddable nodes.
• src/lode/graph.py uses parser output to resolve imports, calls, and inheritance.

Entry points for modification

Add a new language by extending SOURCE_EXTENSIONS, adding parser dispatch in parse_file(), and emitting Node/Edge objects from src/lode/model.py. Add regression tests in tests/test_lode.py for indexing and tests/test_graph.py for resolved relationships.

Key source files

File	Purpose
src/lode/indexer.py	Main indexing and parsing implementation.
src/lode/model.py	Node, Edge, and FileIndex dataclasses.
src/lode/storage.py	Persistence of parsed facts.
tests/test_lode.py	Index and query tests.
tests/test_graph.py	Parser facts used in graph tests.

Related pages

• Storage and search
• Graph and impact
• Repository indexing