Indexing Pipeline

Indexing Pipeline

SenseTree keeps a semantic index of your files in sync with disk, in three cooperating parts:

• Crawler — the past. Walks each root recursively on demand and enqueues work.
• Watchdog — the present. Reacts in real time to create/modify/rename/delete events (debounced).
• Worker — the async background task that turns queued paths into "sense" (extract → hash → chunk → embed → store).

Metadata and the queue live in SQLite (via an r2d2 connection pool); vectors live in LanceDB.

The crawl

For each entry the crawler:

1. Upserts file metadata into the catalog (path, parent, size, mtime).
2. For directories, decides recursive vs. block (below). Blocks are enqueued as a single unit and not descended into.
3. For files, enqueues those whose type isn't Ignored, unless already up-to-date (unchanged mtime/hash).
4. Purges orphans — paths that vanished since the last scan — so their vectors get removed.

One crawler per root at a time. If a scan is requested while one is running (e.g. after changing a model), it is not lost — a re-scan is scheduled, and the running scan does another pass at the end. Indexing can be paused/resumed; the crawler and worker suspend without losing progress.

The three kinds of "sense"

The worker extracts meaning differently by file type (parser.rs routes this):

Kind	Files	What's stored
Textual	PDF, Word, text, code, markup	Real extracted content, chunked and embedded. Scanned PDFs with no text layer fall back to vision OCR (if enabled).
Visual	PNG, JPG, WEBP, GIF, BMP	A caption from the vision model (embedded with filename + folder context). Non-raster formats (.svg, .ico) go to context.
Contextual	opaque binaries, VMs, oversized files	A synthetic doc: filename + parent folder + extension + size. This is the "extraction by context."

Each unit is hashed (SHA-256); if the content is unchanged since last time, embedding is skipped. Summaries are stored after a successful embedding, so a broken embedding endpoint means no summary until it's fixed.

Resilience

The queue has retry_count + last_error with capped retries and backoff. Vision failures are classified: a transient failure (timeout, server busy, model swap) is retried, and only on the last attempt does the image fall back to contextual sense — so a momentary vision hiccup never permanently downgrades a file. See Troubleshooting.

Block vs. recursive folders

Not every folder should be indexed file-by-file. A Python venv, node_modules, an app bundle, or a DAW sample pack is noise — SenseTree treats it as a single opaque block (one summary, not descended into). A folder of your documents/photos/code is recursive (explored and indexed).

The decision cascade (folders.rs):

1. Certain heuristics → block (venv, dependencies, bundles, DAW packs). No AI, ignores the slider.
2. "Dominated by opaque binaries" heuristic → block. Threshold scaled by the slider (below).
3. Trivial folders (< 6 entries) → recursive.
4. Reasoning disabled → recursive (safe default).
5. The LLM decides the rest, nudged by the slider.

The block/recursive slider (block_bias)

A single knob, 0.0–1.0 (default 0.5), from very recursive to very block. It affects the decision in two places:

• The opaque-binary heuristic (no AI). A folder is blocked when its ratio of opaque files clears a threshold:
  • threshold = 0.90 − bias × 0.35 (so 0.90 at the far left → 0.55 at the far right), and
  • the tolerated amount of "rich" content opens up as bias × 0.15.
  • Left: needs ~90% binaries and ~0% useful content to block (very cautious). Right: 55% binaries suffice, even with a little content.
• The LLM classifier. The slider injects a tendency into the prompt — bias ≥ 0.66 → "prefer block", bias ≤ 0.34 → "prefer recursive", middle → no hint — and breaks ambiguous LLM answers by the same bias.

Slider	Effect
Left (recursive)	Explore maximally; block only certain cases. AI leans recursive when unsure.
Middle	Normal heuristics; AI decides the gray areas freely.
Right (block)	Lower binary threshold, AI leans block, doubts become blocks. Lighter, faster index.

The slider does not reclassify retroactively. It applies to folders classified afterward, or after a re-index. To apply a change: move the slider, Save, then Re-index. (Changing the slider or the folder_classify prompt makes SenseTree forget prior classifications automatically.)

A background classifier also retries folders whose decision was deferred (because the AI was momentarily unavailable), so classification completes once the model comes back.