Indexing Pipeline

Indexing Pipeline

How a file on disk becomes something you can search for. Three cooperating parts feed one queue, and one worker drains it through the AI stages.

• Crawler — the past. Walks each root on demand, enqueues work, purges vanished paths.
• Watchdog — the present. Reacts to create/modify/rename/delete events, debounced 2 s.
• Worker — the async background task that turns a queued path into meaning.

Metadata and the queue live in SQLite (r2d2 pool, WAL); vectors live in LanceDB.

The queue

One row per path, with status, priority, retry_count and last_error. The worker pulls the highest-priority pending tasks, and every outcome is recorded:

Status	Meaning
pending_extraction	Waiting for the worker.
completed	Indexed.
failed	Failed, will be retried (up to 3 attempts).
failed_permanent	Retries exhausted. Visible in the queue modal; retryable by hand.

The Indexing queue modal shows what's in flight — the current file, the AI stages it will traverse, what's pending and what failed — and lets you retry, ignore, or retry everything failed. Pause suspends crawler and worker without losing progress, and frees the embedding model.

Per-file pipeline

path from queue
   │
   ├─ gone from disk?      → purge vectors + catalog, done
   ├─ is a directory?      → block-folder indexing (below)
   ├─ parent is a block?   → skip
   │
   ├─ route by type (parser.rs)
   │     Ignored · Text · Document · Image · Media · RequiresAIRouting
   │
   ├─ extract  →  SHA-256 hash  →  unchanged? → skip, just refresh mtime
   │
   ├─ AI stages (below)
   │
   └─ chunk → embed → upsert LanceDB → store sense + extract → completed

Routing

Extension first, magic bytes second — the reverse order misfires (a .docx is a ZIP, and would be routed as a binary archive).

Route	Triggers	Path taken
Ignored	Empty files, ._*, .DS_Store, Thumbs.db, desktop.ini, anything under node_modules, .venv, target, .git	Never indexed.
Document	pdf, docx, pptx, xlsx	Real text extraction.
Text	~60 text/code/data extensions (txt, md, csv, json, yaml, rs, py, ts, sql…)	Direct read.
Image	png, jpg, jpeg, gif, bmp, webp	Vision caption.
Media	~40 audio and video containers	Transcription and/or video description.
AI routing	Everything else — unknown binaries, archives	The LLM decides what can be extracted.

.ts is deliberately not treated as media: it is TypeScript far more often than MPEG-TS here, and sending source code to a transcription server would be worse than missing a rare video. Magic bytes still catch a genuine one.

The three kinds of sense

1. Textual — real content

Format	Extraction
PDF	Standard text layer → encrypted-with-empty-password retry → decrypt via lopdf and re-extract → if still empty, vision OCR (below).
DOCX / PPTX / XLSX	Unzipped, XML text nodes kept (document body, slides, shared strings).
HTML	Tags stripped.
Text / code	Read as-is.

Files above indexing.max_file_mb (default 50 MB) skip extraction and fall back to context. An extraction that panics — a malformed PDF Unicode table, a lone UTF-16 surrogate — also falls back to context immediately rather than burning three identical retries: the failure is deterministic, and a document findable by name beats a document lost.

Scanned PDFs → vision OCR

A PDF with no text layer is not a failure, it's a scan. If the vision slot is on, its first 8 pages are rendered to JPEG (~1600 px long side, quality 82) and sent to the vision model with the vision_ocr prompt.

Rendering, not image extraction, is the point. Scanners commonly produce MRC: the page is split into a background JPEG plus text layers in CCITTFax or JBIG2. Pulling out embedded images then yields the background alone — a photo and a pale ghost, without a single line of readable text — which used to leave the model inventing a description from the filename. Drawing the page composes the layers, and covers raw-pixel, JBIG2, JPEG2000 and purely vector PDFs too. Extracting embedded JPEGs remains only as a fallback for documents the renderer cannot open.

2. Visual — images

The file is base64-encoded and sent to the vision model with the vision_caption prompt. The caption is kept as the extract; a reasoning pass then qualifies it into the sense. What gets embedded is the qualification + caption + context descriptor.

Non-raster formats (.svg, .ico, .cur) go straight to context — vision servers reject them.

3. Contextual — the safety net

For anything unreadable: opaque binaries, VM disks, oversized files, refused media, exhausted retries. SenseTree builds a factual descriptor — filename, parent folder, type, extension, up to 15 neighbouring filenames — and (if qualify_context is on) asks the reasoning model to guess what this most likely is from the path and neighbourhood.

Nothing is ever "not indexed" for lack of readable content. A VM image stays findable as "probably a virtual machine disk for the Windows lab".

Unknown types: the AI router

For RequiresAIRouting files, a 32 KB sample decides:

1. ≥ 85 % printable and within the size cap → treat as text, extract in full.
2. ≥ 30 % printable and reasoning enabled → ask the LLM to extract useful content, or answer exactly NO_CONTENT.
3. Otherwise → contextual.

Media: audio and video

Two independent, complementary sources of meaning, each with its own slot:

• Transcription — what is said, via a Whisper-style server (/audio/transcriptions, multipart).
• Video description — what is seen, via a multimodal server (/chat/completions with a video_url part).

Both are opt-in and off by default. If both are enabled on a video, their outputs are concatenated (Description visuelle: … + Transcription: …) and then run through the normal document path — chunked, qualified, BM25-indexed, contextually enriched. A one-hour recording is therefore searchable on any of its passages, not just its summary.

The app assumes nothing about formats: every routed file is sent as-is and the server accepts or refuses. A refusal is a permanent error → contextual fallback. A transient one is retried. An empty transcription is not an error — it just means no speech (music, ambience, silence).

Hashes are computed streaming (16 KB blocks), and an unchanged media file is never re-transcribed — it is the single most expensive call in the pipeline.

See AI Server Protocol for the exact requests and every configurable field.

Qualification: from extract to sense

Extraction gives content. Qualification gives meaning: a 2–4 sentence answer to what is this? — its nature (invoice, ID card, contract, course notes, bank statement, CV), what it relates to, and its key facts.

Both are kept side by side: the sense and the extract (bounded at 16 000 characters), so you can compare them against the real document in the detail drawer.

Qualification is skipped when the text is under 120 characters (nothing to gain), when the reasoning slot is off, or when its toggle is off. Four independent toggles, all on by default:

Toggle	Applies to
qualify_documents	PDFs, Office, text, code, LLM-extracted files
qualify_images	Vision captions
qualify_media	Audio/video transcriptions — separated deliberately: an hour-long transcript is long and costly to qualify, and you may want to cut that without touching your PDFs
qualify_context	Guesses for unreadable files

indexing.qualify_effort sets the reasoning effort for all of these, defaulting to none — thousands of calls whose answer is a few tokens. It is the only reasoning setting whose default disables thinking; chat and planning stay on Auto.

A pinned sense (one you or the agent corrected) is never regenerated: it survives every re-index, and is re-embedded so the correction takes effect in search too.

You can also qualify on demand, outside indexing: per file, or a whole folder at once (only files whose sense is still a raw excerpt, running in the background and honouring pause).

Sequential vs batch

indexing.pipeline_mode decides how the AI stages are ordered. There is no universally good answer, so it's yours to make.

	Sequential (default)	Batch
Order	One file end to end: vision → reasoning → embedding, then the next	A slice of batch_files files (default 64) through all the LLM work, then all the embedding
Model swaps	One alternation per file	One per slice
Index freshness	A processed file is instantly searchable	Files become searchable at the end of their slice
Best when	Embedding is local (no swap at all), or the server holds several models	Models don't fit in memory together and the server reloads gigabytes on every alternation

In batch mode, SenseTree explicitly unloads the reasoning and vision models from Ollama before the embedding phase, which makes the behaviour deterministic regardless of how the server is configured. If a pause arrives mid-slice, the already-prepared files are still vectorized — the LLM calls were already paid for.

Throughput

The Throughput panel measures the four AI stages separately, because their natural units differ:

• vision, media, reasoning — one call per file, so files/s and ms/call are meaningful.
• embedding — N chunks proportional to volume, so chunks/s and MB/s (a 200-page PDF and a three-line .txt don't compare as "files").

Errors are counted separately, since failed calls would otherwise skew the averages. This measures model speed, not pipeline speed — the pipeline also reads files, extracts PDFs and writes to LanceDB.

Block vs recursive folders

Not every folder deserves file-by-file indexing. A Python venv, node_modules, an app bundle, a DAW sample pack: indexing those individually is noise. SenseTree treats them as a single opaque block — one LLM-written sentence describing the folder plus its facts (item count, dominant extensions, sample names), searchable as a unit, not descended into.

The decision cascade (folders.rs), in order:

1. Certain heuristics → block. Names (venv, site-packages, vendor, pods…), bundle suffixes (.app, .framework, .vst3, .photoslibrary…), or unambiguous technical extensions (Ableton/Kontakt presets, .asd sidecars). No AI call.
2. Dominated by opaque binaries → block. Threshold scaled by the slider (below).
3. Fewer than 6 entries → recursive. Trivially safe.
4. Reasoning disabled → recursive. The safe default.
5. Otherwise the LLM decides, nudged by the slider, from the folder's full path, name, and a sample of its contents.

Hard-ignored everywhere (never explored, never blocked): dotfolders, node_modules, target, AppData, Windows, $RECYCLE.BIN, __pycache__, __MACOSX.

Deferred decisions

The crawler and watchdog never block on the LLM: if a folder needs it, the decision is deferred (marked pending, not indexed) and they move on. Without this, indexing advanced one folder per LLM timeout — roughly one every 25 seconds. A background classifier revisits pending folders, decides once the model is reachable, and launches the scan if the verdict is recursive.

The block/recursive slider (block_bias)

One knob, 0.0–1.0 (default 0.5), from very recursive to very block. It acts in two places:

• The opaque-binary heuristic (no AI). A folder is blocked when opaque_ratio ≥ 0.90 − bias × 0.35 and rich_ratio ≤ bias × 0.15. Far left: ~90 % binaries and virtually no useful content required. Far right: 55 % binaries suffice, even with a little content.
• The LLM classifier. bias ≥ 0.66 injects "prefer block", bias ≤ 0.34 injects "prefer recursive", the middle stays neutral — and ambiguous answers are broken the same way.

The slider does not reclassify retroactively. Changing it (or the folder_classify prompt) makes SenseTree forget existing classifications on save; a scan then reclassifies. Save, then Re-index.

You can also force a folder's mode by hand from the Explorer, which overrides both heuristics and LLM.

Change detection and cleanup

• SHA-256 per file. An unchanged hash skips embedding entirely — only the mtime record is refreshed.
• Orphan purge. Paths not seen during a scan have their vectors, catalog rows and queue entries removed.
• Watchdog syncs deletions and renames live, honouring block folders (a file inside a block is never indexed individually).
• One crawler per root. A scan requested while one runs is not lost: a re-scan is flagged and the running pass repeats at the end. A re-index bumps a scan epoch so in-flight scans abort instead of writing into a state that was just cleared.

Related settings

Everything on this page is configurable — see Configuration for the full field reference, and Prompts for the ten prompts that drive the AI stages.