v1.6.0 — Subagent transcript + tool content indexing
Retrieval coverage and correctness release. On a real ~10,000-file transcript corpus this took indexed documents from 31K to 677K and recall@5 on a 17-query set from 12/17 to 17/17 (recall@1 82.4% → 94.1%).
Coverage
- Nested subagent transcripts are now indexed. Enumeration was non-recursive, so only top-level session files were ever opened — 74% of transcript files were never read. Nested files get a disambiguated
doc_id, because subagent transcripts carry their parent session id and would otherwise collide with and overwrite parent chunks. tool_use,tool_resultandthinkingblocks are now searchable. The extractor kept a content block only if it wastype == "text"or carried a literaltextkey, dropping ~73% of blocks in a real transcript — so decisions made through tools were invisible to search. Prose is preserved ahead of tool content, and turn-pairs split rather than truncate, so adding tool output cannot evict the actual conversation.
Silent-failure fixes
Both of these reported success while doing the wrong thing, which is why they survived so long.
- Enumeration is fault-isolated. An unreadable or dangling path raised
ENOENTduring file collection — upstream of the existing per-file parse isolation — so one bad path out of thousands aborted the entire pass. Worse, the error was caught and converted intoreturn 0, leaving the error list empty and the job reporting"complete"over a corpus it never touched. Bad paths are now skipped and counted, and genuine failures propagate into/index/status. EMBEDDING_MODELis now reachable. The worker launcher overwrote the environment variable with a hardcoded constructor default, so configuring a different embedder silently had no effect.
Embedding configuration
EMBEDDING_DEVICE— selects the embedding device, auto-detecting CUDA and falling back to CPU, and logs the resolved device at startup. ChromaDB's default is"cpu", so embedding silently stayed on CPU on GPU-capable hosts.EMBEDDING_QUERY_PREFIX— applies an asymmetric model's query-side instruction. ChromaDB applies one embedding function to documents and queries alike, but models such as arctic-embed, bge and e5 prefix only the query; without this, query and passage vectors land in different regions of the embedding space.UPSERT_BATCH_SIZE— the previous fixed batch of 20 fit inside a single sentence-transformers internal batch, starving a GPU.- The unused bge-small generator no longer auto-selects CUDA, where it held ~390 MiB of VRAM without ever being used.
Ranking
- Weighted reciprocal-rank fusion with the vector arm down-weighted. Equal-weight fusion measurably regressed recall, and a unit test fails if the weights revert.
Upgrade notes
Indexing behaviour changed substantially — a full reindex is required to pick up subagent transcripts and tool content, and the resulting index will be considerably larger. Existing chunks update in place; the first part of each turn-pair keeps its original doc_id.
Tests: 26 passing.