Full text index is built only once, so every entry inserted after the first batch is invisible to keyword_search

[graysoncooper]

What is wrong

LanceDBVectorStoreBackend builds its Tantivy full text index exactly once, on the first insert, and never rebuilds or refreshes it. Entries written by any later insert are stored and are retrievable through semantic and structured search, but keyword_search cannot see them.

_init_fts_index returns immediately once _fts_initialized is set, and that flag is set the first time create_fts_index(..., use_tantivy=True, ...) succeeds:

SimpleMem/simplemem/core/database/vector_store_backend.py

Lines 149 to 186 in db80b6a

	def _init_fts_index(self) -> None:
	if self._fts_initialized:
	return
	try:
	if self._is_cloud_storage:
	self.table.create_fts_index(
	"lossless_restatement",
	use_tantivy=False,
	replace=True,
	)
	print("FTS index created (native mode for cloud storage)")
	else:
	self.table.create_fts_index(
	"lossless_restatement",
	use_tantivy=True,
	tokenizer_name="en_stem",
	replace=True,
	)
	print("FTS index created (Tantivy mode)")
	self._fts_initialized = True
	except Exception as error:
	print(f"FTS index creation skipped: {error}")
	def insert(self, records: Sequence[VectorStoreRecord]) -> None:
	if not records:
	return
	rows = [
	{
	"entry_id": record.entry_id,
	**record.metadata,
	"vector": list(record.vector),
	}
	for record in records
	]
	self.table.add(rows)
	self._init_fts_index()

insert() calls table.add(rows) and then _init_fts_index(), which is a no-op after the first call. keyword_search at L208-L222 then queries that stale index.

This is specific to the Tantivy branch, which is the one taken for every local db_path. With use_tantivy=False the same sequence returns the later rows.

How it manifests

Ingestion writes one batch per sliding window: MemoryBuilder.process_window calls add_entries per window, and process_remaining and _process_windows_parallel each add more. So any conversation longer than one window performs several inserts, and only the first window's entries end up in the lexical index. The lexical layer of the hybrid retriever then returns empty or partial results for memories that are stored and plainly match the query, with no error surfaced to the caller.

Because _fts_initialized is per instance, a long lived process stays stale for its whole lifetime, and the on disk Tantivy index left behind carries the gap into later query only processes.

Failure scenario

Against the pinned lancedb==0.25.3:

1. Create a table and insert row A.
2. create_fts_index(use_tantivy=True, tokenizer_name="en_stem", replace=True).
3. Insert row B.
4. count_rows() returns 2. search("<term unique to A>") returns A. search("<term unique to B>") returns nothing.

table.optimize() does not repair it. Through the library API the equivalent is: build a memory store from a conversation spanning several windows, then call keyword_search with an exact term that appears only in a later window. It returns no hit, while semantic_search finds the entry.

Suggested fix

Refresh the index after later inserts instead of guarding on a one shot flag: drop the _fts_initialized early return and rebuild with replace=True after each insert, or debounce the rebuild if the cost matters. Switching the local branch to the native index is the other option, since it also searches rows added after index creation.

Notes

This predates PR #38; that PR is only where it surfaced during review. The behavior is not covered by tests because every existing fixture inserts a single batch before querying, so no test exercises a second write.

Automated report: this issue was produced and filed automatically, with no human review before posting. Two independent checks agreed it is a real bug, but if it misreads the code please say so and we will close it.

Found while running Ito (AI code review that runs your application, free for open source) against recently merged PRs. Full analysis.