feat: drop memory index

[mudler]

This PR drops the memory index to use the filesystem instead

[Copilot]

Pull request overview

This PR removes the persisted in-memory “index” from PersistentKB state and shifts chunk lookup/deletion to be driven by engine metadata (source) plus filesystem scanning of assetDir. It also introduces a lightweight in-memory MockEngine and a Ginkgo/Gomega test suite to exercise persistence behavior without embeddings.

Changes:

• Drop CollectionState.Index / PersistentKB.index and derive document listing from the on-disk UUID layout.
• Extend the Engine interface with GetBySource and implement it for Postgres/Chromem/Mock (LocalAI stubbed).
• Add mock-based persistency tests covering store/list/search/reset/remove and external sources.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
rag/persistency.go	Removes state/index map and switches document/chunk operations to filesystem keys + Engine.GetBySource; updates reset/repopulate/migration logic.
rag/engine.go	Extends Engine interface with GetBySource.
rag/engine/postgres.go	Adds GetBySource query implementation for Postgres-backed engine.
rag/engine/chromem.go	Adds GetBySource implementation using a metadata-filtered query.
rag/engine/localai.go	Adds GetBySource stub returning “not implemented”.
rag/engine/mock.go	Adds new in-memory MockEngine for tests (no embeddings/external deps).
rag/persistency_mock_test.go	Adds a comprehensive mock-engine test suite for PersistentKB.

Comments suppressed due to low confidence (1)

rag/persistency.go:200

• PersistentKB.Reset ignores errors from os.RemoveAll/os.MkdirAll/db.save and os.RemoveAll(db.path), so Reset can return nil even when the on-disk state wasn’t actually cleared. Please handle and propagate these errors (and consider ordering so Engine.Reset failures don’t leave disk/state partially reset).

func (db *PersistentKB) Reset() error {
	db.Lock()
	os.RemoveAll(db.assetDir)
	os.MkdirAll(db.assetDir, 0755)
	db.sources = []*ExternalSource{}
	db.save()
	db.Unlock()
	if err := db.Engine.Reset(); err != nil {
		return err
	}
	os.RemoveAll(db.path)
	return nil

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.