DB backend

[robtaylor]

Just out of curiosity, why postgres rather than something more lightweight like sqlite?

[robtaylor]

@parcadei ? For embedded use i suspect sqlite will be faster for this usecase with WAL.

[stephenfeather]

I didn't write it, but use it so I'll throw in my 2 cents.

The pgvector extension is proven and works.
sqlite-vec and the libsql forks have varying degrees of performance.

If you are only going to be working on a single system and work alone, then sqlite is a reasonable solution (and OPC supports this out of the box, right?)

But bring in multiple systems (like I use here), my learnings and memories are stored on one postgres system and my agents have shared knowledge across all the session regardless of which machine I'm on. This also means I can have a system that just spins up subagents for tasks (code writing, research - these are my 2 primary use cases) that gain the advantage of that acquired and stored knowledge.

Add in additional users, and now organizational knowledge from agent interactions are being stored. Why should Jane and Claude have to fuss with something that Jack and Claude solved last week and stored a memory regarding?

My last considerations is my dataset. As the memories increase, sqlite will bounce up against limitations (any database will) but sooner than postgres will because PG has been designed for those medium to large scale datasets. I'v mentioned before that I archive my jsonl files. I may want to retrain, find a better extraction method, or write better extraction prompts. I anticipate a truly large dataset across the work I do, both professionally and personally.

The better I train my agent, the better those memories are, the better the various relational vectors are - the better my agent and I will work together.

---

I'm training my future Jarvis. It won't be an off the shelf clawbot. :)

Yes, eventually I'd love a custom model unique to my needs and interactions. But until then, when OpenAI gets their hooks together and out of the experimental stage, codex will be injected with the same relational memories for my interactions with it. (gemini is ahead of codex in this but behind claude and partially working here for me). I'd have a lot more learning as well if OpenAI didn't encrypt the thinking and reasoning that can be stored in the codex logs. :(

[robtaylor]

Great points, thank you!