Skip to content

v0.3.0 — Benchmarks + Semantic Embeddings

Choose a tag to compare

@snow-ghost snow-ghost released this 10 Apr 19:35
· 35 commits to main since this release

Highlights

Two new public benchmarks and optional semantic-embeddings support
(hybrid BM25 + vector search).

Benchmarks

Benchmark Tests R@5 Runtime
LongMemEval (ICLR 2025) 500 69.4% ~31s
LoCoMo (Snap Research) 1986 88.2% ~11s
ConvoMem sizes 1–6 (Salesforce) 7021 100.0% ~15 min
  • LoCoMo per-category R@5: open-domain 92.7%, temporal 85.0%, single-hop
    80.5%, multi-hop 59.4%, adversarial 93.0%
  • ConvoMem confirms "your first 150 conversations don't need RAG"

Semantic Embeddings (optional)

Set MEM_EMBEDDINGS_URL, MEM_EMBEDDINGS_MODEL, MEM_EMBEDDINGS_API_KEY
pointing at any OpenAI-compatible /v1/embeddings endpoint (OpenAI, Voyage,
Cohere compat, Ollama, LM Studio, LocalAI, llama.cpp). Then:

mem reindex                             # compute vectors for all drawers
mem search "why GraphQL" --mode hybrid  # BM25 + vector fused via RRF
  • Pure Go HTTP client, batch support
  • Embeddings stored as BLOBs in the same SQLite file
  • Hybrid = Reciprocal Rank Fusion (k=60)
  • Entire feature is opt-in; default BM25 behavior unchanged
  • No new module dependencies

Install

# Linux amd64
curl -fsSL https://github.com/snow-ghost/mem/releases/download/v0.3.0/mem-linux-amd64.tar.gz | tar xz
sudo mv mem-linux-amd64 /usr/local/bin/mem

Replace linux-amd64 with linux-arm64, darwin-amd64, or darwin-arm64
for other platforms.

Reproduce benchmarks

go build -o /tmp/longmemeval-bench ./benchmarks/longmemeval/
go build -o /tmp/locomo-bench ./benchmarks/locomo/
go build -o /tmp/convomem-bench ./benchmarks/convomem/

See benchmarks/README.md
for dataset download URLs.