v0.3.0 — LongMemEval v5 hybrid (500 Q)

ACME v0.3.0-longmemeval-v5

Date: 2026-06-26
API: https://acme-api.blackgrass-3076f328.westeurope.azurecontainerapps.io
Image: acme-api:longmemeval-v5-hybrid
Repo: https://github.com/KamilBourouiba/ACME

MemoryBench v3 (13 scenarios)

System	Overall
ACME	0.925
RAG	0.487

Job 3b31e5e3, GPT-4.1, June 2026.

LongMemEval oracle (500 Q, single clean run)

System	Overall	KU	Multi-session	Temporal	Preference	Abstention
ACME	0.876	0.944	0.793	0.803	0.900	0.833
MemGPT	0.786	0.861	0.793	0.630	0.600	0.600
RAG	0.776	0.875	0.793	0.622	0.467	0.600

Job 45623ca0, ~6.2 h, zero errors.

Reproduce: LONGMEMEVAL_TYPES=all bash scripts/run_longmemeval_prod.sh

What's new

• v4: per-type routing (KU, multi-session, temporal, abstention, preference)
• v5: temporal timeline precompute, abstention anchor short-circuit, graph-vector multi-session hybrid
• Paper + benchmark docs updated with canonical 500 Q scores

Docs

• docs/LONGMEMEVAL.md
• docs/PAPER.pdf
• docs/ARXIV_SUBMISSION.md

v0.2.0 — LongMemEval transcript-first

ACME v0.2.0-longmemeval

Date: 2026-06-25
API: https://acme-api.blackgrass-3076f328.westeurope.azurecontainerapps.io
Image: acme-api:longmemeval-transcript
Repo: https://github.com/KamilBourouiba/ACME

MemoryBench v3 (13 scenarios, prod)

System	Overall
ACME	0.925
RAG	0.487

Job 3b31e5e3, 24 June 2026.

LongMemEval oracle (500 Q, prod, June 2026)

System	Overall	KU	Multi-session	Temporal
ACME	0.804	0.897	0.744	0.684
MemGPT	0.780	0.872	0.752	0.609
RAG	0.776	0.859	0.789	0.602

Jobs 705eb2ff (78 KU) + 26e288da (422 other types). Transcript-first path for LongMemEval adapter.

Reproduce: ./azure/configure-premium-ingress.sh && bash scripts/run_longmemeval_prod.sh

What's new

• Official LongMemEval adapter (acme/evaluation/longmemeval.py) + async prod API
• Transcript-first answering (newest-first sessions) for knowledge-update resolution
• Belief demotion on superseded sources during KU ingest
• Paper v1.3: full LongMemEval tables + limitations (multi-session, abstention, routing)

Docs

• docs/LONGMEMEVAL.md
• docs/BENCHMARK_RESULTS.md
• docs/PAPER.pdf — ./scripts/export_paper_pdf.sh
• docs/ARXIV_SUBMISSION.md — checklist (not yet submitted)

ACME v0.1.0-azure

ACME v0.1.0-azure

Date: 2026-06-24
API: https://acme-api.blackgrass-3076f328.westeurope.azurecontainerapps.io
Revision: acme-api--pgflex-* (Azure Postgres Flexible + pgvector, francecentral)

Highlights

• MemoryBench v2: 10 scenarios, sandbox-isolated (Postgres + Neo4j)
• Production scores (GPT-4.1, Azure OpenAI embeddings 256D):
  • ACME overall 0.925 (retention 0.980, groundedness 1.000, belief 0.700)
  • RAG 0.482 · MemGPT 0.482 · LangGraph 0.485
• Azure OpenAI text-embedding-3-small deployment + pgvector on Flexible Server
• Persisted benchmark runs, CI gates (BENCHMARK_MIN_OVERALL=0.85)
• Consolidation worker (6h cron), premium ingress (30 min timeout)
• Compare async endpoint + export

Deploy

./azure/deploy.sh
./azure/embedding-deploy.sh
FALLBACK_LOCATION=francecentral ./azure/postgres-flexible.sh
./azure/configure-premium-ingress.sh
./azure/consolidation-job.sh

Tests

make test
BENCHMARK_MIN_OVERALL=0.85 BENCHMARK_MIN_BELIEF_QUALITY=0.55 pytest tests/test_benchmark_gate.py -q

Docs

• README.md
• docs/PAPER.md
• docs/BASELINES.md
• docs/ARXIV_SUBMISSION.md