Scoring LongMemEval-S retrieval rankings with a standalone -score driver (no tool in the loop)

[vshulcz]

There is now a standalone -score mode in scripts/longmemeval that takes a JSON file mapping each question_id to the session IDs your system ranked ({"gpt4_e5c7d0f1": ["answer_session_3", ...]}) and prints the retrieval table (hit@1, hit@5, hit@10, hit@20, MRR, and the dataset's official per-evidence recall) using the exact same arithmetic: no index, no answering LLM, and no external tool in the loop. It runs in seconds against longmemeval_s_cleaned.json:

go run ./scripts/longmemeval -data longmemeval_s.json -score yours.json -out yours-result.json

Format and rules: docs/benchmarks/SUBMISSION.md.

Disclosure: I maintain deja-vu, which has a lexical retrieval row in that table (88.9% hit@1 / 97.4% hit@5 on the 470-question -skip-abs set, and 88.2% hit@1 / 97.2% hit@5 on all 500 questions, committed in docs/benchmarks/). That is why the standalone scorer is the useful part rather than our own row — a Mem0 retrieval run scored by the same driver on the same 500 questions (separate from end-to-end LLM-judged QA) makes it possible for anyone comparing systems to check both artifacts directly.