Bench'd Independent Benchmark Results for CrewAI Memory

[1vanbui]

Hi team! We're Bench'd — an independent benchmark platform for AI memory systems.

We ran CrewAI Memory through LongMemEval (500 questions).

Results

Benchmark	Score	Questions	Status
LongMemEval v1.0	46.0%	500	Verified

Per-dimension: Recall 74.4% · Temporal 35.5% · Reasoning 29.3%

Full results: benchd.ai/system/crewai-memory

The LLM baseline (no memory) scores 57.6%. CrewAI's recall is strong at 74.4% but temporal and reasoning pull the overall score down.

Run it yourself

```bash
pip install benchd-harness
benchd run -a crewai-memory -b longmemeval-v1 --judge --key ./keys/private.key
```

---

Bench'd — the neutral benchmark standard for AI memory systems.

[0xbrainkid]

These results would be more useful for agent builders if the benchmark output preserved provenance at the memory-item level, not only aggregate scores.

The aggregate split is already interesting: high recall, weaker temporal/reasoning. The next question is which memory entries caused the failures and whether a downstream agent can inspect that path later.

For multi-agent systems, I would want benchmark artifacts to include:

• memory entry/source identity,
• task/run id,
• timestamp or temporal scope,
• retrieved item ids/hashes,
• judge decision and reason code,
• reproducible receipt for the benchmark run.

That turns “CrewAI Memory scored 46%” into evidence another builder can audit or compare across systems.

For AgentFolio/SATP-style reputation, benchmark receipts like that are useful because agent memory quality becomes part of an agent’s portable track record. A score helps ranking; inspectable evidence helps onboarding decisions.

[1vanbui]

@0xbrainkid Spot on — aggregate scores are the ranking layer, but item-level provenance is the engineering layer. Multi-agent systems break down when downstream agents can't verify the temporal scope and source identity of the context they're handed.

This is exactly what we're building into the next version of the Bench'd trace schema. Every benchmark item will produce a structured provenance path:

source event → memory entry → retrieved item → answer → judge decision → signed receipt

Specifically, the trace schema is being extended with:

• Retrieved item IDs/hashes — which specific memory entries were pulled for each query
• Source event identity — which conversation turn or agent action created the memory
• Task/run/agent ID — for multi-agent systems, who created it and in what context
• Temporal scope — when the memory was valid, and whether it was superseded
• Judge reason code — standardized failure taxonomy (retrieval_miss, stale_memory, temporal_confusion, cross_agent_contamination, etc.)
• Signed receipt reference — every run is already Ed25519 signed with full traces

So "CrewAI Memory scored 46%" becomes: "CrewAI failed 54% of questions — 18% retrieval misses, 12% stale memory, 9% wrong entity — with full per-item audit trail in the signed receipt."

For multi-agent specifically, we're scoping a dedicated track that tests memory across agent handoffs: Agent A creates memory → Agent B uses it → Agent C audits why. The scoring dimensions are different: Handoff Recall, Source Attribution, Cross-Agent Contamination, Provenance Integrity.

The AgentFolio/SATP angle is the right framing — Bench'd receipts should be machine-consumable evidence that reputation systems can verify and build on. We're planning a benchd-verify library so any downstream system can programmatically fetch, verify, and consume the structured receipt.

The harness is open source: github.com/benchdai/harness. Would be interested in a longer conversation about what the receipt schema needs to look like for agent reputation systems to consume it directly.

Full results + methodology: benchd.ai

[javierfajardo85-rgb]

Thanks for sharing these results! For deeper analysis of the memory retrieval failures, tools at aiaam.xyz might help — specifically Haystack (pip install haystack-ai) for building evaluation pipelines against LongMemEval, and Firecrawl for scraping additional benchmark data. Isolating which question types hit that 46% could point to specific memory indexing issues worth fixing.

[norika1207-lab]

One practical way to make this easier to evaluate is to turn the report into a small contract test around the runtime boundary, not only the user-facing symptom.

For independent benchmark results for CrewAI Memory, I would check three things: first, the failing input is preserved as a minimal fixture; second, the SDK emits the same observable event/result shape before and after the fix except for the intended change; third, the trace or serialized session contains enough metadata to explain what happened after a resume or retry.

The part that matters for agent systems is durability: if a tool call, memory write, handoff, or structured-output step fails, downstream code should not have to infer state from natural-language output. A focused regression test around that boundary would make the issue much easier to close with confidence.