Benchmarks

Benchmarks

The benchmarks/ directory in the repository measures Noema against two baselines on a large, reproducible, synthetic corpus. Everything is generated from a seed, so no dataset download and no manual labeling is needed, and the corpus scales to any size.

What is compared

System	What it does
noema	KnowledgeBase: vector search over facts, graph expansion, notation under a token budget
naive_chunk_rag	Plain RAG. Documents are split into fixed-size chunks (60 tokens by default) on sentence boundaries, embedded, and the top-k chunks are returned verbatim
full_context	Stuffs the entire corpus into the context for every question

All three are built from the same documents with the same embedding model. full_context bounds the comparison: it cannot miss an answer, and it costs the most tokens.

The synthetic corpus

Each fictional company contributes two documents and four questions.

Profile document:

Nova Robotics was founded by Elena Marsh in 2011. Nova Robotics is headquartered in Pittsburgh. Priya Anand is the CEO of Nova Robotics. Nova Robotics builds warehouse robots.

Acquisition document:

Nova Robotics acquired Delta Labs in 2018.

Question	Type
Who is the CEO of Nova Robotics?	single-hop
Who founded Nova Robotics?	single-hop
Where is Nova Robotics headquartered?	single-hop
Who founded the company that acquired Delta Labs?	multi-hop

The multi-hop question can only be answered by joining a fact in the acquisition document with a fact in the profile document, which is a different text. This is where a graph can beat chunk retrieval. --num-companies 500 means 1,000 documents and 2,000 questions.

How answers are scored

A query is a hit when the retrieved text contains the known answer and the question's subject company, compared case-insensitively. The subject is required because many answers repeat across companies. With only a few cities in the corpus, text about the wrong company can contain the right city.

This is a retrieval proxy. It checks that the context contains what a model would need. It does not run an LLM or grade a generated answer. It also cannot tell whether the answer is stated as a fact about the right company in the retrieved text, only that both strings are present.

Running it

From a clone of the repository:

uv sync
uv run benchmarks/run_benchmark.py --num-companies 50
uv run benchmarks/run_benchmark.py -n 500 -o results/large

Start small to confirm everything works, then scale up. The full-context baseline sends the whole corpus for every question, so large runs take longer, and its tokens are counted only once per run.

Option	Default	Meaning
--num-companies, -n	50	Corpus scale
--seed	42	Seed for a reproducible corpus
--max-tokens	200	Token budget for Noema and the chunk baseline
--top-k	5	Vector search candidates per query
--hops	1	Noema graph expansion distance
--chunk-tokens	60	Chunk size for the chunk baseline
--batch-size	100	Documents canonicalized together
--spacy-model	en_core_web_sm	spaCy pipeline
--embedding-model	all-MiniLM-L6-v2	Embedding model
--output-dir, -o	benchmark_results	Where results are saved

Reading the output

The run prints a table with, for each system:

• avg tokens: mean retrieved tokens per question, and for the baselines how many fewer or more tokens Noema used
• hit rate: overall, single-hop and multi-hop
• avg latency (ms): mean retrieval time per question

It also saves two files in the output directory:

• results.csv: one row per system and question (system, question, answer, hop type, token count, hit, latency, and the retrieved text)
• results.json: the same rows plus the per-system summaries

Saved retrieved text is cut to 300 characters, because the full-context baseline would otherwise repeat the whole corpus on every row.

Interpreting results

• Compare hit rate and tokens together. A system that returns more text hits more often. The useful result is a high hit rate at a low token count.
• Expect full_context to hit on nearly everything at the highest token cost.
• Look at multi-hop separately. It is the case chunk retrieval is weakest at, and the one hops affects.
• Results depend on your machine, models, corpus size and settings. Run it yourself and keep the seed fixed when comparing changes.
• The corpus is synthetic and its sentences are simple. It tests the retrieval design, not extraction quality on messy real text. Validate on your own documents too.

Quick check on your own data

noema benchmark compares token cost on your own bundle and source files, without accuracy scoring. See CLI Reference.

Related

Retrieval, Core Concepts