RagContextBenchmark

Measures RavenDB's retrieval pipeline — chunking + the Context Prefix feature (withContextPrefix) + vector search — on public retrieval benchmarks, so RavenDB's numbers can be placed next to published baseline/contextualized rows.

It compares two arms on the same documents and embedding model:

• Baseline — embed chunk/document content as-is.
• ContextPrefix — prepend static, deterministic document context (BEIR/LoCo title, or a code file's header line) to each chunk before embedding, via RavenDB's withContextPrefix. No LLM-generated context.

Absolute scores will differ from published tables (different embedding model); the Baseline → ContextPrefix delta within one model/dataset is the signal.

Prerequisites

1. A running RavenDB 7.2 server (unsecured dev) at http://127.0.0.1:8080. The server must support the embeddings-generation + withContextPrefix script API.
2. An OpenAI API key in an environment variable the harness reads:
   • RAVEN_AI_INTEGRATION_OPENAI_API_KEY (preferred) or RAVEN_AI_INTEGRATION_OPEN_AI_KEY.
   • The harness creates the per-database OpenAI embeddings connection string from this key. (AI connection strings are per-database, and the harness creates a fresh database per dataset, so it must create the connection string itself.)
3. .NET 10 SDK.

Run

# from the repo root
dotnet run --project src/RagContextBenchmark -- --datasets beir-scifact

# multiple datasets
dotnet run --project src/RagContextBenchmark -- --datasets beir-scifact,beir-nfcorpus,beir-trec-covid

# verify the server round-trip only (creates rcb-smoke, one query)
dotnet run --project src/RagContextBenchmark -- --smoke

Providing the key in a fresh shell

PowerShell:

$env:RAVEN_AI_INTEGRATION_OPENAI_API_KEY = "sk-..."
dotnet run --project src/RagContextBenchmark -- --datasets beir-scifact

Git Bash:

export RAVEN_AI_INTEGRATION_OPENAI_API_KEY="sk-..."
dotnet run --project src/RagContextBenchmark -- --datasets beir-scifact

Options

Flag	Default	Meaning
--server	http://127.0.0.1:8080	RavenDB server URL
--connection	openai-embeddings	connection-string name created per DB
--model	text-embedding-3-small	OpenAI embedding model
--datasets	beir-scifact	comma-separated dataset ids (see below)
--data	./data	download/cache dir
--results	./results	output dir (.txt table + .csv)
--reuse	off	keep existing DB + embeddings instead of drop-and-recreate

Datasets

Each dataset uses its own database (rcb-<dataset>), created fresh each run (drop-and-recreate unless --reuse). Both arms live in the same DB as two embeddings tasks over the BenchDocs collection.

id	source	metric	prefix	notes
beir-scifact / -nfcorpus / -trec-covid	BEIR	nDCG@10	document title	server chunks (256 tok); doc-level qrels
beirgen-<same>	BEIR document-level	nDCG@10	document title	whole-doc, no chunking
loco-<task>	LoCoV1 long-context (HuggingFace)	nDCG@10	task name (constant)	long docs; server chunks (2048 tok)
longembed-<subset>	LongEmbed (HuggingFace)	nDCG@10	subset name (constant)	long docs; server chunks (2048 tok)

Published reference numbers

• Jina late-chunking (jina-embeddings-v2-small, nDCG@10): SciFact naive→late 64.20→66.10; NFCorpus 23.46→29.98.

These used different embedding models than this harness (OpenAI), so compare deltas, not absolute values.

Design

See docs (in the planning repo): IDatasetProvider normalizes each benchmark into a DatasetBundle; BenchmarkStore wraps the RavenDB client (DB lifecycle, per-arm embeddings tasks, wait-for-embeddings, vector query); pure Metrics/Scoring compute Pass@k and nDCG@10 (chunks are mapped back to source docs before nDCG).