Skip to content

Benchmarks

Jump to bottom
antonyrag edited this page Aug 4, 2026 · 1 revision

Benchmarks

All numbers below come from a real, timed run against real infrastructure — never projected or estimated. Full methodology, raw results, and the runnable script live in packages/ragleap-rag/benchmark/.

  • Run date (UTC): 2026-08-03
  • ragleap-rag version: 0.11.1
  • Corpus: 40 synthetic documents (self-contained, no copyright concerns)
  • Queries per config: 15

Ingestion throughput

Config Docs ingested Time (s) Docs/sec
gemini_pgvector 40/40 36.3 1.10
gemini_faiss 40/40 35.4 1.13
ollama_local 40/40 112.7 0.36

Query latency

Config Queries run P50 (ms) P95 (ms) Mean (ms)
gemini_pgvector 8 of 15 (partial — see caveats) 3934 6530 4253
gemini_faiss 0 of 15 (see caveats) — — —
ollama_local 15 of 15 67028 84883 47244

Cost per query

Config Avg cost (USD) Pricing available?
gemini_pgvector $0.0033 Yes
gemini_faiss — No data (see caveats)
ollama_local — No pricing table entry (local, effectively $0)

Honest caveats — read before drawing conclusions

  • Gemini free tier has two separate caps: 5 requests/minute and 20 generation requests/day. The daily cap was the binding constraint — it was exhausted partway through gemini_pgvector's queries, before gemini_faiss could run any at all. The 8 gemini_pgvector numbers above are real, but a partial sample (n=8, not 15).
  • gemini_faiss has zero completed queries — not because FAISS failed (ingestion succeeded 40/40), but because the shared daily quota ran out before it got a turn. A follow-up run is needed once quota resets or a paid tier is used.
  • Grok/xAI was excluded entirely — the API key available during this run failed authentication (confirmed independent of ragleap-rag via a raw API call).
  • Ollama's latency (P50 ~67s) is real, not a bug — `qwen2.5:0.5b running full retrieval + generation on CPU-only hardware, no GPU. This is the genuine tradeoff of fully local, zero-cost inference on modest hardware.

Reproduce this yourself

cd packages/ragleap-rag/benchmark
export GEMINI_API_KEY="..."
python3 run_benchmark.py
python3 render_benchmarks_md.py

Real numbers only get published here after a real run — see Phase 6 of ROADMAP.md for the standard this follows.

Clone this wiki locally