Skip to content

Releases

Run full-20260825: raw records for the six-engine post

Run full-20260825: raw records for the six-engine post Latest
Latest

Choose a tag to compare

View all tags
@EvgeniyPatlan EvgeniyPatlan released this 31 Oct 23:09
run-full-20260825
bb10d92

Raw data for the blog post "Benchmarking vector search in six database engines".

Run full-20260825, profile tuned-complete, tuned resource pass, dbpedia-openai-1000k-angular (990,000 × 1,536, cosine), M = 16, k = 10. Framework commit bb10d92 (this tag). Engines: MariaDB 11.8.8, MariaDB 12.3.2, AliSQL 8.0.44-2, pgvector 0.8.6, Valkey 9.1.1 with valkey-search, Percona Search for MongoDB 1.70.3-1 on PSMDB 8.3.8-2.

What is in the bundle

  • report/records.jsonl: every measurement, one JSON object per line, including all 112 recall/throughput points
  • ops-*.jsonl: raw ops-harness records (build, concurrency, filtered, churn)
  • mem-*.jsonl: server memory and CPU timeseries, 0.25 s
  • logs/: per-phase server and harness logs
  • run-manifest.json: host, resolved settings, image IDs, source commits
  • report/report.html: the report as the framework generated it at the time
  • README.txt: how to read it

The per-configuration ann-benchmarks HDF5 files are not included; the recall points they produced are in report/records.jsonl.

Known problems in this run

The report in this bundle is the original output and contains errors that the post has since corrected. In particular:

  • Concurrent clients were threads in one Python interpreter; the multi-client numbers for MariaDB 11.8, MariaDB 12.3 and AliSQL reflect a client-side limit and are not valid server measurements.
  • The Percona Search row of the recall-floor table comes from the unquantized index; the scalar-quantized index used elsewhere peaks at recall 0.892.
  • MariaDB 12.3 was built with -march=native; the other source-built engines used x86-64-v3.
  • Memory budgets differed per engine; the manifest records only one engine's resolved values.
  • The manifest does not list the ann phases that produced the M = 16 recall files.
  • Concurrency and churn ran at a fixed ef_search of 100, not at equal recall.

The framework fixes for these are tracked in pull requests on the development fork, and the next run will re-measure every engine under the new measurement versions.

Changes made for publication

The host's internal domain name was shortened to beast-node2 in run-manifest.json and README.txt. Nothing else was altered.

SHA-256 of the bundle: 837abb4274276a0b40bf53dd2af0ad7a98766dde0fea3d8c5c1a07dd7c674417

Assets 4
Loading