Conversation
3-way comparison of ember vs chromadb vs pgvector for vector similarity search. benchmarks insert throughput, query latency (p50/p95/p99), query throughput, and memory usage with identical HNSW parameters (M=16, ef_construction=64, cosine similarity). includes: - bench-vector.sh: orchestrator with --ember-only, --quick, --sift flags - bench-vector.py: python harness with client wrappers for all 3 systems - SIFT1M dataset loader for recall accuracy testing - setup-vm-vector.sh: additional VM deps (docker, python, images) - README updates with vector benchmark section
chromadb deprecated the /api/v1/ endpoints. the heartbeat health check now hits /api/v2/heartbeat instead.
psycopg2 sends python lists as numeric[], which pgvector's <=>
operator doesn't accept. format vectors as pgvector string
literals ("[0.1,0.2,0.3]") with explicit ::vector cast.
docker's default shm size is 64MB, but postgres shared_buffers is set to 256MB. add --shm-size=512m to avoid "No space left on device" when building HNSW indexes on larger datasets.
100k random vectors, 128-dim, cosine, k=10 kNN search. ember wins on query throughput (3.2x chromadb, 1.4x pgvector) and memory usage (4-6x less). insert throughput is lower due to per-vector RESP protocol overhead.
formatting changes across concurrent.rs, concurrent_handler.rs, and connection.rs. suppress dead_code warning on cmd_raw test helper (intended for future protobuf binary tests).
the 1M-vector HNSW index exceeds 16GB RAM during construction on c2-standard-8. needs c2-standard-16 or higher.
kacy
added a commit
that referenced
this pull request
Feb 19, 2026
* add vector similarity benchmark suite
3-way comparison of ember vs chromadb vs pgvector for vector
similarity search. benchmarks insert throughput, query latency
(p50/p95/p99), query throughput, and memory usage with identical
HNSW parameters (M=16, ef_construction=64, cosine similarity).
includes:
- bench-vector.sh: orchestrator with --ember-only, --quick, --sift flags
- bench-vector.py: python harness with client wrappers for all 3 systems
- SIFT1M dataset loader for recall accuracy testing
- setup-vm-vector.sh: additional VM deps (docker, python, images)
- README updates with vector benchmark section
* fix chromadb health check to use v2 API
chromadb deprecated the /api/v1/ endpoints. the heartbeat
health check now hits /api/v2/heartbeat instead.
* fix pgvector type casting for vector columns
psycopg2 sends python lists as numeric[], which pgvector's <=>
operator doesn't accept. format vectors as pgvector string
literals ("[0.1,0.2,0.3]") with explicit ::vector cast.
* fix pgvector docker container shared memory limit
docker's default shm size is 64MB, but postgres shared_buffers
is set to 256MB. add --shm-size=512m to avoid "No space left
on device" when building HNSW indexes on larger datasets.
* update vector benchmark results from gcp c2-standard-8
100k random vectors, 128-dim, cosine, k=10 kNN search.
ember wins on query throughput (3.2x chromadb, 1.4x pgvector)
and memory usage (4-6x less). insert throughput is lower due
to per-vector RESP protocol overhead.
* run cargo fmt and fix clippy warning
formatting changes across concurrent.rs, concurrent_handler.rs,
and connection.rs. suppress dead_code warning on cmd_raw test
helper (intended for future protobuf binary tests).
* note sift1m requires larger VM for benchmarking
the 1M-vector HNSW index exceeds 16GB RAM during construction
on c2-standard-8. needs c2-standard-16 or higher.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
summary
3-way benchmark comparing ember's vector similarity search (VADD/VSIM) against chromadb and pgvector. uses identical HNSW parameters across all systems for a fair comparison.
what's measured:
fairness: all systems use M=16, ef_construction=64, cosine similarity. same test vectors generated once and fed to each system sequentially (no CPU contention).
what was tested
bench/bench-vector.sh --quick --ember-onlyruns end-to-end on macOS (no docker needed)--features vectordesign considerations
.bench-venv/when pip install would fail (PEP 668 on modern macOS/linux). avoids requiring the user to manually manage environments.--siftflag downloads the 160MB dataset on first use. random vector mode is the default since it needs no external data.