Repository navigation
CPU vs MPS Query Performance Study
Benchmark hardware: All performance measurements on this page were collected on an Apple M1 Max. Results on other systems will vary.
emailindex uses BAAI/bge-small-en-v1.5 to turn each semantic query into one 384-dimensional vector. This study compares query embedding on the Apple Silicon CPU with Apple's Metal Performance Shaders (MPS) GPU backend after the semantic retriever was migrated to sqlite-vec exact cosine MATCH.
The result is nuanced:
- MPS is measurably faster and uses less CPU for the embedding step.
- Query embedding is only a small part of a complete search.
- sqlite-vec, SQLite joins, FTS5, and Reciprocal Rank Fusion still execute on CPU.
- Warm MPS saves approximately 3-6 ms on ordinary semantic searches and about 2-4 ms on hybrid searches.
- MPS has a substantially slower first encode and slightly higher model-load time.
- CPU remains the preferred MCP default for occasional personal-archive searches. MPS becomes reasonable for sustained semantic-query traffic where reducing CPU contention matters.
- Apple M1 Max
- macOS
- MPS available: yes
- CUDA available: no
- SQLite
3.53.4 - sqlite-vec
v0.1.9 - 81,264 email vectors
- Embedding model:
BAAI/bge-small-en-v1.5 - Vector dimensions: 384, float32
- Retrieval: stable sqlite-vec exact cosine KNN (
vec0 MATCH) - Query result limit: 50
- Each end-to-end comparison used five runs; reported warm latency is the median of runs 2-5
| Component | CPU | MPS | Interpretation |
|---|---|---|---|
| Model load | 3.61 s | 3.89 s | CPU loads about 280 ms faster |
| First query encode | 26.25 ms | 201.13 ms | MPS pays a large one-time initialization cost |
| Warm query encode median | 17.09 ms | 13.95 ms | MPS saves about 3.14 ms per warm encode |
MPS is approximately 18% faster for a warmed single-query embedding. The first MPS encode is roughly 175 ms slower, however, so short-lived MCP processes do not benefit immediately.
The following measurements include query embedding, sqlite-vec retrieval, SQLite joins, response construction, and—where applicable—FTS5 and RRF fusion.
| Query | CPU warm median | MPS warm median | MPS change |
|---|---|---|---|
| Broad semantic search, limit 50 | 84.56 ms | 78.53 ms | 6.03 ms faster (7.7%) |
| Semantic search + date filter | 130.15 ms | 127.23 ms | 2.92 ms faster (2.3%) |
| Semantic search + attachments | 69.99 ms | 64.50 ms | 5.49 ms faster (8.5%) |
Hybrid RRF: project
|
390.53 ms | 388.13 ms | 2.40 ms faster (0.6%) |
Hybrid RRF: meeting
|
391.61 ms | 387.63 ms | 3.98 ms faster (1.0%) |
| Tag substring scalar fallback | 6,272.68 ms | 6,230.54 ms | 42.14 ms faster (0.7%) |
MPS has its largest percentage benefit on ordinary semantic searches, where query embedding represents a meaningful portion of total latency. It barely changes hybrid and scalar-fallback latency because embedding is a small fraction of those workloads.
CPU and MPS produced equivalent retrieval results in every tested workload:
| Query class | Result overlap | Ordering |
|---|---|---|
| Broad semantic | 50/50 | Identical |
| Semantic + date | 50/50 | Identical |
| Semantic + attachments | 50/50 | Identical |
Hybrid project
|
50/50 | Identical |
Hybrid meeting
|
50/50 | Identical |
| Tag fallback | 50/50 | Identical |
No query returned an error. Device selection did not alter the final top-50 membership or ordering in this test set.
To measure resource impact rather than isolated latency, each model was warmed before running:
- 150 consecutive query encodes
- 40 complete broad semantic searches
- Process CPU time and sampled process utilization
On macOS, 100% process CPU represents approximately one fully utilized CPU core.
| Measurement | CPU model | MPS model | MPS effect |
|---|---|---|---|
| Wall time per encode | 26.51 ms | 17.58 ms | 34% faster |
| CPU time per encode | 31.08 ms | 11.88 ms | 62% less CPU time |
| Average CPU cores used | 1.17 | 0.68 | About 0.49 core saved |
| Sampled process CPU mean | 109.6% | 63.6% | 46 percentage points lower |
| Sampled process CPU peak | 117.6% | 68.6% | 49 percentage points lower |
| Peak process RSS | 561 MB | 597 MB | MPS used about 35 MB more process RAM |
MPS meaningfully offloads the embedding operation. CPU embedding briefly consumes roughly one CPU core, while MPS reduces host CPU demand to roughly two-thirds of a core.
| Measurement | CPU model | MPS model | MPS effect |
|---|---|---|---|
| Wall time per search | 106.33 ms | 98.10 ms | 8% faster |
| CPU time per search | 113.95 ms | 93.27 ms | 18% less CPU time |
| Average CPU cores used | 1.07 | 0.95 | About 0.12 core saved |
| Sampled process CPU mean | 106.2% | 91.9% | 14 percentage points lower |
| Sampled process CPU peak | 108.6% | 97.8% | 11 percentage points lower |
| Peak process RSS | 577 MB | 599 MB | MPS used about 22 MB more process RAM |
The resource benefit becomes much smaller for the complete search because only query encoding moves to the GPU. sqlite-vec exact KNN, SQLite filtering and joins, FTS5, and Python RRF remain CPU work.
Using the measured startup and encoding numbers:
- MPS model load starts approximately 278 ms behind CPU.
- MPS first encode adds approximately 175 ms of additional cold-start cost.
- Each warm MPS encode saves approximately 3.14 ms.
(278 ms + 175 ms) / 3.14 ms per search = approximately 144 warm searches
MPS becomes faster cumulatively at approximately the 146th semantic or hybrid query per MCP server process.
| Semantic queries per process | Approximate cumulative outcome |
|---|---|
| 1 | CPU clearly faster because of MPS cold start |
| 50 | CPU still ahead overall |
| 100 | CPU ahead by about 140 ms |
| 146 | Approximately break-even |
| 1,000 | MPS ahead by approximately 2.7 seconds total |
Keyword-only searches do not count toward break-even because they do not generate query embeddings.
If the model is already loaded but neither device has performed its first encode, MPS recovers its first-encode penalty after approximately 57 semantic searches. If both models are already fully warmed, MPS is immediately about 3 ms faster per query.
No. CPU is the better default because:
- It has much lower first-query latency.
- It avoids keeping the embedding model resident on the GPU.
- Warm end-to-end gains are generally only 3-6 ms.
- The CPU spike lasts only about 20-30 ms and consumes roughly one core, not the entire system.
- Personal MCP use is unlikely to sustain enough semantic requests for the cumulative savings to matter.
Possibly. MPS is reasonable when:
- The MCP server remains alive for hundreds of semantic searches.
- Multiple clients or agents generate sustained semantic traffic.
- Reducing CPU contention matters more than cold-start latency.
- Other workloads on the Mac need CPU capacity while searches run.
Under sustained encoding load, MPS reduces CPU time by 62%. For complete semantic searches, however, the reduction is only 18% because the remaining retrieval stack is CPU-bound.
Yes. This conclusion is different from query-time search. Ingestion embeds many documents in batches, allowing MPS to remain busy and amortize launch overhead. Separate local benchmarks showed materially higher throughput for MPS during bulk ingestion, especially with larger batches. emailindex therefore retains automatic MPS/CUDA selection for ingestion while configuring MCP query embeddings on CPU.
Keep CPU as the MCP query default for the current personal email archive workload.
Keep MPS enabled for bulk ingestion, where batching produces a substantial payoff.
Consider an opt-in MCP MPS setting if the server evolves into a long-lived, multi-client service handling hundreds or thousands of semantic searches per process. Even in that scenario, expect only modest end-to-end latency improvement: the GPU accelerates the single query embedding, while vector retrieval and hybrid fusion remain CPU-side.
The highest-value remaining performance work is not GPU offload. It is replacing slow scalar fallback behavior for tag, project, sender-name, and recipient substring filters with filter structures that can constrain exact KNN efficiently.