Question: remote (proxy) embedding throughput during autoindex is latency-bound — one file per request, 4 in flight

[thomas-villani]

Context. Measuring xerj autoindex with --embed-mode proxy against OpenRouter (sentence-transformers/all-minilm-l6-v2), v1.0.0-rc.85, through a logging pass-through. (Agent-written; every number below is from a run.)

What we observed (82-file Python repo, 1,938 chunks):

• 83 embedding requests — about one per file. Mean 23 inputs per request; the largest was one file's 192 chunks.
• Never more than 4 requests in flight (the proxy semaphore: max_concurrent: 4, engine/crates/xerj-engine/src/index.rs:35612, xerj-ai/src/embed.rs:154), with about 3.8 in flight on average. That is about 38 inputs/s.
• Raising onnx_scheduling_window to 256, with or without neural_window_concurrency = 4, gave the identical 83 requests. The engine does batch across documents within one _bulk (20 docs → 1 request), but autoindex's graph path sends one file per _bulk, and windows don't span bulks.
• The provider's latency is mostly fixed cost (direct calls, 512-char code chunks): 64 inputs 0.85 s (75/s), 256 inputs 1.68 s (152/s), 1,024 inputs 3.96 s (259/s).

So a remote embedder runs at a fraction of what the provider can take, and there's no setting to change that: batch_size is dead (separate issue, see #1190) and the concurrency is hard-coded.

Question. Is this worth changing, and if so, which direction would you prefer?

1. Make the proxy's concurrency configurable (smallest change).
2. Let the proxy embedder coalesce windows from concurrent bulks into larger requests.
3. Let autoindex pack several small files into one _bulk when the backend is remote.

Or is per-file bulking load-bearing (journaling, back-pressure) in a way that rules out 3? I'm happy to measure whichever option you'd consider, but didn't want to propose an indexing change without asking.