v0.9.0 — Queue hardening, robustness and batch-2 serving

v0.9 hardens the server under load: requests beyond the server's capacity now get clear, immediate answers instead of waiting forever, identical concurrent requests are computed once, shutdown drains what is in flight, a broken compute path fails loudly instead of returning poisoned numbers — and short-input embedding throughput can be raised ~25% with an opt-in batch-2 artifact.

Admission control and queue timeout

Inference requests pass admission control. [server] max_pending_requests (default 500, 0 = unlimited) caps how many are accepted at once — running and waiting combined — and the excess is rejected immediately with 429 and a Retry-After header (measured rejection time ~0.09 s even under a full queue; failed authentication never consumes a slot). A request that waits longer than [server] queue_timeout (default 600 s, 0 = disabled) for its turn is answered with 503 and Retry-After. A request whose inference has started is never interrupted: the timeout only bounds waiting.

[server]
# max_pending_requests = 500   # 0 = unlimited
# queue_timeout = 600          # seconds; 0 = no timeout

The defaults are sized well above real ingestion bursts: a document-ingestion style burst of 33 concurrent requests (448k tokens) completes all-200 with default settings.

Request coalescing

Identical concurrent requests — same model, same input — are served from a single computation ([server] coalesce_requests, default on). Eight identical requests fired at once completed in 2.75 s against 20.2 s with coalescing off (~7.3x) on a development Mac, with byte-identical responses. Only requests overlapping in time are merged; nothing is cached.

Graceful shutdown

On SIGTERM/Ctrl-C the server stops accepting connections and finishes every accepted request before exiting (measured under load: all 32 in-flight requests completed, clean exit). [server] graceful_shutdown_timeout bounds that wait; unset, the server waits as long as it takes.

Batch-2 serving (opt-in)

Compiling a second artifact with eeane compile <model> --buckets <S> --batch 2 lets the server predict two same-bucket inputs of one embedding request per Neural Engine call. A single 200-short-input request went from 5,792 to 7,234 tokens/s (+24.9%) on a development Mac, with per-text agreement between the batch-1 and batch-2 paths at cosine ≥ 0.9999999. Grouping never crosses requests and never reorders results; configurations without a batch-2 artifact behave exactly as before. Id-only [[models]] entries pick a compiled batch-2 artifact up from the cache automatically. The compiled-model record now keeps the batch-1 and batch-2 families separate, fixing a latent issue where a --batch 2 compile could overwrite the batch-1 record.

Non-finite output guard

A model output containing NaN or infinite values is now answered with a clear 500 error and an error log naming the model and bucket, instead of being passed on to poison a similarity search. This matters because eeANE requires the Apple Neural Engine: CPU-only execution of the compiled models was measured to produce non-finite outputs across all supported architectures, so this error usually means the ANE was not available to the process. The READMEs now state this under known limitations.

Read-ahead tokenization

A multi-input request tokenizes its next input on a worker thread while the current one is being predicted. Measured effect is modest (+1.6% on a concurrent burst, within noise for single requests — tokenization is small next to prediction); results, ordering and accuracy are unchanged.

Load-test client

tools/load_test.py builds deterministic, seeded request sets from the bundled corpus and fires them concurrently: an ingestion-style burst, a short-input sweep, an admission-limit probe (--expect-429) and a duplicate-request probe that checks responses are byte-identical.

Behavior changes

• Admission cap (500) and queue timeout (600 s) are active by default; previously the queue was unbounded. Set max_pending_requests = 0 and queue_timeout = 0 to restore the old behavior.
• New 429/503 responses carry Retry-After; clients that treat any non-200 as fatal will surface these as errors instead of hanging.
• No API or configuration compatibility is broken: every v0.8 config file remains valid.

Verification: 811 tests pass, CI green; default-config accuracy verification passes end to end with a 36-document rerank median of 5.612 s, level with v0.8.