v2.13.0 — Client-scoped collections (RAG multi-tenancy)

InferHub v2.13.0 — Client-scoped collections (RAG multi-tenancy)

v2.7 gave an API key an identity, a budget and a bill. RAG collections stayed fleet-global: any
client key that could ingest could, in principle, read or write any collection. For a single owner
that is fine. For an agency running one mesh behind several end-clients it is a data-isolation gap.
This release closes it — without a second store, a second truth, or a migration.

What changed

• Auth:Clients[].Collections scopes a key to a set of collections. Each entry is an exact
  collection name or a single trailing-* prefix (tenant-a-*). Absent or empty = every
  collection, which is what every key could do before v2.13, so an existing config is unchanged.
• The scope is enforced on every path that names a collection: document ingest, list, get,
  chunks, delete; POST /api/collections/{c}/search; the raw /api/vector/{c}/* data plane; and
  the X-InferHub-Retrieve inline-RAG header on both /api and /v1.
• Out of scope is 404, not 403 — byte-identical to a collection that does not exist, with the
  same sentence. The check runs before the store is consulted, so a name outside your scope reads
  the same whether or not it exists. A tenant is never told another tenant's collections are there.
• Provisioning is just ingesting. A scoped client posting a document to a collection inside its
  scope that doesn't exist yet creates it. No separate create ceremony, no admin round trip per
  tenant. Unscoped clients keep the pre-v2.13 contract: collections are an admin's to create.
• Admin keys stay fleet-wide. GET /api/admin/vector/collections gains a scopes block naming
  which clients can reach each collection — the view that makes a tenancy misconfiguration visible
  before a tenant finds it.

Invariants

• Rule 4 holds. This is an authorization filter over the one vector store, not a store per
  tenant. One collection namespace, one source of truth; a scope only decides which names a key may
  say.
• One place decides. CollectionAccessPolicy answers "may this key touch this collection", and
  the route groups carry it as a filter rather than each handler carrying a copy — the ingestion
  group alone has five routes, and the one that gets forgotten is the isolation hole.
• A scoped-out retrieval header is an error, not a passthrough. It deliberately does not go
  through Retrieval:OnMissing: answering without the context the caller asked for, silently, is the
  wrong failure on a tenancy boundary.
• Auto-provision does not guess the dimension. Phase 23 refused to auto-create collections for
  two reasons — it would guess the dimension, and it would route around the admin scope that owns
  collection lifecycle. The second dissolves for a client whose config names its scope; that list
  is the provisioning grant. The first doesn't, so creation is deferred until the first batch
  comes back embedded and the dimension is measured from the vectors. An embed that never
  succeeds therefore leaves no empty collection behind to be misread as provisioned.
• Rule 5 holds: zero new dependencies.
• Backwards compatible. A config with only Auth:ApiKeys, or clients with no Collections,
  behaves byte-identically to v2.12.

Upgrading

Nothing to do. Scoping is opt-in per client:

"Auth": {
  "Clients": [
    { "Id": "acme",   "Collections": ["acme-*", "shared-glossary"] },
    { "Id": "globex", "Collections": ["globex-docs"] },
    { "Id": "internal" }                          // no list = all collections, as before
  ]
}

569 tests green (17 Postgres-gated, skipped without a database).

v2.12.0 — Stable-node affinity + optional persistence

InferHub v2.12.0 — Stable-node affinity + optional persistence

Sticky routing keeps a conversation warm on the node that already loaded its model. Until now that
mapping was keyed to a SignalR connectionId and held only in memory, which had two costs: a
coordinator restart dropped every warm conversation, and — the non-obvious one — a connectionId
is not stable across a node's own reconnect, so a node bouncing its connection lost its warm
conversations even while it stayed up. This release re-keys affinity onto the stable node id and
then, opt-in, lets it survive a coordinator restart.

What changed

• Affinity keys on the stable nodeId, not the connection id. Record(conversationKey, nodeId);
  GetNodeFor returns a node id; the router resolves it to a live candidate at dispatch time. A
  sticky node that is disconnected, cordoned, or no longer holds the model is simply absent from the
  candidate set, so a stale hint is a clean miss and routing falls through to the best fresh node.
• A node reconnecting keeps its warm conversations. A disconnect no longer forgets affinity: the
  node comes back with a new connection id but the same node id, and its conversations resume on it.
  Node eviction doesn't forget either — an evicted node that re-registers picks its conversations
  back up, and the sliding window bounds the map for one that never returns. Only an explicit
  admin deregister — the operator saying a node is gone for good — forgets a node's affinity.
• Optional file persistence, off by default. Affinity:Persistence = none (default) | file.
  With file, the map is written to Affinity:DataDirectory as an append-log + periodic compacted
  snapshot — the same raw-store discipline the local vector store uses — and reloaded on startup with
  any entry past its sliding expiry dropped on load. Restart the coordinator and a still-fresh
  conversation stays pinned to its node with no cold reload.
• Observability. Live affinity entry count is on /api/status (affinityEntries) and /metrics
  (inferhub_affinity_entries, a fleet gauge present at zero).

Invariants

• Rule 4 holds. Persistence is opt-in and off by default; with it off, behaviour is byte-identical
  to v2.11. When on, the file store is a derived cache of routing hints, never a source of truth:
  a lost or stale entry costs at most one cold model load, never a wrong answer, so it does not become
  a third authority alongside the vector store and the usage ledger. A torn last line from a crash
  mid-append is skipped on load, not treated as corruption.
• Rule 7 holds. The affinity key is still a conversation header value or a hash of the opening
  message — never content. The persisted record is (conversationKey, nodeId, lastUsed) and nothing
  more.
• Rule 5 holds: zero new dependencies. The file store is hand-rolled, exactly like the vector raw
  store.
• Docker (D7). The container points Affinity__DataDirectory at /data/affinity, under the
  chown app:app /data mount that USER app can write — the same trap the vector store hit, headed
  off in the same place. Inert unless persistence is turned on.

Why this matters for v3.0

The upcoming warm-failover coordinator (phase 32) needs warm routing to survive a hub switch. That is
only possible once affinity keys on a stable identity and can be reloaded — which is exactly what this
release lays down. Phase 30 is the prerequisite, shipped on its own so it can be verified on its own.

543 tests green (17 Postgres-gated, skipped without a database). Verified live: a warm conversation
stays pinned to the same node across a coordinator restart with Affinity:Persistence=file, and
across a node reconnect without persistence.

v2.10.0 — Observability export: Prometheus /metrics

InferHub v2.10.0 — Observability export: Prometheus /metrics

Everything the mesh measured lived in-memory and was reachable only as /api/status JSON and
the status page. That is a snapshot: no Grafana, no alerting, no capacity history, no way to
answer "was the fleet saturated at 3am?" the morning after. GET /metrics fixes that.

This phase exposes numbers; it does not measure any. Every series here already existed in
Metrics, ThroughputTracker, RequestQueue and AdmissionControl. Nothing was added to the
request path.

What changed

• GET /metrics returns the Prometheus text exposition format
  (text/plain; version=0.0.4). PrometheusFormatter is a pure function from a gathered
  PrometheusScrape to a string — no services, no clock, no I/O — so a test can assert the
  exact bytes.
• What is exposed, all namespaced inferhub_*: fleet counters (requests total / in-flight /
  completed / failed, failovers, evictions, OpenAI-surface requests, uptime, build_info),
  cloud burst (fallback_dispatched_total and an info-style fallback_last_model{model}),
  per-node (node=) up/cordoned/models/local-in-flight/heartbeat-age plus the routed request
  counters and node_tokens_per_second{node,model}, vector and per-collection (collection=)
  queries, latency, documents, chunks and failures, the request queue's depth and outcomes and
  median wait, and per-client (client=) live windows against their configured limits.
• ThroughputTracker.Snapshot() — the EWMA map had no way to be enumerated; now it has a
  read-only, ordered one.
• deploy/docker/compose.observability.yml — an optional Prometheus + Grafana overlay with
  a provisioned datasource and a starter InferHub mesh dashboard. It stays an overlay: the
  base stack is still two services, and nobody who just wants a mesh has to run a monitoring
  stack to get one.

Auth — recorded, because it was a real decision

/metrics is operational, like /health, which is deliberately open. But unlike /health it
exposes node names, model names, client ids and the shape of your traffic. So it is
admin-key-guarded by default — AdminApiKeyMiddleware now guards a small prefix set
rather than one constant — and Metrics:OpenScrape=true drops the guard on this one
endpoint for a trusted network. It never opens /api/admin/*, and there is a test that says
so.

It is deliberately not under the bearer inference guard. A scraper is not a client, and
handing a monitoring system a token that can spend GPU time would be the wrong trade.

Counts, never content

The per-client series come from the in-memory admission windows — the same source
/api/admin/clients already reads — and never from the usage ledger. The ledger is append-only
history and is never read to drive anything; that is the reasoning that keeps it from being a
second source of truth (rule 4 / phase-25 D2), and a metrics endpoint reading it would have
quietly ended that.

Absence is a fact, so absence is what you get

An unmeasured (node, model) produces no node_tokens_per_second series rather than a
zero. The router treats an unmeasured node as average, never as slow (phase 26, D4) — a 0 on
a dashboard would be a lie that invites an alert against a node that has simply not been asked
yet. Same for a client limit that is unset (unlimited is the absence of a series, not a 0 and
not a -1 sentinel a dashboard would happily plot) and for the queue's median before anything
has ever queued. The fleet counters, by contrast, are always present at 0 — there, a zero is a
statement rather than an absence.

Not a migration

/api/status and the status page are unchanged. This adds a surface; it does not replace
one.

Tests

539 total (522 passed, 17 skipped — the gated Postgres integration tests, as always). New:
PrometheusMetricsTests parses the output back with a minimal in-test exposition parser rather
than string-matching it — substring assertions would pass happily on output no Prometheus could
read, which is the exact failure this endpoint exists to avoid. It covers # HELP/# TYPE on
every series, counter-vs-gauge typing, values matching what Metrics recorded, label
cardinality (a node id is a label, never baked into a metric name), label-value escaping,
invariant decimal separators on every value line (a decimal comma would be a locale bug
that only appears on a Bulgarian or German host and would sink the whole scrape), the
absent-not-zero cases, and the four auth cases including OpenScrape=true not unlocking
/api/admin.

Also fixed: the node image has been dead at startup since v2.3.0

Not a phase-28 change, and not a regression — a shipped bug found by running the container,
which is the only way this class is ever found (D7). Bringing the documented compose stack up to
verify the Grafana overlay, the node died before it ever reached the coordinator:

System.UnauthorizedAccessException: Access to the path '/app/.inferhub-node-id' is denied.

FileNodeIdentity persists the stable node id to Node:DataDirectory, which defaults to the
content root — /app, which USER app cannot write. This is the same root cause as the
v2.5.1 fix, whose two lines covered only the vector replica half. That half was conditional on
a feature being enabled, so it looked like the whole bug; the identity write is unconditional, so
the node image has in fact been broken on every run since v2.3.0.

Two lines, matching the existing pattern:

• src/InferHub.Node/Dockerfile — ENV Node__DataDirectory=/data (the already-chowned dir).
• deploy/docker/docker-compose.yml — the node's volume moves from node_replicas:/data/vector-replicas
  to node_data:/data, so the node id and the replicas share the one mount point the image
  creates and owns. A volume at a path the image does not contain is created root-owned, which is
  the same trap a second time.

Upgrade note. The volume rename means an existing stack drops its old node_replicas
volume; the coordinator re-pushes the replicas, and the node takes a new id on first start. Both
are correct and both are one-time. docker volume rm inferhub_node_replicas to reclaim the space.

Verified live

A running coordinator from source: curl localhost:5080/metrics returns valid exposition text
with the correct content type and build_info carrying the real informational version.

Then the full documented stack, built and run: coordinator + node + Prometheus + Grafana. The
Prometheus target reports health: up, scraping /metrics with the admin key over the bridge
network — so the non-loopback auth path is proved, not assumed. After one real
/api/chat against a qwen2.5:0.5b node:

inferhub_build_info{version="2.10.0"} 1
inferhub_node_up{node="58e35f0d-…",name="docker-node"} 1
inferhub_node_tokens_per_second{node="58e35f0d-…",model="qwen2.5:0.5b"} 166.94889
inferhub_node_requests_completed_total{node="58e35f0d-…"} 1

Grafana provisioned the InferHub mesh dashboard and its panels resolve real data through the
datasource. The datasource uid is now pinned in the provisioning file (inferhub-prometheus)
and referenced by the dashboard JSON — left unset, Grafana generates a different uid per install
and every panel of a provisioned dashboard comes up "datasource not found".

Zero new dependencies

Rule 5 holds. The exposition format is # HELP / # TYPE / name{labels} value — the same
"three lines of string formatting" reasoning that kept the NDJSON framing (phase 9) and the SSE
framing (phase 21) dependency-free. prometheus-net would have been a permanent dependency and
a registry abstraction on the hot path in exchange for code that fits on a screen. An OTLP
push exporter would genuinely need a package; it stays deferred and opt-in, if demand appears.

v2.9.0 — Streaming tool_calls deltas

InferHub v2.9.0 — Streaming tool_calls deltas

A half-finished feature, finished. Tool calls were mapped in blocking mode only: on the
streaming /v1/chat/completions path the delta carried role and content and nothing else,
so the terminal chunk resolved finish_reason=tool_calls with no call attached. A streaming
agent loop (OpenAI SDK function-calling, LangChain, LlamaIndex) got a finish reason it could
not act on. Now the call streams too.

What changed

• The streaming delta grows a tool_calls slot (ChatCompletionDelta.ToolCalls), and
  ResponseTranslator.ToChatChunk fills it when the Ollama chunk carries
  message.tool_calls. Each entry gets the 0-based index the streaming spec requires, a
  synthesized id, an explicit type: function, and arguments serialized to a string —
  the same shape the OpenAI Python SDK parses into a ChoiceDeltaToolCall.
• A tool-call delta carries content: null, not "" — a tool-call frame and a text frame
  are not the same frame, matching OpenAI.
• The streamed call reuses the blocking extractor (ExtractToolCalls), so the function name
  and stringified arguments are byte-equivalent (synthesized id aside) to what blocking
  ToChatCompletion produces for the same Ollama body. OpenAiTranslationTests pins the parity.

The honest part

The whole tool call arrives in one delta — we deliberately do not fabricate
OpenAI-style argument-fragment streaming that Ollama never sent. It is marked in a comment so
nobody "improves" it into fake fragmentation; if a future Ollama actually streams partial
arguments, that is a separate change.

Two things the tests didn't catch — the live node did

The phase brief assumed Ollama emits message.tool_calls on its final chunk (done:true).
Verified against a real qwen2.5 node, it does not: the call lands on a non-terminal
chunk (done:false), and the terminal chunk carries no tool_calls. A stateless per-chunk
translator would emit finish_reason: stop on that terminal frame — and a strict agent loop
keys tool execution off finish_reason == "tool_calls", so the call would stream but never
fire. The stream formatter now remembers that a call was seen and resolves the terminal
frame to tool_calls regardless of which chunk carried it.

The second half of the loop was also broken, in the request translator (untouched since
phase 21): OpenAI clients serialize function.arguments as a JSON string (as does our new
streamed delta), but Ollama emits and expects an object. A prior assistant turn's tool call
therefore reached the model as a string it could not read, and the model answered empty —
so the streamed call → tool result → answer loop never closed. The translator now parses the
arguments string back to an object. Both fixes were found by running the real loop, not by a
green suite — the unit tests passed happily while the live round-trip returned nothing.

Not a regression

Text-only streaming is byte-identical to v2.8 — ordinary deltas carry no
tool_calls: null field (the slot is omitted when null). The node-facing job protocol is
unchanged; the node keeps sending Ollama-shaped chunks.

Tests

527 total (510 passed, 17 skipped — the gated Postgres integration tests, as always). New
coverage: OpenAiStreamingTests (a terminal tool_calls chunk yields a delta.tool_calls
frame with a synthesized id, index: 0, function.name, stringified arguments and
finish_reason: tool_calls; a call on a non-terminal chunk still resolves the terminal
frame to tool_calls, not stop; ordinary text deltas carry no tool_calls field), and
OpenAiTranslationTests (the streamed call is byte-equivalent to the blocking one, id aside; a
text delta serializes without a tool_calls key; a prior tool call's string arguments are
parsed back to an object, and an object passes through unchanged).

Verified live

A real coordinator + qwen2.5 Ollama node: stream:true with a get_weather tool → a
delta.tool_calls frame then finish_reason: tool_calls, and a second turn carrying the tool
result → a grounded streamed answer. The full call → tool-result → answer loop closes with the
arguments in the spec's string form.

Zero new dependencies

Rule 5 holds: System.Text.Json did all of it, and the SSE framing is still hand-written.

v2.8.0 — Fleet operations: remote model management & measured routing

InferHub v2.8.0 — Fleet operations: remote model management & measured routing

Two unglamorous features that decide whether a mesh is pleasant to run: the hub can now act on
the fleet's models, and the router finally knows that a 4090 and a laptop are not the same node.

Remote model management

The coordinator has always seen every model on every node and been able to do nothing about any of
them. Now it can pull, delete and warm a model on any node — from the console or the admin API, with a
progress bar, over the connection the node already holds open. The node still has no inbound surface;
commands travel down the same outbound SignalR channel inference jobs already use.

• POST /api/admin/nodes/{id}/models/{model}/pull — stream a pull, progress relayed on the existing
  SSE /api/admin/stream as model-progress events.
• DELETE /api/admin/nodes/{id}/models/{model} — delete a model from one node.
• POST /api/admin/nodes/{id}/models/{model}/warm — load a model into memory ahead of first use.
• POST /api/admin/models/{model}/ensure?replicas=N — pull onto the N most suitable capable nodes that
  don't already have it, skipping cordoned ones, and report what it decided and why.
• GET /api/admin/models — the fleet-wide model × node matrix.

A duplicate command for the same node+model coalesces onto the running one. Every command lands in the
audit log. And not every backend can do this: an OpenAI-compatible upstream (vLLM, llama.cpp, a
hosted provider) has its model fixed at launch, so it declares SupportsModelManagement = false and the
console greys out its controls — a backend asked to do the impossible refuses with a clean error frame,
never a 500.

Measured routing

Since 1.0 the router has picked the "least-busy" node — fewest jobs in flight. That is a fine proxy on a
uniform fleet, and InferHub's whole premise is that your fleet is not uniform. A 4090 and a laptop with
an eGPU both report one job in flight; they will not finish at the same time.

Nodes now carry a measured throughput — an EWMA of tokens/second per model, computed from the
eval_count/eval_duration every completed job already reported. No new measurement plumbing. With
Router:Strategy=throughput, routing considers expected completion time rather than raw queue depth. A
node with no measurement is treated as average, never as slow — otherwise a fresh node never earns
a measurement and stays frozen out. Sticky conversation affinity still wins where it applies; throughput
is a tiebreak among candidates, not a replacement for the thing that was already right.

The default is unchanged, bit for bit. Router:Strategy=least-busy is the default; measured routing
is opt-in for this release. Measured tokens/sec is on /api/status per node and on the status page.

Config

Router:Strategy — least-busy (default) | throughput. Nothing else changed; see appsettings.json.

Tests

520 total (503 passed, 17 skipped — the gated Postgres integration tests, as always). New coverage:
ModelCommandTests (pull/delete/warm progress, coalescing, a backend that cannot manage refuses cleanly
— not a 500), PlacementTests (ensure skips cordoned and already-present nodes, counts a non-manageable
holder toward N, and says so when candidates run short), ThroughputRoutingTests (a fast node wins, an
unmeasured node is not starved, affinity still wins, and least-busy is byte-identical to v2.7).

Verified live end-to-end against a real coordinator + Ollama node: warm/pull/delete relayed over the SSE
stream as model-progress frames (including a clean error frame for a missing model, no stack trace);
the model matrix; ensure reporting a shortfall honestly; and measured tokens/sec appearing on
/api/status after real chat traffic.

Zero new dependencies

Rule 5 holds: HttpClient, SignalR and System.Text.Json did all of it.

InferHub v2.7.0 — Clients, quotas & usage accounting

InferHub v2.7.0 — Clients, quotas & usage accounting

Phase 25. Auth:ApiKeys has been a flat list of strings since 0.6: every key anonymous,
unlimited, and indistinguishable from every other key. Fine for one person and their own GPUs;
not fine the moment a second party is behind one of those keys. This release gives a key an
identity, a budget, and a bill — and it does so without storing a single prompt anywhere.

Everything is backwards compatible: a config with only the flat Auth:ApiKeys list runs
byte-identically to v2.6, its keys resolving as anonymous unlimited clients.

What's new

• Named clients — Auth:Clients is a list of { Id, Key, Limits }. Every limit is
  nullable (null = unlimited): MaxConcurrent, RequestsPerMinute, TokensPerMinute,
  TokensPerDay, AllowedModels. A duplicate key across two clients fails startup — key →
  client attribution must never depend on list order.
• Usage accounting — every completed request is metered per client and per model:
  requests, prompt tokens, completion tokens, and whether it was served by cloud burst (the one
  that costs actual money). Embeddings and document ingestion count too. The counts were already
  in every response since 2.3/2.4 — this release attaches a name to numbers we already had.
  • GET /api/admin/usage?from&to&clientId&model — aggregates you could put on an invoice.
  • GET /api/admin/clients — configured clients with live window consumption against limits.
  • A Clients & usage console panel with a date range and CSV export.
• Honest, standard rejection — over a rate limit → 429 with a window-accurate
  Retry-After; over the daily budget → 402 Payment Required with Retry-After pointing at
  UTC midnight (checked before the rate limits — "waiting a minute will help" would be a
  lie). A model outside a client's allowlist → 404 byte-identical to a model that does not
  exist. On /v1 these use the OpenAI error envelope (rate_limit_exceeded,
  insufficient_quota), so SDK retry logic works out of the box.
• Bounded queueing — when every node holding a model is at its declared MaxConcurrency,
  a request waits up to Queue:MaxWaitSeconds (default 30), at most Queue:MaxDepth waiting
  (default 64), then 503 + Retry-After. Nodes with no declared cap never queue. Queue depth
  and median wait are on /api/status and the status page — a queue you cannot see is a queue
  you will not notice filling. With cloud burst on no-node-or-saturated, saturation overflows
  to the upstream instead of queueing; the precedence is explicit and tested.
• Optional durable ledger — Usage:Persistence=postgres writes each record to an
  append-only table over its own connection string (deliberately independent of the vector
  store's). Default stays none: in-memory, reset on restart, like every other counter.

Counts, never text

A usage record is a client id, a model, a kind, two integers, a fallback flag and a timestamp.
It does not contain the prompt, the completion, a hash of either, or a "representative sample"
— and there is no flag to change that, because a flag is an invitation. Streaming counts
come from the terminal chunk; a stream that never delivers it (mid-stream disconnect) records
nothing rather than guessing. A test pins the record's shape.

Zero new dependencies

The durable ledger reuses Npgsql, already a recorded dependency since phase 20. Everything
else is System.Text.Json and a dictionary.

Config

New sections: Auth:Clients, Usage (Persistence, Postgres:*), Queue (MaxWaitSeconds,
MaxDepth). All optional; see appsettings.json and the README's
Clients, quotas & usage.

Tests

501 total, all passing with the Postgres gate open (the 19 gated integration tests now include
the usage ledger). New coverage: ClientRegistryTests, AdmissionControlTests,
UsageLedgerTests (including the mid-stream-disconnect contract and the record-shape pin),
QueueTests (including the fallback-vs-queue precedence and the uncapped-node regression), and
PostgresUsageLedgerTests. Verified live end-to-end: a real coordinator + real node over
SignalR against an OpenAI-dialect upstream — blocking and streaming usage metered, 429/402/
404/503 each observed on the wire with correct envelopes per dialect, one client's
saturation leaving another untouched, and the Postgres ledger surviving a coordinator restart.

v2.6.0 — Hybrid search, reranking & eval harness

InferHub v2.6.0 — Hybrid search, reranking & an eval harness

Phase 24. Pure vector search is excellent at "what is this about" and poor at "find the exact
thing I named" — ask for an error code, a SKU, or a surname and cosine similarity returns the general
topic, not the line you wanted. This release adds keyword search, hybrid (keyword + vector) retrieval
fused by rank, an opt-in reranker, and — because "this improved retrieval" is a claim, not a feeling —
an evaluation harness that measures whether any of it helped on your corpus.

Everything here is per-request and off by default: a deployment that sends no new headers and
changes no config behaves byte-identically to v2.5.

What's new

• Retrieval modes via X-InferHub-Retrieve-Mode: vector | keyword | hybrid (an unknown value is a
  400, not a silent fallback).
  • keyword — classic BM25 over the same chunks. Provider-native: an in-memory inverted index under
    local, Postgres full-text (tsvector + GIN + ts_rank_cd) under postgres.
  • hybrid — runs both branches and fuses them by Reciprocal Rank Fusion (by rank, not by
    blending scores that live on different scales). This is the one you usually want.
• Reranking via X-InferHub-Rerank: true — an opt-in pass that hands the top candidates to a chat
  model already on your fleet with a scoring prompt and reorders them. It costs a round trip, so it is
  off unless asked, and hard-capped by Retrieval:RerankCandidates / Retrieval:RerankTimeoutSeconds;
  past the timeout the un-reranked order is kept. A dedicated cross-encoder fits behind the same seam
  later.
• POST /api/collections/{c}/search — a query playground endpoint: run a query in a mode and get
  the ranked chunks back directly. The admin console gains a panel that shows what every mode retrieves
  for a query, side by side.
• tools/InferHub.Eval — a standalone eval harness (not built into the images). Point it at a
  live coordinator with a golden set and it reports Recall@k, MRR, nDCG@k and median latency for every
  mode.

Zero new dependencies

Keyword search is provider-native — Npgsql already reaches Postgres full-text, and the local BM25
index is a dictionary. The reranker reuses a model already on the fleet. Nothing was added.

Defaults are unchanged

Retrieval:Mode defaults to vector and Retrieval:Rerank to none. No headers, no config: identical
to v2.5. A feature that silently changes existing results is a regression wearing better clothes, and
the test suite asserts the default equals vector-only — plus the exact-term case (an error code) that
vector search misses and hybrid recovers.

Measured, not asserted

The blog post's numbers must come from the harness on a real corpus, run against a live coordinator with
an embedding fleet. This release ships the harness; the numbers get filled in from a real run (see the
harness README — and its warning that a golden set written by the model you are about to evaluate is a
mirror, not evidence).

Config

New keys under VectorStore:Retrieval: — Mode, CandidatesPerBranch, Rerank, RerankModel,
RerankCandidates, RerankTimeoutSeconds. All optional; see appsettings.json and the README.

Tests

457 total (442 passed, 15 skipped — the gated Postgres integration tests, as always). New coverage:
InvertedIndexTests, HybridSearchTests, RerankerTests, the mode/rerank pipeline tests including the
byte-identical-default regression, and Postgres keyword SQL shape tests. Verified live end-to-end against
a real coordinator: keyword retrieval through the HTTP surface (header → pipeline → BM25 →
X-InferHub-Sources), vector/hybrid degrading cleanly to 424 without an embedding node, and mode
validation returning 400/404.

v2.5.1 — the local vector store actually works in Docker

InferHub v2.5.1 — the vector store actually works in Docker now

A patch release with one fix, and it is a serious one.

The bug

Enabling the local vector store in a published container crashed the coordinator at startup.
Not degraded — dead:

System.UnauthorizedAccessException: Access to the path '/app/data' is denied.
  at InferHub.Coordinator.Vector.LocalVectorStore..ctor(...)

Both images run as USER app, which cannot write /app. VectorStore:DataDirectory defaults to
./data/vectors, which inside the container resolves to /app/data. So the moment you set
VectorStore__Enabled=true, the host failed to start.

Mounting a volume did not save you either. Docker seeds a fresh named volume from the image's
contents at the mount point — including its ownership — and a mount point that does not exist in
the image is created root-owned. -v vol:/data therefore failed for exactly the same reason,
which means the documented compose stack was broken too. Nobody noticed because
INFERHUB_VECTORS_ENABLED defaults to false.

The node had the same latent fault: its Vector:ReplicaDirectory defaults to
./data/vector-replicas → /app/data, so a containerised node would have died the moment the
coordinator assigned it a vector replica.

This has been shipped and broken since v2.3, when the images first landed. It made v2.5's
headline feature — document ingestion, which is useless without a vector store — unusable in Docker,
the recommended install.

The fix

Two lines in each Dockerfile, and both are load-bearing:

RUN mkdir -p /data && chown app:app /data      # makes the volume case work
ENV VectorStore__DataDirectory=/data/vectors   # makes the bare-image case work

The compose stack also gains a node_replicas volume, so a node no longer re-pulls the whole corpus
from the hub on every restart.

How it was found, and the lesson

By pulling the published image on a clean machine and turning the feature on. Nothing else would
have caught it: the unit tests pass, and a from-source end-to-end passes, because from source the
working directory is writable. The artefact users actually install was dead on arrival while every
signal we were watching stayed green.

Phase 22 left "GHCR images pulled and smoke-tested from a clean machine" as an unticked box. This is
what was behind it.

Verified

The published-style image, pulled clean, with the store enabled: starts healthy, creates a
collection, serves the ingestion routes, persists to disk, correctly returns 500 partial with an
honest error when no node advertises the embedding model, and rejects a scanned PDF with 422.
Both the bare-image path and the named-volume path.

Upgrade

docker compose pull — or bump your image tag to 2.5.1. No config change is needed: the image now
declares its own writable data location. If you had explicitly set VectorStore__DataDirectory to
something under /app, move it (or just drop the setting and take the default).

v2.5.0 — Document ingestion pipeline

InferHub v2.5.0 — Document ingestion pipeline

Phase 23. The vector store has existed since v1.5 and inline retrieval since v2.0. What InferHub
never had was a way to fill the store with anything but pre-computed vectors or hand-pasted text.
That made RAG a primitive rather than a feature. This release closes it: documents in — text,
Markdown, HTML, JSON, PDF text layer — chunks and embeddings out, embedded on the GPU fleet through
the dispatcher that already exists.

What's new

• POST /api/collections/{c}/documents — upload via multipart/form-data or JSON. The
  coordinator extracts the text, chunks it (paragraph → sentence → character, with overlap), embeds
  the chunks on the fleet, and upserts them into the vector store.
• GET /api/collections/{c}/documents — list documents: chunk count, bytes, content hash,
  ingested-at, status.
• GET /api/collections/{c}/documents/{id}/chunks — the chunks in order, with page numbers.
• DELETE /api/collections/{c}/documents/{id} — removes every chunk of that document.
• Console: a Documents panel. Pick a collection, drop a file, watch the chunk count climb,
  preview chunks, delete a document.
• /api/status gains a per-collection ingestion block: documents ingested, chunks embedded,
  failures, last ingest, embedding model.

All four endpoints are guarded by the client key (Auth:ApiKeys), not the admin key. Ingesting
is a client action, and forcing an admin key on it would push people toward using one key for
everything.

⚠ Breaking: X-InferHub-Sources changed shape

It used to carry bare chunk ids:

X-InferHub-Sources: ["5d981c…","0b72c7…"]

It now carries objects that name where each chunk came from:

X-InferHub-Sources: [{"id":"5d981c…","documentId":"employee-handbook.pdf","page":1},
                     {"id":"0b72c7…","documentId":"policy.md"}]

A chunk id alone identifies the row we retrieved but tells the reader nothing about where the
answer came from, and a citation that cannot name a document and a page is not a citation.
documentId and page are omitted (not null) for records written straight through /api/vector,
which never had a document. If you parse this header, you need to update.

Decisions worth reading

• No OCR, and there never will be. A PDF whose text layer yields under ~50 characters per page
  is rejected with an error saying it looks like a scan. Bolting on an OCR pass would produce
  something that usually works — and a bad extraction does not fail. It succeeds, quietly, and
  fills the corpus with near-gibberish that retrieves plausible nonsense, surfacing months later as
  a model that is subtly and unaccountably wrong. If a document genuinely needs OCR, that is a
  decision its owner should make deliberately, with a tool they chose, before it reaches InferHub.
• Your file is not kept. Chunk text, a content hash, metadata. Not the document. A retrieval
  system that quietly becomes a document store has two sources of truth and a data-retention
  question its owner never agreed to answer.
• Re-ingesting is idempotent. Chunk ids are sha256(documentId + ":" + chunkIndex), so a
  revision replaces its chunks in place rather than layering a second copy underneath the first —
  and a shorter revision has its orphaned tail chunks swept, because a stale chunk retrieves as
  confidently as a live one. Identical bytes twice: the second call does no work and says
  "status": "unchanged".
• A partial ingest is a failure and says so. Lose the fleet mid-document and the response is
  HTTP 500 with "status": "partial" and the chunk counts. The chunks that landed are real and
  visible, the document lists as partial, and re-posting the same bytes resumes rather than
  no-ops — the content-hash short-circuit deliberately does not fire on a partial document, because
  "you already have this" would be a lie about a document that is half-missing.
• Embedding runs through the fleet, batched, with at most Ingestion:EmbeddingBatchSize chunks
  in flight, so a 300-page manual queues behind itself instead of starving interactive chat. The
  coordinator grows no embedding path of its own.

Under the hood

IVectorStore gained two operations, implemented by both providers and proven to agree by
VectorProviderParityTests:

• ScanAsync(collection, filter, limit, afterId) — metadata scan ordered by id, without the
  embeddings (a new VectorEntry record: "not fetched" must not be confusable with "not there").
• DeleteByFilterAsync(collection, filter) — bulk delete by metadata. The filter must be non-empty.

These are what let ingestion keep its promise that it writes to the vector store and nowhere
else: there is no documents table, no blob directory, no second lifecycle. A document is the set
of chunks sharing a documentId in their metadata. Rule 4 ("one source of truth per deployment")
survives untouched.

LocalVectorStore.DeleteByFilterAsync deliberately loops over the ordinary per-id delete rather than
bulk-removing under the lock: the per-id path is what raises RecordDeleted, and that is the only
way a deletion reaches the node replicas. A faster bulk delete would leave every node in the fleet
still serving the chunks of a document the hub thinks is gone.

Dependencies

One new package: PdfPig (0.1.15, Apache-2.0). The second recorded exception to the
no-new-dependencies rule, after Npgsql/Pgvector in phase 20. It is coordinator-only (never in
InferHub.Shared or InferHub.Node), lives behind IPdfTextExtractor, is referenced by exactly one
file, and no code path reaches it unless a PDF is actually uploaded. Hand-rolling a PDF text-layer
parser is a bad use of a week.

Verification

• 418 tests, 403 passed, 15 skipped (the skips are the gated Postgres integration tests).
  PdfExtractionTests builds real PDFs with PdfPig's writer and parses them back — a stub would
  prove only that the seam is wired, and the parts this dependency was spent on are exactly the ones
  a stub cannot reach.
• Real end-to-end, run against a live coordinator + a live node (Backend:Type=openai) over real
  SignalR, against a real HTTP server speaking the OpenAI dialect. Verified: a Markdown document and
  a real two-page PDF ingested; identical bytes re-posted → unchanged with zero embed calls; a
  scanned PDF → 422 with the "this looks like a scan" error; chunks carrying their page number;
  a /v1/chat/completions call with X-InferHub-Retrieve answered from the PDF with
  X-InferHub-Sources naming the document and page 1; the chunks replicating to the node and the
  retrieval query being served from the node replica, not the hub; document delete removing every
  chunk of one document and none of the other; the /api/status ingestion block.

Upgrade

Nothing to do unless you parse X-InferHub-Sources (see above). Ingestion is inert unless
VectorStore:Enabled=true, and every Ingestion key has a working default.

v2.4.0 — OpenAI-compatible node backend & cloud burst

InferHub v2.4.0 — OpenAI-compatible node backend & cloud burst

Phase 22. Two features, one translation layer, mirrored — and zero new dependencies.

A node is no longer synonymous with Ollama

IInferenceBackend has had exactly one implementation since phase 3. It now has two.

Set Backend:Type=openai and a node drives any server that speaks the OpenAI wire format:

Upstream	Why you'd want it
vLLM	Continuous batching and paged attention — the difference between one GPU serving four users and one serving forty.
llama.cpp server	GGUF quants, CPU/Metal, tiny footprint.
LM Studio	A desktop model runner people already have open.
TGI	HuggingFace's inference server.
Hosted providers	OpenAI, or anything wearing the same dialect.

{
  "Backend": { "Type": "openai" },
  "OpenAi": {
    "BaseUrl": "http://localhost:8000/v1",
    "Models": { "Include": ["meta-llama/Llama-3.1-8B-Instruct"] }
  }
}

One implementation covered all five, because they all converged on the same format — three
roadmap items ("vLLM backend", "llama.cpp backend", "hosted backend") collapsed into one, and
the collapse was possible only because v2.3's translation code could be run backwards.

Against a hosted provider, OpenAi:Models:Include is effectively mandatory: the catalogue is
hundreds of models the node cannot serve, and reporting them all makes the coordinator route
anything to it. Digest and SizeBytes come back null, because an OpenAI-compatible server
reports a name and nothing else and inventing values would be worse.

A node with Backend:Type=openai and no BaseUrl refuses to start, naming the key. A node
that boots and then 500s on every job it is handed is worse than one that doesn't boot.

Cloud burst

The failure nobody plans for is that the GPU box is switched off. A coordinator with no capable
node has returned a flat 404 since 1.0. It can now fall back to a configured
OpenAI-compatible upstream instead — a hosted provider, a second mesh, anything speaking the
dialect.

This is the feature most likely to do something its owner never agreed to, so it is fenced in:

• Off by default. Upgrading changes nothing.
• Mapped models only. Fallback:ModelMap is the consent. There is no wildcard.
• Always tagged. Every fallback response carries X-InferHub-Served-By: fallback.
  Node-served responses carry node.
• Always counted. /api/status and the status page report cloud burst and its counter
  even when it is off — "is this thing sending my prompts anywhere?" should not require
  reading a config file.
• Never stored. The coordinator forwards in flight and streams straight through. It retains
  neither the prompt nor the answer.

Trigger: no-node-or-saturated also bursts when every node holding the model is at its
declared MaxConcurrency. A node that declared no cap is never saturated — there is no
number to compare against, and guessing would burst to a paid upstream on a hunch. If a
saturation burst fails and a node exists, the request goes to the node: bursting on saturation
is an optimisation, not a promise.

The bit that looks silly, and why it stays

An OpenAI request to /v1/chat/completions is translated into an Ollama body, sent to a node,
and translated back into an OpenAI body for vLLM. Two JsonSerializer round-trips on a call
that is about to occupy a GPU for two seconds.

The alternative was a polymorphic job payload carrying a dialect tag, which would have pushed
dialect-awareness into the dispatcher, the router, the affinity keys, the retrieval pipeline,
and every test that touches them. The mesh's internal protocol stays exactly one shape. That is
worth more than the round-trips cost, and the reason is written into UpstreamTranslator so
nobody "fixes" it later.

Fixed

• A node reported Ollama:Endpoint as its endpoint at registration regardless of backend,
  so an OpenAI-backed node would have advertised localhost:11434 on the status page while
  driving vLLM. IInferenceBackend now carries Endpoint. The NodeRegistration.OllamaEndpoint
  field name is unchanged — it is a SignalR payload field and an /api/status field, and
  renaming it would break a mixed-version fleet for a cosmetic gain.

Internal

• The OpenAI DTOs and shape mappers moved from InferHub.Coordinator/OpenAi/ to
  InferHub.Shared/OpenAi/: both ends of the mesh speak the dialect now, and two copies of a
  wire format drift silently. Only the ASP.NET-bound pieces (OpenAiEndpoints,
  OpenAiStreamingResult) stayed behind. Pure refactor — not one test file changed.
• OpenAiUpstreamClient (in InferHub.Shared) is the single implementation of the upstream
  dialect: HTTP, SSE parsing, error mapping. The node's OpenAiBackend and the coordinator's
  FallbackDispatcher both drive it. One dialect, one implementation, two callers.
• Zero new NuGet packages. HttpClient and System.Net.Http.Json ship in the shared
  framework; the SSE parser is written by hand, exactly as the SSE writer was in v2.3.

Tests

361 total, 347 passing, 14 skipped (the gated Postgres integration suite).
New: OpenAiBackendTests (17), FallbackTests (18) — most of the latter being assertions
about when cloud burst must not fire.