Shared LLM inference layer for the docs-as-tests toolchain: schema-constrained completion across Anthropic, OpenAI-compatible, Claude CLI, and in-process local (llama.cpp) providers, with result caching, cost accounting, and an LLM-as-judge ensemble on top.
Extracted from three projects that had each grown their own copy — docevals, dockg, and agentevals — so a provider fix lands once instead of three times.
npm install @hawkeyexl/inferenceRequires Node 24+. Three runtime dependencies, plus one optional peer dependency for local models.
ESM only. The exports map has no require condition, so
require("@hawkeyexl/inference") fails with ERR_PACKAGE_PATH_NOT_EXPORTED. From CommonJS, use
await import("@hawkeyexl/inference").
Every consumer wants the same narrow thing: send a system prompt, a user prompt, and a JSON Schema; get back JSON that validates against that schema, or a recorded error.
(system, user, schema, temperature) -> JSONNo streaming, no multi-turn, no tool loops. If you need a conversation, this is the wrong package. Widening the provider contract requires an ADR.
Two layers, one entry point:
- Completion — the provider contract, five providers, a content-addressed cache, a price table, and a validate-and-retry wrapper. All that structured extraction needs.
- Judge — the canonical verdict schema, an N-run ensemble, consensus math, and confidence-zone routing. Built on the completion layer; ignore it if you do not need it.
No API key required — MockProvider is exported for exactly this.
import { MockProvider, completeValidatedJSON } from "@hawkeyexl/inference";
const run = await completeValidatedJSON({
provider: new MockProvider([{ json: { summary: "Covers authentication." } }]),
system: "You summarize documentation pages.",
user: pageBody,
schema: {
type: "object",
required: ["summary"],
properties: { summary: { type: "string" } },
additionalProperties: false,
},
});
if (run.error) console.error(run.error);
else console.log(run.result.summary, run.usage);completeValidatedJSON never throws on a model failure and never coerces a bad response. It retries
once, then returns a run with error set and result absent.
Point it at a real model by swapping the provider — or omit provider entirely and let the library
detect one this machine can use, ending at the free local model:
const provider = await makeProviderAsync({});provider |
Structured output via | Credential | Reports usage |
|---|---|---|---|
anthropic |
forced tool call | ANTHROPIC_API_KEY |
yes |
openai |
strict json_schema, falls back to json_object |
OPENAI_API_KEY |
yes |
claude-cli |
schema in the prompt, --output-format json |
local claude auth |
no |
llama-cpp |
GBNF grammar compiled from the schema | — (runs locally) | yes |
mock |
scripted responses | — | synthetic |
Omit provider and the highest-priority one this machine can actually use is detected, ending at
llama-cpp — which needs no key, and whose native binding is installed on demand into
~/.hawkeyexl-inference/runtime if it is missing. That install warns once and is refused by
INFERENCE_NO_AUTO_INSTALL; it never touches your package.json, lockfile or node_modules.
Usage reporting is the column that decides whether cost accounting works: a provider that reports no tokens makes a budget gate inert. See Choose a provider.
| Track | What it covers |
|---|---|
| Get started | Install, one validated call with no key, choosing a provider |
| Judge & consensus | Ensembles, consensus math, confidence zones, caching, budgets |
| Structured extraction | One schema-constrained call, honest failures, the subprocess seam |
| Run models locally | GGUF weights in-process, model selection, managing weights on disk |
| Keep it working | Testing without a network, upgrading without losing a cache |
| Reference | Full signatures for every export |
Who the docs serve and why each page exists lives in docs/content-strategy/.
Recorded as ADRs in adrs/:
- 01000 — a library-owned
ProviderSpec, not consumer config objects - 01001 — one entry point; a canonical verdict schema with a per-consumer override seam
- 01002 — which fork won for each merged file, so the losing variants are not reintroduced
- 01003 — in-process local models via node-llama-cpp, why selectors need an async factory, and why the catalog pins exact blob paths
- 01004 — detect an available provider when none is specified, ending at the local model
- 01005 — a CUJ-first documentation set, with samples that CI executes
- 01006 — document failure and orchestration, and gate both against the source
- 01007 — harden two operational failure paths: non-JSON CLI output, and an unsupported Node
- 01008 — auto-install the local runtime into a
library-owned prefix, why the shim beats
createRequire, and why a model without a provider is now an error
MIT