proofbook-sentinel

Find out how many LLM calls your tracing is silently missing.

The sentinel counts every outbound request your Node process makes to a model provider, at the transport layer, and checks each one against your existing OpenTelemetry instrumentation. At exit it tells you how many calls your tracing actually captured, and names the ones it did not.

proofbook-sentinel  claims-api  4m12s

  api.openai.com  /v1/chat/completions        8,412   captured
  api.anthropic.com  /v1/messages             5,786   captured
  api.deepseek.com  /v1/chat/completions          5   UNCAPTURED
  llm.internal.acme.com  /v1/chat/completions     93   unknown provider

  14,296 calls, 14,198 captured (99.3%), 5 uncaptured, 93 unknown

  5 calls to api.deepseek.com are not covered by any
  instrumentation. Add an adapter or these are invisible to your traces.

It is not a proxy, not a tracer, not a guardrail, and not a compliance tool. It records no content and produces no evidence. It answers exactly one question: how many calls is your instrumentation missing?

Why this number is invisible to your tracing

Semantic LLM instrumentation (OpenLLMetry, OpenInference, the OTel GenAI packages) works by patching SDK methods. It only ever sees calls that went through the method it patched. It misses services nobody instrumented, apps that were bundled with Webpack or esbuild so the loader hooks never fired, raw fetch calls someone wrote because it was quicker, and providers added after instrumentation was set up.

Every one of those produces a clean-looking trace with a hole in it and no signal that the hole exists. A dashboard with 14,198 spans looks identical whether the true number of calls was 14,198 or 20,000, because the numerator and the denominator come from the same blind mechanism.

The sentinel measures the denominator independently, one layer down, where every call has to pass no matter how it was made.

Install

npm install proofbook-sentinel

Node 18.19 or newer. ESM and CJS both work. @opentelemetry/api is an optional peer dependency: without it the sentinel still counts calls and reports coverage as unknown.

Quick start

Zero code changes, works in Docker, PM2, systemd, Kubernetes and most PaaS:

NODE_OPTIONS='--import proofbook-sentinel/register' node app.js

Or one import, for platforms that do not expose runtime flags:

import 'proofbook-sentinel/register';  // first line of the entry file

Or programmatic, when configuration must live in code:

import { start } from 'proofbook-sentinel';

const sentinel = start({
  serviceName: 'claims-api',
  extraHosts: ['llm.internal.acme.com'],
  outFile: './sentinel.json',
});

This survives bundling. The sentinel patches runtime globals (globalThis.fetch, the undici dispatcher, node:http), not module imports, so it needs no loader hooks and does not care what your bundler did to your application code. That is the case the tool exists for.

What it never records

Request bodies. Response bodies. Prompts. Completions. Headers. API keys. Query strings. Full URLs. Model names read from bodies. User identifiers. IP addresses. Timings that could fingerprint content.

What it keeps, in memory, aggregated: host, normalised path (identifiers stripped, so /v1/messages/msg_abc123 becomes /v1/messages/:id), provider name, coverage state, a counter and a status class.

The sentinel makes no network calls of its own, ever. No telemetry, no version check, no update ping. Audit it; the whole thing reads in one sitting and has zero runtime dependencies.

How coverage is decided

At the moment a call leaves the process, the sentinel asks whether an OpenTelemetry span that looks like LLM instrumentation is active. Three answers are possible, and the third one matters:

• captured: a span with gen_ai.* or llm.* attributes, or an instrumentation-shaped name, was active. Your tracing saw this call.
• uncaptured: no span at all was active. Nothing saw this call.
• ambiguous: a span was active but did not look like LLM instrumentation, for example an HTTP server span wrapping an uninstrumented call. This is never reported as a confirmed miss.

The report then speaks with two voices. An endpoint where not a single call was captured gets the plain statement: nothing is covering it. An endpoint with a mix of captured and uncaptured calls gets a softer note, because that pattern can mean an uninstrumented code path or async context lost in a queue, a batch or a worker, and the sentinel does not accuse when it cannot tell. Run proofbook-sentinel doctor to find out which one you have.

When the number could be wrong, and what the sentinel does about it

The coverage check depends on Node's AsyncLocalStorage carrying trace context to the point where the request leaves. In some setups it does not: certain worker pools, some batching libraries, some bundler output. In those cases a captured call can look uncaptured, which would be a false accusation.

So the sentinel tests its own footing at startup and degrades honestly:

• @opentelemetry/api not installed: calls are counted, coverage is reported as unknown. Bare counting is still useful.
• OTel present but context propagation provably broken (the startup smoke test fails): calls are counted, coverage is reported as unknown, and the reason is printed. The sentinel would rather withdraw the claim than print a confident wrong number.
• Everything healthy: full per-call coverage.

If a report surprises you, proofbook-sentinel doctor shows what was patched, what was already patched by something else, whether OTel is present, and whether context propagation actually works.

Known limits, stated here rather than discovered by you: calls made over gRPC (some Google Vertex SDKs) are not seen, since the sentinel counts REST traffic. SDKs that construct their own undici Agent and pass it explicitly bypass the global dispatcher and are not seen either. Both undercount, never overcount. SDK retries count as separate calls, so the sentinel's total can legitimately exceed the span count in your tracing UI.

One more, and it matters for reading the report: "captured" means an LLM-shaped span was active in the calling context at the moment the request left. The sentinel cannot see your exporter, so it infers coverage from context rather than observing what your backend received. Instrumentation that creates spans without making them active (plain startSpan with no context.with) is invisible to that check and reads as uncaptured. Every mainstream GenAI instrumentation sets its spans active, because nested spans depend on it, but if the report accuses an endpoint you know is traced, a hand-rolled wrapper doing exactly this is the first thing to look for.

Configuration

Environment variables, all optional:

PROOFBOOK_SENTINEL_HOSTS      comma-separated extra hosts to count:
                              self-hosted models, gateways, proxies.
                              vLLM, Ollama, LiteLLM and TGI live here.
PROOFBOOK_SENTINEL_SERVICE    service name in the report
                              (falls back to OTEL_SERVICE_NAME)
PROOFBOOK_SENTINEL_JSON       write a structured report to this path on exit
PROOFBOOK_SENTINEL_QUIET      1 to suppress the exit summary, for CI
PROOFBOOK_SENTINEL_METRICS    1 to emit proofbook.sentinel.calls as an OTel
                              counter to your existing metrics backend

The same options exist on start() for programmatic use. If you run a self-hosted model and do not set PROOFBOOK_SENTINEL_HOSTS, the sentinel will truthfully report zero calls and you will conclude it is broken, so set it.

CLI

proofbook-sentinel doctor        what will be patched, what is already
                                 patched, whether OTel and async context
                                 propagation are working
proofbook-sentinel hosts         the endpoint registry this build knows
proofbook-sentinel report x.json summarise a report written by
                                 PROOFBOOK_SENTINEL_JSON

Safety

The sentinel must never break your application, so every patch site is wrapped, errors are swallowed and counted, and after ten internal errors it disables itself entirely, prints one warning and leaves your app running untouched. If another library already wrapped fetch, the sentinel patches on top and says so in the report; it never unwraps anyone else's patch.

Overhead is one classification, one context read and one counter increment per intercepted call, benchmarked under 50 microseconds.

Runnable examples

The examples/ directory has three self-contained examples. Each starts a local mock provider, so nothing needs an API key and nothing leaves your machine. Clone the repository, npm install && npm run build, then:

sh examples/01-zero-code/run.sh       # attached via NODE_OPTIONS only
node examples/02-programmatic/app.mjs # start() in code, JSON report on exit
node examples/03-with-tracing/app.mjs # real OTel tracing, partial coverage

The third one is the whole tool in one screen. It instruments six chat calls, leaves one raw call next to them, and never instruments the embeddings endpoint at all, which is exactly how production drifts:

proofbook-sentinel  tracing-example  0s

  127.0.0.1  /v1/chat/completions          7   mixed
  127.0.0.1  /v1/embeddings                4   UNCAPTURED

  11 calls, 6 captured (54.5%), 5 uncaptured

  4 calls to 127.0.0.1 /v1/embeddings are not covered by any
  instrumentation. Add an adapter or these are invisible to your traces.

  1 of 7 calls to 127.0.0.1 /v1/chat/completions had no active
  trace context. That is either an uninstrumented code path or async
  context lost in a queue, batch or worker. Run proofbook-sentinel doctor.

Adding a provider

The registry maps provider hosts and model API path shapes. When the sentinel sees a known path shape on a host it does not recognise, it counts the call as unknown provider and names the host in the report. If that host is a real provider, open a PR against src/registry.ts; that is exactly how the registry is meant to grow.

Why does this exist

We build Proofbook, which turns agent telemetry into audit evidence. The first question an auditor asks about telemetry is "how do you know that is all of them", and we could not find any tool that could answer it, so we built the measurement layer and are shipping it standalone. If the sentinel shows you a gap, you have found the exact problem the larger product solves; if it shows you 100% coverage, you have a number nobody else can show, and the tool has cost you nothing.

License

MIT