vern-llm@3.0.0

3.0.0

Major Changes

• 9121676: Stopping a stream early cancels it, tool arguments come back as the argumentsSchema output, and invalid rate limits throw.

  Streaming: leaving a for await over chunks before the end, by break, return, a throw in the loop body, or return(), cancels the provider stream. finalResult rejects with LLMError('aborted') and the rate limiter slot is freed. A stream that already finished is left alone. On a cachedCall() miss, only that caller leaves: the shared stream keeps running, and is cached, while another caller waits on it.

  Tools: ToolCall.arguments is now the argumentsSchema output, so defaults, coercion and transforms apply. Tools without a schema are unchanged.

  Rate limiting: RateLimiter throws LLMError('invalid_params') at construction for a negative, NaN or Infinity limit, a requestsPerMinute or tokensPerMinute between 0 and 1, a fractional maxConcurrent or maxQueueSize, a maxQueueMs past 2147483647, a bad aimd.proactiveFloor, and aimd without requestsPerMinute. 0 still means unlimited.

• df1d501: Adapters import from vern-llm/adapters, fromGemini takes the top level client, and fromBedrock moves to its own package, vern-llm-bedrock.

  Adapters: every from* factory, their option types, ModelCapabilityOverride, parseSseStream and SSE_PING moved off the root entry to the new vern-llm/adapters subpath. LLMClient, the wire types and everything else stay on vern-llm. Both entries share one LLMError class, so instanceof checks keep working. The rules adapter packages share with the built-in adapters are exported there too: supportsNativeStructuredOutput, rejectsForcedToolChoice, assertForcedToolChoiceSupported, throwMissingForcedJsonSchemaTool, assertForcedJsonSchemaToolInputIsObject, assertSupportedImageMimeType, the Claude reasoning budget helpers, and planClaudeStructuredOutput and resolveClaudeThinking, which apply Claude's structured output and thinking rules to a request.

  Gemini: pass ai, the GoogleGenAI instance. fromGemini(ai.models) throws a plain Error at construction that says to pass ai, and GeminiClient now describes the top level client, with the model methods under models.

  Bedrock: fromBedrock now ships in vern-llm-bedrock. Install it with @aws-sdk/client-bedrock-runtime, import fromBedrock from it, and pass a BedrockRuntimeClient. The hand-written { converse, converseStream } client form and BedrockConverseClient are gone. vern-llm no longer loads the AWS SDK at all.

• 5ec7a11: promptTokens is now every input token the provider processed, cache reads and writes included, and TokenUsage reports the cache split.

  Usage: fromAnthropic adds cache_read_input_tokens to promptTokens and totalTokens, on top of the cache writes it already counted. Streams take the input counts from the final message_delta when it sends them, since they are cumulative totals, as Anthropic's own SDK does. OpenAI compatible adapters and fromGemini already worked this way, and fromFetch reports what mapResponse returns. TokenUsage gains cacheReadTokens, cacheWriteTokens and cacheWriteTokensByTtl, each a subset of promptTokens and undefined when the provider doesn't report it. The wire usage gains prompt_tokens_details with cached_tokens, cache_write_tokens and cache_write_tokens_by_ttl. fromOpenAICompatible fills cached_tokens from DeepSeek's prompt_cache_hit_tokens, and fromFetch's mapResponse can return the three cache counts.

  Rate limiting: LLMClient gains cacheReadsCountTowardRateLimit, default true. fromAnthropic sets it to false, since Anthropic's input limit skips cache reads, so the token rate limiter reconciles against the total less cacheReadTokens and its numbers don't change.

  // Before: cache reads were left out of promptTokens on Anthropic
  const total = usage.promptTokens + (usage.cacheReadTokens ?? 0);
  
  // After: promptTokens already includes them
  const total = usage.promptTokens;

  Existing code keeps compiling. At runtime, promptTokens and totalTokens are higher on Anthropic calls that read the prompt cache, in onUsage, the 'usage' event and OpenTelemetry's gen_ai.usage.input_tokens. Code that added cache reads to promptTokens by hand now counts them twice. See Migration Notes.

Minor Changes

• d02c2a7: Caching, rate limiting, retry, fallback, middleware, and streaming fixes, plus a new response_truncated error code.

  Caching: NormalizedCacheAdapter no longer strips punctuation or case, so "2+2" and "2-2" no longer share an answer. It only normalizes Unicode composition, line endings, and outer whitespace. InMemoryCacheAdapter copies values on write and read, and doesn't cache a value it can't copy faithfully (a class instance or function anywhere inside it). A missing, NaN, zero, or negative ttl stores nothing instead of an entry that never expires. cachedCall warns when ttl is missing or NaN and skips the write. TieredCacheAdapter no longer serves a value after L2 has expired it.

  Rate limiting: the default token estimate counts each image as 1,600 tokens instead of reading its base64 as text. AIMD shrinks at most once per 60 seconds. A blank rate limit header is treated as missing, and all zero usage keeps the estimate. An error thrown by a limiter's estimate or acquire never counts toward the circuit breaker.

  Retries: a negative or NaN maxRetries means one attempt instead of none, with a warning. Output cut off at max_tokens that fails to parse throws LLMError('parse') with the new code response_truncated, which is retried and never counts toward the breaker. fromAnthropic, fromGemini, and fromBedrock report when they stopped at max_tokens. Custom clients can set finish_reason: 'length' on a choice.

  Fallback: a per call model override applies to the primary target only. Fallback targets run their own model, and every report of the model names the one each target ran.

  Middleware: ctx.own persists for one middleware across all its hooks within a logical call. An unnamed middleware is labeled [i] everywhere, including ctx.registeredMiddlewareNames, and a name that matches another entry's [i] label throws at construction. The response body in a fromFetch error passes through redact. A circuit transition with no call behind it reaches middleware onEvent.

  Streaming: a stream that fails before its first content chunk is retried and falls back, even after a keep-alive ping opened it. call() still resolves on the first ping, so a long thinking phase stays outside timeoutMs. chunks and finalResult follow whichever attempt produces content, and meta.current is updated in place if a different target answers.

  Existing code keeps compiling unless it has an exhaustive switch over LLMErrorCode, which needs a case for response_truncated. At runtime: prompts that only matched through case or punctuation stop sharing a cache entry, code that matched an unnamed middleware in registeredMiddlewareNames by its bare index must use [i], and a failure between a ping and the first content chunk now retries instead of rejecting finalResult.

• 63c976f: A call can seed ctx.state before any middleware runs, with the new state param and stateEntry helper.

  const routeDecisionKey = createStateKey<RouteDecision>('router.decision');
  
  await llm.call({ userContent, state: [stateEntry(routeDecisionKey, decision)] });

  stateEntry checks the value against the key's type. Raw [key, value] pairs also work, and a later entry wins over an earlier one with the same key. createMiddlewareStateBag accepts the same entries. The seed is never sent to the provider, never reported on events or usage, and not part of the cachedCall cache key. With cachedCall, each caller's wrap sees that caller's state, while the shared request sees the state of the caller that started it. Anything that isn't an array of [key, value] pairs throws invalid_params with the new code invalid_state, before any hook runs.

  Existing code keeps compiling, and nothing changes at runtime until a call passes state.

• f3ab557: A call takes a context, plain JSON that middleware, events and usage carry.

  await llm.call({
    userContent,
    context: { tenantId: 't1', routing: { only: ['bedrock'] } },
  });

  Middleware reads it as ctx.context in every hook and stage, fallback attempts included. Every event the call reports has it as event.context, and so does the TokenUsage in onUsage and the 'usage' event, so a cost middleware can attribute spend to a tenant. It is a frozen copy made when the call starts, is never sent to the provider, and is not part of the cachedCall cache key. VernLLM defines no keys: whatever is built on it owns its own.

  Anything that isn't a plain JSON object throws LLMError of type invalid_params with the new code invalid_context, before any hook runs or request is sent.

  Existing code keeps compiling unless it has an exhaustive switch over LLMErrorCode, which needs a case for invalid_context, or builds AttemptContext/PreDispatchContext objects by hand, which need context. At runtime nothing changes until a call passes context.

• f1ce144: Target names must be unique.

  VernLLM now throws at construction when two targets, the primary or a fallback, share a name. Defaults count, so a fallback named primary collides with an unnamed primary. Usage and events identify a target by name, so a shared name merged their data.

• 987189c: Middleware dispatch hook, adapter context, a configurable Retry-After cap, an opt-in reader stall timeout, Claude thinking blocks in tool loops, and request fixes for newer ...

vern-llm-redis@0.3.0

0.3.0

Minor Changes

• 76331c9: Circuit breaker and rate limiter fixes for many processes sharing one Redis, a vern-llm-redis/clients subpath, and stricter option validation.

  Circuit breaker: onStateChange fires once per real change in each process, however many replies, messages or polls carry it, and a stale reply no longer rolls the local state back. A call let through never takes a second trial slot, a burst of rejected calls asks for one slot at a time, releaseTrial only gives its slot back, and slots held for an abandoned trial are dropped. A running trial call renews its lease, so a slow trial is never handed over mid call. cooldownBackoff uses core's formula, so a cooldown is never shorter than cooldownMs unless maxMs is. With isolateByModel, calls with no model use keyPrefix itself, apart from a model named default, and the startup scan no longer reads another adapter's keys under a longer prefix.

  Rate limiter: maxQueueSize counts waiters across every process. The AIMD ceiling lives in its own key, {keyPrefix:rpm}:aimd, which survives an idle bucket, starts no higher than aimd.maxCapacity, and shrinks at most once a minute across every process, like core.

  Both: a Redis failure in acquire, prepare or readState rejects with LLMError('network') and code connection_failed. Several adapters can share one subscriber, and dispose() removes its own listener. Scripts are sent by SHA1 after the first call. RedisClient gains optional evalsha and RedisSubscriber optional off, both already on ioredis and now on the node-redis wrappers.

  Requires vern-llm 3.0.0 as a peer. See the Redis migration notes.

  Breaking: fromIoredis, fromIoredisSubscriber, fromNodeRedis and fromNodeRedisSubscriber are exported from vern-llm-redis/clients, no longer from the root entry. redisRateLimit validates limits with core's rules and messages, and redisCircuitBreaker throws LLMError('invalid_params') for every bad option, including a rolling window mistake that used to throw RangeError and a threshold that is not an integer of at least 1. Calls with no model under isolateByModel, and the AIMD ceiling, start fresh once after upgrading.

Patch Changes

• Updated dependencies: vern-llm@3.0.0

vern-llm-otel@0.2.0

0.2.0

Minor Changes

• f3ab557: Cache token attributes and custom span events. Needs vern-llm 3.0.0.

  Attempt spans set gen_ai.usage.cache_read.input_tokens and gen_ai.usage.cache_write.input_tokens when the provider reports them. Both are subsets of gen_ai.usage.input_tokens, as the GenAI conventions define. The token usage metric still records input and output only.

  Each event a middleware reports through ctx.emit becomes a span event on the call span, named after the event, with vernllm.event.source. The event's data is whatever the middleware passed, so it is left out unless you opt in, and no metric is recorded for custom events.

  otelMiddleware({ customEvents: { data: true, maxLength: 2048 } }); // default: on, no data
  otelMiddleware({ customEvents: false }); // record none

  Existing code keeps compiling. At runtime, spans gain the two cache attributes on calls that use the prompt cache, and a call whose middleware emits events gains span events. Set customEvents: false to keep spans as they were.

• 82ffcb4: Attempt spans now follow the provider request, and time to first chunk skips keep-alive pings. Needs vern-llm 3.0.0.

  An attempt span starts when the request is sent, after the rate limiter and every transform, and is the active span while it runs, so an HTTP client span nests under it. Its duration no longer includes rate limit waiting or later transforms. A call rejected before sending, by an open circuit, a full rate limit queue, a throwing transform, or a capability the model lacks, gets no attempt span, no duration point, and no count in vernllm.total_attempts.

  Time to first chunk runs from sending the request to the first content chunk. Pings, including those Claude sends while reasoning, no longer count, so reasoning models report a later, accurate value.

  gen_ai.provider.name uses the provider the adapter names before guessing from the model id. providerNames still wins.

  Input capture reads the request as sent, after every transform, so it is never skipped for ordering and vernllm.content.skipped_reason is gone. The default priority is -1000 whether or not capture is on.

  Existing code keeps compiling. At runtime, dashboards see shorter attempt spans, fewer attempt spans, and later time to first chunk values for streams that ping first.

Patch Changes

• 8846fbb: Telemetry fixes.

  gen_ai.provider.name no longer uses the target label. Unmapped targets are inferred from the model id (Claude with an @version suffix is gcp.vertex_ai), and _OTHER is used when that fails. Dashboards filtering on primary or fallback[0] need updating, or set providerNames.
  Input capture is skipped when any middleware sorts after it, so a later redactor can never be bypassed. The span gets vernllm.content.skipped_reason and a warning is logged once. Middleware without a transform counts too, so give otelMiddleware the highest priority when capturing.
  maxLength now caps the whole attribute, JSON and marker included. Older messages that don't fit are dropped whole, not left as marker only parts. An attribute whose structure alone doesn't fit is left out instead of exceeding the limit.
  The normalizeModel cache is bounded.
  gen_ai.request.temperature is omitted for Anthropic, and for Claude on Bedrock, when thinking is on, since it is never sent.
  An attempt whose closing signal was missed ends with vernllm.attempt.outcome set to unknown instead of an error, and records no duration.

• Updated dependencies: vern-llm@3.0.0

vern-llm-bedrock@0.1.0

0.1.0

Minor Changes

• 5ec7a11: promptTokens now includes cache reads and writes, and the cache split is reported on TokenUsage.

  fromBedrock adds cacheReadInputTokens and cacheWriteInputTokens to Converse's inputTokens, which excludes them. Before, promptTokens plus completionTokens fell short of Bedrock's totalTokens whenever prompt caching was on. Now they agree, and totalTokens is still what Bedrock reports. TokenUsage gains cacheReadTokens, cacheWriteTokens and cacheWriteTokensByTtl, the last read from cacheDetails and keyed by its TTL label.

  // Before: inputTokens alone
  usage.promptTokens; // 10
  
  // After: inputTokens plus cache reads and writes
  usage.promptTokens; // 10010
  usage.cacheReadTokens; // 8000
  usage.cacheWriteTokens; // 2000

  Existing code keeps compiling. At runtime, promptTokens is higher on calls that use the prompt cache, in onUsage, the 'usage' event and OpenTelemetry's gen_ai.usage.input_tokens. The token rate limiter is unchanged, since Bedrock's total already included cache tokens. See Migration Notes.

• df1d501: Add vern-llm-bedrock, the AWS Bedrock Converse adapter for vern-llm, built on the AWS SDK's BedrockRuntimeClient. @aws-sdk/client-bedrock-runtime and vern-llm 3.0.0 are peer dependencies.

  import { BedrockRuntimeClient } from '@aws-sdk/client-bedrock-runtime';
  import { VernLLM } from 'vern-llm';
  import { fromBedrock } from 'vern-llm-bedrock';
  
  const llm = new VernLLM({
    client: fromBedrock(new BedrockRuntimeClient({ region: 'us-east-1', maxAttempts: 1 })),
    model: 'anthropic.claude-sonnet-4-5-20250929-v1:0',
  });

  fromBedrock takes a BedrockRuntimeClient and sends ConverseCommand and ConverseStreamCommand through its send. Requests and responses are typed with the SDK's own types, and stream events the adapter doesn't model, including the SDK's generated $unknown member, are skipped.

  It carries everything fromBedrock did in vern-llm 2.x: native structured output, forced tool calls for jsonSchema elsewhere, the forcedToolChoiceUnsupportedModels rule for newer Claude models, Claude reasoning kept in history for tool loops, keep-alive pings while reasoning streams, and a local error for toolChoice: 'none' with tools. The adapter names itself bedrock with provider aws.bedrock on LLMClient.adapter.

Patch Changes

• 5a5ff4b: Warn when the AWS SDK retries on its own.

  fromBedrock logs a warn at VernLLM construction, once per process, when the client's resolved maxAttempts is above 1, since those retries are hidden from VernLLM's retries, circuit breaker, rate limiter, and events. The client is never modified. Pass maxAttempts: 1 to the BedrockRuntimeClient to silence it. An explicit retryStrategy overrides maxAttempts, so remove it or configure it for one attempt. The warning reads the attempt count of the SDK's own StandardRetryStrategy and AdaptiveRetryStrategy, including one you pass in. A fully custom strategy exposes no count, so it can't be checked and gets no warning even if it still retries.

• Updated dependencies: vern-llm@3.0.0

vern-llm@2.9.1

2.9.1

Patch Changes

• 805908b: Fix circuit breaker recovery with the rolling policy: a successful half open trial now resets the tripping window, so old failures no longer reopen the circuit right after recovery. Fix withTimeout to race the call against the timer, so a client that ignores the abort signal or throws its own abort error type still produces a timeout error.

vern-llm-redis@0.2.1

0.2.1

Patch Changes

• 34dc0c0: Fix a missed wake in the rate limiter. A release published before the subscriber's SUBSCRIBE was confirmed was lost, so a waiter only woke on the next poll. acquire now waits for the subscription first, bounded by the poll interval.
• Updated dependencies [805908b]
  • vern-llm@2.9.1

vern-llm@2.9.0

What's Changed

• feat(event)!: usage by @LakBud in #161
• feat(test): improve coverage to 100% by @LakBud in #164
• fix(bedrock): validate legacy jsonSchema tool calls by @LakBud in #165
• feat(circuit-breaker): add pluggable adapter support by @LakBud in #166
• feat(packages): redis by @LakBud in #167
• fix(vern-llm): release stuck half-open breaker trial by @LakBud in #169
• feat(vern-llm-redis): crash safe shared state for the breaker, limiter and cache by @LakBud in #170
• fix(vern-llm): deliver middleware onEvent synchronously by @LakBud in #175
• feat: vern-llm-otel by @LakBud in #176
• chore(deps): Update aws bedrock provider by @renovate[bot] in #171
• chore(deps): Update dependency @anthropic-ai/sdk to ^0.127.0 by @renovate[bot] in #172
• chore(deps): Update dependency @google/genai to ^2.23.0 by @renovate[bot] in #173
• chore(deps): Update all non-major dependencies by @renovate[bot] in #168
• chore(deps): migrate Renovate config by @renovate[bot] in #177
• chore(deps): Update github actions by @renovate[bot] in #162
• chore: version packages by @github-actions[bot] in #163

Full Changelog: https://github.com/LakBud/vernLLM/compare/vern-llm@2.8.0...vern-llm@2.9.0

vern-llm-redis@0.2.0

What's Changed

• feat(event)!: usage by @LakBud in #161
• feat(test): improve coverage to 100% by @LakBud in #164
• fix(bedrock): validate legacy jsonSchema tool calls by @LakBud in #165
• feat(circuit-breaker): add pluggable adapter support by @LakBud in #166
• feat(packages): redis by @LakBud in #167
• fix(vern-llm): release stuck half-open breaker trial by @LakBud in #169
• feat(vern-llm-redis): crash safe shared state for the breaker, limiter and cache by @LakBud in #170
• fix(vern-llm): deliver middleware onEvent synchronously by @LakBud in #175
• feat: vern-llm-otel by @LakBud in #176
• chore(deps): Update aws bedrock provider by @renovate[bot] in #171
• chore(deps): Update dependency @anthropic-ai/sdk to ^0.127.0 by @renovate[bot] in #172
• chore(deps): Update dependency @google/genai to ^2.23.0 by @renovate[bot] in #173
• chore(deps): Update all non-major dependencies by @renovate[bot] in #168
• chore(deps): migrate Renovate config by @renovate[bot] in #177
• chore(deps): Update github actions by @renovate[bot] in #162
• chore: version packages by @github-actions[bot] in #163

Full Changelog: https://github.com/LakBud/vernLLM/compare/vern-llm@2.8.0...vern-llm-redis@0.2.0

vern-llm-otel@0.1.0

What's Changed

• feat(event)!: usage by @LakBud in #161
• feat(test): improve coverage to 100% by @LakBud in #164
• fix(bedrock): validate legacy jsonSchema tool calls by @LakBud in #165
• feat(circuit-breaker): add pluggable adapter support by @LakBud in #166
• feat(packages): redis by @LakBud in #167
• fix(vern-llm): release stuck half-open breaker trial by @LakBud in #169
• feat(vern-llm-redis): crash safe shared state for the breaker, limiter and cache by @LakBud in #170
• fix(vern-llm): deliver middleware onEvent synchronously by @LakBud in #175
• feat: vern-llm-otel by @LakBud in #176
• chore(deps): Update aws bedrock provider by @renovate[bot] in #171
• chore(deps): Update dependency @anthropic-ai/sdk to ^0.127.0 by @renovate[bot] in #172
• chore(deps): Update dependency @google/genai to ^2.23.0 by @renovate[bot] in #173
• chore(deps): Update all non-major dependencies by @renovate[bot] in #168
• chore(deps): migrate Renovate config by @renovate[bot] in #177
• chore(deps): Update github actions by @renovate[bot] in #162
• chore: version packages by @github-actions[bot] in #163

Full Changelog: https://github.com/LakBud/vernLLM/compare/vern-llm@2.8.0...vern-llm-otel@0.1.0

vern-llm@2.8.0

What's Changed

• feat(detectSoftFailure): usage data by @LakBud in #155
• feat(middleware)!: ref for runsBefore and runsAfter instead of name by @LakBud in #158
• feat(middleware): requireRef by @LakBud in #159
• chore(deps): Update typescript tooling by @renovate[bot] in #147
• chore: version packages by @github-actions[bot] in #156

Full Changelog: https://github.com/LakBud/vernLLM/compare/vern-llm@2.7.0...vern-llm@2.8.0