en plugin dev apis llm requests

LLM request configuration

These options require a Core build containing Core #2661. A merged PR does not imply a published release. Pass them through the existing extra_args; plugin and Runner signatures, message types and authorization remain unchanged. See LangBot API and Context APIs.

Retries and timeouts

The following non-streaming example configures the request and reads complete usage:

from langbot_plugin.api.entities.builtin.provider.message import Message

result = await self.plugin.invoke_llm_with_usage(
    llm_model_uuid=model_uuid,
    messages=[Message(role="user", content="Hello!")],
    timeout=110,
    extra_args={
        "request_policy": {
            "retries": 1,
            "total_timeout": 90,
            "first_event_timeout": 30,
            "stream_idle_timeout": 30,
            "connect_timeout": 10,
            "read_timeout": 60,
        },
    },
)
print(result.message.content)
if result.usage is not None:
    print(result.usage.model_dump())

Runner ctx.api.invoke_llm* accepts the same extra_args, subject to run authorization and remaining time. Calls through self.plugin also inherit these restrictions inside a run context.

timeout= is the SDK/RPC waiting limit (120 seconds by default for non-streaming calls). Core's request_policy controls the inner request. Keep its total budget within the enclosing RPC/Runner deadline; raising it cannot extend that deadline. Streaming methods have no timeout= argument and remain bounded by SDK/Runner waiting limits.

request_policy field	Default	Meaning
retries	0	Additional attempts after the first, integer 0–5
total_timeout	300	Whole invocation budget, including backoff
first_event_timeout	120	Wait for the first upstream event, not the first visible text
stream_idle_timeout	120	Idle time between stream events
connect_timeout	10	Connection timeout
read_timeout	120	Transport read timeout
write_timeout	30	Transport write timeout
pool_timeout	10	Connection pool acquisition timeout
backoff_base / backoff_max	0.5 / 8	Exponential backoff base and cap, with full jitter
attribution	True	Automatic application attribution headers
conversation_header	None	Generated session header name, described below

Times are seconds; timeouts must be finite and positive. Unknown policy fields are rejected. Precedence: instance llm_request_policy, model extra_args, then call extra_args. Within one layer, request_policy.retries wins over legacy num_retries / max_retries. A call-level legacy value still overrides model policy; 0 disables retries. Legacy numeric extra_args["timeout"] sets read, first-event and stream-idle limits without extending the total budget.

Authentication, invalid-request and exhausted-quota failures are not retried. Retries stop after any upstream event, including reasoning or usage. Retry-After is bounded by the remaining budget. Core disables LiteLLM and provider SDK retries for managed calls; gateways may independently retry. This adds no tool replay, model fallback or embedding/rerank retries. Legacy third-party requesters still own their transport configuration.

Reasoning budgets

Existing reasoning_level behavior is unchanged. For a numeric budget, pass this object as extra_args or save it in model advanced parameters. Omitting level means enabled:

{
  "reasoning_config": {"budget_tokens": 2048},
  "max_tokens": 4096
}

Do not combine a budget with an effort level such as medium, disabled, or native reasoning parameters. Direct Anthropic calls require explicit max_tokens greater than the budget; Core never raises that limit automatically.

Precedence: persisted model reasoning_config, model extra_args.reasoning_config, call extra_args.reasoning_config, then an explicit Runner level override. Each object replaces the previous one; an effort override clears a lower-priority budget.

Numeric budgets map to native Anthropic, Gemini 2.5, supported Qwen models and OpenRouter routes. Direct OpenAI/Codex and adaptive-only Claude models retain effort control. Check the model's reasoning_capabilities.budget_tokens for support and bounds. Unsupported budgets fail before HTTP. Providers/gateways may impose stricter limits; OpenRouter may translate a numeric budget into effort.

Prompt caching

extra_args.prompt_caching is consumed by Core, not forwarded unchanged. Call configuration replaces model configuration. The default {"mode": "provider_default"} adds no fields. This is not a local response cache.

Configuration	Behavior
{"mode": "openai", "retention": "in_memory"}	Generates a model-and-conversation-scoped prompt_cache_key when conversation context exists; optional retention also accepts 24h if supported by the target
{"mode": "anthropic", "ttl": "5m"}	Adds an ephemeral breakpoint to the last text block of the leading system/developer prefix; ttl accepts 5m or 1h

OpenAI mode applies to compatible routes and OpenRouter OpenAI models supporting these options. Explicit native prompt_cache_key, prompt_cache_retention and prompt_cache_options win. Anthropic mode supports native Claude and OpenRouter Claude. Existing native cache markers are preserved without adding duplicates. Core does not reorder tools/history or mark user messages, and inserts nothing without an eligible prefix.

Hints do not guarantee hits. Inspect usage.model_dump() for provider fields such as prompt_tokens_details.cached_tokens or cache_read_input_tokens; missing counts are not zero. Minimum prefix lengths, retention eligibility and pricing are provider-controlled. Codex's restricted Responses body does not accept this configuration object; existing server caching remains. General Responses migration and Gemini explicit cache-resource management are outside this change.

Application and session headers

Direct OpenRouter calls receive HTTP-Referer: https://langbot.app, X-OpenRouter-Title: LangBot, and, with conversation context, x-session-id: lb-<opaque scoped hash>. Custom gateway addresses do not automatically inherit these headers. request_policy.attribution=False disables only automatic application attribution. Explicit extra_headers remain and take precedence case-insensitively.

A gateway can use an explicitly configured session header:

{
  "request_policy": {"conversation_header": "x-litellm-session-id"},
  "extra_headers": {
    "HTTP-Referer": "https://langbot.app",
    "X-OpenRouter-Title": "LangBot"
  }
}

Supported names: x-session-id, x-litellm-session-id, x-langbot-conversation-id. The last is a custom convention, not a universal provider standard, and provides no automatic routing or caching benefit. Generated identifiers are scoped by workspace/bot/pipeline/session and contain no raw session ID or prompt. They are pseudonymous, not a guarantee of anonymization, and are omitted without context. New API and other gateways may filter or rewrite headers; verify forwarding rules and actual outbound requests.

Stream cancellation and compatibility

Use aclosing when leaving a stream early. Allow task cancellation exceptions to propagate:

from contextlib import aclosing
from langbot_plugin.api.entities.builtin.provider.message import Message

async with aclosing(self.plugin.invoke_llm_stream_events(
    llm_model_uuid=model_uuid,
    messages=[Message(role="user", content="Hello!")],
    extra_args={"request_policy": {"retries": 0, "total_timeout": 90}},
)) as events:
    async for event in events:
        if event.usage is not None:
            print(event.usage.model_dump())
        if event.chunk is not None and event.chunk.content:
            print(event.chunk.content)
            break

This example intentionally stops after the first nonempty content chunk and may miss final usage. For a complete response, remove break; a final event can have usage without a chunk. invoke_llm still returns Message, and invoke_llm_stream still yields MessageChunk. Choose invoke_llm_with_usage or invoke_llm_stream_events for accounting. Usage may be None; nested cache/reasoning counts and provider-specific fields are preserved.

SDK ≤0.7.11 has a proxy-layer defect that delays upstream cancellation on early close. Deterministic close requires langbot-plugin ≥0.7.12, which publishes the fix from SDK #177. A Core dependency pin does not automatically upgrade a separately deployed Runtime or plugin development environment; upgrade those executing environments too. Ordinary signatures, payloads and return types remain compatible. Total deadlines include consumer time, but cannot interrupt a consumer while the generator is suspended at yield; expiration is enforced on the next pull.

Diagnostics

Core monitoring records expose details.timing, complete usage and structured error; legacy SDK exception transport is unchanged. Structured errors include code, status_code, retryable, retry_after, request_id and phase. Do not assume these fields exist on legacy exceptions received by plugins.

Millisecond timings include preparation_ms, transport_setup_ms, first_event_ms, first_reasoning_ms, first_content_ms, first_visible_chunk_ms, total_ms, plus attempts and retry_wait_ms. The first event may be reasoning/control, and the first visible chunk may include reasoning. request_started_ms marks the SDK call, not a packet leaving the socket. Missing fields mean unobserved, not zero. Monitoring retains partial usage received before cancellation but cannot reconstruct missing upstream accounting.