Debug and Diagnostics

ChatHub provides several ways to enable diagnostic logging. This page explains each switch, what it does, and the safety boundaries.

Debug by Component

Component	How to Enable	What You See	Safe for Production?
tRPC API (lambda, edge, async)	`CHATHUB_DEBUG=1`	Request path + response timing	Yes
Database	`CHATHUB_DEBUG=1`	Connection timing	Yes
MCP Service	`CHATHUB_DEBUG=1`	Tool call timing + counts	Yes
Chat Router	`CHATHUB_DEBUG=1`	Message routing timing	Yes
MCP OAuth	`DEBUG=lobe-mcp:oauth-service`	OAuth flow status	Yes
Tool + MCP lifecycle	`CHATHUB_TOOLS_DEBUG=1` / `=verbose`	Structured batch/call outcomes for HTTP MCP, delegated, and server tools	Yes
Image generation lifecycle	`CHATHUB_IMAGE_DEBUG=1` / `=verbose`	Structured `/image` workspace submission, dispatch, provider, upload, and task state. Chat Image tool tiles use `CHATHUB_GENERATION_DEBUG`.	Yes
Knowledge Base lifecycle	`CHATHUB_KNOWLEDGE_DEBUG=1` / `=verbose`	Structured upload, chunk/embed, retrieval, and prompt-injection outcomes	Yes
Durable generation send path	`CHATHUB_GENERATION_DEBUG=1`	Structured client send/attach/sync + server enqueue/worker events for background conversation generation, including chat Image tool tile create/persist/attach	Yes
Topic compaction / memory dream	`CHATHUB_COMPACTION_DEBUG=1` / `=verbose`	Why topic compaction ran or skipped, estimated token parts, the context-window number used, and why the scheduled memory dream ran or skipped (`dream_scheduler_tick` / `dream_scheduler_settled`)	Yes
ChatGPT / Codex OAuth + chat	`CHATHUB_OPENAI_CODEX_DEBUG=1`	Structured device-login, refresh, credential overlay, and `chatgpt.com/backend-api/codex` HTTP/SSE outcomes (status, media type, SSE event count, stream lifecycle). No tokens, device codes, emails, or prompts.	Yes
SuperGrok OAuth	`CHATHUB_XAI_OAUTH_DEBUG=1`	Structured xAI device-login, refresh, overlay, and billing-fetch outcomes. No tokens, device codes, emails, or prompts.	Yes
Background conversation generation	`[conversation-generation]` logs; `DISABLE_CONVERSATION_WORKER=1`	Worker enqueue, heartbeat, sweep, and cancel. The same process still honors `CHATHUB_TOOLS_DEBUG`, `CHATHUB_KNOWLEDGE_DEBUG`, and `DEBUG_*_CACHE` on worker model/tool/RAG calls. Image `async_tasks` debug is separate. See Background Conversation Generation.	Yes
Model Runtime	`DEBUG=lobe-model-runtime:*`	Prompts + responses	No
Context Engine	`DEBUG=context-engine:*`	System roles + templates	No
Search	`DEBUG=lobe-search:*`	Search request bodies	No
OIDC Adapter	`DEBUG=lobe-oidc:adapter`	Session payloads	No
Group Chat (browser)	`NEXT_PUBLIC_CHATHUB_DEBUG=1`	Instructions + decisions	No
Chat input / text suggestions (browser)	visit any page with `?replacement_debug=1`	On-page overlay of raw input events + suggestion routing decisions	Yes
Auth Adapter	`DEBUG=lobe-next-auth:api:auth:adapter`	Adapter request data	No
Provider Streams	`DEBUG_OPENAI_CHAT_COMPLETION=1`	Raw stream chunks	No
Provider Request Shape	provider-specific `DEBUG_*_CHAT_COMPLETION=1`	Redacted request-shape summary plus raw debug logs	No
Provider Cache	provider-specific `DEBUG_*_CACHE=1`	Keyed request/usage correlations and normalized cache status	Yes
Image Generation (legacy)	`DEBUG=lobe-image:*`	Image prompts + URLs	No

ChatGPT / Codex stream outcomes

CHATHUB_OPENAI_CODEX_DEBUG=1 records chat_request_settled for the HTTP exchange and chat_stream_settled after the Responses lifecycle finishes. chat_request_settled ok only means HTTP 200. sseEventCount is a metric, not success: a failed or truncated stream can still parse one or more SSE frames.

`chat_stream_settled.outcome`	Meaning
`ok`	Completed stream (`onCompletion`)
`empty`	HTTP 200 with zero SSE events
`failed`	Provider `response.failed`
`incomplete`	Provider `response.incomplete` (`reason` is `max_output_tokens`, `content_filter`, or `response_incomplete`)
`unexpected_end`	Events arrived, then EOF without a terminal event (`reason=missing_terminal_event`)
`parse_error`	Malformed JSON or HTML (`reason` is `invalid_json` or `html_response`)
`cancelled`	Consumer cancelled the stream
`error`	Other terminal stream error (`reason` is the allowlisted terminal reason)

This flag does not dump Platform Chat Completions or OpenAI-compatible traffic. Those remain DEBUG_OPENAI_* and DEBUG_OPENAICOMPATIBLE_*.

Background generation: enqueue noise vs worker death

tRPC UNPROCESSABLE_CONTENT lines that mention "kagi", "lobe-artifacts", DALL·E, or non-HTTP MCP are expected deferrals. The server refuses to enqueue that turn because those tools cannot run on Graphile; the connected tab handles it instead. They are not a sign that the worker is down. Enabling Sandbox (lobe-sandbox) is not a deferral — see Sandbox (OpenSandbox). See Background Conversation Generation.

A dead worker looks different:

logs Cannot find module 'tslib' while loading graphile-worker (upgrade the image; DISABLE_CONVERSATION_WORKER=1 does not avoid that import)
no Worker connected / [conversation-generation] lifecycle lines after boot
Graphile tasks never succeed

Development-only API auth bypass

Local API debugging has two opt-in paths, both ignored unless NODE_ENV=development:

# Headerless local mock identity
ENABLE_MOCK_DEV_USER=1
MOCK_DEV_USER_ID=DEV_USER

# Or require both request headers:
# lobe-auth-dev-backend-api: 1
# lobe-auth-dev-secret: <matching value>
AUTH_DEV_BYPASS_SECRET=replace-with-a-random-local-secret

Lambda tRPC and WebAPI authentication, including /webapi/files, resolve the same trimmed MOCK_DEV_USER_ID and fall back to DEV_USER. Never enable these variables in a production deployment; production requests cannot use this bypass even if the variables or headers are present.

CHATHUB_DEBUG=1 (Server — Safe Default)

Set this environment variable to change the server-side Pino logger from info to debug.

CHATHUB_DEBUG=1

What it does

Enables pino.debug() logs in key server flows. These logs contain only operational metadata — timing, counts, and IDs. No user content, prompts, or auth tokens.

Example logs you will see:

tRPC lambda request: POST /trpc/lambda
tRPC lambda response in 145ms
Server DB connection established in 23ms
MCP listTools completed in 312ms, count: 5
MCP callTool "search" completed in 890ms
sendMessageInServer completed in 1204ms (sessionId=abc, model=gpt-4o)

What it does NOT do

Does NOT log	Why not
User prompts or message content	Not included in any pino.debug call
Model responses	Not included in any pino.debug call
Auth tokens or API keys	Pino redacts top-level secret keys
MCP tool arguments or results	Only timing is logged, not payloads
Raw stream chunks	Separate explicit flags required
`debug()` package namespaces	Not auto-enabled (see below)

Pino redaction (with caveats)

Pino redacts these exact top-level keys:

apiKey, authorization, Authorization, accessToken, refreshToken,
password, secret, token, cookie, x-api-key, api_key

Important limitations:

Redaction only covers top-level keys. Nested objects like { user: { apiKey: '...' } } are not redacted.
String-formatted secrets like "Authorization: Bearer abc" inside a message are not redacted.
The debug logs added by ChatHub only log strings (timing, counts, IDs), not objects, so these limitations do not apply to CHATHUB_DEBUG=1 specifically. But they matter if you add your own pino logs or enable DEBUG namespaces.

Override with explicit LOG_LEVEL

If you set LOG_LEVEL, it takes precedence over CHATHUB_DEBUG:

LOG_LEVEL=warn # Pino stays at warn even if CHATHUB_DEBUG=1

CHATHUB_TOOLS_DEBUG (Server — Tool + MCP, PII-Safe)

A single server-side switch for structured, privacy-safe lifecycle events across HTTP MCP, application built-ins, default/markdown/standalone plugins, delegated provider tools, and server-agent tools. It works standalone (no need to also set CHATHUB_DEBUG=1). It does not lower Pino or auto-enable broad legacy debug() namespaces. Set it as a container env var and recreate the service; no rebuild is needed.

CHATHUB_TOOLS_DEBUG=1       # safe set
CHATHUB_TOOLS_DEBUG=verbose # safe set + sanitized tool args/results

Values

Value	Effect
unset / `0` / `off`	Off — no auto-enable (default)
`1` / `true` / `on` / `safe`	Safe structured metadata events
`verbose` / `2`	Safe events plus bounded payload fingerprints

An unrecognized value is treated as off with a one-line startup warning, so a typo does not silently disable diagnostics.

What the safe set logs

Each batch emits tool_batch_started, one tool_completion_reported per call, and tool_batch_settled. Records include synthetic batchId/continuationId, call-set and per-call hashes, expected/result/failure counts, runtime type, terminal outcome, timing, and bounded result shape. Terminal outcomes are completed, failed, cancelled, skipped, persistence_failed, and handed_off. No prompts, arguments, result content, secrets, or full request/response bodies are logged.

Provider-native OpenAI web search, Moonshot $web_search, Anthropic server tools, and Google grounding are delegated/handed_off, not completed ChatHub tools. ChatHub's own web-browsing tool remains builtin. Diagnostic reporting is best-effort and fire-and-forget, so it cannot delay the next model turn.

Each isolated MCP request inherits its per-call diagnosticId in call_tool_complete or call_tool_failed. The validated bounded MCP correlation also retains batchId, continuationId, the call-set hash, and settled result/failure counts. These fields join MCP server execution to tool_completion_reported and the following model cache event without exposing raw tool-call IDs, arguments, or results.

Server-routed MCP calls carry an opaque invocationId and persist the normalized result before responding. If the browser loses the mcp.callTool response, it reads the matching recovery record immediately and, if the result is still pending or that read fails transiently, retries once after an abortable 500 ms delay. Both reads use the matching tool-message ID and invocation ID. Recovery returns only the result committed for that invocation, so a stale retry cannot overwrite or recover a newer attempt. The captured invocation signal is checked before recovery and failure persistence; browser cancellation reported as TypeError: Load failed is not recorded as a gateway failure. This follows the AbortSignal contract, where aborted records the operation's cancellation state. Assistant tool calls are launched only after final message persistence is confirmed. The save is attempted up to three times with a short backoff; if every attempt fails, the client reads the message back from the server by ID to check whether the write actually landed (this recovers lost-response cases in every conversation type, including group chats). Only when that verification also fails does the client report the diagnostic failure and stop before invoking external tools.

"Tool call was not saved" with an HTML 401/403 response

The warning means the browser received an HTML page instead of the API's JSON response. That usually indicates something in front of the application — a reverse proxy, WAF, or CDN access layer — answered the request itself, but it is not proof of where execution stopped: an origin-side auth or middleware layer can produce the same shape. Treat the classification as a lead to correlate, not a verdict. Typical candidates are a web application firewall rule matching the request body (tool-call arguments can contain program code, which code-injection rules often flag) or an expired access session on the edge (for example Cloudflare Access).

To diagnose, establish the source first:

Enable CHATHUB_TOOLS_DEBUG=1 and look for client_rpc_response_failed records in the server logs. This report is best-effort: the browser sends it as a separate request and drops it silently on failure, so the same interception (or an expired access session) can block the report itself — an absent record proves nothing. The warning shown in the UI does not display the diagnostic ID.
If a report exists, use its diagnostic ID, timestamp, request path, and response fingerprint to search the edge provider's security event log for POSTs to /trpc/lambda/message.update. A matching edge event confirms interception.
If there is no report or no matching edge event, treat the absence as inconclusive — an intermediary or a telemetry gap may simply not appear in the inspected view. Instead, take the failure time from the browser and inspect that exact POST across the origin logs and every intermediary (CDN, WAF, reverse proxy, access layer) before assigning the source.

Remedies, only after the source is confirmed:

Confirmed WAF false positive: add a least-privilege exception scoped to the exact host, the POST method, the /trpc/lambda/message.update path, and the specific rule ID/signature that fired. Keep access-control, rate-limit, bot-management, and all unrelated managed rules active. Do not create blanket exclusions such as /trpc/* or /webapi/* — those paths also carry data import/export, provider chat, the plugin gateway, and other sensitive APIs.
Expired access session: renew the session / re-authenticate at the edge; no rule change is needed.

Do not paste raw intercepted response bodies or credentials into issue reports — the diagnostic records above are sufficient.

The browser receives two independent capabilities from the server. toolLifecycleEnabled follows CHATHUB_TOOLS_DEBUG; cacheContinuationEnabled requires both a provider-specific DEBUG_*_CACHE=1 switch and KEY_VAULTS_SECRET or NEXT_AUTH_SECRET. This matches the fail-closed keyed fingerprint requirement. Cache-only mode retains the batch correlation, result/failure counts, and bounded result summaries for the following model request without sending per-call lifecycle telemetry, while clients do not collect continuation metadata when cache diagnostics are unusable. Capability discovery waits at most 250 ms, caches successful responses for 30 seconds, and retries after rejected or timed-out requests.

Client-provided x-chathub-tools-diagnostic-id headers are not trusted log labels. Tools and Lambda ingress replace valid values with deployment-keyed HMAC fingerprints before logging. If no fingerprint secret is configured, ChatHub uses a fresh server-owned ID instead. The response header contains only the protected or server-owned value. Authenticated client failure reports use the same boundary: without a fingerprint secret, the event retains the report request's server-owned route ID and never reuses the raw client ID. Correlation back to the original failed RPC requires KEY_VAULTS_SECRET or NEXT_AUTH_SECRET.

What verbose adds

Verbose records pass tool payloads through sanitizeToolDebugPayload. Secret-keyed fields (token, secret, password, apiKey, authorization, cookie, etc.) are omitted. Other strings become length plus SHA-256 fingerprint metadata; arrays, object width, depth, and total record size are bounded.

Relationship to DEBUG=...

CHATHUB_TOOLS_DEBUG does not auto-enable chathub-tools:*, lobe-mcp:*, lobe-search:*, lobe-chat:*, provider stream flags, or other DEBUG=... namespaces. Explicit DEBUG=chathub-tools:safe|verbose remains a legacy fallback only. Set other namespaces separately and only after reviewing their privacy risk.

CHATHUB_IMAGE_DEBUG (Server — Image Generation, PII-Safe)

A server-side switch for structured image-generation diagnostics. It works standalone, does not lower Pino level, and does not auto-enable legacy lobe-image:* namespaces.

CHATHUB_IMAGE_DEBUG=1       # safe lifecycle metadata
CHATHUB_IMAGE_DEBUG=verbose # safe metadata + bounded keyed fingerprints

Values

Value	Effect
unset / `0` / `off`	Off — no image diagnostic records (default)
`1` / `true` / `on` / `safe`	Safe structured lifecycle metadata
`verbose` / `2`	Safe metadata plus bounded keyed payload hashes

Unknown values are treated as off and emit one [chathub-image-debug:config_warning] record without echoing the invalid value.

What it logs

Records use [chathub-image-debug:<event>] {JSON} with schemaVersion: 1. The core event chain includes submission_accepted, batch_persisted, dispatch_started, dispatch_settled, async_route_started, async_route_settled, async_task_started, provider_call_started, provider_call_settled, transform_settled, upload_settled, and task_status_settled.

This switch is the Image workspace (/image) path. Chat Image tool tiles in a conversation (lobe-image-designer) emit chat_image_* events on CHATHUB_GENERATION_DEBUG instead.

Safe records may include counts, dimensions, provider/model/task/batch hashes, HTTP status, media type, body classification, elapsed time, outcome, and bounded response fingerprints. Provider values are always represented by keyed hash and length metadata, including custom user-controlled provider IDs. Verbose records add keyed prompt/config fingerprints. Oversized strings are hashed from bounded head/middle/tail samples plus their original length; arrays, object width, depth, and emitted record size are also bounded. Records never include raw prompts, URLs, image data, response bodies, user IDs, database IDs, credentials, headers, cookies, environment values, arbitrary error messages, or stacks.

The async dispatch correlation header is internal only: x-chathub-image-diagnostic-id. ChatHub accepts it only when the same request has the valid internal bearer secret; external or unauthorized values are not logged or returned. The ID is not stored in the database.

Response diagnostics do not clone or pre-consume async response streams. The route records status/header metadata, then the dispatch client's normal JSON read produces one bounded body sample. bodyKind=json therefore remains valid when fingerprintTruncated=true; truncation describes the sample, not malformed JSON. If the body stream fails after headers arrive, a failed dispatch_http_parse record uses failurePhase=response_read and includes only status/media and sanitized error metadata; intentional request aborts are not reported as read failures. Cancellation requires an aborted request signal or the exact AbortError class; response/error message text is not used. Recognized runtime error classes remain readable, while arbitrary Error.name values are reported as OtherError. Each authenticated async request has one async_route_started record and one async_route_settled record; the terminal outcome is failed when tRPC reports an error or the response status is not successful.

Axiom query

['chathub'] | where ['debug_namespace'] == 'chathub-image-debug'
| extend payload=parse_json(extract("(\\{.*\\})$", 1, ['message']))
| extend diagnosticId=tostring(payload.diagnosticId),
  spanId=tostring(payload.spanId),
  eventSequence=tolong(payload.eventSequence),
  phase=tostring(payload.phase),
  outcome=tostring(payload.outcome),
  providerHash=tostring(payload.provider.hash),
  bodyKind=tostring(payload.response.bodyKind),
  httpStatus=tolong(payload.response.httpStatus)
| project _time, diagnosticId, spanId, eventSequence, debug_event, phase,
  outcome, providerHash, httpStatus, bodyKind
| sort by _time desc | take 50

Use verbose only temporarily. If no records appear, verify the live container environment with /proc/1/environ and recreate the container after changing the variable.

CHATHUB_KNOWLEDGE_DEBUG (Server — Knowledge Base/RAG, PII-Safe)

This standalone server switch emits structured lifecycle records without changing Pino level or enabling raw debug namespaces:

CHATHUB_KNOWLEDGE_DEBUG=1       # safe counts, timing, outcomes, and scores
CHATHUB_KNOWLEDGE_DEBUG=verbose # safe set + bounded keyed fingerprints/shapes

Unset, 0, false, or off disables it; 1, true, on, or safe enables the safe set; 2 or verbose enables verbose records. An unknown value fails closed and emits one configuration warning without echoing the value.

Records use [chathub-knowledge-debug:<event>] {JSON} with schemaVersion: 1. The lifecycle covers document registration/object repair, Knowledge Base association, chunk/embed task dispatch, object reads, parsing, embedding batches/provider calls, reindexing, query embedding, scope expansion, vector search/retrieval, prompt injection, and client preparation failures. Opaque diagnosticId (kb_...), spanId, and event sequence fields correlate the stages. The internal x-chathub-knowledge-diagnostic-id header is trusted only with the matching server bearer secret; relevant async-task and chat errors show the same ID.

Safe records may contain counts, timings, dimensions, enumerated outcomes, candidate/selected counts, thresholds, and bounded similarity scores. Verbose adds HMAC fingerprints and structural summaries keyed by KEY_VAULTS_SECRET or NEXT_AUTH_SECRET; if neither is available, verbose additions are skipped and a warning is written. Both levels omit raw filenames, database IDs, document/chunk/query/prompt text, URLs, headers, credentials, request/response bodies, arbitrary error messages, and stacks. Every record is capped at 16 KiB, and diagnostics never alter RAG selection or task behavior.

At the browser preparation boundary, prompt_injection_reported includes countMode: exact for successful worker tokenization, estimated for the non-blocking estimator fallback, or character for the final character-length fallback. A client_preparation_failed record includes one safe failurePhase: retrieval, prompt_assembly, token_accounting, or message_metadata. Raw browser exception text is deliberately omitted.

The normal correlated chat chain is retrieval start/settle, prompt injection, the provider response, and assistant finalization. Retrieval followed directly by client_preparation_failed identifies a client preparation failure; the persisted assistant error carries the same kb_... ID instead of remaining at .... A tokenizer problem by itself should produce an estimated or character prompt-injection event and continue to the provider.

After reproducing a failure, copy its kb_... ID from the task tooltip or chat error and filter the application logs, for example:

docker logs lobe-chat 2>&1 | grep 'kb_1234567890abcdef'

Recreate the container after changing the variable. Use verbose only while needed. Do not enable raw provider completion logging for this lifecycle check; those switches can contain full prompts and responses.

CHATHUB_GENERATION_DEBUG (Server + Client — Durable Generation Send Path, PII-Safe)

A single switch for structured diagnostics of the durable (background) conversation-generation send path: what the browser decided (send, attach, sync, regenerate guards) and what the server did with the enqueue (operation row, Graphile job, worker execution). Use it when a send appears to do nothing, a reply never appears, Regenerate seems dead, or a background generation looks stuck — the plain [conversation-generation] logs do not cover the client decisions or the enqueue internals.

CHATHUB_GENERATION_DEBUG=1 # safe metadata records (server + re-emitted client events)

Values

Value	Effect
unset / `0` / `off`	Off — no generation diagnostic records (default)
`1` / `true` / `on` / `safe`	Safe structured metadata events
`verbose` / `2`	Same record set today (reserved for future use)

How client events reach the logs

The server switch is exposed to the browser through the global server config. When it is on, the client records send-path decisions locally and flushes them fire-and-forget through the conversationGeneration.reportClientDebug tRPC procedure, which re-sanitizes the fields as untrusted input and re-emits them into the same log stream with side=client (server events carry side=server). A per-browser override exists for support sessions: localStorage['chathub.generationDebug'] = '1' (or 'true'/'on') forces client collection on; '0'/'false'/'off' forces it off regardless of the server flag. Diagnostics are best-effort and can never interrupt a send.

What it logs

Records use [chathub-generation-debug:<event>] {JSON} with schemaVersion: 1. Each send or regenerate creates a client spanId (gd_...) that is forwarded with the enqueue, so client and server events for one action share the same span.

Key events:

Event	Side	Meaning
`send_started`	client	A durable send entered the RPC phase
`send_rpc_settled`	client	Send RPC outcome (`ok`/`error`, tRPC code, operation id present)
`send_recovery`	client	Recovery lookup by idempotency key after an RPC error
`send_failure_ui`	client	Whether an error toast/temp-row cleanup was shown
`durable_attach` / `durable_attach_skipped`	client	Client attached to the server operation, or why it did not (fenced cancel, stale conversation)
`browser_path_started` / `exec_runtime_settled`	client	Browser-fallback turn (unsupported tool) started/settled
`enqueue_client_settled`	client	Direct enqueue outcome (`ok`, `nonRecoverable`, `recoveredByKey`, `lost`)
`regenerate_started` / `regenerate_early_return`	client	Regenerate began, or which guard made it a no-op (`noAnchor`, `retrying`, `notCurrent`)
`regenerate_enqueue_settled`	client	Regenerate enqueue outcome and browser fall-through
`sync_summary`	client	Per-sync counts: active, attached, detached, fenced cancels, orphans deleted; `resumedTools` / `resumedModel` when return sync restarted tools or the model
`deferred_lane_resumed`	client	Return to a browser-fallback topic (`resume_tools` / `resume_model` / `finalize` / `still_producing` / `loading_flat`). `still_producing` includes `chatLoading` / `toolsCalling` / `streamActive` / `pendingModelContinue`
`tool_loop_continue` / `tool_loop_continue_skipped`	client	After MCP/tools settle, whether the model started the follow-up (`reason=no_resumable_tool` / `batch_gated` / `not_alive` / … plus outcome counts)
`builtin_tool_settled`	client	A builtin (web search, Sandbox, DALL·E, …) finished. Uses the tool's conversation + deferred `spanId`, not the topic you clicked (`visible=false` after leave is OK)
`chat_image_run_started` / `chat_image_item_settled` / `chat_image_run_settled`	client	Chat Image tool tiles on the same send `spanId` (`toolName=lobe-image-designer`). `visible=false` after leave is OK. Run `outcome`: `ok` / `partial` / `failed` / `no_origin` / `already_done` / `no_model` / `persist_unproven` / `not_current`. Item `outcome`: `attached` / `failed` / `not_current` / `persist_unproven` / `skipped`. Stop-marker persist failure: `kind=stop_mark` `outcome=persist_failed` plus the hashed assistant/message id and `spanId` (`gd_…`) of the affected tile — never raw task/message/prompt ids. Counts and hashes only — never prompts.
`invalidate_preserved`	client	Topic/session switch kept vs aborted in-flight plugin / RAG / search / chat-loading ids (counts only)
`rag_retrieve_settled`	client	Knowledge Base retrieve finished: `ok` / `empty` / `hard_cancelled` / `error` plus `chunkCount`
`message_persist_skipped`	client	A message write returned before persisting (`hard_cancelled` = Stop; `not_visible` should be rare)
`orphan_deleted`	client	A stale `...` placeholder row was cleaned up (hash + age)
`event_dropped`	client	Browser could not apply an event for an operation this tab had attached (`hadAttached=true`; `not_attached` / `stale_revision`). Snapshot/`status` drops are throttled per operation; `done`/`error` always log. SSE replay of other operations is not this event
`event_drop_summary`	client	Count of suppressed replay drops (`suppressedCount`, `notAttachedDone`, `distinctOps`). Flushed at 50 suppressed, SSE end, cursor reset, or tab hide; hide also drains the client debug queue so the summary is not left on the 2s timer
`event_applied_terminal`	client	A terminal (`done`/`error`) event was applied — proves end-to-end delivery
`sse_client_stream_ended` / `sse_client_stream_failed`	client	SSE stream ended or failed; reconnect episode counter
`sse_client_poll_failed`	client	Fallback polling failed (first failure per episode only)
`sse_client_reset_replay`	client	Event-log cursor reset triggered a full replay (replayed count)
`enqueue_received` / `enqueue_rejected`	server	Enqueue entered validation, or was rejected (reason + tRPC code)
`enqueue_persisted`	server	Operation row written; `jobAdded` says whether the Graphile job was added
`sweep_reenqueued`	server	Sweeper re-queued an orphaned operation (`jobAdded=false` means the job still could not be added)
`worker_started` / `worker_start_failed` / `worker_stopped`	server	Graphile runner lifecycle; a boot without `worker_started` means no job will run
`job_received`	server	Worker claimed a job; `queueAgeMs` measures queue dwell
`job_malformed`	server	A queued payload lacked `operationId`/`userId` and was dropped
`sweep_failed`	server	A sweeper pass failed (`phase=initial`/`periodic`)
`sse_opened` / `sse_closed`	server	SSE connection lifecycle (`deliveredCount`, `reason=abort`/`error`)
`sse_reset`	server	SSE cursor reset triggered a replay
`sse_poll_failed`	server	SSE route's event poll failed
`execute_started` / `execute_skipped`	server	Worker claimed the operation, or why it did not (not found, not active, claim failed); `queueAgeMs` measures operation-row age at claim
`execute_transcript_loaded`	server	Transcript size plus content hashes of the triggering user message and the last transcript user message
`execute_settled`	server	Terminal worker outcome (`succeeded`, `failed`, `cancelled`, ...)
`sandbox_run_started` / `sandbox_run_settled`	server	Sandbox run on OpenSandbox (`provider` `opensandbox`, counts, `httpStatus`, `exitCode`, `outcome`; never code/stdout/filenames)
`sandbox_collect_failed`	server	OpenSandbox output download failed, including a body past the size cap. Stdout and stderr are still returned; files for that run are empty
`sandbox_cleanup_failed`	server	OpenSandbox sandbox delete failed. The server TTL still reaps it
`browser_tool_stubbed`	server	DALL·E (and similar) still cannot run on Graphile; CI uses `sandbox_run_*` instead
`chat_image_task_created` / `chat_image_task_rejected`	server	Chat Image billable insert (`inserted` / `idempotent`) or refused create (`uncorrelated` / `unproven` / `stopped` / `conflict`). `unproven` is a UUID that does not derive from this request's principal aliases (`user:<raw>` / mapped `user:<userId>` / `local` / `anonymous` / `guest`) plus message/index/attempt. `scopeKind` on create is `canonical` / `mapped` / `local` / `anonymous` / `guest`; on `unproven` it is `none`. `stopped` is only a `ChatImageTaskCancelled` placeholder — a duplicate create against a generic provider/task error is `created` (`idempotent`) and does not dispatch. Hashed message/task ids only.

A healthy durable send shows, in order and sharing one spanId: send_started → send_rpc_settled(outcome=ok) → durable_attach → enqueue_received → enqueue_persisted(jobAdded=true) → job_received → execute_started → execute_settled(outcome=succeeded) → event_applied_terminal (client, when the browser was open).

A healthy search+Sandbox send is the same, plus sandbox_run_settled(outcome=ok) when the model actually called the tool. enqueue_rejected with lobe-sandbox (or the legacy lobe-code-interpreter) must not appear. Sandbox does not emit browser_tool_stubbed.

Notes:

enqueue_persisted(jobAdded=false) cannot happen on the transactional send path (a failed job insert rolls the whole enqueue back). The jobAdded=false signal lives in sweep_reenqueued: the operation row exists but no Graphile job could be added; the sweeper retries on its next pass.
In execute_transcript_loaded, a parentUserHash that differs from lastTranscriptUserHash proves a transcript/binding race: the worker would answer a different user turn than the one that triggered it.
Failed execute_settled rows may include errorParamClass (an enum such as web_search_disabled / temperature_out_of_range / other) and errorParamHash. Provider error.param text is not copied into this channel — it can echo prompt or customer data. The chat UI error can still show Xiaomi's param string.
"Server finished but the reply never appeared": look for execute_settled(outcome=succeeded) without a matching event_applied_terminal. The gap is explained by event_dropped(hadAttached=true, type=done|error) (this tab had attached the operation and then could not apply the terminal event), sse_client_stream_failed / sse_client_poll_failed (delivery broken), or the browser simply being closed (the next sync_summary reattaches and refreshes). High event_drop_summary.suppressedCount on reconnect is expected SSE history replay, not a missing reply.
MCP/tools returned JSON but no summary: look for call_tool_complete (tools debug) then tool_loop_continue on the same generation spanId. If you instead see tool_loop_continue_skipped or deferred_lane_resumed stuck on still_producing with pendingModelContinue=true, the model never started the follow-up.
Off-screen RAG / search / persist: on the same spanId, invalidate_preserved should show kept (not aborted) plugin/RAG/search/chat-loading counts, then rag_retrieve_settled(outcome=ok|empty) and/or builtin_tool_settled (visible=false). Leave during retrieve must show deferred_lane_left(producerAlive=true). producerAlive=false plus rag_retrieve_settled(ok) is the empty Reference Source hang. rag_retrieve_settled(outcome=hard_cancelled) is Stop, not leave. message_persist_skipped(reason=not_visible) after leave is the persist-context regression; hard_cancelled is Stop.
Chat Image tool tiles: on the same spanId, a turn that actually started generating should show chat_image_run_started, then chat_image_item_settled(outcome=attached) per tile and chat_image_run_settled(outcome=ok) (visible=false after you switch topics is expected). Server chat_image_task_created(outcome=inserted) is the billable insert. Live creates copy the send spanId (gd_…) onto the unresolved tile so a remount or reload can still join chat_image_task_created after the in-memory deferred lane is gone. Filter with | where spanId == '<gd_...>' — never join on raw task, message, or prompt ids. fetch_stream_interrupted(errorKind=webkit_load_failed) with no chat_image_run_started means Safari aborted during Deep Thinking before any cards appeared. persist_unproven or missing attached after leave is a persist problem, not a Safari abort. chat_image_run_settled(outcome=ok, attachedCount=N) while the topic still shows Prompt-only cards means a later message refresh dropped imageId (overlapping getMessages / focus revalidate). Current builds merge those ids on fetch and adopt the finished file from the deterministic task id on remount without creating a new paid task.
Large queueAgeMs on job_received/execute_started with a healthy worker_started means queue backlog, not a stuck job.

Privacy

Message content is never logged. Identifiers (operations, lanes, idempotency keys, messages) appear only as SHA-256-16 fingerprints; content comparisons use the same fingerprinting. Records contain only error classes, tRPC codes, counts, ages, and enumerated outcomes, are capped at 16 KiB, and client-reported fields are re-sanitized server-side.

Axiom query

['chathub'] | where ['debug_namespace'] == 'chathub-generation-debug'
| extend payload=parse_json(extract("(\\{.*\\})$", 1, ['message']))
| extend spanId=tostring(payload.spanId), side=tostring(payload.side),
  kind=tostring(payload.kind), outcome=tostring(payload.outcome),
  reason=tostring(payload.reason), trpcCode=tostring(payload.trpcCode),
  jobAdded=tobool(payload.jobAdded),
  operationHash=tostring(payload.operationHash)
| project _time, spanId, side, debug_event, kind, outcome, reason, trpcCode,
  jobAdded, operationHash
| sort by _time asc

Filter to one send with | where spanId == '<gd_...>'. If no records appear, verify the live container environment (/proc/1/environ) and recreate the container after changing the variable, as with the other switches.

CHATHUB_COMPACTION_DEBUG (Server + Client — Topic Compaction and Memory Dream, PII-Safe)

A dedicated switch for why topic compaction ran and why the scheduled memory dream ran or skipped. Token auto-compact does not compare the full topic to the vendor's advertised context window. It compares a client estimate of the sliced request (system role, tools, assistant memory, history summary, sliced messages, input) to enabledAiModels[].contextWindowTokens for the active chat model card (including user overrides; OpenAI-compatible cards are forced to 258k). The 80% high-watermark gate applies only to token_threshold. message_count (history overflow — also run pre-send before truncate on server-mode sends) and manual (Compact now) ignore that gate and can compact a short chat. There is no browser daily-topic scheduler; planner_settled with trigger=scheduled on a topic path is a regression. Dream runs emit path=assistant_memory_rollup instead.

CHATHUB_GENERATION_DEBUG already logs Graphile kind=memory_compaction execute_started / execute_settled. It does not log client not_needed / ineligible outcomes, estimate parts, or the window used. This switch fills that gap. It does not change when compaction runs.

CHATHUB_COMPACTION_DEBUG=1       # safe metadata
CHATHUB_COMPACTION_DEBUG=verbose # safe set plus flag booleans

Values

Value	Effect
unset / `0` / `off`	Off (default)
`1` / `true` / `on` / `safe`	Safe structured metadata events
`verbose` / `2`	Same records plus compaction-flag booleans

Set it as a container env var and recreate the service; no image rebuild is needed. The server flag is exposed as GlobalServerConfig.compactionDebug. The client also honors localStorage['chathub.compactionDebug'] (1/true/on force-on; 0/false/off force-off). Client events flush through conversationGeneration.reportCompactionDebug (not reportClientDebug) and are re-sanitized as untrusted input.

Events

Records use [chathub-compaction-debug:<event>] {JSON} with schemaVersion: 1. A planner spanId (cd_...) is forwarded on durable enqueue so worker events join generation-debug enqueue/execute records.

Event	Side	Meaning
`watcher_armed`	client	Token auto-compact watcher scheduled an attempt (not every 750ms skip). Includes `totalToken`, `maxTokens`, `highWatermark`, `ratio`, `knowledgeBaseToken`. The watcher ratio includes in-flight Knowledge Base tokens; the planner estimate does not.
`planner_settled`	client	Every compaction planner return, including `not_needed` / `ineligible`. `trigger`, `status`, `reason`, `path` (`pre_send` / `durable_enqueued` / `client_inline`), watermarks, `totalToken` / `maxTokens` / `ratio`, estimate parts (`systemRoleToken`, `toolsToken`, `inputToken`, `chatsToken`, `memoryToken`, `historySummaryToken`), and window fields (`topicMessageCount`, `slicedMessageCount`, `effectiveHistoryCount` as the history-limit setting rather than included rows, `excludedByHistoryCount`, `preSendMessageCountCompact`). `trigger=scheduled` on these topic paths is a regression.
`worker_settled`	server	Graphile compaction outcome: `trigger`, `outcome`, `candidateCount`, and `contentChars` / `reasoningChars` when the summary was empty.
`dream_scheduler_tick`	server	Memory-dream dispatcher sweep (every agent considered): `frequency`, `scheduleTime` (UTC `HH:mm`), `due`, `skippedReason` (`before_time` / `already_ran` / `disabled` / `off` / `backoff`), `markerKeyHash` (hashed `assistant-memory-dream:<agentId>:<periodStamp>` job key), `path=assistant_memory_rollup`, `trigger=scheduled`.
`dream_scheduler_settled`	server	Terminal memory-dream event — emitted when a Graphile job finishes, a per-card Regenerate finishes (`trigger=manual`), or when the dispatcher fails to enqueue one. `status` (`success` / `skipped` / `failed`), `reason` (including `already_has_card`, `invalid_regenerate`, `not_found`, `mismatch`, `enqueue_failed`, `completion_failed`, `stale_conflict`, `no_changes`, `no_summaries`, `no_active_topics_yesterday`, `stale_job`, `malformed_job`, …). Worker and regenerate settles also include `historyDate`, `maxEntries`, keep-N counts (`keepCount`, `foldCount`, `singleDayCount`, `overflowCount`, …), `topicsWithSummary` / `topicsWithExcerpt` (how many topics fed the dream from a compaction summary vs a bounded recent-message excerpt), `overflowEnvelope` (`none` / `overflow_v1` / `opaque_v3` / `opaque_payload`), `foldPath` (`none` / `llm` / `llm_rewrite` / `concat_fallback`; `foldFallbackReason` only on concat), `cardKind` on regenerate (`single_day`), and `activityWindowStart` / `activityWindowEnd` (UTC dates). `path=assistant_memory_rollup`. `markerKeyHash` is present only when `reason=enqueue_failed` (join to the matching tick); normal worker outcomes and `malformed_job` omit it. Never logs card bodies, overflow text, or marker strings.

How to read “compacted before 80%”

['chathub'] | where debug_namespace == 'chathub-compaction-debug'
| extend payload=parse_json(extract("(\\{.*\\})$", 1, ['message']))
| extend event=tostring(debug_event), trigger=tostring(payload.trigger),
  status=tostring(payload.status), reason=tostring(payload.reason),
  ratio=todouble(payload.ratio), maxTokens=tolong(payload.maxTokens)

trigger=message_count / manual — expected; not an 80% bug
trigger=scheduled on a topic path (pre_send / durable_enqueued / client_inline) — regression; the daily topic note scheduler was removed
Dream runs show path=assistant_memory_rollup with dream_scheduler_tick / dream_scheduler_settled
Dated-card settle: reason=already_has_card is the usual skip when that UTC day already has a card. Unknown reasons are stripped by the allowlist, so a missing reason is not proof the job had none. foldPath is llm / llm_rewrite / concat_fallback / none; overflowEnvelope is none / overflow_v1 / opaque_v3 / opaque_payload. Never query overflow text.
Filter one settle with | where debug_event == 'dream_scheduler_settled' | project _time, status, reason, foldPath, overflowEnvelope, historyDate, maxEntries, keepCount, foldCount
dream_scheduler_settled with reason=enqueue_failed — dispatcher could not enqueue the Graphile job (no worker execution; no activityWindow* dates); markerKeyHash on the settled event matches the preceding tick
Worker dream_scheduler_settled events (success, skip, or failure after enqueue) do not include markerKeyHash
trigger=token_threshold and ratio>=0.8 but maxTokens much smaller than the vendor window — wrong or overridden model card (especially OpenAI-compatible 258k)
memoryToken or historySummaryToken dominating totalToken — memory/archives, not chat length
watcher_armed.ratio high then planner_settled.reason=below_high_watermark — RAG tokens on the watcher only

Privacy: counts, ratios, and allowlisted labels only. Built-in model ids stay readable; custom ids are fingerprinted. Session/topic ids are hashed. Summary text, message content, and archives are never logged. The in-app memoryDebugLog strip is separate and is not copied into Axiom.

OpenAI-Compatible Responses Stream EOF

When an OpenAI-compatible Responses request shows Stream parsing error with reason=unexpected_end, ChatHub received a stream that closed without an explicit Responses terminal event. A successful Responses stream must finish with response.completed and status=completed; ChatHub does not treat a clean connection close after partial output as success.

With DEBUG_OPENAICOMPATIBLE_CACHE=1, the matching terminal_error event contains bounded fields that separate the failure paths:

`terminalSource`	Meaning
`missing_terminal_event`	The stream closed without `response.completed`, failed, incomplete, or error
`upstream_iterator_exception`	The SDK could not open/read the stream or parse its SSE JSON
`provider_terminal_event`	The provider explicitly emitted a failed, incomplete, or error event
`request_cancelled`	The browser or downstream consumer cancelled the response

terminalReason further identifies safe categories such as unexpected_end, html_response, invalid_json, response_failed, max_output_tokens, or consumer_cancelled. Provider-specific codes may appear separately in the bounded errorCode field. These diagnostics do not include prompts, raw stream chunks, URLs, response IDs, provider messages, or stacks.

If unexpected_end repeats at a consistent elapsed boundary, inspect the configured gateway, reverse proxy, and upstream model-service streaming/request timeouts. ChatHub's ten-second SSE heartbeats protect the browser-facing connection but cannot extend an upstream provider connection. Raising the OpenAI Node SDK timeout does not fix a stream-body cutoff after response headers.

As an immediate workaround, increase the gateway/upstream timeout. If the gateway cannot preserve /v1/responses typed terminal events, select Chat Completions under OpenAI-compatible API route in provider settings until the gateway's Responses implementation is corrected. Raw DEBUG_OPENAICOMPATIBLE_RESPONSES=1 logs can help confirm the final upstream events, but they may contain user content and should be enabled only briefly in a controlled environment.

DEBUG=... (Server — Explicit Opt-In)

The standard Node.js debug package environment variable. Use this to enable specific namespaces.

DEBUG=lobe-async:*,lobe-server:discover

Critical safety warning

The codebase contains 70+ debug namespaces. Many use %O or %o to print full objects, which can include:

Auth tokens and API keys
User prompts and model responses
File contents (base64)
OIDC session data and JWTs
Search request bodies

Only enable namespaces you have individually audited. Do not use broad wildcards like lobe-* or * in production.

Safe namespaces (audited)

Namespace	What it logs
`lobe-async:*`	Async tRPC init / middleware / errors
`lobe-server:discover`	Plugin discovery operations
`lobe-mcp:oauth-discovery`	MCP OAuth metadata discovery (URLs only)
`lobe-mcp:oauth-service`	MCP OAuth token exchange/refresh (status/timing only)
`config-router`	Config router operations
`lobe-cost:*`	Cost calculation numbers

Unsafe namespaces (never enable in production)

Namespace	What it logs	Risk
`lobe-model-runtime:*`	Prompts, responses, tool inputs	Full user content
`lobe-image:*`	Image prompts, URLs, generation params	Prompts
`lobe-search:*`	Search request bodies	API keys in requests
`lobe-oidc:adapter`	Full OIDC session / payload objects	Tokens, user data
`lobe-trpc:*`	Context params including auth headers	Auth tokens
`lobe-chat:*`	Instructions, agent config, supervisor decisions	System roles, prompts
`context-engine:*`	System roles, input templates, vision descriptions	Prompt fragments

NEXT_PUBLIC_CHATHUB_DEBUG=1 (Client — Browser)

Enables client-side debug() logs in the browser for store-level debugging (group chat, supervisor, etc.).

NEXT_PUBLIC_CHATHUB_DEBUG=1

Warning

This enables lobe-chat:* in the browser, which logs:

User instructions (prompts)
Agent configuration (system roles, model settings)
Supervisor tool parameters and decisions

These are prompt-adjacent data. Only enable this in development or when actively debugging chat routing issues. Never enable in production environments where user privacy matters.

Chat Input Replacement Debug (Client — On-Page Overlay)

Diagnoses OS text-suggestion / autocorrect problems in the chat input (word duplication, dropped suggestions), including inside isolated / remote-isolation browsers (e.g. Menlo "Safeview"-class thin clients) where normal DevTools debugging is impossible: the app runs in a remote browser, so the local console never shows the app's logs and local localStorage commands never reach it.

The debug channel is therefore designed to cross an isolation boundary:

Inbound — the URL: visit the app with ?replacement_debug=1 (or #replacement_debug=1) appended. The flag persists in the app's own localStorage, so it survives navigation. Disable with replacement_debug=0. In a regular browser, localStorage.setItem('lobe_replacement_debug','1') also works.
Outbound — the page itself: output appears in a fixed overlay at the bottom-right of the page (#replacement-debug-overlay), which is streamed back through DOM mirroring even under browser isolation. The same lines also go to console.debug for regular browsers.

Each line describes one beforeinput / input / textInput / keydown / composition* event on the editor root:

beforeinput insertReplacementText data=null plain="forever " dt=[text/plain] trust=0 cancel=0 comp=0 edComp=0 ranges=[#text:0-#text:0(collapsed)]
decision: controlled (replace=4, synth=1)
input suppressed (synthetic pair)

Key fields: trust (0 = synthetic event from an injector/isolation layer — it has no native default action), cancel (0 = preventDefault is a no-op), plain (the dataTransfer text/plain payload), ranges (the browser's target ranges, often bogus in isolated browsers), plus the plugin's routing decision (controlled / native-fallback / guard / input suppressed).

Safe for production: it logs only input-event metadata and the suggestion payload the user is typing, renders only for users who opt in via the URL flag, and never sends data anywhere.

When reporting a suggestion bug that cannot be reproduced locally, capture a screenshot of the overlay right after accepting a suggestion — it identifies the exact event mechanism the environment uses.

Provider Raw Stream Debugging (Privacy Risk)

These flags dump raw HTTP request/response stream chunks to the console. They are never enabled by CHATHUB_DEBUG=1.

DEBUG_OPENAI_CHAT_COMPLETION=1
DEBUG_OPENAICOMPATIBLE_CHAT_COMPLETION=1
DEBUG_OPENAICOMPATIBLE_RESPONSES=1
DEBUG_ANTHROPIC_CHAT_COMPLETION=1
DEBUG_ANTHROPICCOMPATIBLE_CHAT_COMPLETION=1
DEBUG_MOONSHOT_CHAT_COMPLETION=1
DEBUG_MINIMAX_CHAT_COMPLETION=1
DEBUG_DEEPSEEK_CHAT_COMPLETION=1
DEBUG_MIMO_CHAT_COMPLETION=1
DEBUG_GOOGLE_CHAT_COMPLETION=1
DEBUG_AZURE_CHAT_COMPLETION=1
DEBUG_AZURE_AI_CHAT_COMPLETION=1

Avoid in production. These logs contain full prompts, responses, and token streams with no redaction.

Structured Provider Request Debugging

Some provider debug flags also emit a redacted one-line request summary before the upstream call:

DEBUG_MOONSHOT_CHAT_COMPLETION=1
DEBUG_MINIMAX_CHAT_COMPLETION=1
DEBUG_DEEPSEEK_CHAT_COMPLETION=1
DEBUG_MIMO_CHAT_COMPLETION=1
DEBUG_ANTHROPICCOMPATIBLE_CHAT_COMPLETION=1

Look for:

[provider-debug:request] {"provider":"moonshot",...}

This summary is designed for endpoint and request-shape debugging, especially errors such as url.not_found. It includes:

provider, redacted baseURL, full effectiveURL, upstream route, model, and stream
payloadFingerprint
turnShape.count and role/content sequence
tools.count, tool names, and tool fingerprint
key parameter presence such as temperature, top_p, thinking, reasoning_effort, and stream_options

It does not log API keys, auth tokens, raw prompt text, tool arguments, image URLs, or full schemas. The same debug flags still enable raw [requestPayload] and stream chunk logs, so avoid using them in privacy-sensitive production sessions.

Cross-Provider Cache Debugging

Prefer cache-only diagnostics before raw provider streams:

DEBUG_OPENAI_CACHE=1
DEBUG_AZURE_CACHE=1
DEBUG_AZUREAI_CACHE=1
DEBUG_MOONSHOT_CACHE=1
DEBUG_DEEPSEEK_CACHE=1
DEBUG_MINIMAX_CACHE=1
DEBUG_MIMO_CACHE=1
DEBUG_ANTHROPIC_CACHE=1
DEBUG_ANTHROPICCOMPATIBLE_CACHE=1
DEBUG_GOOGLE_CACHE=1
DEBUG_OPENAICOMPATIBLE_CACHE=1
DEBUG_ZHIPU_CACHE=1
DEBUG_XAI_CACHE=1

These switches emit compact request, usage, usage_missing, and terminal_error JSON. All providers except OpenAI-compatible use [model-cache-debug:<event>]; OpenAI-compatible keeps [openai-compatible-cache-debug:<event>] for existing Axiom queries.

Records include trusted provider/runtime family, API and stream mode, server-derived cache-policy booleans, input/tool counts, deployment-keyed request/response/model correlations, continuation metadata, normalized cache read/write/miss counters, and hit, mixed, write, miss, not_reported, or unsupported.

They require KEY_VAULTS_SECRET or NEXT_AUTH_SECRET for HMAC-SHA-256 fingerprints. They do not log prompts, raw model/response IDs, cache keys, API keys, URLs, headers, schemas, tool names/arguments/results, or response content.

Reported zero cache counters remain zero. Omitted counters remain absent: not_reported is unavailable telemetry, not a miss. Azure AI Inference is observability-only and reports unobservable; Moonshot/Kimi, MiniMax, and Google/Vertex do not receive invented cache request fields. DeepSeek reports supported cache hit/miss counters from its native token usage. Anthropic request events include the final cache_control breakpoint count and effective 5m, 1h, or mixed TTL; an omitted ephemeral TTL means the documented 5m default.

Successful JSON and streaming requests emit exactly one terminal usage or usage_missing event. Provider rejection, stream protocol failure, and downstream cancellation emit exactly one terminal_error, while cancellation continues to abort the owned upstream stream.

Native OpenAI and Azure OpenAI ignore client-supplied prompt_cache_key. Eligible GPT-5.6-or-later requests receive a stable opaque key derived server-side from authenticated user plus topic/session scope. OpenAI-compatible continues to use its configured provider matrix.

Zhipu (GLM) uses implicit prefix caching: caching is automatic, needs no request field, and reports hits in usage.prompt_tokens_details.cached_tokens. Matching requires a byte-stable prompt prefix, so ChatHub skips volatile system-prompt placeholders ({{time}}, {{datetime}}, {{timestamp}}, {{random*}}, {{uuid*}}, etc.) for zhipu — an agent system role that injects a fresh time/random value every round would otherwise never hit. A hit also requires a repeated prefix of at least 512 tokens within the provider's cache TTL; short conversations may never reach that threshold. Use DEBUG_ZHIPU_CACHE=1 and compare requestHash across rounds: a changing requestHash means the prefix is drifting; a stable requestHash with cachedTokens: 0 points at the provider (TTL, shard/warmup, or sub-threshold prefix).

GLM through a custom gateway: intermittent 400s (`thinking` / `reasoning_effort` / `tool_stream`)

If GLM runs through a non-native OpenAI-compatible gateway (e.g. LiteLLM → vLLM) rather than open.bigmodel.cn / api.z.ai, some GLM-native fields can be rejected with 400 Bad Request — and the rejection is intermittent: the same gateway has been observed to accept a field on one request while rejecting it on another, and to change behavior when its backend pool rotates. ChatHub avoids the entire class by never sending the rejected shapes (probe-verified 2026-08-06):

thinking on → the thinking field is omitted (the official API defaults to enabled); off → thinking: { "type": "disabled" }
Preserved Thinking → thinking: { "clear_thinking": false } with no type
Reasoning Effort Skip → reasoning_effort: "none" (not "minimal")
tool_stream → never sent (the strict backend rejects it with Extra inputs are not permitted, field: 'tool_stream'; tool-call arguments arrive as one chunk instead of being incrementally streamed)

If a custom gateway still 400s on GLM, capture the payload with DEBUG_ZHIPU_CHAT_COMPLETION=1 and compare it against the shapes above — anything beyond them is coming from the gateway side, not ChatHub.

If the redacted fingerprints still do not explain a cache miss, use DEBUG_OPENAICOMPATIBLE_CHAT_COMPLETION=1 or DEBUG_OPENAICOMPATIBLE_RESPONSES=1 for exact payload/stream inspection. Those full debug modes can expose prompt, tool, and file context.

Observability Tools (Separate Opt-Ins)

Tool	Env Var	Purpose
Langfuse	`ENABLE_LANGFUSE=1`	LLM tracing and observability
OpenTelemetry	`ENABLE_TELEMETRY=1`	Distributed tracing

These are not affected by CHATHUB_DEBUG=1.

What Can I Actually Debug?

With CHATHUB_DEBUG=1 alone (safe)

Problem	What to look for
Chat feels slow	`sendMessageInServer completed in Xms` — if >2s, check MCP callTool timing
MCP tools not appearing	`MCP listTools completed in Xms, count: Y` — if count=0 or timeout, MCP server is down
DB operations lagging	`Server DB connection established in Xms` — if >100ms, check PostgreSQL
tRPC endpoints slow	`tRPC lambda response in Xms` — identifies slow endpoints

With DEBUG=... (requires audit)

Problem	Namespace
MCP OAuth flow failing	`lobe-mcp:oauth-discovery,lobe-mcp:oauth-service`
Async tRPC errors	`lobe-async:*`
Plugin discovery issues	`lobe-server:discover`

For errors and crashes

CHATHUB_DEBUG=1 does not make errors more visible. Errors are already logged at info level or higher:

tRPC errors: visible via pino.info() in onError handler
MCP exceptions: visible via console.error() before TRPCError throw
Model failures: visible via tRPC onError

For production error tracking, use Sentry or similar tools rather than debug logging.

Quick Reference

Goal	Variable	Safe?
Safe server diagnostics	`CHATHUB_DEBUG=1`	Yes
Tool + MCP lifecycle diagnostics	`CHATHUB_TOOLS_DEBUG=1` / `=verbose`	Yes (sanitized)
Image workspace diagnostics	`CHATHUB_IMAGE_DEBUG=1` / `=verbose`	Yes (sanitized)
Knowledge Base diagnostics	`CHATHUB_KNOWLEDGE_DEBUG=1` / `=verbose`	Yes (sanitized)
Durable generation send-path diagnostics	`CHATHUB_GENERATION_DEBUG=1`	Yes (sanitized)
ChatGPT / Codex OAuth + chat diagnostics	`CHATHUB_OPENAI_CODEX_DEBUG=1`	Yes (sanitized)
SuperGrok OAuth diagnostics	`CHATHUB_XAI_OAUTH_DEBUG=1`	Yes (sanitized)
Cross-provider cache diagnostics	provider-specific `DEBUG_*_CACHE=1`	Yes (keyed and bounded)
Specific server debug namespaces	`DEBUG=namespace1,namespace2`	Only if audited
Browser store debugging	`NEXT_PUBLIC_CHATHUB_DEBUG=1`	No (prompt-adjacent data)
Raw provider streams	`DEBUG_*_CHAT_COMPLETION=1`	No (full prompts/responses)
Override Pino level	`LOG_LEVEL=debug` / `LOG_LEVEL=warn`	Yes
All debug namespaces	`DEBUG=*`	Never in production

Debug and Diagnostics

Debug and Diagnostics

Debug by Component

ChatGPT / Codex stream outcomes

Background generation: enqueue noise vs worker death

Development-only API auth bypass

CHATHUB_DEBUG=1 (Server — Safe Default)

What it does

What it does NOT do

Pino redaction (with caveats)

Override with explicit LOG_LEVEL

CHATHUB_TOOLS_DEBUG (Server — Tool + MCP, PII-Safe)

Values

What the safe set logs

"Tool call was not saved" with an HTML 401/403 response

What verbose adds

Relationship to DEBUG=...

CHATHUB_IMAGE_DEBUG (Server — Image Generation, PII-Safe)

Values

What it logs

Axiom query

CHATHUB_KNOWLEDGE_DEBUG (Server — Knowledge Base/RAG, PII-Safe)

CHATHUB_GENERATION_DEBUG (Server + Client — Durable Generation Send Path, PII-Safe)

Values

How client events reach the logs

What it logs

Privacy

Axiom query

CHATHUB_COMPACTION_DEBUG (Server + Client — Topic Compaction and Memory Dream, PII-Safe)

Values

Events

How to read “compacted before 80%”

OpenAI-Compatible Responses Stream EOF

DEBUG=... (Server — Explicit Opt-In)

Critical safety warning

Safe namespaces (audited)

Unsafe namespaces (never enable in production)

NEXT_PUBLIC_CHATHUB_DEBUG=1 (Client — Browser)

Warning

Chat Input Replacement Debug (Client — On-Page Overlay)

Provider Raw Stream Debugging (Privacy Risk)

Structured Provider Request Debugging

Cross-Provider Cache Debugging

GLM through a custom gateway: intermittent 400s (thinking / reasoning_effort / tool_stream)

Observability Tools (Separate Opt-Ins)

What Can I Actually Debug?

With CHATHUB_DEBUG=1 alone (safe)

With DEBUG=... (requires audit)

For errors and crashes

Quick Reference

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

GLM through a custom gateway: intermittent 400s (`thinking` / `reasoning_effort` / `tool_stream`)