feat(llm): configurable SSE buffer caps and profiling tracing coverage#4823
Merged
Conversation
github-actions
Bot
added
documentation
This was
linked to
issues
Jun 5, 2026
…mentation Closes #4750, #4808, #4790. - Add `StreamLimits` to `zeph-config` with `max_tool_json_bytes` (4 MiB), `max_thinking_bytes` (1 MiB), and `max_compaction_bytes` (32 KiB) defaults. Wire through `LlmConfig`, `ClaudeProvider.with_stream_limits()` builder, `provider_factory`, and `src/init`. Migration step 56 injects a commented `[llm.stream_limits]` hint into existing configs. - Add `#[cfg_attr(feature = "profiling", tracing::instrument(...))]` to `chat_with_tools` on Claude, OpenAI, Gemini, and Compatible providers; to `embed_batch` on Compatible; and to all five `LlmProvider` trait methods plus `bandit_chat` and `cascade_chat` on `RouterProvider`. Span names follow `llm.<provider>.<method>` convention.
…ethods
The 5 LlmProvider impl methods on RouterProvider (chat, chat_stream, embed,
embed_batch, chat_with_tools) return `impl Future` via `Box::pin(async move
{...})`, not `async fn`. A #[tracing::instrument] attribute on such a fn only
wraps the synchronous future-construction call and exits the span before the
future is polled, making the spans no-ops.
Replace the #[cfg_attr(..., tracing::instrument(...))] attributes on those 5
methods with manual span + .instrument() applied to the returned Box::pin
future, guarded by #[cfg(feature = "profiling")]. The two async fn helpers
(bandit_chat, cascade_chat) are correct as-is and unchanged.
Also use section_header_present helper in migrate_llm_stream_limits guard
to match the idempotency pattern introduced by #4812.
bug-ops
force-pushed
the
4750-llm-sse-caps-tracing
branch
from
884d4f0 to
f9ff9a9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
StreamLimitsconfig struct ([llm.stream_limits]) withmax_tool_json_bytes(4 MiB),max_thinking_bytes(1 MiB),max_compaction_bytes(32 KiB). Wire throughLlmConfig,ClaudeProvider.with_stream_limits()builder,provider_factory, andsrc/init. Migration step 56 injects a commented[llm.stream_limits]hint. Defaults preserve existing behavior exactly.#[cfg_attr(feature = "profiling", tracing::instrument(name = "llm.chat_with_tools", ...))]tochat_with_toolson Claude, OpenAI, Gemini, and Compatible providers; also adds missingembed_batchinstrument to Compatible.RouterProviderhot-path methods. The 5fn -> impl Futuremethods (chat, chat_stream, embed, embed_batch, chat_with_tools) use manual.instrument(span)on the returnedBox::pinfuture to correctly capture async execution. The 2async fnhelpers (bandit_chat, cascade_chat) use the standard#[cfg_attr]attribute.LLM serialization gate
Not applicable — no
#[derive(Serialize, Deserialize)]structs,MessagePart,Message,ToolDefinition, or context assembly paths were modified. Changes are limited to compile-time tracing annotations (profilingfeature only) and SSE buffer size config (no request/response format changes).Test plan
cargo +nightly fmt --check— cleanRUSTFLAGS="-D warnings" cargo check --workspace --all-targets --features desktop,ide,server,chat,pdf,scheduler --locked— cleancargo clippy -p zeph-llm -p zeph-config -p zeph-core --features profiling -- -D warnings— cleancargo nextest run --workspace --lib --bins— 10496 PASS, 21 skippedcargo test --doc -p zeph-config -p zeph-llm— 15 PASSStreamLimits::default()values == old hardcoded constants (4 MiB / 1 MiB / 32 KiB)Follow-up
OllamaProvider::chat_with_toolsis the only backend still missing the profiling instrument (out of refactor(llm): chat_with_tools missing profiling-gated instrument in claude, openai, and gemini backends #4808 scope). File as P4.StreamLimits::default()values and custom-cap SSE path. File as P4.Closes #4750, #4808, #4790.