LLM Streaming

LLM Streaming

Edge adaptation of the upstream model streaming and provider abstraction layer.

Upstream reference: LLM Streaming

What Upstream Provides

The LLM subsystem (ctx.llm) provides the complete model interaction stack:

• LlmRuntime — provider registration, model catalog, call preparation, retry policy, and the llm/stream waterfall that dispatches every request.
• Adapter contract — each provider implements stream(), emitting a closed union of StreamChunk types (text-delta, reasoning-delta, tool-call-delta, usage, finish). Strict ordering: usage before finish, no chunks after termination.
• BlockAssembler — incrementally assembles raw chunks into complete ContentBlock objects and a frozen assistant Message.
• Model selection — three-level resolution: per-session log → deployment default → adapter default. Catalog is advisory — adapters accept unlisted model ids.
• Token accounting — disjoint counts: input, output, cache-read, cache-write, reasoning (informational only).

What Edge Changed

LlmRuntime

LlmRuntime is installed as-is. Streaming protocol, block assembly, retry policy, model discovery, and the llm/stream waterfall are entirely upstream code.

DeepSeek adapter

The upstream dsh-llm-deepseek adapter is installed as-is. It registers the deepseek-official provider, implements the streaming contract over the DeepSeek API, handles context-window overflow mapping, and provides model catalog and reasoning effort metadata. No Edge-specific adapter code.

Configuration via Worker environment

Edge configures the adapter through Worker environment variables instead of upstream's CLI flags or config files:

| Environment Variable | Default | Purpose|

DEEPSEEK_API_KEY	—	Provider authentication (BYOK)
DEEPSEEK_BASE_URL	https://api.deepseek.com	API endpoint (custom deployments)
---	---	---
DEEPSEEK_MODEL	deepseek-v4-flash	Default model for new sessions
---	---	---
DEEPSEEK_MAX_OUTPUT_TOKENS	256000	Output token cap per request
---	---	---
DEEPSEEK_REASONING_EFFORT	high	Thinking depth (off/low/high/max)
---	---	---
DEEPSEEK_STREAM_IDLE_TIMEOUT_MS	120000	Max silence before stream abort

All variables are validated at DO initialization (deepseek.ts). Invalid values throw immediately — no silent fallbacks.

Model selection persistence

EdgeModelSelectionBridge persists per-session model selections to DO KV (dsh-edge:session-model:{id}). Upstream stores selections in the local filesystem. The bridge provides an in-memory cache with async KV fallback, matching upstream's three-level resolution: session log → saved selection → deployment default.

File upload index

DurableObjectUploadIndex + DeepSeekFileStore track image attachments uploaded to the DeepSeek API. Upstream uses a local filesystem index; Edge uses DO KV (dsh-edge:file-upload-index).

Attachment storage tiers

Edge supports two image upload backends, selected at first use based on whether an R2 binding is configured:

| Plan | Backend | Storage | Limit|

Free (no R2)	temporary-do	DO SQL table	64 MB per workspace
Paid (R2 binding)	private-r2	Cloudflare R2	Unlimited

Both tiers accept the same image formats and sizes at upload. The difference is durability and capacity — DO storage is tied to the Durable Object's lifecycle, while R2 provides independent object storage.

What Edge Did NOT Change

• Streaming protocol (StreamChunk union, ordering invariants)
• Adapter contract (10 invariants, single-attempt-per-call)
• Block assembly and message freezing
• Retry policy (default: 5 retries, exponential backoff)
• Token accounting and billing calculation
• Model catalog and reasoning effort resolution
• llm/stream waterfall interception

Performance Characteristics

Streaming latency

The streaming path is: browser → Worker → DO → DeepSeek API → SSE chunks back through the same chain. Edge adds one network hop (Worker → DO) vs upstream's direct connection, but this is within Cloudflare's internal network (sub-millisecond). The dominant latency is the DeepSeek API's time-to-first-token (typically 500ms–2s depending on model and reasoning effort).

Idle timeout

The stream idle timeout (DEEPSEEK_STREAM_IDLE_TIMEOUT_MS, default 120s) aborts the request if no chunks arrive within the window. This protects against hung API connections. The value is clamped to MAX_TIMER_DELAY_MS (2,147,483,647ms) — a V8 timer limit.

BYOK model

Edge uses a Bring Your Own Key model — the user provides their DeepSeek API key via DEEPSEEK_API_KEY env var or the credentials UI. No Edge-side token metering or billing. API costs are between the user and DeepSeek directly.

Architecture Summary

| Component | Category | Edge Code|

LlmRuntime		One ctx.plugin() call
DeepSeek adapter		One ctx.plugin() call + config
---	---	---
Env var configuration		deepseek.ts (~80 lines)
---	---	---
Model selection bridge		model-selection-bridge.ts (~60 lines)
---	---	---
Upload index		do-upload-index.ts (~40 lines)

Key observation: The LLM subsystem is the most critical upstream capability and requires zero business-logic adaptation. All Edge-owned code is configuration mapping (env vars → plugin config) and storage delegation (filesystem → DO KV). The streaming protocol, adapter, assembly, and retry logic are 100% upstream.

LLM 流式传输

上游模型流式传输和提供者抽象层在 Edge 中的适配。

上游参考：LLM Streaming

上游提供了什么

LLM 子系统（ctx.llm）提供完整的模型交互栈：

• LlmRuntime — 提供者注册、模型目录、调用准备、重试策略和分发每个请求的 llm/stream 瀑布。
• Adapter 契约 — 每个提供者实现 stream()，发出封闭联合的 StreamChunk 类型（text-delta、reasoning-delta、tool-call-delta、usage、finish）。严格排序：usage 在 finish 之前，终止后无 chunk。
• BlockAssembler — 增量式将原始 chunk 组装为完整的 ContentBlock 和冻结的 assistant Message。
• 模型选择 — 三级解析：session 日志 → 部署默认 → 适配器默认。目录是建议性的——适配器接受未列出的模型 id。
• Token 计量 — 不重叠计数：input、output、cache-read、cache-write、reasoning（仅信息性）。

Edge 改了什么

LlmRuntime

LlmRuntime 原封安装。流式协议、块组装、重试策略、模型发现和 llm/stream 瀑布完全是上游代码。

DeepSeek 适配器

上游 dsh-llm-deepseek 适配器原封安装。注册 deepseek-official 提供者，基于 DeepSeek API 实现流式契约，处理上下文窗口溢出映射，提供模型目录和推理强度元数据。无 Edge 专属适配器代码。

通过 Worker 环境变量配置

Edge 通过 Worker 环境变量配置适配器，替代上游的 CLI 参数或配置文件：

| 环境变量 | 默认值 | 用途|

DEEPSEEK_API_KEY	—	提供者认证（BYOK）
DEEPSEEK_BASE_URL	https://api.deepseek.com	API 端点（自定义部署）
---	---	---
DEEPSEEK_MODEL	deepseek-v4-flash	新 session 的默认模型
---	---	---
DEEPSEEK_MAX_OUTPUT_TOKENS	256000	每请求输出 token 上限
---	---	---
DEEPSEEK_REASONING_EFFORT	high	思考深度（off/low/high/max）
---	---	---
DEEPSEEK_STREAM_IDLE_TIMEOUT_MS	120000	流无数据时的最大等待

所有变量在 DO 初始化时验证（deepseek.ts）。无效值立即抛错——无静默降级。

模型选择持久化

EdgeModelSelectionBridge 将每 session 的模型选择持久化到 DO KV（dsh-edge:session-model:{id}）。上游存在本地文件系统。该桥提供内存缓存 + 异步 KV 降级，匹配上游的三级解析：session 日志 → 保存的选择 → 部署默认。

文件上传索引

DurableObjectUploadIndex + DeepSeekFileStore 追踪上传到 DeepSeek API 的图片附件。上游用本地文件系统索引；Edge 用 DO KV（dsh-edge:file-upload-index）。

附件存储分层

Edge 支持两种图片上传后端，在首次使用时根据是否配置了 R2 binding 自动选择：

| 方案 | 后端 | 存储 | 限制|

免费（无 R2）	temporary-do	DO SQL 表	每 workspace 64 MB
付费（R2 binding）	private-r2	Cloudflare R2	无限制

两种方案在上传时接受相同的图片格式和大小。差异在于持久性和容量——DO 存储绑定于 Durable Object 的生命周期，R2 提供独立的对象存储。

Edge 没有改什么

• 流式协议（StreamChunk 联合、排序不变量）
• Adapter 契约（10 条不变量、单次尝试）
• 块组装和消息冻结
• 重试策略（默认：5 次重试、指数退避）
• Token 计量和计费计算
• 模型目录和推理强度解析
• llm/stream 瀑布拦截

性能特征

流式延迟

流式路径：浏览器 → Worker → DO → DeepSeek API → SSE chunk 原路返回。Edge 比上游多一跳（Worker → DO），但在 Cloudflare 内部网络中（亚毫秒）。主导延迟是 DeepSeek API 的首 token 时间（通常 500ms–2s，取决于模型和推理强度）。

空闲超时

流空闲超时（DEEPSEEK_STREAM_IDLE_TIMEOUT_MS，默认 120s）在窗口内无 chunk 时中止请求。防御挂起的 API 连接。值上限为 MAX_TIMER_DELAY_MS（2,147,483,647ms）——V8 定时器限制。

BYOK 模式

Edge 使用自带密钥模式——用户通过 DEEPSEEK_API_KEY 环境变量或凭据 UI 提供 DeepSeek API 密钥。Edge 侧无 token 计量或计费。API 费用由用户直接与 DeepSeek 结算。

架构总结

| 组件 | 分类 | Edge 代码|

LlmRuntime		一行 ctx.plugin() 调用
DeepSeek 适配器		一行 ctx.plugin() 调用 + 配置
---	---	---
环境变量配置		deepseek.ts（约 80 行）
---	---	---
模型选择桥		model-selection-bridge.ts（约 60 行）
---	---	---
上传索引		do-upload-index.ts（约 40 行）

关键观察：LLM 子系统是最关键的上游能力，需要零业务逻辑适配。所有 Edge 代码都是配置映射（环境变量 → 插件配置）和存储委托（文件系统 → DO KV）。流式协议、适配器、组装和重试逻辑 100% 上游。