Skip to content

feat: configurable per-node first-token timeout for LLM nodes#38586

Draft
lin-snow wants to merge 3 commits into
mainfrom
feat/llm-ft-timeout
Draft

feat: configurable per-node first-token timeout for LLM nodes#38586
lin-snow wants to merge 3 commits into
mainfrom
feat/llm-ft-timeout

Conversation

@lin-snow

@lin-snow lin-snow commented Jul 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Adds an opt-in, per-node first-token timeout for LLM-family nodes. While a stream waits for its first token, a wall-clock watchdog aborts the transport connection after the configured timeout and raises FirstTokenTimeoutError, which surfaces as a node failure and is handled by the node's existing error strategy (retry / fail-branch / default). Default 0 disables it, so current behavior is unchanged.

  • Backend (api) — transport-layer enforcement in the plugin-daemon stream. The value is carried from the graphon LLM adapter (DifyPreparedLLM) down to _stream_request through a ContextVar, so the intermediate model-runtime layers stay untouched. On timeout the raw socket is shut down (socket.shutdown) to unblock the in-flight read; response.close() alone does not interrupt a blocked sync iter_lines().
  • Frontend (web) — a first_token_timeout field (milliseconds; 0 = off) in the LLM node config panel, written to node.data.invocation.first_token_timeout (graphon's LLMInvocationConfig), with i18n across all locales.

Dependency (blocking). Requires graphon first-token-timeout support (langgenius/graphon#212 — the FirstTokenTimeoutError type and the protocol first_token_timeout parameter). This stays a draft until that is released and the graphon pin in api/pyproject.toml is bumped to the released version.

Screenshots

Before After
no first-token control on the LLM node "first token timeout" field in the retry/error panel

Checklist

  • This change requires a documentation update, included: Dify Document
  • I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
  • I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
  • I've updated the documentation accordingly.
  • I ran make lint && make type-check (backend) and cd web && pnpm exec vp staged (frontend) to appease the lint gods

From Claude Code

@lin-snow
feat(workflow): enforce per-node first-token timeout in the plugin tr…
bbd0171 
…ansport

Adds transport-layer enforcement for the LLM node's first_token_timeout, which
graphon carries down the LLM invoke path. While a stream waits for its first
token, a wall-clock watchdog fires after the configured timeout and shuts the
underlying socket down to abort the in-flight read; the caller raises
FirstTokenTimeoutError, which bubbles up as a node failure and is handled by the
node's existing error strategy (retry / fail-branch / default).

The value is tunneled from the graphon-facing adapter (DifyPreparedLLM) to
_stream_request through a ContextVar, so the intermediate model-runtime layers
that don't take the parameter stay untouched. It is disarmed on the first token,
so later tokens are never gated. Default 0 (disabled) keeps today's behavior.

response.close() does not interrupt a blocked sync iter_lines(); the cut is
socket.shutdown(SHUT_RDWR) reached via httpx' network_stream extension. The
daemon emits no pre-first-token heartbeats, so first-token timing is clean.

Requires graphon with langgenius/graphon#212 (FirstTokenTimeoutError + the
protocol first_token_timeout parameter); the graphon pin bump is release-gated.
@github-actions github-actions Bot added the web This relates to changes on the web. label Jul 9, 2026
@github-actions

github-actions Bot commented Jul 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Pyrefly Diff

base → PR 
--- /tmp/pyrefly_base.txt	2026-07-09 10:53:20.499361213 +0000
+++ /tmp/pyrefly_pr.txt	2026-07-09 10:53:05.030505160 +0000
@@ -1,3 +1,5 @@
+ERROR Could not import `FirstTokenTimeoutError` from `graphon.model_runtime.errors.invoke` [missing-module-attribute]
+  --> core/plugin/impl/base.py:36:5
 ERROR Argument `SimpleNamespace` is not assignable to parameter `dataset` with type `Dataset` in function `dify_vdb_alibabacloud_mysql.alibabacloud_mysql_vector.AlibabaCloudMySQLVectorFactory.init_vector` [bad-argument-type]
   --> providers/vdb/vdb-alibabacloud-mysql/tests/unit_tests/test_alibabacloud_mysql_factory.py:42:38
 ERROR Argument `SimpleNamespace` is not assignable to parameter `dataset` with type `Dataset` in function `dify_vdb_alibabacloud_mysql.alibabacloud_mysql_vector.AlibabaCloudMySQLVectorFactory.init_vector` [bad-argument-type]
@@ -4502,6 +4504,8 @@
    --> tests/unit_tests/core/ops/test_utils.py:107:41
 ERROR Argument `None` is not assignable to parameter `project` with type `str` in function `core.ops.utils.validate_project_name` [bad-argument-type]
    --> tests/unit_tests/core/ops/test_utils.py:136:40
+ERROR Could not import `FirstTokenTimeoutError` from `graphon.model_runtime.errors.invoke` [missing-module-attribute]
+  --> tests/unit_tests/core/plugin/impl/test_first_token_timeout.py:15:49
 ERROR Missing required key `content` for TypedDict `MultimodalRerankInput` [bad-typed-dict-key]
    --> tests/unit_tests/core/plugin/impl/test_model_client.py:472:19
 ERROR Missing required key `content_type` for TypedDict `MultimodalRerankInput` [bad-typed-dict-key]
@@ -6533,53 +6537,57 @@
 ERROR `SimpleNamespace` is not assignable to attribute `graph_runtime_state` with type `GraphRuntimeState` [bad-assignment]
     --> tests/unit_tests/core/workflow/test_node_factory.py:1130:39
 ERROR `object | None` is not assignable to attribute `model_runtime` with type `LLMModelRuntime` [bad-assignment]
-  --> tests/unit_tests/core/workflow/test_node_runtime.py:68:30
+  --> tests/unit_tests/core/workflow/test_node_runtime.py:69:30
 ERROR Class member `_ModelTypeInstanceStub.get_model_schema` overrides parent class `LargeLanguageModel` in an inconsistent manner [bad-override-param-name]
-  --> tests/unit_tests/core/workflow/test_node_runtime.py:69:14
+  --> tests/unit_tests/core/workflow/test_node_runtime.py:70:14
 ERROR Object of class `ExternalRecipient` has no attribute `reference_id` [missing-attribute]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:136:12
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:137:12
 ERROR Argument `object` is not assignable to parameter `method` with type `EmailDeliveryMethod | InteractiveSurfaceDeliveryMethod` in function `core.workflow.node_runtime.apply_dify_debug_email_recipient` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:151:45
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:152:45
 ERROR Argument `_ModelInstanceStub` is not assignable to parameter `model_instance` with type `ModelInstance` in function `core.workflow.node_runtime.DifyPreparedLLM.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:174:32
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:175:32
 ERROR Object of class `FunctionType` has no attribute `assert_called_once_with` [missing-attribute]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:193:5
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:194:5
 ERROR Argument `_ModelInstanceStub` is not assignable to parameter `model_instance` with type `ModelInstance` in function `core.workflow.node_runtime.DifyPreparedLLM.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:207:32
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:219:32
 ERROR Argument `_ModelInstanceStub` is not assignable to parameter `model_instance` with type `ModelInstance` in function `core.workflow.node_runtime.DifyPreparedLLM.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:215:32
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:252:32
+ERROR Argument `_ModelInstanceStub` is not assignable to parameter `model_instance` with type `ModelInstance` in function `core.workflow.node_runtime.DifyPreparedLLM.__init__` [bad-argument-type]
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:271:32
+ERROR Argument `_ModelInstanceStub` is not assignable to parameter `model_instance` with type `ModelInstance` in function `core.workflow.node_runtime.DifyPreparedLLM.__init__` [bad-argument-type]
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:279:32
 ERROR Argument `SimpleNamespace` is not assignable to parameter `model_instance` with type `ModelInstance` in function `core.workflow.node_runtime.DifyPreparedLLM.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:241:32
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:305:32
 ERROR Argument `_ModelInstanceStub` is not assignable to parameter `model_instance` with type `ModelInstance` in function `core.workflow.node_runtime.DifyPreparedPollingLLM.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:266:39
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:330:39
 ERROR Argument `_ModelInstanceStub` is not assignable to parameter `model_instance` with type `ModelInstance` in function `core.workflow.node_runtime.DifyPreparedPollingLLM.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:323:39
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:387:39
 ERROR Argument `Literal['chat']` is not assignable to parameter `model_mode` with type `LLMMode` in function `core.workflow.node_runtime.DifyPromptMessageSerializer.serialize` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:340:20
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:404:20
 ERROR Argument `SimpleNamespace` is not assignable to parameter `file_reference_factory` with type `FileReferenceFactoryProtocol` in function `core.workflow.node_runtime.DifyRetrieverAttachmentLoader.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:375:32
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:439:32
 ERROR Object of class `NoneType` has no attribute `storage_key` [missing-attribute]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:386:12
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:450:12
 ERROR Argument `SimpleNamespace` is not assignable to parameter `file_reference_factory` with type `FileReferenceFactoryProtocol` in function `core.workflow.node_runtime.DifyRetrieverAttachmentLoader.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:451:32
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:515:32
 ERROR Argument `SimpleNamespace` is not assignable to parameter `file_reference_factory` with type `FileReferenceFactoryProtocol` in function `core.workflow.node_runtime.DifyRetrieverAttachmentLoader.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:477:32
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:541:32
 ERROR Argument `object` is not assignable to parameter `variable_pool` with type `VariablePool | None` in function `core.workflow.node_runtime.DifyToolNodeRuntime.get_runtime` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:575:23
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:639:23
 ERROR Object of class `object` has no attribute `tool` [missing-attribute]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:579:12
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:643:12
 ERROR Object of class `object` has no attribute `parent_trace_context` [missing-attribute]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:580:12
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:644:12
 ERROR Object of class `object` has no attribute `trace_session_id` [missing-attribute]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:603:5
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:667:5
 ERROR Argument `object` is not assignable to parameter `variable_pool` with type `VariablePool | None` in function `core.workflow.node_runtime.DifyToolNodeRuntime.get_runtime` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:620:23
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:684:23
 ERROR Object of class `object` has no attribute `trace_session_id` [missing-attribute]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:624:12
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:688:12
 ERROR Argument `object` is not assignable to parameter `variable_pool` with type `VariablePool | None` in function `core.workflow.node_runtime.DifyToolNodeRuntime.get_runtime` [bad-argument-type]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:652:23
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:716:23
 ERROR Object of class `object` has no attribute `tool` [missing-attribute]
-   --> tests/unit_tests/core/workflow/test_node_runtime.py:655:12
+   --> tests/unit_tests/core/workflow/test_node_runtime.py:719:12
 ERROR Argument `TestWorkflowEntry.test_single_step_run_injects_code_limits.StubWorkflow` is not assignable to parameter `workflow` with type `Workflow` in function `core.workflow.workflow_entry.WorkflowEntry.single_step_run` [bad-argument-type]
    --> tests/unit_tests/core/workflow/test_workflow_entry.py:145:22
 ERROR Object of class `_FakeModelNodeMixin` has no attribute `node_data` [missing-attribute]

@github-actions

github-actions Bot commented Jul 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Pyrefly Type Coverage

Metric Base PR Delta
Type coverage 52.64% 52.66% +0.02%
Strict coverage 52.16% 52.18% +0.03%
Typed symbols 32,618 32,663 +45
Untyped symbols 29,619 29,630 +11
Modules 2986 2988 +2

@codecov

codecov Bot commented Jul 9, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.88889% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 85.35%. Comparing base (cee88de) to head (4b4987c).
⚠️ Report is 17 commits behind head on main.

Files with missing lines Patch % Lines
web/app/components/workflow/nodes/llm/panel.tsx 50.00% 1 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main   #38586      +/-   ##
==========================================
+ Coverage   85.34%   85.35%   +0.01%     
==========================================
  Files        5018     5021       +3     
  Lines      266995   267054      +59     
  Branches    50824    50835      +11     
==========================================
+ Hits       227861   227957      +96     
+ Misses      34665    34629      -36     
+ Partials     4469     4468       -1
Flag Coverage Δ
dify-ui 94.61% <ø> (ø)
web 84.89% <88.88%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@lin-snow
feat(web): add first-token timeout to the LLM node config
5929092 
Adds a numeric first_token_timeout (milliseconds; 0 = off, max 30s) to the LLM
node's invocation config (node.data.invocation.first_token_timeout), matching
graphon's LLMInvocationConfig. Rendered as a field in the LLM node panel, with
i18n across all locales.

The value is enforced in the backend plugin transport; this is only the config
surface.
@lin-snow lin-snow force-pushed the feat/llm-ft-timeout branch from af4bec1 to 5929092 Compare July 9, 2026 10:08
@lin-snow
refactor(llm): harden first-token timeout and fix ms unit i18n
4b4987c 
Follow-ups from code review:

- Transport: treat a non-positive timeout (0/negative) as disabled so a 0-second
  watchdog can never arm; add a post-stream backstop that raises
  FirstTokenTimeoutError when aborting the socket ends the stream with a clean EOF
  instead of a transport error; only raise when no token was delivered.
- Watchdog: make fire/disarm mutually exclusive with a lock, so a token arriving on
  the deadline is never both yielded and aborted and `fired` reliably means
  "timed out before the first token".
- Adapter: extract the shared set/reset-ContextVar logic into one helper.
- i18n: fix nodes.common.retry.ms, mistranslated as the "Ms./Mrs." honorific in
  9 locales (de/es/hi/th/pl/tr/vi/id/ro; uk lowercased).
- Tests: real-socket test proving socket.shutdown unblocks a blocked httpx
  iter_lines and surfaces a real RequestError; a clean-EOF backstop test; a
  0=disabled test; and NaN/negative-clamp + preserve-other-fields assertions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@iamjoel iamjoel Awaiting requested review from iamjoel iamjoel will be requested when the pull request is marked ready for review iamjoel is a code owner
@zxhlyh zxhlyh Awaiting requested review from zxhlyh zxhlyh will be requested when the pull request is marked ready for review zxhlyh is a code owner
@WH-2099 WH-2099 Awaiting requested review from WH-2099 WH-2099 will be requested when the pull request is marked ready for review WH-2099 is a code owner
@laipz8200 laipz8200 Awaiting requested review from laipz8200 laipz8200 will be requested when the pull request is marked ready for review laipz8200 is a code owner
@QuantumGhost QuantumGhost Awaiting requested review from QuantumGhost QuantumGhost will be requested when the pull request is marked ready for review QuantumGhost is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

web This relates to changes on the web.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@lin-snow