Skip to content

feat(workflow): resume runs across rolling updates - #39706

Open
zyssyz123 wants to merge 15 commits into
mainfrom
song/snp-559-workflow-run-pauseresume-in-rolling-update
Open

feat(workflow): resume runs across rolling updates#39706
zyssyz123 wants to merge 15 commits into
mainfrom
song/snp-559-workflow-run-pauseresume-in-rolling-update

Conversation

@zyssyz123

@zyssyz123 zyssyz123 commented Jul 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Implements durable pause/resume for workflow runs during planned rolling updates, covering the full product surface selected in SNP-559: Workflow, Advanced Chat/Chatflow, snippet/one-step runs, triggered/scheduled runs, and RAG pipelines across console, web, service API, OpenAPI, streaming, and blocking modes.

The same workflow_run_id continues on an upgraded worker. Completed nodes are not replayed. Clients can reconnect by run ID and Last-Event-ID; the original TCP/SSE connection is not treated as durable.

Runtime protocol

  • Adds a durable PREPARING -> PREPARED -> READY -> CLAIMED -> RESUMED/FAILED handoff protocol with leases, retries, acknowledgement, snapshot GC, and terminal compensation.
  • Uses a source-side flush barrier before activation and a capability-isolated Celery queue so N workers cannot steal N+1 resume work during adjacent-version rolling updates.
  • Preserves monotonic terminal state and makes Stop win through creator-scoped durable cancellation tombstones.
  • Keeps the public run status RUNNING during maintenance; records handoff duration separately while excluding handoff wait from execution-timeout accounting.
  • Adds RAG batch/group sealing, tenant-slot release, dispatch leases, and done fences for nested/parallel pipeline work.
  • Persists and replays workflow events through Redis Streams, including SSE IDs, bounded cursor validation, gap detection, snapshots, and frontend reconnect/de-duplication.

Safety and rollout

  • Feature flag defaults to off: WORKFLOW_HANDOFF_ENABLED=false.
  • Planned warm SIGTERM is supported; OOM/SIGKILL is intentionally not claimed as recoverable.
  • Unsafe, incompatible, exhausted, or timed-out resumes fail closed to STOPPED; the workflow is never restarted from the beginning.
  • Enabling requires shared object storage, EVENT_BUS_REDIS_CHANNEL_TYPE=streams, stream retention of at least drain timeout + 60s, and a process-shared Celery pool (prefork is rejected).
  • Default drain timeout is 600s; Gunicorn and Compose graceful-stop defaults are 660s.
  • Deploy schema and upgraded resume consumers first, then enable handoff creation. Keep upgraded consumers running until old handoffs have reached terminal state.

Validation

  • Backend changed/new unit regression: 1,609 passed across 86 changed test files with four xdist workers; the complete controller suite passed 3,575 tests serially.
  • CI completed with 44 successful and 7 intentionally skipped checks; Codecov patch coverage passed at 89.85% (target 86.13%).
  • Container integration regression: 123 passed across Redis Streams, handoff repositories, pause persistence, RAG dispatch, cleanup, and controllers.
  • Production typing: Mypy passed for 1,660 source files; production, strict unit, and container Pyrefly passed with 0 errors.
  • Backend architecture/contracts: Ruff and format passed for 200 changed Python files; import boundaries passed; response contract lint reported 472 valid / 0 mismatches; no-new-getattr and no-controller-SQLAlchemy guards passed.
  • Frontend: the complete Vitest suite passed with 23,556 tests (2 skipped); TypeScript type-check, 6,008 TSS checks, Knip, production Knip, and production-unused passed.
  • Real Gunicorn processes: feature disabled/enabled health checks returned 200; enabled SIGTERM entered the warm-shutdown hook and exited cleanly.
  • Real two-worker Celery handoff: SIGTERM interrupted the source worker mid-workflow; an upgraded survivor resumed the same run ID and completed it once, with unique continuous node indices, one terminal stream event, retained event cursors, and a measured 64.704 ms handoff.
  • Alembic: one head; isolated PostgreSQL full upgrade, downgrade to the parent, and re-upgrade passed with tables, columns, indexes, and checks verified. PostgreSQL and MySQL offline upgrade/downgrade SQL compilation passed.
  • Compose/config: generated compose is in sync; compose and template parse successfully; API/worker grace is 660s; handoff + prefork is rejected; the capability queue remains consumed after handoff creation is disabled.

Screenshots

Not applicable (runtime continuity and reconnect behavior; no visual redesign).

Checklist

  • Tests were added for each introduced behavior and failure path.
  • Backend lint/type checks and frontend staged-equivalent checks were run locally.
  • Configuration examples, Docker operator documentation, and rollout guidance were updated.
  • Migration, runtime-process, and real cross-worker handoff regressions were run locally.
  • I understand that this PR may be closed if there was no prior discussion or issue.

From Codex

@zyssyz123
zyssyz123 requested a review from a team July 28, 2026 18:43
@dosubot dosubot Bot added the size:XXL This PR changes 1000+ lines, ignoring generated files. label Jul 28, 2026
@github-actions github-actions Bot added the web This relates to changes on the web. label Jul 28, 2026
@github-actions

github-actions Bot commented Jul 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Pyrefly Type Coverage

Metric Base PR Delta
Type coverage 56.78% 57.64% +0.85%
Strict coverage 56.31% 57.16% +0.85%
Typed symbols 37,092 38,626 +1,534
Untyped symbols 28,468 28,627 +159
Modules 3091 3134 +43

@github-actions

github-actions Bot commented Jul 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Pyrefly Diff

base → PR 
--- /tmp/pyrefly_base.txt	2026-07-30 04:27:55.101598246 +0000
+++ /tmp/pyrefly_pr.txt	2026-07-30 04:27:39.090647606 +0000
@@ -657,19 +657,19 @@
 ERROR Argument `dict[tuple[str, str], dict[str, str] | str]` is not assignable to parameter `variables` with type `dict[tuple[str, str], object] | None` in function `TestPauseStatePersistenceLayerTestContainers._create_graph_runtime_state` [bad-argument-type]
    --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:334:23
 ERROR Argument `dict[tuple[str, str], dict[str, str] | list[int] | str]` is not assignable to parameter `variables` with type `dict[tuple[str, str], object] | None` in function `TestPauseStatePersistenceLayerTestContainers._create_graph_runtime_state` [bad-argument-type]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:398:23
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:399:23
 ERROR Object of class `FixtureFunctionDefinition` has no attribute `_workflow_run_repo` [missing-attribute]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:410:24
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:412:24
 ERROR Object of class `AdvancedChatAppGenerateEntity` has no attribute `workflow_execution_id` [missing-attribute]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:424:16
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:426:16
 ERROR Argument `dict[str, LiteralString]` is not assignable to parameter `outputs` with type `dict[str, object] | None` in function `TestPauseStatePersistenceLayerTestContainers._create_graph_runtime_state` [bad-argument-type]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:465:21
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:468:21
 ERROR Object of class `AdvancedChatAppGenerateEntity` has no attribute `workflow_execution_id` [missing-attribute]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:489:16
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:493:16
 ERROR Object of class `FixtureFunctionDefinition` has no attribute `_workflow_run_repo` [missing-attribute]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:551:24
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:556:24
 ERROR Object of class `AdvancedChatAppGenerateEntity` has no attribute `workflow_execution_id` [missing-attribute]
-   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:554:16
+   --> tests/test_containers_integration_tests/core/app/layers/test_pause_state_persist_layer.py:559:16
 ERROR Object of class `NoneType` has no attribute `id` [missing-attribute]
   --> tests/test_containers_integration_tests/core/rag/retrieval/test_dataset_retrieval_integration.py:38:23
 ERROR Object of class `NoneType` has no attribute `id` [missing-attribute]
@@ -739,7 +739,7 @@
 ERROR Argument `Literal['owner']` is not assignable to parameter `role` with type `SQLCoreOperations[TenantAccountRole] | TenantAccountRole` in function `models.account.TenantAccountJoin.__init__` [bad-argument-type]
   --> tests/test_containers_integration_tests/helpers/execution_extra_content.py:48:14
 ERROR Returned type `int | str` is not assignable to declared return type `str` [bad-return]
-   --> tests/test_containers_integration_tests/libs/broadcast_channel/redis/test_sharded_channel.py:363:16
+   --> tests/test_containers_integration_tests/libs/broadcast_channel/redis/test_sharded_channel.py:379:16
 ERROR Object of class `TypeEngine` has no attribute `length` [missing-attribute]
    --> tests/test_containers_integration_tests/models/test_types_enum_text.py:104:16
 ERROR Object of class `NoneType` has no attribute `user_type` [missing-attribute]
@@ -750,11 +750,11 @@
    --> tests/test_containers_integration_tests/models/test_types_enum_text.py:146:20
 ERROR Object of class `NoneType` has no attribute `user_type_nullable` [missing-attribute]
    --> tests/test_containers_integration_tests/models/test_types_enum_text.py:147:20
-ERROR Cannot instantiate `DifyAPISQLAlchemyWorkflowNodeExecutionRepository` because the following members are abstract: `save`, `save_synchronously`, `save_execution_data`, `get_by_workflow_execution` [bad-instantiation]
+ERROR Cannot instantiate `DifyAPISQLAlchemyWorkflowNodeExecutionRepository` because the following members are abstract: `save`, `save_synchronously`, `save_execution_data`, `get_by_workflow_execution`, `get_max_index` [bad-instantiation]
   --> tests/test_containers_integration_tests/repositories/test_sqlalchemy_api_workflow_node_execution_repository.py:72:70
-ERROR Cannot instantiate `DifyAPISQLAlchemyWorkflowNodeExecutionRepository` because the following members are abstract: `save`, `save_synchronously`, `save_execution_data`, `get_by_workflow_execution` [bad-instantiation]
+ERROR Cannot instantiate `DifyAPISQLAlchemyWorkflowNodeExecutionRepository` because the following members are abstract: `save`, `save_synchronously`, `save_execution_data`, `get_by_workflow_execution`, `get_max_index` [bad-instantiation]
   --> tests/test_containers_integration_tests/repositories/test_sqlalchemy_api_workflow_node_execution_repository.py:89:70
-ERROR Cannot instantiate `DifyAPISQLAlchemyWorkflowNodeExecutionRepository` because the following members are abstract: `save`, `save_synchronously`, `save_execution_data`, `get_by_workflow_execution` [bad-instantiation]
+ERROR Cannot instantiate `DifyAPISQLAlchemyWorkflowNodeExecutionRepository` because the following members are abstract: `save`, `save_synchronously`, `save_execution_data`, `get_by_workflow_execution`, `get_max_index` [bad-instantiation]
    --> tests/test_containers_integration_tests/repositories/test_sqlalchemy_api_workflow_node_execution_repository.py:151:70
 ERROR Class member `_TestWorkflowRunRepository.save` overrides a member in a parent class but is missing an `@override` decorator [missing-override-decorator]
   --> tests/test_containers_integration_tests/repositories/test_sqlalchemy_api_workflow_run_repository.py:45:9
@@ -1456,7 +1456,7 @@
    --> tests/test_containers_integration_tests/services/workflow/test_workflow_converter.py:659:16
 ERROR Argument `Literal['owner']` is not assignable to parameter `role` with type `SQLCoreOperations[TenantAccountRole] | TenantAccountRole` in function `models.account.TenantAccountJoin.__init__` [bad-argument-type]
   --> tests/test_containers_integration_tests/services/workflow/test_workflow_deletion.py:38:18
-ERROR Cannot instantiate `DifyAPISQLAlchemyWorkflowNodeExecutionRepository` because the following members are abstract: `save`, `save_synchronously`, `save_execution_data`, `get_by_workflow_execution` [bad-instantiation]
+ERROR Cannot instantiate `DifyAPISQLAlchemyWorkflowNodeExecutionRepository` because the following members are abstract: `save`, `save_synchronously`, `save_execution_data`, `get_by_workflow_execution`, `get_max_index` [bad-instantiation]
   --> tests/test_containers_integration_tests/services/workflow/test_workflow_node_execution_service_repository.py:21:64
 ERROR Argument `Literal['active']` is not assignable to parameter `status` with type `AccountStatus | SQLCoreOperations[AccountStatus]` in function `models.account.Account.__init__` [bad-argument-type]
   --> tests/test_containers_integration_tests/tasks/test_add_document_to_index_task.py:56:20
@@ -2069,7 +2069,7 @@
 ERROR Argument `scoped_session[flask_sqlalchemy.session.Session]` is not assignable to parameter `session` with type `sqlalchemy.orm.session.Session` in function `_add_binding` [bad-argument-type]
   --> tests/unit_tests/controllers/console/datasets/test_data_source.py:83:28
 ERROR Object of class `FunctionType` has no attribute `assert_called_once_with` [missing-attribute]
-   --> tests/unit_tests/controllers/console/explore/test_trial.py:189:5
+   --> tests/unit_tests/controllers/console/explore/test_trial.py:190:5
 ERROR Argument `str` is not assignable to parameter `view` with type `(InstalledApp) -> str` in function `controllers.console.explore.wraps.decorator` [bad-argument-type]
   --> tests/unit_tests/controllers/console/explore/test_wraps.py:73:18
 ERROR Argument `str` is not assignable to parameter `view` with type `(InstalledApp) -> str` in function `controllers.console.explore.wraps.decorator` [bad-argument-type]
@@ -2315,27 +2315,25 @@
 ERROR Missing argument `session` in function `services.conversation_service.ConversationService.pagination_by_last_id` [missing-argument]
    --> tests/unit_tests/controllers/service_api/app/test_conversation.py:480:59
 ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `AdvancedChatAppGenerateEntity | WorkflowAppGenerateEntity` in function `core.app.apps.common.workflow_response_converter.WorkflowResponseConverter.__init__` [bad-argument-type]
-   --> tests/unit_tests/controllers/service_api/app/test_hitl_service_api.py:125:37
+   --> tests/unit_tests/controllers/service_api/app/test_hitl_service_api.py:126:37
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow` with type `WorkflowSnapshot` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/controllers/service_api/app/test_hitl_service_api.py:485:22
+   --> tests/unit_tests/controllers/service_api/app/test_hitl_service_api.py:490:22
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/controllers/service_api/app/test_hitl_service_api.py:486:27
+   --> tests/unit_tests/controllers/service_api/app/test_hitl_service_api.py:491:27
 ERROR Argument `SimpleNamespace` is not assignable to parameter `conversation` with type `ConversationSnapshot` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/controllers/service_api/app/test_hitl_service_api.py:487:26
-ERROR Argument `SimpleNamespace` is not assignable to parameter `message` with type `MessageSnapshot` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/controllers/service_api/app/test_hitl_service_api.py:488:21
+   --> tests/unit_tests/controllers/service_api/app/test_hitl_service_api.py:492:26
 ERROR Argument `(**kwargs: Unknown) -> None` is not assignable to parameter `draft_var_saver_factory` with type `DraftVariableSaverFactory` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/controllers/service_api/app/test_hitl_service_api.py:498:37
+   --> tests/unit_tests/controllers/service_api/app/test_hitl_service_api.py:503:37
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow` with type `Workflow` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/controllers/service_api/app/test_hitl_service_api.py:578:22
+   --> tests/unit_tests/controllers/service_api/app/test_hitl_service_api.py:583:22
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/controllers/service_api/app/test_hitl_service_api.py:579:27
+   --> tests/unit_tests/controllers/service_api/app/test_hitl_service_api.py:584:27
 ERROR Argument `SimpleNamespace` is not assignable to parameter `user` with type `Account | EndUser` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/controllers/service_api/app/test_hitl_service_api.py:580:18
+   --> tests/unit_tests/controllers/service_api/app/test_hitl_service_api.py:585:18
 ERROR Argument `(**kwargs: Unknown) -> None` is not assignable to parameter `draft_var_saver_factory` with type `DraftVariableSaverFactory` in function `core.app.apps.workflow.generate_task_pipeline.WorkflowAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/controllers/service_api/app/test_hitl_service_api.py:582:37
+   --> tests/unit_tests/controllers/service_api/app/test_hitl_service_api.py:587:37
 ERROR Argument `SimpleNamespace` is not assignable to parameter `graph_runtime_state` with type `GraphRuntimeState` in function `core.app.apps.common.workflow_response_converter.WorkflowResponseConverter.workflow_pause_to_stream_response` [bad-argument-type]
-   --> tests/unit_tests/controllers/service_api/app/test_hitl_service_api.py:674:33
+   --> tests/unit_tests/controllers/service_api/app/test_hitl_service_api.py:679:33
 ERROR Argument `list[FromClause]` is not assignable to parameter `tables` with type `Sequence[Table] | None` in function `sqlalchemy.sql.schema.MetaData.create_all` [bad-argument-type]
   --> tests/unit_tests/controllers/service_api/conftest.py:40:16
 ERROR No matching overload found for function `typing.MutableMapping.update` called with arguments: (dict[str, object]) [no-matching-overload]
@@ -2915,161 +2913,165 @@
    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:307:18
 ERROR Object of class `object` has no attribute `single_iteration_run` [missing-attribute]
    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:318:16
-ERROR Object of class `object` has no attribute `extras` [missing-attribute]
+ERROR Object of class `object` has no attribute `workflow_run_id` [missing-attribute]
    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:319:16
+ERROR Object of class `object` has no attribute `extras` [missing-attribute]
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:320:16
 ERROR Argument `SimpleNamespace` is not assignable to parameter `app_model` with type `App` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.single_loop_generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:368:23
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:369:23
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow` with type `Workflow` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.single_loop_generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:369:22
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:370:22
 ERROR Argument `SimpleNamespace` is not assignable to parameter `user` with type `Account | EndUser` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.single_loop_generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:371:18
-ERROR Argument `SimpleNamespace` is not assignable to parameter `args` with type `LoopNodeRunPayload` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.single_loop_generate` [bad-argument-type]
    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:372:18
+ERROR Argument `SimpleNamespace` is not assignable to parameter `args` with type `LoopNodeRunPayload` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.single_loop_generate` [bad-argument-type]
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:373:18
 ERROR Object of class `object` has no attribute `single_loop_run` [missing-attribute]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:382:16
-ERROR Object of class `object` has no attribute `extras` [missing-attribute]
    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:383:16
+ERROR Object of class `object` has no attribute `workflow_run_id` [missing-attribute]
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:384:16
+ERROR Object of class `object` has no attribute `extras` [missing-attribute]
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:385:16
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow` with type `Workflow` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:470:22
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:479:22
 ERROR Argument `SimpleNamespace` is not assignable to parameter `user` with type `Account | EndUser` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:471:18
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:480:18
 ERROR Argument `SimpleNamespace` is not assignable to parameter `session` with type `Session` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:474:21
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:483:21
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow_execution_repository` with type `WorkflowExecutionRepository` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:475:43
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:484:43
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow_node_execution_repository` with type `WorkflowNodeExecutionRepository` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:476:48
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:485:48
 ERROR Argument `SimpleNamespace` is not assignable to parameter `pause_state_config` with type `PauseStateLayerConfig | None` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:480:32
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:489:32
 ERROR Cannot index into `Generator[Mapping[str, Any] | str]` [bad-index]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:483:16
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:492:16
 ERROR Cannot index into `object` [bad-index]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:487:33
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:496:33
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow` with type `Workflow` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:577:22
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:593:22
 ERROR Argument `SimpleNamespace` is not assignable to parameter `user` with type `Account | EndUser` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:578:18
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:594:18
 ERROR Argument `SimpleNamespace` is not assignable to parameter `session` with type `Session` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:581:21
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:597:21
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow_execution_repository` with type `WorkflowExecutionRepository` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:582:43
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:598:43
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow_node_execution_repository` with type `WorkflowNodeExecutionRepository` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:583:48
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:599:48
 ERROR Argument `SimpleNamespace` is not assignable to parameter `conversation` with type `Conversation | None` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:584:26
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:600:26
 ERROR Argument `SimpleNamespace` is not assignable to parameter `message` with type `Message | None` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:585:21
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:601:21
 ERROR Argument `SimpleNamespace` is not assignable to parameter `flask_app` with type `Flask` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:645:27
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:661:27
 ERROR Argument `SimpleNamespace` is not assignable to parameter `context` with type `Context` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:650:25
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:666:25
 ERROR Argument `SimpleNamespace` is not assignable to parameter `variable_loader` with type `VariableLoader` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:651:33
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:667:33
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow_execution_repository` with type `WorkflowExecutionRepository` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:652:47
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:668:47
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow_node_execution_repository` with type `WorkflowNodeExecutionRepository` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:653:52
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:669:52
 ERROR Argument `SimpleNamespace` is not assignable to parameter `flask_app` with type `Flask` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:709:27
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:725:27
 ERROR Argument `SimpleNamespace` is not assignable to parameter `context` with type `Context` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:714:25
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:730:25
 ERROR Argument `SimpleNamespace` is not assignable to parameter `variable_loader` with type `VariableLoader` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:715:33
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:731:33
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow_execution_repository` with type `WorkflowExecutionRepository` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:716:47
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:732:47
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow_node_execution_repository` with type `WorkflowNodeExecutionRepository` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:717:52
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:733:52
 ERROR Argument `SimpleNamespace` is not assignable to parameter `flask_app` with type `Flask` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:781:23
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:797:23
 ERROR Argument `SimpleNamespace` is not assignable to parameter `context` with type `Context` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:786:21
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:802:21
 ERROR Argument `SimpleNamespace` is not assignable to parameter `variable_loader` with type `VariableLoader` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:787:29
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:803:29
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow_execution_repository` with type `WorkflowExecutionRepository` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:788:43
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:804:43
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow_node_execution_repository` with type `WorkflowNodeExecutionRepository` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:789:48
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:805:48
 ERROR Argument `SimpleNamespace` is not assignable to parameter `flask_app` with type `Flask` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:865:23
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:881:23
 ERROR Argument `SimpleNamespace` is not assignable to parameter `context` with type `Context` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:870:21
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:886:21
 ERROR Argument `SimpleNamespace` is not assignable to parameter `variable_loader` with type `VariableLoader` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:871:29
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:887:29
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow_execution_repository` with type `WorkflowExecutionRepository` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:872:43
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:888:43
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow_node_execution_repository` with type `WorkflowNodeExecutionRepository` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:873:48
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:889:48
 ERROR Argument `SimpleNamespace` is not assignable to parameter `flask_app` with type `Flask` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:946:27
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:962:27
 ERROR Argument `SimpleNamespace` is not assignable to parameter `context` with type `Context` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:951:25
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:967:25
 ERROR Argument `SimpleNamespace` is not assignable to parameter `variable_loader` with type `VariableLoader` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:952:33
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:968:33
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow_execution_repository` with type `WorkflowExecutionRepository` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:953:47
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:969:47
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow_node_execution_repository` with type `WorkflowNodeExecutionRepository` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:954:52
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:970:52
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._handle_advanced_chat_response` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1003:31
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1019:31
 ERROR Argument `SimpleNamespace` is not assignable to parameter `user` with type `Account | EndUser` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._handle_advanced_chat_response` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1012:22
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1028:22
 ERROR Argument `(**kwargs: Unknown) -> None` is not assignable to parameter `draft_var_saver_factory` with type `DraftVariableSaverFactory` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._handle_advanced_chat_response` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1013:41
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1029:41
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._handle_advanced_chat_response` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1051:35
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1067:35
 ERROR Argument `SimpleNamespace` is not assignable to parameter `user` with type `Account | EndUser` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._handle_advanced_chat_response` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1060:26
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1076:26
 ERROR Argument `(**kwargs: Unknown) -> None` is not assignable to parameter `draft_var_saver_factory` with type `DraftVariableSaverFactory` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._handle_advanced_chat_response` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1061:45
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1077:45
 ERROR Argument `SimpleNamespace` is not assignable to parameter `flask_app` with type `Flask` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1138:23
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1154:23
 ERROR Argument `SimpleNamespace` is not assignable to parameter `context` with type `Context` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1143:21
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1159:21
 ERROR Argument `SimpleNamespace` is not assignable to parameter `variable_loader` with type `VariableLoader` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1144:29
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1160:29
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow_execution_repository` with type `WorkflowExecutionRepository` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1145:43
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1161:43
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow_node_execution_repository` with type `WorkflowNodeExecutionRepository` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator._generate_worker` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1146:48
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1162:48
 ERROR No matching overload found for function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.generate` called with arguments: (app_model=SimpleNamespace, workflow=SimpleNamespace, user=Account, args=dict[str, dict[@_, @_] | str], invoke_from=Literal[InvokeFrom.DEBUGGER], workflow_run_id=Literal['run-id'], streaming=Literal[False], session=MagicMock) [no-matching-overload]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1218:36
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1234:36
 ERROR Object of class `NoneType` has no attribute `show_retrieve_source` [missing-attribute]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1230:16
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1246:16
 ERROR No matching overload found for function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.generate` called with arguments: (app_model=SimpleNamespace, workflow=SimpleNamespace, user=EndUser, args=dict[str, dict[@_, @_] | str], invoke_from=Literal[InvokeFrom.SERVICE_API], workflow_run_id=Literal['run-id'], streaming=Literal[False], session=MagicMock) [no-matching-overload]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1298:27
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1314:27
 ERROR Argument `SimpleNamespace` is not assignable to parameter `app_model` with type `App` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1365:23
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1381:23
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow` with type `Workflow` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1366:22
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1382:22
 ERROR Argument `SimpleNamespace` is not assignable to parameter `user` with type `Account | EndUser` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1367:18
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1383:18
 ERROR Argument `SimpleNamespace` is not assignable to parameter `conversation` with type `Conversation` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1368:26
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1384:26
 ERROR Argument `SimpleNamespace` is not assignable to parameter `message` with type `Message` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1369:21
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1385:21
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow_execution_repository` with type `WorkflowExecutionRepository` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1372:43
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1388:43
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow_node_execution_repository` with type `WorkflowNodeExecutionRepository` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1373:48
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1389:48
 ERROR Argument `SimpleNamespace` is not assignable to parameter `graph_runtime_state` with type `GraphRuntimeState` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1374:33
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1390:33
 ERROR Object of class `Generator` has no attribute `ok`
 ERROR Argument `SimpleNamespace` is not assignable to parameter `app_model` with type `App` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1413:23
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1429:23
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow` with type `Workflow` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1414:22
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1430:22
 ERROR Argument `SimpleNamespace` is not assignable to parameter `user` with type `Account | EndUser` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1415:18
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1431:18
 ERROR Argument `SimpleNamespace` is not assignable to parameter `conversation` with type `Conversation` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1416:26
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1432:26
 ERROR Argument `SimpleNamespace` is not assignable to parameter `message` with type `Message` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1417:21
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1433:21
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow_execution_repository` with type `WorkflowExecutionRepository` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1420:43
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1436:43
 ERROR Argument `SimpleNamespace` is not assignable to parameter `workflow_node_execution_repository` with type `WorkflowNodeExecutionRepository` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1421:48
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1437:48
 ERROR Argument `SimpleNamespace` is not assignable to parameter `graph_runtime_state` with type `GraphRuntimeState` in function `core.app.apps.advanced_chat.app_generator.AdvancedChatAppGenerator.resume` [bad-argument-type]
-    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1422:33
+    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_generator.py:1438:33
 ERROR Object of class `Generator` has no attribute `ok`
 ERROR Argument `object` is not assignable to parameter `variable` with type `VariableBase` in function `models.workflow.ConversationVariable.from_variable` [bad-argument-type]
   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_runner_conversation_variables.py:66:22
@@ -3077,110 +3079,130 @@
   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_runner_conversation_variables.py:99:25
 ERROR Argument `list[VariableBase]` is not assignable to parameter `workflow_variables` with type `list[object]` in function `_runner` [bad-argument-type]
    --> tests/unit_tests/core/app/apps/advanced_chat/test_app_runner_conversation_variables.py:116:25
+ERROR Class `WorkflowEntry` has no class attribute `call_args` [missing-attribute]
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_runner_input_moderation.py:138:24
+ERROR Class `WorkflowEntry` has no class attribute `call_args` [missing-attribute]
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_app_runner_input_moderation.py:168:24
 ERROR Cannot index into `str` [bad-index]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_response_converter.py:129:16
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_response_converter.py:137:16
 ERROR Cannot index into `str` [bad-index]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_response_converter.py:130:16
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_response_converter.py:138:16
 ERROR Cannot index into `str` [bad-index]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_response_converter.py:131:16
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_response_converter.py:139:16
 ERROR `SimpleNamespace` is not assignable to attribute `_application_generate_entity` with type `AdvancedChatAppGenerateEntity` [bad-assignment]
    --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:109:45
 ERROR Argument `SimpleNamespace` is not assignable to parameter `application_generate_entity` with type `AdvancedChatAppGenerateEntity` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:176:37
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:177:37
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:178:23
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:179:23
 ERROR Argument `SimpleNamespace` is not assignable to parameter `draft_var_saver_factory` with type `DraftVariableSaverFactory` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:184:33
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:185:33
 ERROR `SimpleNamespace` is not assignable to attribute `_application_generate_entity` with type `AdvancedChatAppGenerateEntity` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:218:45
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:221:45
 ERROR `SimpleNamespace` is not assignable to attribute `_application_generate_entity` with type `AdvancedChatAppGenerateEntity` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:240:45
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline.py:243:45
 ERROR Argument `SimpleNamespace` is not assignable to parameter `queue_manager` with type `AppQueueManager` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:103:23
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:112:23
 ERROR Argument `(**kwargs: Unknown) -> None` is not assignable to parameter `draft_var_saver_factory` with type `DraftVariableSaverFactory` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline.__init__` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:109:33
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:122:33
+ERROR Argument `SimpleNamespace` is not assignable to parameter `message` with type `Message` in function `core.app.apps.advanced_chat.generate_task_pipeline.MessageSnapshot.from_message` [bad-argument-type]
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:218:49
+ERROR Argument `SimpleNamespace` is not assignable to parameter `session` with type `Session` in function `core.app.apps.advanced_chat.generate_task_pipeline.MessageSnapshot.from_message` [bad-argument-type]
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:218:66
+ERROR Object of class `WorkflowMaintenancePausedBlockingResponse` has no attribute `data` [missing-attribute]
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:337:16
+ERROR Object of class `WorkflowMaintenancePausedBlockingResponse` has no attribute `data` [missing-attribute]
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:338:16
+ERROR `() -> Generator[SimpleNamespace, Unknown, Never]` is not assignable to attribute `listen` with type `(self: AppQueueManager) -> Generator[MessageQueueMessage | WorkflowQueueMessage, Unknown]` [bad-assignment]
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:371:61
+ERROR `() -> Iterator[MessageQueueMessage | WorkflowQueueMessage] | Iterator[Any]` is not assignable to attribute `listen` with type `(self: AppQueueManager) -> Generator[MessageQueueMessage | WorkflowQueueMessage, Unknown]` [bad-assignment]
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:382:61
+ERROR `(event: Unknown) -> Generator[WorkflowMaintenancePausedStreamResponse, Unknown]` is not assignable to attribute `_handle_workflow_maintenance_paused_event` with type `(self: AdvancedChatAppGenerateTaskPipeline, event: QueueWorkflowMaintenancePausedEvent, **kwargs: Unknown) -> Generator[StreamResponse]` [bad-assignment]
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:394:62
 ERROR `SimpleNamespace` is not assignable to attribute `_message_cycle_manager` with type `MessageCycleManager` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:224:43
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:457:43
 ERROR Argument `SimpleNamespace` is not assignable to parameter `event` with type `QueueTextChunkEvent` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_text_chunk_event` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:232:60
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:465:60
 ERROR Argument `SimpleNamespace` is not assignable to parameter `publisher` with type `AppGeneratorTTSPublisher | None` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._listen_audio_msg` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:277:57
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:510:57
 ERROR `(err: Unknown) -> Unknown` is not assignable to attribute `error_to_stream_response` with type `(self: BasedGenerateTaskPipeline[Unknown], e: Exception) -> ErrorStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:292:65
-ERROR `(**kwargs: Unknown) -> Literal['started']` is not assignable to attribute `workflow_start_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_run_id: str, workflow_id: str, reason: WorkflowStartReason) -> WorkflowStartStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:306:83
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:525:65
+ERROR `(**kwargs: Unknown) -> Literal['started']` is not assignable to attribute `workflow_start_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_run_id: str, workflow_id: str, reason: WorkflowStartReason, logical_started_at: datetime | None = None, handoff_duration: float = 0.0) -> WorkflowStartStreamResponse` [bad-assignment]
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:539:83
+ERROR `(**kwargs: Unknown) -> Literal['started']` is not assignable to attribute `workflow_start_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_run_id: str, workflow_id: str, reason: WorkflowStartReason, logical_started_at: datetime | None = None, handoff_duration: float = 0.0) -> WorkflowStartStreamResponse` [bad-assignment]
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:558:83
 ERROR `TestAdvancedChatGenerateTaskPipeline.test_handle_output_moderation_chunk_publishes_stop._Moderation` is not assignable to attribute `output_moderation_handler` with type `OutputModeration | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:340:66
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:606:66
 ERROR `SimpleNamespace` is not assignable to attribute `queue_manager` with type `AppQueueManager` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:341:54
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:607:54
 ERROR `(outputs: Unknown) -> list[dict[str, str]]` is not assignable to attribute `fetch_files_from_node_outputs` with type `(cls: type[WorkflowResponseConverter], outputs_dict: Mapping[str, Any] | None) -> Sequence[Mapping[str, Any]]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:354:79
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:620:79
 ERROR `(**kwargs: Unknown) -> Literal['done']` is not assignable to attribute `workflow_node_finish_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent, task_id: str) -> NodeFinishStreamResponse | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:357:89
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:623:89
 ERROR Argument `SimpleNamespace` is not assignable to parameter `event` with type `QueueNodeSucceededEvent` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_node_succeeded_event` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:367:64
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:633:64
 ERROR `(outputs: Unknown) -> list[@_]` is not assignable to attribute `fetch_files_from_node_outputs` with type `(cls: type[WorkflowResponseConverter], outputs_dict: Mapping[str, Any] | None) -> Sequence[Mapping[str, Any]]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:374:79
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:640:79
 ERROR `(**kwargs: Unknown) -> Literal['done']` is not assignable to attribute `workflow_node_finish_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent, task_id: str) -> NodeFinishStreamResponse | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:375:89
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:641:89
 ERROR Argument `SimpleNamespace` is not assignable to parameter `event` with type `QueueNodeSucceededEvent` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_node_succeeded_event` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:385:52
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:651:52
 ERROR `(outputs: Unknown) -> list[@_]` is not assignable to attribute `fetch_files_from_node_outputs` with type `(cls: type[WorkflowResponseConverter], outputs_dict: Mapping[str, Any] | None) -> Sequence[Mapping[str, Any]]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:391:79
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:657:79
 ERROR `(**kwargs: Unknown) -> Literal['done']` is not assignable to attribute `workflow_node_finish_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent, task_id: str) -> NodeFinishStreamResponse | None` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:392:89
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:658:89
 ERROR Argument `SimpleNamespace` is not assignable to parameter `event` with type `QueueNodeSucceededEvent` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_node_succeeded_event` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:404:52
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:670:52
 ERROR Argument `SimpleNamespace` is not assignable to parameter `event` with type `QueueNodeSucceededEvent` in function `core.app.apps.advanced_chat.generate_task_pipeline.AdvancedChatAppGenerateTaskPipeline._handle_node_succeeded_event` [bad-argument-type]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:405:52
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:671:52
 ERROR `(**kwargs: Unknown) -> Literal['iter_start']` is not assignable to attribute `workflow_iteration_start_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_execution_id: str, event: QueueIterationStartEvent) -> IterationNodeStartStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:412:93
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:678:93
 ERROR `(**kwargs: Unknown) -> Literal['iter_next']` is not assignable to attribute `workflow_iteration_next_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_execution_id: str, event: QueueIterationNextEvent) -> IterationNodeNextStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:415:92
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:681:92
 ERROR `(**kwargs: Unknown) -> Literal['iter_done']` is not assignable to attribute `workflow_iteration_completed_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_execution_id: str, event: QueueIterationCompletedEvent) -> IterationNodeCompletedStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:416:97
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:682:97
 ERROR `(**kwargs: Unknown) -> Literal['loop_start']` is not assignable to attribute `workflow_loop_start_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_execution_id: str, event: QueueLoopStartEvent) -> LoopNodeStartStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:419:88
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:685:88
 ERROR `(**kwargs: Unknown) -> Literal['loop_next']` is not assignable to attribute `workflow_loop_next_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_execution_id: str, event: QueueLoopNextEvent) -> LoopNodeNextStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:420:87
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:686:87
 ERROR `(**kwargs: Unknown) -> Literal['loop_done']` is not assignable to attribute `workflow_loop_completed_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_execution_id: str, event: QueueLoopCompletedEvent) -> LoopNodeCompletedStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:421:92
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:687:92
 ERROR `(**kwargs: Unknown) -> Literal['finish']` is not assignable to attribute `workflow_finish_to_stream_response` with type `(self: WorkflowResponseConverter, *, task_id: str, workflow_id: str, status: WorkflowExecutionStatus, graph_runtime_state: GraphRuntimeState, error: str | None = None, exceptions_count: int = 0) -> WorkflowFinishStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:488:84
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:754:84
 ERROR `(**kwargs: Unknown) -> list[str]` is not assignable to attribute `workflow_pause_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueWorkflowPausedEvent, task_id: str, graph_runtime_state: GraphRuntimeState) -> list[StreamResponse]` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:489:83
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:755:83
 ERROR `(err: Unknown) -> Unknown` is not assignable to attribute `error_to_stream_response` with type `(self: BasedGenerateTaskPipeline[Unknown], e: Exception) -> ErrorStreamResponse` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:494:65
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:760:65
 ERROR `(**kwargs: Unknown) -> SimpleNamespace` is not assignable to attribute `_get_message` with type `(self: AdvancedChatAppGenerateTaskPipeline, *, session: Session) -> Message` [bad-assignment]
-   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:495:33
+   --> tests/unit_tests/core/app/apps/advanced_chat/test_generate_task_pipeline_core.py:761:33
 ERROR `(**kwargs: Unknown) -> Literal['node_finish']` is not assignable to attribute `workflow_node_finish_to_stream_response` with type `(self: WorkflowResponseConverter, *, event: QueueNodeExceptionEvent | QueueNodeFailedEvent | QueueNodeSucceededEvent, task_id: str) -> NodeFinishStreamResponse | None` [bad-assi\n\n... (truncated) ...

@codecov

codecov Bot commented Jul 28, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.96687% with 424 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.35%. Comparing base (97a13b6) to head (492e37e).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
...e/app/apps/advanced_chat/generate_task_pipeline.py 70.86% 25 Missing and 12 partials ⚠️
api/tasks/rag_pipeline/rag_pipeline_run_task.py 75.75% 13 Missing and 11 partials ⚠️
api/core/rag/pipeline/queue.py 68.91% 11 Missing and 12 partials ⚠️
.../core/app/layers/workflow_handoff_persist_layer.py 84.29% 10 Missing and 9 partials ⚠️
...sks/rag_pipeline/priority_rag_pipeline_run_task.py 80.61% 11 Missing and 8 partials ⚠️
..._api/dataset/rag_pipeline/rag_pipeline_workflow.py 63.26% 15 Missing and 3 partials ⚠️
api/core/app/layers/pause_state_persist_layer.py 68.96% 9 Missing and 9 partials ⚠️
api/services/workflow_handoff_resume_routes.py 92.10% 6 Missing and 12 partials ⚠️
...sitories/sqlalchemy_workflow_handoff_repository.py 98.00% 6 Missing and 9 partials ⚠️
...e/app/apps/pipeline/generate_response_converter.py 65.78% 10 Missing and 3 partials ⚠️
... and 45 more
Additional details and impacted files 
@@            Coverage Diff             @@
##             main   #39706      +/-   ##
==========================================
+ Coverage   86.18%   86.35%   +0.17%     
==========================================
  Files        5087     5265     +178     
  Lines      287747   295793    +8046     
  Branches    57364    59149    +1785     
==========================================
+ Hits       247985   255426    +7441     
- Misses      34961    35427     +466     
- Partials     4801     4940     +139
Flag Coverage Δ
api 86.51% <89.96%> (+0.15%) ⬆️
cli 89.33% <ø> (?)
dify-ui 93.98% <ø> (ø)
web 85.97% <ø> (+0.09%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@laipz8200 laipz8200 Awaiting requested review from laipz8200 laipz8200 is a code owner
@hyoban hyoban Awaiting requested review from hyoban hyoban is a code owner
@CodingOnStar CodingOnStar Awaiting requested review from CodingOnStar CodingOnStar is a code owner
@iamjoel iamjoel Awaiting requested review from iamjoel iamjoel is a code owner
@zxhlyh zxhlyh Awaiting requested review from zxhlyh zxhlyh is a code owner
@WTW0313 WTW0313 Awaiting requested review from WTW0313 WTW0313 is a code owner
@snakevash snakevash Awaiting requested review from snakevash snakevash is a code owner
@MRZHUH MRZHUH Awaiting requested review from MRZHUH MRZHUH is a code owner
@Mairuis Mairuis Awaiting requested review from Mairuis Mairuis is a code owner
@CourTeous33 CourTeous33 Awaiting requested review from CourTeous33 CourTeous33 is a code owner
@JohnJyong JohnJyong Awaiting requested review from JohnJyong JohnJyong is a code owner
@QuantumGhost QuantumGhost Awaiting requested review from QuantumGhost QuantumGhost is a code owner
@crazywoola crazywoola Awaiting requested review from crazywoola crazywoola is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

size:XXL This PR changes 1000+ lines, ignoring generated files. web This relates to changes on the web.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@zyssyz123