refactor: shell provider protocol

[wylswz]

Important

1. Make sure you have read our contribution guidelines
2. Ensure there is an associated issue and you have been assigned to it
3. Use the correct syntax to link this PR: Fixes #<issue number>.

Summary

Added an abstraction layer for 3rd party shell provider support in the future. Providers should implement

• provision a new session
• attach to an existing session
• destroy a session

Testing

The agent is reading file correctly, but the download link in webapp is not correctly rendered, I'm not sure if it's related to this PR.

Screenshots

Before	After
...	...

Checklist

• This change requires a documentation update, included: Dify Document
• I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
• I've added a test for each change that was introduced, and I tried as much as possible to make a single atomic change.
• I've updated the documentation accordingly.
• I ran make lint && make type-check (backend) and cd web && pnpm exec vp staged (frontend) to appease the lint gods

[dosubot]

📄 Knowledge review

✏️ Suggested updates

1 page suggestion needs review.

Page	Library	Status
Dify Agent Server 模块分析 (Commit 55f95dbc)	dify	🟡 Review

📝 Dify Agent Server 模块分析 (Commit 55f95db)

@@ -258,7 +258,7 @@
 
 #### 6. Dify Shell Layer
 
-**`DifyShellLayer`**（type_id: `dify.shell`）：基于 shellctl 的有状态工具层，暴露 `shell_run`、`shell_wait`、`shell_input` 和 `shell_interrupt` 四个工具。该层仅在 `runtime_state` 中持久化 JSON 安全的 shell 会话状态，而在 `resource_context()` 活跃时将活跃 shellctl HTTP 客户端保存在 Layer 实例上。Agenton 在 `on_context_create` 或 `on_context_resume` 之前进入该资源作用域，并在 `on_context_suspend` 或 `on_context_delete` 之后退出，因此业务钩子和 shell 工具可以依赖活跃客户端而无需将其序列化到快照中。该层具有可选依赖 `DifyShellLayerDeps`（包含 `drive: DifyDriveLayer | None` 和 `execution_context: DifyExecutionContextLayer | None`），并可通过 `build_shell_agent_stub_env` 和 `ShellAgentStubTokenFactory` 向用户可见的 `shell.run` 作业注入 Agent Stub 环境变量（`DIFY_AGENT_STUB_API_BASE_URL`、`DIFY_AGENT_STUB_AUTH_JWE` 和 `DIFY_AGENT_STUB_DRIVE_BASE`），仅当 shell layer 构建时传入了有效的 `agent_stub_api_base_url` 和 `agent_stub_token_factory` 参数。当运行包含 `dify.drive` 依赖时，注入的 drive base 从 Agent Stub drive 挂载和 drive 引用计算（如 `/mnt/drive/agent-123`）。没有 drive 依赖时，CLI 保留历史 `/mnt/drive` 回退值。内部生命周期脚本不注入 Agent Stub 环境。
+**`DifyShellLayer`**（type_id: `dify.shell`）：shell 适配器提供者/执行器机制支持的有状态工具层，暴露 `shell_run`、`shell_wait`、`shell_input` 和 `shell_interrupt` 四个工具。该层仅在 `runtime_state` 中持久化 JSON 安全的 shell 会话状态，而在 `resource_context()` 活跃时将活跃 `ShellctlHandle` 保存在 Layer 实例上。Agenton 在 `on_context_create` 或 `on_context_resume` 之前进入该资源作用域，并在 `on_context_suspend` 或 `on_context_delete` 之后退出，因此业务钩子和 shell 工具可以依赖活跃资源而无需将其序列化到快照中。该层将工作区生命周期委托给 `ShellProvisionProtocol`：`provision()` 分配新工作区，`reattach()` 从描述符重建句柄，`destroy()` 拆除工作区。内部工作区分配逻辑（`_allocate_workspace`、`_workspace_mkdir_script`、`_workspace_cleanup_script`）已移除，现由提供者处理。会话 ID 验证已放宽：现在仅拒绝包含 `/`、`..` 或单引号的值（不再要求精确的 5+2 十六进制格式）。该层具有可选依赖 `DifyShellLayerDeps`（包含 `drive: DifyDriveLayer | None` 和 `execution_context: DifyExecutionContextLayer | None`），并可通过 `build_shell_agent_stub_env` 和 `ShellAgentStubTokenFactory` 向用户可见的 `shell.run` 作业注入 Agent Stub 环境变量（`DIFY_AGENT_STUB_API_BASE_URL`、`DIFY_AGENT_STUB_AUTH_JWE` 和 `DIFY_AGENT_STUB_DRIVE_BASE`），仅当 shell layer 构建时传入了有效的 `agent_stub_api_base_url` 和 `agent_stub_token_factory` 参数。当运行包含 `dify.drive` 依赖时，注入的 drive base 从 Agent Stub drive 挂载和 drive 引用计算（如 `/mnt/drive/agent-123`）。没有 drive 依赖时，CLI 保留历史 `/mnt/drive` 回退值。服务器拥有的脚本（如 `run_remote_script`）现在使用来自 shell 适配器的 `ShellctlExecutor` 进行执行、输出排空和临时作业清理，不再在层中内联实现。
 
 **Shell Layer 配置模型**（`dify-agent/src/dify_agent/layers/shell/configs.py`）：
 
@@ -276,7 +276,108 @@
 
 **`_shell_config_export_lines` 辅助函数**：返回 Agent Soul shell 环境的 export 语句列表，包括环境变量、敏感凭证引用和沙箱配置。该函数同时被引导脚本（用于 CLI 工具安装）和 `_wrap_user_script`（用于用户命令）复用，提供临时的按命令注入（ephemeral per-command injection），而不持久化工作区文件。
 
-**`run_remote_script` 方法**：与模型可见的 `shell.run` 不同，该服务器拥有的边界不注入 Agent Soul shell 环境。将用户控制的 shell 环境排除在此路径之外，防止沙箱代码在 `dify-agent file upload` 执行前覆盖受信任的 Agent Stub 环境值。
+**`run_remote_script` 方法**：与模型可见的 `shell.run` 不同，该服务器拥有的边界不注入 Agent Soul shell 环境。执行、输出排空和临时作业清理委托给来自 shell 适配器的 `ShellctlExecutor`；层只负责可选的 Agent Stub 环境注入和 `RemoteCommandResult` 映射。将用户控制的 shell 环境排除在此路径之外，防止沙箱代码在 `dify-agent file upload` 执行前覆盖受信任的 Agent Stub 环境值。
+
+**`environment_descriptor()` 方法**：返回用于 shell 适配器的可序列化工作区种子。桥接层的 `runtime_state` 到 `dify_agent.adapters.shell`：返回的描述符标识会话工作区，以便适配器的 `ShellProvisionProtocol.reattach` 可以重建指向它的活跃句柄而无需重新分配，并且无需重新进入该层。如果会话标识缺失或不一致，则抛出 `ValueError`。
+
+#### Shell 适配器协议层
+
+**Shell 适配器架构**（`dify-agent/src/dify_agent/adapters/shell/`）：引入了提供者无关的 shell 层抽象，支持当前的 shellctl 后端和未来的第三方 shell 提供者。
+
+**协议层次结构**（`protocols.py`）：
+
+- **`ShellProvisionProtocol`**：创建、重新连接和销毁 shell 环境
+  - `provision() -> ShellHandle`：分配新的 shell 环境
+  - `reattach(descriptor: ShellEnvironmentDescriptor) -> ShellHandle`：从持久化的描述符重建活跃句柄而不重新分配
+  - `destroy(handle: ShellHandle) -> None`：拆除环境
+
+- **`ShellHandle`**：对已配置 shell 环境的活跃引用
+  - `descriptor() -> ShellEnvironmentDescriptor`：返回可序列化的工作区标识种子
+  - `get_executor() -> ShellExecutorProtocol`：返回命令执行器
+  - `get_file_transfer() -> ShellFileTransferProtocol`：返回文件传输处理器
+
+- **`ShellEnvironmentDescriptor`**：可序列化的工作区标识
+  - `workspace_cwd: str`：工作区目录路径
+  - `session_id: str`：会话标识符
+
+- **`ShellExecutorProtocol`**：在已配置环境内运行命令
+  - `execute(command, cwd, env) -> ShellExecutionHandle`：启动命令
+  - `wait(handle) -> ShellExecutionResult`：等待命令完成
+
+- **`ShellExecutionHandle`**：对已启动 shell 命令的不透明引用
+  - `job_id: str`：后端定义的作业标识符
+
+- **`ShellExecutionResult`**：已完成的 shell 命令结果
+  - `stdout() -> str`：标准输出
+  - `stderr() -> str`：标准错误
+  - `exit_code() -> int | None`：退出代码
+  - `truncated() -> bool`：输出是否被截断
+
+- **可选协议**：
+  - **`SupportsShellInput`**：`input(handle, text) -> ShellExecutionResult`
+  - **`SupportsShellInterrupt`**：`interrupt(handle) -> ShellExecutionResult`
+
+- **`ShellFileTransferProtocol`**：文件传输操作
+  - `upload(content, remote_path) -> None`：上传字节到工作区
+  - `download(remote_path) -> bytes`：从工作区下载字节
+
+**Shellctl 实现**（`shellctl.py`）：
+
+- **`ShellctlProvisioner`**：实现 `ShellProvisionProtocol`，管理 shellctl 工作区
+  - 使用 `client_factory: ShellctlClientFactory` 为每个环境创建新客户端
+  - `provision()` 生成随机会话 ID 并创建 `~/workspace/<session_id>` 目录
+  - `reattach()` 为现有工作区构建新客户端（用于快照恢复）
+  - `destroy()` 运行 `rm -rf` 清理脚本并关闭客户端
+
+- **`ShellctlHandle`**：shellctl 工作区的活跃引用
+  - 持有 `client: ShellctlClientProtocol`、`workspace_cwd: str` 和 `session_id: str`
+  - `get_executor()` 返回绑定到该工作区的新 `ShellctlExecutor`
+  - `get_file_transfer()` 返回新 `ShellctlFileTransfer`
+
+- **`ShellctlExecutor`**：在 shellctl 工作区内执行命令
+  - 有状态：跟踪每个启动作业的分页偏移量和输出部分
+  - `execute()` 启动命令并记录初始结果
+  - `wait()` 排空输出窗口直到完成（最多 64 个窗口），然后删除作业
+  - 实现可选的 `input()` 和 `interrupt()` 功能
+
+- **`ShellctlExecutionResult`**：已完成的 shellctl 命令结果
+  - shellctl 将 stderr 合并到单个输出流中，因此 `stderr()` 始终为空
+  - `truncated()` 在最终窗口仍有未读输出时返回 `True`
+
+- **`ShellctlFileTransfer`**：基于 base64 的文件传输
+  - `upload()` 将 base64 编码的内容通过管道传输到 `base64 -d`
+  - `download()` 在帧标记之间发出文件的 base64 并提取有效负载
+
+**工厂模式**（`factory.py`）：
+
+- **`create_shell_provisioner(settings)`**：基于 `DIFY_AGENT_SHELL_PROVIDER` 选择提供者
+  - 当前仅支持 `"shellctl"` 提供者
+  - 要求为 shellctl 设置 `DIFY_AGENT_SHELLCTL_ENTRYPOINT`
+  - 返回 `ShellProvisionProtocol` 实现
+
+**配置**（`config.py`）：
+
+- **`ShellAdapterSettings`**：shell 适配器环境变量
+  - `shell_provider: str`（默认 `"shellctl"`）
+  - `shellctl_entrypoint: str | None`
+  - `shellctl_auth_token: str | None`
+
+**DifyShellLayer 变更**：
+
+- 初始化：`from_config_with_settings` 现在接受单个 `shell_provisioner: ShellProvisionProtocol` 参数，而不是 `shellctl_entrypoint` 和 `shellctl_client_factory`
+- 层在 `resource_context()` 活跃时保持活跃的 `ShellctlHandle`（而不是原始的 `ShellctlClient`）
+- 工作区生命周期委托给 `ShellProvisionProtocol`：
+  - `on_context_create()` 调用 `provision()` 分配新工作区
+  - `on_context_resume()` 调用 `reattach()` 从描述符重建句柄
+  - `on_context_delete()` 调用 `destroy()` 拆除工作区
+- 内部工作区分配逻辑（`_allocate_workspace`、`_workspace_mkdir_script`、`_workspace_cleanup_script`）已移除
+- 会话 ID 验证放宽：现在仅拒绝 `/`、`..` 或单引号（不再要求精确的 5+2 十六进制格式）
+- 新增 `environment_descriptor()` 方法返回 `ShellEnvironmentDescriptor`，用于桥接到 shell 适配器
+
+**RemoteCommandResult 变更**：从数据类中移除了几个字段，因为它们是调用者不使用的内部 shellctl 分页详细信息：
+
+- **已移除**：`job_id`、`done`、`offset`、`output_path`
+- **保留**：`status`、`exit_code`、`output`、`truncated`
 
 #### 7. Dify Drive Layer
 

Accept · Edit · Decline

---

[Copilot]

Pull request overview

This PR refactors the Dify Agent shell layer to route workspace provisioning, remote command execution, and file transfer through a new provider-style adapter boundary (with an initial shellctl implementation), aiming to make future third-party shell backends pluggable.

Changes:

• Introduces dify_agent.adapters.shell protocols, settings, a factory, and a shellctl backend (provisioner/executor/file transfer).
• Refactors DifyShellLayer to use a ShellProvisionProtocol + live handle instead of directly owning shellctl entrypoint/client factory.
• Updates/extends local tests to validate provisioning/execution/file-transfer behavior with structural fake clients.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
dify-agent/src/dify_agent/adapters/shell/init.py	Exposes the new adapter API surface and re-exports key types/errors.
dify-agent/src/dify_agent/adapters/shell/config.py	Adds env-backed settings (DIFY_AGENT_*) for provider selection and shellctl config.
dify-agent/src/dify_agent/adapters/shell/factory.py	Implements env/settings-driven provisioner selection (create_shell_provisioner).
dify-agent/src/dify_agent/adapters/shell/protocols.py	Defines provider-agnostic protocols for provisioning, execution, and file transfer.
dify-agent/src/dify_agent/adapters/shell/shellctl.py	Adds shellctl-backed provisioner/executor/file transfer implementation.
dify-agent/src/dify_agent/layers/shell/layer.py	Refactors the shell layer to use a provisioner + handle; updates session validation and remote execution mapping.
dify-agent/src/dify_agent/runtime/compositor_factory.py	Wires the shell layer to construct a provisioner via adapter settings instead of a raw shellctl client factory.
dify-agent/tests/local/dify_agent/adapters/shell/test_shellctl.py	New tests for shellctl adapter behavior (provision/execute/transfer/factory).
dify-agent/tests/local/dify_agent/layers/drive/test_layer.py	Adjusts drive-layer shell remote result construction to the new RemoteCommandResult shape.
dify-agent/tests/local/dify_agent/layers/shell/test_layer.py	Updates shell-layer tests for provisioner/handle lifecycle behavior.
dify-agent/tests/local/dify_agent/server/test_app.py	Updates server app wiring assertions to the new ShellctlProvisioner injection.
dify-agent/tests/local/dify_agent/server/test_sandbox_files.py	Updates sandbox file service tests to inject a ShellctlProvisioner instead of shellctl entrypoint/client factory.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[zyssyz123]

I took a pass over this PR. The provider/provisioner direction looks reasonable, but I think there are a few issues to address before this is safe to merge:

1. Some dify-agent tests still use the old DifyShellLayer.from_config_with_settings() signature.

DifyShellLayer.from_config_with_settings() now expects shell_provisioner, but a few tests still pass shellctl_entrypoint / shellctl_client_factory. These fail immediately at runtime:

• dify-agent/tests/local/dify_agent/layers/drive/test_layer.py:20
• dify-agent/tests/local/dify_agent/runtime/test_runner.py:1347

I reproduced this locally with:

cd dify-agent
uv run pytest tests/local/dify_agent/layers/drive/test_layer.py -q
uv run pytest tests/local/dify_agent/runtime/test_runner.py::test_runner_rejects_duplicate_tool_names_between_shell_and_other_layers -q

2. test_compositor_factory.py is still patching a removed symbol.

dify_agent.runtime.compositor_factory.create_shellctl_client_factory no longer exists after this refactor, but the test still monkeypatches it:

• dify-agent/tests/local/dify_agent/runtime/test_compositor_factory.py:32
• dify-agent/tests/local/dify_agent/runtime/test_compositor_factory.py:63

Local repro:

cd dify-agent
uv run pytest tests/local/dify_agent/runtime/test_compositor_factory.py -q

This currently fails with AttributeError: ... has no attribute 'create_shellctl_client_factory'.

3. The new shell adapter should validate reattached descriptors at the adapter boundary.

ShellctlProvisioner.reattach() currently trusts ShellEnvironmentDescriptor as-is, and destroy() later interpolates handle.session_id into a shell cleanup script:

• dify-agent/src/dify_agent/adapters/shell/shellctl.py:308
• dify-agent/src/dify_agent/adapters/shell/shellctl.py:327
• dify-agent/src/dify_agent/adapters/shell/shellctl.py:413

DifyShellLayer validates its runtime state today, but this PR introduces dify_agent.adapters.shell as a reusable provider boundary. The adapter should not rely on every caller to perform the same validation before calling reattach(). I would add validation in ShellctlProvisioner.reattach() / handle construction to enforce something like:

• session_id matches the expected lowercase hex pattern.
• workspace_cwd == ~/workspace/<session_id>.

That keeps the new abstraction safe even if a future caller uses the provider directly.

[wylswz]

I took a pass over this PR. The provider/provisioner direction looks reasonable, but I think there are a few issues to address before this is safe to merge:

1. Some dify-agent tests still use the old DifyShellLayer.from_config_with_settings() signature.

DifyShellLayer.from_config_with_settings() now expects shell_provisioner, but a few tests still pass shellctl_entrypoint / shellctl_client_factory. These fail immediately at runtime:

• dify-agent/tests/local/dify_agent/layers/drive/test_layer.py:20
• dify-agent/tests/local/dify_agent/runtime/test_runner.py:1347

I reproduced this locally with:

cd dify-agent
uv run pytest tests/local/dify_agent/layers/drive/test_layer.py -q
uv run pytest tests/local/dify_agent/runtime/test_runner.py::test_runner_rejects_duplicate_tool_names_between_shell_and_other_layers -q

2. test_compositor_factory.py is still patching a removed symbol.

dify_agent.runtime.compositor_factory.create_shellctl_client_factory no longer exists after this refactor, but the test still monkeypatches it:

• dify-agent/tests/local/dify_agent/runtime/test_compositor_factory.py:32
• dify-agent/tests/local/dify_agent/runtime/test_compositor_factory.py:63

Local repro:

cd dify-agent
uv run pytest tests/local/dify_agent/runtime/test_compositor_factory.py -q

This currently fails with AttributeError: ... has no attribute 'create_shellctl_client_factory'.

3. The new shell adapter should validate reattached descriptors at the adapter boundary.

ShellctlProvisioner.reattach() currently trusts ShellEnvironmentDescriptor as-is, and destroy() later interpolates handle.session_id into a shell cleanup script:

• dify-agent/src/dify_agent/adapters/shell/shellctl.py:308
• dify-agent/src/dify_agent/adapters/shell/shellctl.py:327
• dify-agent/src/dify_agent/adapters/shell/shellctl.py:413

DifyShellLayer validates its runtime state today, but this PR introduces dify_agent.adapters.shell as a reusable provider boundary. The adapter should not rely on every caller to perform the same validation before calling reattach(). I would add validation in ShellctlProvisioner.reattach() / handle construction to enforce something like:

• session_id matches the expected lowercase hex pattern.
• workspace_cwd == ~/workspace/<session_id>.

That keeps the new abstraction safe even if a future caller uses the provider directly.

Makes sense, I'll improve the interface design.

[wylswz]

There are still 11 test failures, which are not introduced by this PR, will ignore them.